Research method for load transfer of electric heating gas integrated energy system based on reinforcement learning

Abstract

The invention provides an electric heating gas comprehensive energy system load transfer method based on reinforcement learning, and belongs to the field of artificial intelligence. The method comprises the following steps: acquiring related data from actual operation of the integrated energy system, and taking the collected data as training sample data; constructing an intelligent agent-environment model, and simulating and constraining the operation of the system; establishing a transfer trigger condition mechanism of different uncertain events occurring in the integrated energy system, and determining various uncertain events occurring in the integrated energy system; constructing a DDQN model, and generating an optimal load transfer strategy of the electric heating gas comprehensive energy system in different uncertain events after multiple iterations through a DDQN algorithm; and realizing an optimal load transfer strategy of the electric heating gas integrated energy system containing different load types by utilizing the trained model. According to the method, the limitations of too long calculation time, low strategy error-tolerant rate and the like of a traditional load transfer method of the electric heating gas comprehensive energy system containing various different types of loads are overcome, and the complex dynamic behaviors of the electric heating gas comprehensive energy system are accurately described; and a real-time and accurate load transfer strategy of the electric heating gas comprehensive energy system containing different types of loads is realized.

Description

Technical Field

[0001] The application of the present invention belongs to the field of integrated energy, and the technical means of the present invention belongs to the field of machine learning and also belongs to the field of artificial intelligence, especially the design of reinforced learning for load transfer path analysis of integrated energy systems. Background Art

[0002] Driven by the "dual carbon goals" and the goal of building a new power system, in order to solve the contradiction between supply and demand of a single energy source during peak load in the traditional power system, it has become an inevitable trend to build a new integrated energy system with multi-energy coupling of power system, thermal system, and natural gas system. With the grid connection of photovoltaic and wind power in the integrated energy system and the continuous increase in the scale of the system, the integrated energy system faces a series of uncertain events such as photovoltaic and wind power consumption, sudden surges and decreases in loads in different regions, and line failures. There are huge challenges in ensuring the stability of load operation in different regions of different integrated energy systems. Accurate and real-time formulation of integrated energy system load transfer plans can provide reliable decision-making information for the stable operation of loads in different regions of the integrated energy system, thereby ensuring the safety, stability, economy, and operation of the integrated energy system.

[0003] How to realize the transfer of loads in different regions of the integrated energy system has become a focal issue. The traditional power system, thermal system and natural gas system realize the load transfer through the heuristic algorithm (genetic algorithm, particle swarm algorithm, etc.) network, such as the connecting switch, section switch or connecting valve, to achieve system safety. This transfer method has the following limitations: First, the energy mode of the traditional power system, thermal system and natural gas system is single, which cannot meet the needs of new multi-type load transfer; second, different regions in the integrated energy system have different load types and different flexibility. The current research method lacks the research on the load transfer scheme of different types of integrated energy systems; third, since the realization of the load transfer of the integrated energy system involves the switching actions of each switch valve, and the system topology is complex, traditional heuristic algorithms such as genetic algorithms and particle swarm algorithms are difficult to provide real-time and accurate load transfer route strategies; fourth, the current load transfer solution usually only considers the safety of load transfer after the system fails, lacks the load transfer scheme that considers the economy when the system is not faulty, and it is difficult to achieve both safety and economy.

[0004] In view of the fact that it is difficult to achieve real-time and accurate load transfer strategies for integrated energy systems, the reinforcement learning technology has become increasingly mature, providing the necessary conditions for developing real-time and accurate load transfer strategies for future integrated energy systems. However, this technology has not been widely used in integrated energy systems. Therefore, it is necessary to make full use of the topological structure of the integrated energy system and a large number of load transfer strategies, deeply explore the relationship between different uncertain events and load transfer strategies, and effectively reduce the time for formulating load transfer strategies. At the same time, since the integrated energy system is highly sensitive to topological operations, incorrect operations may lead to serious consequences. It is also necessary to establish load transfer strategies corresponding to different uncertain events to improve the accuracy of load transfer strategies. Summary of the invention

[0005] The purpose of the present invention is to provide a load transfer method for an electric, heat and gas integrated energy system based on reinforcement learning. Since the load types and flexibility in different areas of the integrated energy system are different, and it is difficult to formulate accurate and real-time load transfer strategies when uncertain events occur in the integrated energy system, this method mainly learns a large number of integrated energy system load transfer strategies through neural networks.

[0006] The purpose of the present invention is to provide a load transfer method for an electric, heat and gas integrated energy system based on reinforcement learning to make up for the deficiencies of the prior art. The goal of the present invention is to minimize the system operating cost while ensuring the stable operation of the system load when an uncertain event occurs in the integrated energy system. A model-based and data-driven approach is adopted. By establishing an integrated energy system model containing different load types, the limitations of traditional integrated energy system modeling are overcome, and the complex operating state of the integrated energy system can be accurately described; the integrated energy system data is trained through reinforcement learning to achieve a fast and accurate load transfer strategy in highly complex scenarios. The specific steps of the method are as follows:

[0007] S1, establish an electric, heat and gas integrated energy system model containing different load types from the actual integrated energy system, and collect system load transfer related data after different uncertain events occur, and use the collected data as training samples;

[0008] S2, build an agent-environment model to simulate the operating environment of the integrated energy system. In this environment, constraints are imposed on the actions of the agent based on the operating constraints of the integrated energy system to ensure the physical laws and actual conditions of the system's operating load;

[0009] S3, build a reinforcement learning model, clarify the state function, action function, and state transition variables in the algorithm agent, and formulate a reporting function by designing a reward and punishment mechanism to optimize the agent strategy;

[0010] S4, establish a load transfer triggering mechanism for different uncertain events in the integrated energy system, clarify various uncertain events that occur in the integrated energy system, so as to enable the intelligent agent to make load transfer strategies more efficiently and accurately for different uncertain events;

[0011] S5, training the agent, updating the agent strategy through a large amount of load transfer data in the integrated energy system, using the Dueling Deep-Q-Learning (DDQN) algorithm for training, and generating the optimal load transfer strategy for the integrated energy system under different uncertain events after multiple iterations of training the agent;

[0012] S6, uses the trained reinforcement learning model to formulate a real-time and accurate load transfer strategy for the integrated energy system.

[0013] Furthermore, in said S1,

[0014] The integrated energy system modeling part includes: power system line model, heat network, natural gas network pipeline model, compressor model; power-to-gas, power-to-heat, and cogeneration coupling equipment models between various systems.

[0015] Pipeline Model:

[0016] Natural Gas Pipeline Model:

[0017]

[0018] Where: f ij,t is the steady-state flow rate of natural gas pipeline i, j; K ij is the pipeline constant, which depends on many factors such as the inner diameter, length, friction coefficient and temperature of the pipeline; ρ i,t , j,t are the pressures at nodes i and j respectively; s ij Characterizes the direction of natural gas flow, when ρ i,t >ρ j,t +1 if yes, -1 otherwise.

[0019] Thermal pipe model:

[0020]

[0021] In the formula, Q loss (t) is the total heat loss of the pipe at time t, Q loss,i (t) is the heat loss of pipeline segment i, and N is the total number of pipeline segments in the heating network.

[0022]

[0023] In the formula, ΔP is the pressure loss of the pipeline section; f is the friction factor, which is usually related to the flow velocity, pipeline roughness and fluid properties; L is the length of the pipeline section; D is the diameter of the pipeline; ρ is the density of the fluid; and v is the flow velocity of the fluid.

[0024] Compressor Model:

[0025]

[0026] R com,i,j =ρ j,t / ρ i,t (5)

[0027]

[0028] Where: W com,ij,t is the power consumed by the compressor; f com,j,t,t is the air flow through the compressor; B com,ij,t 、z cm , i,t is the compressor parameter, which depends on the compressor temperature, efficiency and compression factor, etc.; R com,i,j is the compression ratio; τ com,ij is the gas flow consumed by the gas turbine; α com , β com , γ com is the energy conversion efficiency constant.

[0029] Power-to-gas model:

[0030] f P2G,itt =η P2G P P2G,i,t / G HV (7)

[0031] In the formula η P2G is the conversion efficiency of electricity to gas; G HV The high calorific value of natural gas is 39MJ / m3.

[0032] Electric boiler model:

[0033] H EB (t) = μ EB (t)·η EB ·P EB (t) (8)

[0034] In the formula, μ EB (t) is the start and stop state of the electric boiler; η EB is the electrothermal conversion efficiency.

[0035] Gas boiler:

[0036] H GB (t) = ηGB ·G GB (t)·LHV (9)

[0037] In the formula η GB is the gas-to-heat conversion efficiency coefficient of the gas boiler.

[0038] Modeling of different types of loads:

[0039] P i (t) = P i,0 ·f(t) (10)

[0040] In the formula, P i,0 is the base power of the equipment; f(t) is a function that describes load changes, which is usually related to the equipment's operating cycle, load changes, and process requirements. For example, f(t) can be a time-based sinusoidal function to simulate periodic fluctuations, or a function driven by process requirements (such as production volume, production time, etc.).

[0041] The relevant data collected include: the topology of the integrated energy system, the network initial node voltage, pressure and node temperature after uncertain events such as load surge reduction, line failure, line maintenance, photovoltaic and wind power generation, user power, heat and gas loads, real-time electricity prices, hourly information, and energy storage equipment information

[0042] Further, as described in S2:

[0043] S2-1: First, the goal is to achieve economic operation of the integrated energy system on the basis of ensuring stable operation of the system load when different uncertain events such as load surge and decrease, line failure, and line maintenance occur in the integrated energy system of electricity, heat, and gas. The load transfer strategy formulated focuses on load recovery and the minimum operating cost of the integrated energy system; the optimization goal can be described by the following formula:

[0044]

[0045] C E (t) = C C / E (t)+C M / E (t)+C S / E (t)+C B / E (t)

[0046] C H (t) = C C / H (t)+C M / H (t)+C S / H (t)+C B / H (t)

[0047] C G (t) = C C / G (t)+CM / G (t)+C S / G (t)+C B / G (t)

[0048] S2-2: Construct the operation constraints of the electric, thermal and gas integrated energy system. The integrated energy system couples the power system, thermal system and natural gas system, and constructs the energy flow balance constraints of each system pipeline network and the constraints of each energy coupling equipment;

[0049] Conditions for power system flow constraints:

[0050]

[0051] Where: P G,i,t , Q G,i,t is the active and reactive power of the generator set at node i at time t; P L,j,t , Q L,i,t is the active and reactive loads of node i at time t; P STO,i,t is the active power of random energy such as wind power and photovoltaic power; P P2G,i,t is the active power of the power-to-gas unit; P SOC,i,t U is the charging (discharging) power of the energy storage; i,t is the voltage modulus of node i; θ ij,t is the phase angle difference between the two nodes; G ij , B ij is the conductance and susceptance between nodes i and j.

[0052] Natural Gas System:

[0053]

[0054] In the formula, f N,i,t 、f L,i,t 、f P2G,i,t 、f GT,i,t are the natural gas source injection volume, natural gas load, P2G device injection volume and gas turbine gas consumption of node i at time t; f s,in,i,t 、f s,out,i,t is the natural gas injection and output flow of the gas storage tank; when node i is the compressor inlet, s com,j,i,t Take +1, otherwise take -1.

[0055] Thermal system:

[0056] H buy (t)+H GB (t)+H EB (t)+H dis (t) = H cha (t)+H load (t)

[0057] Where: H buy (t) is the heat purchased from the outside by the park; H GB (t), H GB (t) are the thermal output power of gas boiler and electric boiler respectively; H cha (t), H dis (t) are the heat storage power and heat release power of the heat storage equipment; H load (t) is the heat load demand, the unit is kW.

[0058] Power-to-gas equipment:

[0059] f N,i,min ≤f N,i,t ≤f N,i,max (13)

[0060] Energy storage equipment:

[0061]

[0062] in the formula Respectively λ The maximum charging and discharging rate of the energy storage device; Respectively λ The minimum and maximum load states of energy storage equipment; C λ is the rated storage capacity of the energy storage device; E λ (0), E λ (T) are the storage amounts at the start and end of the energy storage period, respectively.

[0063] Furthermore, the S3 specifically includes:

[0064] Neural network construction. Using the agent-environment training framework, the DDQN algorithm is used to establish the input and output model of the system, the agent strategy network and the value network. The input parameters are environmental variables, and the output parameters are the integrated energy system network contact switch contact valve control strategy.

[0065] Furthermore, the strategy network of the above neural network model is configured, and a three-layer neural network is used, including an input layer, a hidden layer, and an output layer. The input layer includes 128 nodes, the hidden layer is 256 nodes, and the output layer is 128 nodes. The input variables of the input layer are the node voltage, node pressure, set node temperature, electric and thermal load demand, photovoltaic and wind power generation, and electricity price information at the current moment after the failure of the integrated energy system. The ReLU function is selected as the activation layer, and the output layer is the switching action of each interconnection switch, segmented switch, and interconnection valve in the integrated energy system at the next moment. The loss function of the strategy network is optimized based on the stochastic gradient descent method, and the Adam optimizer is used to find the optimal solution of gradient descent.

[0066] Furthermore, the DDQN model specifically models the control strategy problem as a Markov decision process:

[0067] (4) The observation state S is set to describe the state information characteristics of the intelligent agent at time t. These characteristics are the node voltage, air pressure, temperature, electric, thermal and gas load demand, photovoltaic and wind power generation, and electricity price information of the integrated energy system at the current moment.

[0068] (5) Setting of action space A: The action space of the intelligent agent is the switch action of the interconnecting switch, section switch of the power system and the interconnecting valve in the natural gas and heat system in the integrated energy system. Each switch action is (0,1), where 0 means the switch is open and 1 means the switch is closed.

[0069] (6) Setting of reward function R. The present invention adopts a design of reward function that includes the safety and economy of the integrated energy system. That is, in the process of training the agent, the safety and economy of the agent's actions for the integrated energy system are taken into account. The proposed reward function consists of two parts: 1) the load recovery amount of the integrated energy system, and 2) the minimum operating cost of the integrated energy system. The reward function is defined as follows:

[0070]

[0071] In the formula: N op is the total number of actions; P res,k C is the load restored by the kth action; C / E (t), C M / E (t), C S / E (t), C B / E (t) are the initial construction cost of the power system, operation and maintenance cost, unit start-up and shutdown cost, and cost of purchasing electricity from the external main grid; C C / H (t), C M / H (t), C S / H (t), C B / H (t) are the initial construction cost depreciation, operation and maintenance cost, unit start-up and shutdown cost and external heat purchase cost of the thermal system. The calculation formulas for the initial construction cost depreciation, operation and maintenance cost and unit start-up and shutdown cost can be compared with the power system; C C / G (t), C M / G (t), C S / G (t), C B / G (t) are the depreciation of the initial construction cost, operation and maintenance cost, unit start-up and shutdown cost, and external gas purchase cost of the gas supply system. The calculation formulas for the initial construction cost, operation and maintenance cost, and unit start-up and shutdown cost are analogous to those of the power system.

[0072] Furthermore, the transfer trigger condition mechanism mentioned in S4 is as follows: in each time step, the intelligent agent simulates the comprehensive energy system result when there is no operation, and uses the trigger condition W defined by formula (16) to determine whether the intelligent agent should act.

[0073] W=W1+W2+W3 (16)

[0074] In the formula, W is the load surge and decrease sign, W2 is the fault sign, and W3 is the maintenance sign, which correspond to different event drivers of load transfer.

[0075]

[0076] In the formula, LF is the power flow of line i, TL i is the limit value of line i, i is the line number in the system, and N is the total number of lines in the system. If the load level of a line in the power grid is higher than the predetermined threshold λ, the power flow over-limit flag is set to 1.

[0077] The calculation formulas for W2 and W3 are as follows:

[0078]

[0079] By adding a transfer triggering condition mechanism, when any of the different types of uncertain events occur in the system, the intelligent agent will generate a corresponding load transfer strategy and perform corresponding operations on the integrated energy system to improve the accuracy of the load transfer strategy.

[0080] Furthermore, for the DDQN training model described in S3 and S4: the agent continuously updates the weights of the policy network and the value network according to the scope of the environment, the agent continuously interacts with the environment, continuously trains the strategy, and thus formulates the optimal strategy. During the training process, the cumulative reward curve is used to reflect the training progress and intuitively reflect the training status and convergence status of the agent.

[0081] Select the number of training times as N, the length of the sample time series in data collection as T, the state space as S, and the action space as A:

[0082] Step 1: Initialize the weights of the strategy network and the value network respectively;

[0083] Step 2: The agent obtains the samples (time, load, etc.) generated by each operation;

[0084] Step 3: The agent performs the current operation at (device output) and obtains the reward for the current operation;

[0085] Step 4: The agent executes the transfer starting conditions

[0086] Step 5: The agent performs the switch action;

[0087] Step 6: Randomly extract data from the experience playback area;

[0088] Step 7: Input parameter generation in the policy network;

[0089] Step 8: Calculate and update the network using stochastic gradient descent method;

[0090] Step 9: Input parameters into the value network to generate targets;

[0091] Step 11: Update the value network.

[0092] Compared with the prior art, the advantages and benefits of the present invention are:

[0093] The present invention proposes a load transfer strategy method for an electric, heat and gas integrated energy system based on reinforcement learning, which overcomes the limitations of traditional load transfer methods for electric, heat and gas integrated energy systems containing multiple different types of loads, such as long calculation time and low strategy fault tolerance, accurately describes the complex dynamic behavior of the electric, heat and gas integrated energy system, and realizes real-time and accurate load transfer strategies for electric, heat and gas integrated energy systems containing different types of loads.

[0094] The present invention proposes a load transfer strategy method for an electric, heat and gas integrated energy system based on reinforcement learning, which overcomes the limitations of traditional load transfer methods for electric, heat and gas integrated energy systems containing multiple different types of loads, such as long calculation time and low strategy fault tolerance, accurately describes the complex dynamic behavior of the electric, heat and gas integrated energy system, and realizes real-time and accurate load transfer strategies for electric, heat and gas integrated energy systems containing different types of loads. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] Figure 1 This is a diagram of the basic principles of reinforcement learning.

[0096] Figure 2 It is a framework structure diagram of the present invention.

[0097] Figure 3 It is a structural diagram of the electric, heat and gas integrated energy system.

[0098] Figure 4 This is the comprehensive energy system topology adopted by the present invention.

[0099] Figure 5 This is the flow chart of the DDQN algorithm.

[0100] Figure 6 It is the DDQN model training curve. DETAILED DESCRIPTION

[0101] The technical solution of the present invention is further described below in conjunction with the embodiments. This method expresses the load transfer strategy process of the integrated energy system as a Markov decision process, which can effectively handle the complex switching problems of the load transfer process of the electric, heat and gas integrated energy system containing different types of loads. Reinforcement learning continuously interacts and tries errors in the intelligent agent Yuhua environment, thereby learning the optimal load transfer strategy from complex strategies. At the same time, the trained reinforcement learning model has good generalization ability and can adapt to different environments through a large amount of data-driven learning without the need for detailed physical models, which greatly simplifies the complexity of modeling. A load transfer path method for an electric, heat and gas integrated energy system based on reinforcement learning is shown in the figure. The method comprises the following steps:

[0102] (1) The data in this example comes from the integrated energy system of China Electric, Heat and Gas in California, USA. The equipment model includes the network pipeline model, power-to-gas equipment, electric boilers, compressors, and gas turbines; the collected data includes the topology of the integrated energy system, the network initial node voltage, pressure, and node temperature after uncertain events such as load surge reduction, line failure, and line maintenance, photovoltaic and wind power generation, user power, heat, and gas loads, real-time electricity prices, hourly information, and energy storage equipment information.

[0103] (2) Construct an agent-environment model to simulate the operating environment of the integrated energy system. In this environment, constraints are imposed on the actions of the agent based on the operating constraints of the integrated energy system to ensure that the physical laws and actual conditions of the system's operating load are met:

[0104] Conditions for power system flow constraints:

[0105]

[0106] Natural Gas System:

[0107]

[0108] Thermal system:

[0109] H buy (t)+H GB (t)+H EB (t)+H dis (t) = H cha (t)+H load (t)

[0110] Power-to-gas equipment:

[0111] f N,i,min ≤f N,i,t ≤f N,i,max (twenty one)

[0112] Energy storage equipment:

[0113]

[0114] (3) Build a reinforcement learning model, clarify the state function, action function, and state transition variables in the algorithm agent, and formulate a reporting function by designing a reward and punishment mechanism to optimize the agent strategy;

[0115] Construction of neural network. Adopting the agent-environment training framework, through the DDQN algorithm, the input and output model of the system is established, and the agent strategy network and value network are established. The input parameters are environmental variables, and the output parameters are the control strategy of the network contact switch and contact valve of the integrated energy system. A three-layer neural network is adopted, with input layer, hidden layer, and output layer. The input layer includes 128 nodes, the hidden layer is 256 nodes, and the output layer is 128 nodes. The input variables of the input layer are the node voltage, node pressure, set node temperature, electric and thermal gas load demand, photovoltaic and wind power generation, and electricity price information at the current moment after the integrated energy system fails. The ReLU function is selected as the activation layer, and the output layer is the switch action of each contact switch, segment switch, and contact valve in the integrated energy system at the next moment. The loss function of the strategy network is optimized based on the stochastic gradient descent method, and the Adam optimizer is used to find the optimal solution of gradient descent.

[0116] The DDQN model specifically models the control strategy problem as a Markov decision process:

[0117] (1) The observation state S is set to describe the state information characteristics of the intelligent agent at time t. These characteristics are the node voltage, air pressure, temperature, electric, thermal and gas load demand, photovoltaic and wind power generation, and electricity price information of the integrated energy system at the current moment.

[0118] (2) Setting of action space A: The action space of the intelligent agent is the switch action of the interconnecting switch and section switch of the power system and the interconnecting valve in the natural gas and heat system in the comprehensive energy system. Each switch action is (0,1), where 0 means the switch is open and 1 means the switch is closed.

[0119] (3) Setting of reward function R. The present invention adopts a design of reward function that includes the safety and economy of the integrated energy system. That is, in the process of training the agent, the safety and economy of the agent's actions for the integrated energy system are taken into consideration. The proposed reward function consists of two parts: 1) the load recovery amount of the integrated energy system, and 2) the minimum operating cost of the integrated energy system. The reward function definition

[0120] as follows:

[0121]

[0122] C E (t) = CC / E (t)+C M / E (t)+C S / E (t)+C B / E (t)

[0123] C H (t) = C C / H (t)+C M / H (t)+C S / H (t)+C B / H (t)

[0124] C G (t) = C C / G (t)+C M / G (t)+C S / G (t)+C B / G (t)

[0125] (4) Establish a load transfer triggering mechanism for different uncertain events in the integrated energy system, clarify the various uncertain events that occur in the integrated energy system, and enable the intelligent agent to make load transfer strategies more efficiently and accurately for different uncertain events;

[0126] Finally, the DDQN algorithm is used for training through the stochastic gradient descent method. The agent continuously updates the weights of the policy network and the value network according to the scope of the environment. The agent continuously interacts with the environment and continuously trains the strategy to formulate the optimal strategy. During the training process, the cumulative reward curve is used to reflect the training progress and intuitively reflect the training and convergence of the agent. After the model training is completed, the model is evaluated using the test environment. The evaluation process includes multiple test rounds to measure the stability of the model. Focus on observing whether the agent can make fast and accurate decisions in an uncertain environment that has not occurred before. If the model performance is not ideal during the evaluation process, the model can be tuned by adjusting the hyperparameters. For example, the learning rate, the number of layers of the neural network, the number of neurons in the hidden layer, the exploration rate, etc. can be adjusted to further improve the model performance.

Claims

1. A research method for load transfer of electric, thermal and gas integrated energy system based on reinforcement learning is proposed. The method includes the following steps: S1, establish an electric, heat and gas integrated energy system model containing different load types from the actual integrated energy system, and collect system load transfer related data after different uncertain events occur, and use the collected data as training samples; S2, build an agent-environment model to simulate the operating environment of the integrated energy system. In this environment, constraints are imposed on the actions of the agent based on the operating constraints of the integrated energy system to ensure the physical laws and actual conditions of the system's operating load; S3, build a reinforcement learning model, clarify the state function, action function, and state transition variables in the algorithm agent, and formulate a reporting function by designing a reward and punishment mechanism to optimize the agent strategy; S4, establish a load transfer triggering mechanism for different uncertain events in the integrated energy system, clarify the various uncertain events that occur in the integrated energy system, so that the intelligent agent can make load transfer strategies more efficiently and accurately for different uncertain events; S5, training the agent, updating the agent strategy through a large amount of load transfer data in the integrated energy system, using the Dueling Deep-Q-Learning (DDQN) algorithm for training, and generating the optimal load transfer strategy for the integrated energy system under different uncertain events after multiple iterations of training the agent; S6, uses the trained reinforcement learning model to formulate a real-time and accurate load transfer strategy for the integrated energy system.

2. The integrated energy system modeling part includes: power system line model, heat network, natural gas network pipeline model, compressor model; power-to-gas, power-to-heat, and cogeneration coupling equipment models between various systems, Natural Gas Pipeline Model: Where: f ij,t is the steady-state flow rate of natural gas pipeline i, j; K ij is the pipeline constant, which depends on many factors such as the inner diameter, length, friction coefficient and temperature of the pipeline; ρ i,t , j,t are the pressures at nodes i and j respectively; s ij Characterizes the direction of natural gas flow, when ρ i,t >ρ j,t +1 if yes, -1 otherwise. Thermal Pipe Model: In the formula, Q loss (t) is the total heat loss of the pipe at time t, Q loss,i (t) is the heat loss of pipeline segment i, and N is the total number of pipeline segments in the heating network. In the formula, ΔP is the pressure loss of the pipeline section; f is the friction factor, which is usually related to the flow velocity, pipeline roughness and fluid properties; L is the length of the pipeline section; D is the diameter of the pipeline; ρ is the density of the fluid; and v is the flow velocity of the fluid. Compressor Model: R com,i,j =ρ j,t / r i,t (5) Where: W com,ij,t is the power consumed by the compressor; f com,j,t,t is the air flow through the compressor; B com,ij,t 、z cm, i,t are compressor parameters, which depend on compressor temperature, efficiency, and compression factor; R com,i,j is the compression ratio; τ com,ij is the gas flow consumed by the gas turbine; α com , β com , γ com is the energy conversion efficiency constant. Power-to-gas model: f P2G,itt Hη P2G P.S P2G,i,t / G HV (7) In the formula η P2G is the conversion efficiency of electricity to gas; G HV The high calorific value of natural gas is 39MJ / m3. Electric boiler model: H EB (t)=μ EB (t)·h EB ·P EB (t) (8) In the formula, μ EB (t) is the start and stop state of the electric boiler; η EB is the electrothermal conversion efficiency. Gas boiler: H GB (t)=η GB ·G GB (t)·LHV (9) In the formula: η GB is the gas-to-heat conversion efficiency coefficient of the gas boiler. Modeling of different types of loads: P i (t)=P i,0 ·f(t) (10) In the formula, P i,0 is the base power of the equipment; f(t) is a function that describes load changes, which is usually related to the equipment's operating cycle, load changes, and process requirements. For example, f(t) can be a time-based sinusoidal function to simulate periodic fluctuations, or a function driven by process requirements (such as production volume, production time, etc.). The relevant data collected include: the topology of the integrated energy system, the network's initial node voltage, pressure and node temperature after uncertain events such as load surge reduction, line failure, line maintenance, photovoltaic and wind power generation, user electricity, heat and gas loads, real-time electricity prices, hourly information, and energy storage equipment information. 3.S2-1: First, the goal is to achieve economic operation of the integrated energy system on the basis of ensuring stable operation of the system load when different uncertain events such as load surge and decrease, line failure, and line maintenance occur in the integrated energy system of electricity, heat, and gas. The load transfer strategy formulated focuses on the load recovery amount and the minimum operating cost of the integrated energy system; the optimization goal can be described by the formula: C E (t)=C C / E (t)+C M / E (t)+C S / E (t)+C B / E (t) C H (t)=C C / H (t)+C M / H (t)+C S / H (t)+C B / H (t) C G (t)=C C / G (t)+C M / G (t)+C S / G (t)+C B / G (t) S2-2: Construct the operation constraints of the electric, thermal and gas integrated energy system. The integrated energy system couples the power system, thermal system and natural gas system, and constructs the energy flow balance constraints of each system pipeline network and the constraints of each energy coupling equipment; Conditions for power system flow constraints: Where: P G,i,t , Q G,i,t is the active and reactive power of the generator set at node i at time t; P L,j,t , Q L,i,t is the active and reactive loads of node i at time t; P STO,i,t is the active power of random energy such as wind power and photovoltaic power; P P2G,i,t is the active power of the power-to-gas unit; P SOC,i,t U is the charging (discharging) power of the energy storage; i,t is the voltage modulus of node i; θ ij,t is the phase angle difference between the two nodes; G ij , B ij is the conductance and susceptance between nodes i and j. Natural Gas System: In the formula, f N,i,t 、f L,i,t 、f P2G,i,t 、f GT,i,t are the natural gas source injection volume, natural gas load, P2G device injection volume and gas turbine gas consumption of node i at time t; f s,in,i,t 、f s,out,i,t is the natural gas injection and output flow of the gas storage tank; when node i is the compressor inlet, s com,j,i,t Take +1, otherwise take -1. Thermal system: H buy (t)+H GB (t)+H EB (t)+H dis (t)=H cha (t)+H load (t) Where: H buy (t) is the heat purchased from the outside by the park; H GB (t), H GB (t) are the thermal output power of gas boiler and electric boiler respectively; H cha (t), H dis (t) are the heat storage power and heat release power of the heat storage equipment; H load (t) is the heat load demand, the unit is kW. Power-to-gas equipment: f N,i,min ≤f N,i,t ≤f N,i,max (13) Energy storage equipment: in the formula Respectively λ The maximum charging and discharging rate of the energy storage device; Respectively λ The minimum and maximum load states of energy storage equipment; C λ is the rated storage capacity of the energy storage device; E λ (0), E λ (T) are the storage amounts at the start and end of the energy storage period, respectively.

4. Claim 1 includes: Construction of neural network. Adopting the agent-environment training framework, through the DDQN algorithm, the input and output model of the system is established, and the agent strategy network and value network are established. The input parameters are environmental variables, and the output parameters are the control strategy of the network contact switch and contact valve of the integrated energy system. A three-layer neural network is adopted, with input layer, hidden layer, and output layer. The input layer includes 128 nodes, the hidden layer is 256 nodes, and the output layer is 128 nodes. The input variables of the input layer are the node voltage, node pressure, set node temperature, electric and thermal gas load demand, photovoltaic and wind power generation, and electricity price information at the current moment after the integrated energy system fails. The ReLU function is selected as the activation layer, and the output layer is the switch action of each contact switch, segment switch, and contact valve in the integrated energy system at the next moment. The loss function of the strategy network is optimized based on the stochastic gradient descent method, and the Adam optimizer is used to find the optimal solution of gradient descent. The DDQN model specifically models the control strategy problem as a Markov decision process: (1) The observation state S is set to describe the state information characteristics of the intelligent agent at time t. These characteristics are the node voltage, air pressure, temperature, electric, thermal and gas load demand, photovoltaic and wind power generation, and electricity price information of the integrated energy system at the current moment. (2) Setting of action space A: The action space of the intelligent agent is the switch action of the interconnecting switch and section switch of the power system and the interconnecting valve in the natural gas and heat system in the comprehensive energy system. Each switch action is (0,1), where 0 means the switch is open and 1 means the switch is closed. (3) Setting of reward function R. The present invention adopts a design of reward function that includes the safety and economy of the integrated energy system. That is, in the process of training the intelligent agent, the safety and economy of the intelligent agent's actions for the integrated energy system are taken into consideration. The proposed reward function consists of two parts: 1) the load recovery amount of the integrated energy system, and 2) the minimum operating cost of the integrated energy system.

5. The transfer trigger condition mechanism mentioned in claim 1: In each time step, the intelligent agent simulates the results of the integrated energy system when there is no operation, and uses the trigger condition W defined by formula (15) to determine whether the intelligent agent should act. W=W1+W2+W3 (15) In the formula, W is the load surge and decrease sign, W2 is the fault sign, and W3 is the maintenance sign, which correspond to different event drivers of load transfer. In the formula, LF is the power flow of line i, TL i is the limit value of line i, i is the line number in the system, and N is the total number of lines in the system. If the load level of a line in the power grid is higher than the predetermined threshold λ, the power flow over-limit flag is set to 1. The calculation formulas for W2 and W3 are as follows: By adding a transfer triggering condition mechanism, when any of the different types of uncertain events occur in the system, the intelligent agent will generate a corresponding load transfer strategy and perform corresponding operations on the integrated energy system to improve the accuracy of the load transfer strategy.

6. The DDQN training model in claim 1: The agent continuously updates the weights of the policy network and the value network according to the scope of the environment, and the agent continuously interacts with the environment and continuously trains the strategy to formulate the optimal strategy. During the training process, the cumulative reward curve is used to reflect the training progress and intuitively reflect the training status and convergence status of the agent. Select the number of training times as N, the length of the sample time series in data collection as T, the state space as S, and the action space as A: Step 1: Initialize the weights of the strategy network and the value network respectively; Step 2: The agent obtains the samples (time, load, etc.) generated by each operation; Step 3: The agent performs the current operation at (device output) and obtains the reward for the current operation; Step 4: The agent executes the transfer starting conditions Step 5: The agent performs the switch action; Step 6: Randomly extract data from the experience playback area; Step 7: Input parameter generation in the policy network; Step 8: Calculate and update the network using stochastic gradient descent method; Step 9: Input parameters into the value network to generate targets; Step 11: Update the value network.

Images