Electric-thermal integrated energy system energy management strategy based on noise network and self-attention mechanism DQN-CE algorithm

Abstract

The invention discloses an energy management strategy of an electric-thermal integrated energy system based on a noise network and a self-attention mechanism DQN-CE algorithm. The energy management strategy comprises the following steps: establishing an electric-thermal integrated energy system model; designing a reinforcement learning state space St, an action space At and a reward function R according to variables and constraints on the basis of the established electric-thermal integrated energy system model; a current Q network and a target Q network are established based on a noise network and a self-attention mechanism DQN-CE algorithm, the current Q network is updated through an improved loss function, and then parameters of the current Q network are periodically copied to the target Q network; based on the current Q network, the target Q network and the improved loss function, the current Q network model and the target Q network model are trained through interaction with the electric-heat integrated energy system model, and stable and economical operation of the electric-heat integrated energy system is achieved. The model trained by the method can respond to the uncertainty of renewable energy sources in real time and manage the energy of the electric-thermal integrated energy system containing the renewable energy sources on line.

Description

Technical Field

[0001] The present invention relates to the technical field of energy management of an electric-thermal integrated energy system, and in particular to an energy management strategy for an electric-thermal integrated energy system based on a NoisyNet and a Deep Q network cross entropy (DQN-CE) algorithm with a self-attention mechanism. Background Art

[0002] Integrated energy systems can effectively improve energy utilization through multi-energy coupling, thus providing an effective means to solve energy and environmental problems. However, the uncertainty of renewable energy and multi-energy loads has brought challenges to the energy management of integrated energy systems. At present, the methods for dealing with source and load uncertainty mainly include robust optimization, stochastic optimization and intelligent algorithms. Robust optimization methods are highly robust because they consider the worst scenario of uncertain variables, but the optimization results are relatively conservative and have poor economic performance. Stochastic optimization methods are heavily dependent on the probability distribution of random variables, requiring modelers to have considerable skills and experience, resulting in inaccurate descriptions of uncertainty. Algorithms such as particle swarms and whales are model predictive control methods. They are relatively dependent on the accuracy of parameters and models. Simulation results are usually greatly affected by the prediction of uncertain factors. Since uncertainty changes over time, models, predictors and solvers must be adjusted and redesigned accordingly. Deep reinforcement learning algorithms are data-driven and do not rely on specific models, which can effectively deal with the source and load uncertainty problems of integrated energy systems.

[0003] The traditional deep Q network (DQN) algorithm has the disadvantages of slow convergence, easy to fall into local optimality, and overestimation. In the early stage of training, the DQN algorithm requires a lot of exploration to collect enough data for learning. However, the exploration strategy (ε-greedy strategy) usually leads to the selection of a large number of suboptimal actions, which slows down the convergence speed.

[0004] The traditional DQN algorithm solves the problem of comprehensive energy management of electric and thermal energy. The algorithm's own overestimation problem may cause the neural network to choose suboptimal actions during the training phase, resulting in increased training time and unnecessary time costs. When updating the Q network, the output of the current Q network is used to estimate the target Q value. This method will lead to an overestimation of the Q value because in each update, the deep Q network algorithm may tend to choose those actions that currently appear to be optimal. Although the deep Q network algorithm introduces a target Q network to stabilize training, the update lag of the target Q network still leads to the accumulation of estimation errors. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides an energy management strategy for an electric-thermal integrated energy system based on a noise network and a self-attention mechanism DQN-CE algorithm, with the goal of minimizing the operating cost of the electric-thermal integrated energy system to achieve energy management of the integrated energy system. The model trained by this method can respond to the uncertainty of renewable energy in real time and manage the energy of the electric-thermal integrated energy system containing renewable energy online.

[0006] The technical solution adopted by the present invention is:

[0007] The energy management strategy of the electric-thermal integrated energy system based on the noise network and self-attention mechanism DQN-CE algorithm includes the following steps:

[0008] Step 1: Establish an electric-thermal integrated energy system model, in which the energy supply side includes photovoltaic, wind turbine and natural gas sources; the energy conversion device includes combined heat and power unit (CHP), electric boiler (EB) and gas boiler (GB); the energy demand side includes electric load and thermal load;

[0009] Step 2: Based on the electric-thermal integrated energy system model established in step 1, clarify the system variables and constraints; build a deep reinforcement learning training model framework based on the variables and constraints, that is, design the reinforcement learning state space S t , action space A t , reward function R;

[0010] Step 3: Based on the noise network and self-attention mechanism DQN-CE algorithm, build the current Q network and the target Q network. First, update the current Q network through the improved loss function, and then periodically copy the parameters of the current Q network to the target Q network.

[0011] Step 4: Based on the current Q network, the target Q network and the improved loss function, the current Q network and the target Q network model are trained by interacting with the electric-thermal integrated energy system model, so that they can learn to make the best decisions in different situations to maximize the reward function and achieve stable and economic operation of the electric-thermal integrated energy system.

[0012] In the step 1, an electric-thermal integrated energy system model is established, specifically including:

[0013] 1.1. Constant temperature control load:

[0014] Thermostatic control load refers to the load that controls the indoor temperature through thermostatic equipment. The change of working state is achieved by switching on and off. At time t, the action u of the i-th thermostatic control load i,control,t Defined as:

[0015]

[0016] In formula (1): T i,t is the current ambient temperature; T i,max and T i,min are the highest and lowest temperatures respectively; u i,control For the action of constant temperature control load, the present invention sets 2 gears, and the value range of subscript i is i∈N TCLs , N TCLs The number of thermostatically controlled loads.

[0017] The power load P of the i-th constant temperature control load at time t TCL,i,t The expression is:

[0018] P TCL,i,t =S TCL u i,control,t (2);

[0019] Where: S TCL is the nominal power of the thermostatically controlled load.

[0020] 1.2. Price Response Load:

[0021] Price-responsive load refers to the load that cannot be directly controlled in the electric-thermal integrated energy system. It is affected by the electricity price and its load expression is:

[0022] P load,v,t =P b,t -P v,SL,t +P v,PB,t (3);

[0023] In formula (3): P load,v,t is the load power of the vth price-responsive load at time t; p b,t is the base load power at time t, P v,SL,t is the transfer load power at time t; P v,PB,t is the redemption load power at time t;

[0024] P v,SL,t =β v δ t P b,t , where β v ∈[0,1] is the sensitivity coefficient of the vth price-responsive load; δ t ∈[-2,-1,0,1,2] is the price level at time t, which directly determines the transfer load P v,SL,t ;

[0025] Among them, ω v,j is the state variable of whether v transfers the load from time j; the value range of subscript v is v∈N loads , N loadsis the quantity of price-responsive load; t is an hour in a 24-hour day.

[0026] 1.3 Energy storage components:

[0027] The battery expression is:

[0028]

[0029] In formula (4): The amount of electricity stored in the battery for the next hour; The amount of electricity stored in the battery; η bt The charging and discharging efficiency of the battery; P t bt,dis is the discharge power of the battery at time t; P t bt,chr is the charging power of the battery at time t, is the maximum storage capacity of the battery; is the state variable of the battery, is the maximum charge and discharge power of the battery, It is the minimum charging and discharging power of the battery.

[0030] The heat storage tank expression is:

[0031]

[0032] In formula (5): The heat stored in the heat storage tank for the next hour, is the heat stored in the heat storage tank at time t; η ts The charging and discharging efficiency of the heat storage tank; and The heat stored and released by the heat storage tank respectively; is the maximum storage capacity of the heat storage tank; is the state variable of the heat storage tank, is the maximum charging and discharging power of the heat storage tank, is the minimum charging and discharging power of the heat storage tank.

[0033] 1.4. Energy conversion device:

[0034] The energy conversion device converts different energies through electric boilers, gas boilers and cogeneration units to achieve economic regulation of the electric-thermal integrated energy system. The conversion constraint expressions of electric boilers, gas boilers and cogeneration units are as follows:

[0035]

[0036] In formula (6): P chp,t is the electrical output of the cogeneration unit at time t; chpis the gas-to-electricity conversion efficiency of the cogeneration unit; Q chp,t is the gas volume of the cogeneration unit at time t; H chp,t is the thermal output of the cogeneration unit at time t; k chp is the heat-to-electricity ratio of the cogeneration unit; and are the upper and lower limits of the output of the cogeneration unit respectively; P eb,t is the electric output of the electric boiler at time t; P gb,t is the electrical output of the gas boiler at time t; k eb , k gb They are the electricity-heat conversion efficiency of the heat pump and the gas-heat conversion efficiency of the gas boiler; and They are the upper and lower limits of the heat pump output respectively; and H is the upper and lower limits of the gas boiler output; chp,t , H eb,t and H gb,t are the thermal outputs of the cogeneration unit, heat pump and gas boiler at time t respectively.

[0037] In the step 2,

[0038] 2.1: The power balance constraint of the electric-thermal integrated energy system is expressed as follows:

[0039]

[0040]

[0041] Q gas,t =Q chp,t +Q gb,t (9);

[0042] Where: P grid,t is the power purchased from the external power grid at time t; P pv,t is the power generation of the photovoltaic power generation equipment at time t; P w,t P is the power generation capacity of the wind turbine generator at time t; chp,t is the power output of the cogeneration unit at time t; P t bt,dis is the discharge power of the battery at time t; P t bt,chr is the charging power of the battery at time t; P eb,t is the electric output of the electric boiler at time t; P TCL,i,t P is the electrical load of the constant temperature control load at time t; load,v,t is the electricity load of the price response load at time t; N TCLs is the number of constant temperature control loads; N loadsis the number of price-responsive loads; i is N TCLs The price response load in; v is N loads The price response load in H chp,t , H eb,t and H gb,t are the heat outputs of the combined heat and power unit, heat pump and gas boiler at time t; H load,t is the total thermal power at time t; H t ts,chr The heat stored in the heat storage tank; Q gas,t is the gas purchasing power of the natural gas source at time t; Q chp,t and Q gb,t are the gas power consumed by the cogeneration unit and the gas boiler at time t respectively.

[0043] 2.2: State space S of the electric-thermal integrated energy system t , which is expressed as follows:

[0044]

[0045] Where: price,t is the electricity price purchased from the power grid at time t; is the amount of electricity stored in the battery at time t; is the amount of heat stored in the heat storage tank at time t; t is an hour in a 24-hour day.

[0046] 2.3: Action space A of the electric-thermal integrated energy system t , which is expressed as follows:

[0047]

[0048] Where: a tcl 4 actions for constant temperature control load; a load 5 actions for price response load; P t bt is the power of the battery at time t; P t ts is the power of the heat storage tank at time t.

[0049] 2.4: The reward function R of the electric-thermal integrated energy system is expressed as follows:

[0050] R=C profile -C cost (12);

[0051]

[0052] C cost =ρ w P w,t △t+ρpv P pv,t △t+ρ price,t P grid,t +ρ gas Q gas,t (14);

[0053] Where: C profile The benefit obtained from executing the action; C cost is the cost required to perform the action; ρ load is the electricity price of the constant temperature control load; ρ TCL is the electricity price of the constant temperature control load; ρ w The cost price of generating electricity for wind power generation equipment; ρ pv The cost price of electricity generated by photovoltaic power generation equipment; ρ gas is the price of natural gas; Q gas,t is the amount of natural gas purchased from the external network at time t; △t is a certain hour in a 24-hour day.

[0054] In the step 3, based on the noise network and self-attention mechanism DQN-CE algorithm, the current Q network and the target Q network are built; the current Q network is a neural network used to generate an action value function, and the target Q network is used to calculate the target Q value. The target Q network has the same structure as the current Q network, but different parameters; the current Q network and the target Q network both include an input layer, a self-attention mechanism layer, a noise network layer, a noise network layer, and an output layer; first, the current Q network is updated by an improved loss function, wherein the improved loss function includes a mean square error loss and a cross entropy loss, and then, the parameters of the current Q network are periodically copied to the target Q network.

[0055] In step 3, the current Q network and target Q network structure in the noise network and self-attention mechanism DQN-CE algorithm are constructed, and Figure 3 The composition of the current Q network and the target Q network can be seen in detail. The self-attention mechanism layer is the second layer in the current Q network and the target Q network, and its expression is as follows:

[0056]

[0057] Where: Attention(Q,K,V) means mapping a query vector Q, a set of key-value pairs K and V into an output, and the output weight vector is calculated by weighting the values ​​and the softmax function. Q, K and V are the designed state feature vectors; K T is the transpose of the state eigenvector; d k is the dimension of the eigenvector.

[0058] The noise network layer is the third layer in the current Q network and the target Q network, and is derived from Figure 2The composition of the noise network layer can be seen in detail. ω is the weight of the neural network, ω = μ + σ × ε, where μ is the mean of the neural network parameter, σ is the standard deviation, ε is the random noise parameter, ε ~ N (0, 1), N (0, 1) represents a normal distribution with a mean of 0 and a standard deviation of 1. The improved loss function expression is as follows:

[0059]

[0060]

[0061] a t+1 =softmax(Q(s t+1 ,a,θ)) (18);

[0062]

[0063]

[0064]

[0065] Where: s t+1 is the state at the next moment; a is the action at the next moment; θ' is the parameter of the target Q network; Q' is the target Q network; Q is the current Q network; the softmax(x) function is a normalization function that converts a vector containing unnormalized values ​​into a normalized probability distribution vector; Perform a natural exponential function transformation on each element i of the input vector x; a t+1 is the next action of the current Q network; CrossEntropyLoss is the cross entropy loss function, which is used to measure the difference between the predicted probability distribution and the actual distribution; MSELoss is the mean square error loss function; is the probability value of the target action; m is the total number of samples; a target is the action of the target Q network; L DQN-CE is the improved loss function; is the target Q value of the i-th sample; is the current Q value of the i-th sample; The log probability that the predicted value matches the target value.

[0066] The step 4 includes the following steps:

[0067] Step 4.1: Based on steps 1 to 3, build the current Q network, the target Q network and the improved loss function, interact with the electric-thermal integrated energy system model, so as to train the current Q network, periodically copy the parameters of the current Q network to the target Q network, and save the current Q network model after the training is completed.

[0068] Step 4.2: By loading the trained current Q network model and testing data, a stable and economical operation level of the electric-thermal integrated energy system can be achieved.

[0069] The present invention provides an energy management strategy for an electric-thermal integrated energy system based on a noise network and a self-attention mechanism DQN-CE algorithm, and the technical effects are as follows:

[0070] 1) By establishing a mechanism model of the electric-thermal integrated energy system and using deep reinforcement learning, the system can more effectively allocate and utilize various energy resources, including wind power, photovoltaics, batteries, and main grid electricity. This helps to improve the overall efficiency of the system, reduce energy waste, and improve the economic benefits of the energy system.

[0071] 2) The model based on the noise network and self-attention mechanism DQN-CE algorithm in the present invention enables the system to make automated and intelligent decisions. By interacting with the electric-thermal integrated energy system environment, it continuously learns and improves its decisions to adapt to different operating conditions. This reduces the need for manual intervention and improves the autonomy of the system.

[0072] 3) Through intelligent scheduling and management of electrical resources, the present invention enables the system to autonomously learn and optimize energy production and distribution to ensure efficient operation of the system, reduce energy costs, and improve the stability and reliability of the energy system. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 This is a schematic diagram of the structure of the electric-thermal integrated energy system.

[0074] Figure 2 Schematic diagram of the noisy neural network structure.

[0075] Figure 3 Schematic diagram of the current Q network and target Q network structure.

[0076] Figure 4 This is a flow chart of energy management of an integrated energy system according to the algorithm of the present invention.

[0077] Figure 5 This is a reward convergence curve diagram during the training process of the present invention.

[0078] FIG6( a ) is a diagram showing the power system optimization results of the algorithm of the present invention;

[0079] FIG6( b ) is a diagram showing the results of the thermal system optimized by the algorithm of the present invention. DETAILED DESCRIPTION

[0080] The energy management strategy of the electric-thermal integrated energy system based on the noise network and self-attention mechanism DQN-CE algorithm includes establishing an integrated energy system model, in which the energy supply side includes photovoltaic, wind turbine and natural gas sources, the energy conversion device includes a combined heat and power unit (CHP), an electric boiler (EB) and a gas boiler (GB), and the energy demand side includes electric load and thermal load; according to the electric-thermal integrated energy system model, the system variables and constraints are clarified; according to the variables and constraints, a deep reinforcement learning training model framework is constructed, that is, a reinforcement learning state space S is designed. t , action space A t , reward function R;

[0081] Build the network structure of the noise network and self-attention mechanism DQN-CE algorithm, set the parameters, buffer size, and discount factor of the algorithm to train the network structure; through interaction with the electric-thermal integrated energy system, train the current Q network and target Q network model to enable them to learn how to make the best decisions in different situations to maximize the reward function, thereby achieving a stable economic operation level of the electric-thermal integrated energy system.

[0082] Example:

[0083] by Figure 1 Taking the electric-thermal integrated energy system shown in FIG. 1 as an example, the capacity / specifications of CHP, gas boiler, electric boiler, battery, heat storage tank, and gas storage tank are shown in Table 1 and Table 2. The current Q network and the target Q network both have 3 noise layers, each with 256 neurons. The initial value of the standard deviation of the noise network is set to 0.4. In order to avoid the gradient disappearance of the model during training, which leads to the convergence of the results to the local optimum, the activation function of the hidden layer is set to ReLU. In addition, the discount factor reflecting the importance of long-term returns is set to 0.95, the experience pool size is set to 10,000, the learning rate and mini-batch size are 128, and the Adam optimizer is used to update the weights.

[0084] Table 1 Energy conversion equipment operating parameters

[0085]

[0086] Table 2 Energy storage device operating parameters

[0087]

[0088] according to Figure 5The reward convergence curve during the training process is shown, and the moving average reward is obtained by calculating the average cumulative reward every 5 epochs. Each epoch randomly selects one day's data from 60 days for training; therefore, the convergence curve fluctuates continuously. It can be noted that 1000 trainings can ensure the training performance and convergence. Before 200 epochs, the agent is in the exploration stage. This marks the first interaction with the electric-thermal integrated energy system simulation environment. Therefore, the agent cannot generate appropriate actions to meet the constraints and obtain high rewards. In 200-400 epochs, the model in the electric-thermal integrated energy system is in the learning stage. In the continuous interaction with the electric-thermal integrated energy system simulation environment, the model in the electric-thermal integrated energy system can gain experience to enhance its decision-making ability, thereby finding the optimal energy management strategy with high rewards. After 1000 epochs, the model in the electric-thermal integrated energy system converges because the changing trend of the moving average reward curve is stable. Finally, the reward converges to the maximum value, and the model in the electric-thermal integrated energy system learns the information in the data that can be used to perform real-time control. The small probability of random action selection causes oscillations in the late convergence period.

[0089] The output results of electric energy and thermal energy are given in Figure 6(a) and Figure 6(b). Since the electricity purchase price from 0 to 6 o'clock reaches the valley value and is lower than the gas price, the power system increases the power purchased from the external power grid. In view of the low electricity demand at this time, the electric boiler converts electricity into heat, and the system increases the storage capacity of the battery to achieve the effect of "filling the valley" electricity demand. In the thermal system during the same period, the heat load demand is large, CHP has the characteristics of efficient heat production and low heat cost, and EB has high conversion efficiency and environmental protection. The thermal system gradually increases the output power of CHP and EB to maintain the balance of the system. From 7 to 13 hours, the power purchase price of the power grid rebounds, the output of photovoltaic and wind turbines is sufficient, and the excess electricity is converted into thermal energy through GB. During the same period, the thermal system ensures power balance by increasing the heat production of GB and EB. From 15 to 17 hours, the supply and demand gap caused by the reduction of battery power is used to achieve the effect of "peak shaving" electricity demand; from 18 to 23 hours, both electricity prices and electricity demand fall. In order to consider economic benefits, CHP and GB are used to produce heat. In the electronic block, the power purchased by the power grid changes significantly with the peak and valley periods of electricity prices. The power system adjusts the storage / discharge of batteries to effectively achieve the "peak shaving and valley filling" of the electric load and promote the consumption of some wind power. In the thermal sub-block, the thermal system mainly adjusts the output ratio of CHP and EB to meet the supply and demand balance of thermal energy.

[0090] The present invention proposes a reinforcement learning method based on energy management of an electric-thermal integrated energy system, with the goal of minimizing the operating cost of the electric-thermal integrated energy system to achieve energy management of the integrated energy system. First, an economic dispatch model of an electric-thermal integrated energy system is established; then, the energy management process of the electric-thermal integrated energy system containing renewable energy is transformed into a Markov decision process (MDP), and the Deep Q Network (DQN-CE) algorithm integrating the NoisyNet and the Self-attention mechanism is used to interactively learn and train the model of the electric-thermal integrated energy system. Finally, the results of the example analysis show that the model of the electric-thermal integrated energy system trained by the proposed method can respond to the uncertainty of renewable energy in real time and manage the energy of the electric-thermal integrated energy system containing renewable energy online.

Claims

1. Energy management strategy for electric-thermal integrated energy system based on noise network and self-attention mechanism DQN-CE algorithm, characterized by The following steps are involved: Step 1: Establish an electric-thermal integrated energy system model, in which the energy supply side includes photovoltaic, wind turbine and natural gas sources; the energy conversion device includes combined heat and power unit (CHP), electric boiler (EB) and gas boiler (GB); the energy demand side includes electric load and thermal load; Step 2: Based on the electric-thermal integrated energy system model established in step 1; construct a deep reinforcement learning training model framework based on variables and constraints, that is, design the reinforcement learning state space S t , action space A t , reward function R; Step 3: Based on the noise network and self-attention mechanism DQN-CE algorithm, build the current Q network and the target Q network. First, update the current Q network through the improved loss function, and then periodically copy the parameters of the current Q network to the target Q network. Step 4: Based on the current Q network, the target Q network and the improved loss function, the current Q network and the target Q network model are trained by interacting with the electric-thermal integrated energy system model to make the best decision to maximize the reward function and achieve stable and economic operation of the electric-thermal integrated energy system.

2. The energy management strategy of the electric-thermal integrated energy system based on the noise network and self-attention mechanism DQN-CE algorithm according to claim 1 is characterized by: In the step 1, an electric-thermal integrated energy system model is established, specifically including: 1.

1. Constant temperature control load: Thermostatic control load refers to the load that controls the indoor temperature through thermostatic equipment. The change of working state is achieved by switching on and off. At time t, the action u of the i-th thermostatic control load i,control,t Defined as: In formula (1): T i,t is the current ambient temperature; T i,max and T i,min are the highest and lowest temperatures respectively; u i,control The action of constant temperature control load, the value range of subscript i is i∈N TCLs , N TCLs The number of loads for constant temperature control; The power load P of the i-th constant temperature control load at time t TCL,i,t The expression is: P TCL,i,t =S TCL u i,control,t (2); Where: S TCL is the nominal power of the thermostatically controlled load; 1.

2. Price Response Load: Price-responsive load refers to the load that cannot be directly controlled in the electric-thermal integrated energy system. It is affected by the electricity price and its load expression is: P load,v,t =P b,t -P v,SL,t +P v,PB,t (3); In formula (3): P load,v,t is the load power of the vth price-responsive load at time t; p b,t is the base load power at time t, P v,SL,t is the transfer load power at time t; P v,PB,t is the redemption load power at time t; P v,SL,t =β v δ t P b,t , where β v ∈[0,1] is the sensitivity coefficient of the vth price-responsive load; δ t ∈[-2,-1,0,1,2] is the price level at time t; Among them, ω v,j is the state variable of whether v transfers the load from time j; the value range of subscript v is v∈N loads , N loads is the quantity of price-responsive load; t is an hour in a 24-hour day; 1.3 Energy storage components: The battery expression is: In formula (4): The amount of electricity stored in the battery for the next hour; The amount of electricity stored in the battery; η bt The charging and discharging efficiency of the battery; P t bt,dis is the discharge power of the battery at time t; P t bt,chr is the charging power of the battery at time t, is the maximum storage capacity of the battery; is the state variable of the battery, is the maximum charge and discharge power of the battery, is the minimum charging and discharging power of the battery; The heat storage tank expression is: In formula (5): The heat stored in the heat storage tank for the next hour, is the heat stored in the heat storage tank at time t; η ts The charging and discharging efficiency of the heat storage tank; and The heat stored and released by the heat storage tank respectively; is the maximum storage capacity of the heat storage tank; is the state variable of the heat storage tank, is the maximum charging and discharging power of the heat storage tank, is the minimum charging and discharging power of the heat storage tank; 1.

4. Energy conversion device: The energy conversion device converts different energies through electric boilers, gas boilers and cogeneration units to achieve economic regulation of the electric-thermal integrated energy system. The conversion constraint expressions of electric boilers, gas boilers and cogeneration units are as follows: In formula (6): P chp,t is the power output of the cogeneration unit at time t; γ chp is the gas-to-electricity conversion efficiency of the cogeneration unit; Q chp,t is the gas volume of the cogeneration unit at time t; H chp,t is the thermal output of the cogeneration unit at time t; k chp is the heat-to-electricity ratio of the cogeneration unit; and are the upper and lower limits of the output of the cogeneration unit respectively; P eb,t is the electric output of the electric boiler at time t; P gb,t is the electrical output of the gas boiler at time t; k eb , k gb They are the electricity-heat conversion efficiency of the heat pump and the gas-heat conversion efficiency of the gas boiler; and They are the upper and lower limits of the heat pump output respectively; and H is the upper and lower limits of the gas boiler output; chp,t , H eb,t and H gb,t are the thermal outputs of the cogeneration unit, heat pump and gas boiler at time t respectively.

3. The energy management strategy of the electric-thermal integrated energy system based on the noise network and self-attention mechanism DQN-CE algorithm according to claim 1 is characterized by: In the step 2, 2.1: The power balance constraint of the electric-thermal integrated energy system is expressed as follows: Q gas,t =Q chp,t +Q gb,t (9); Where: P grid,t is the power purchased from the external power grid at time t; P pv,t is the power generation of the photovoltaic power generation equipment at time t; P w,t P is the power generation capacity of the wind turbine generator at time t; chp,t is the power output of the cogeneration unit at time t; P t bt,dis is the discharge power of the battery at time t; P t bt,chr is the charging power of the battery at time t; P eb,t is the electric output of the electric boiler at time t; P TCL,i,t P is the electrical load of the constant temperature control load at time t; load,v,t is the electricity load of the price response load at time t; N TCLs is the number of constant temperature control loads; N loads is the number of price-responsive loads; i is N TCLs The price response load in; v is N loads The price response load in H chp,t , H eb,t and H gb,t are the heat outputs of the combined heat and power unit, heat pump and gas boiler at time t; H load,t is the total thermal power at time t; The heat stored in the heat storage tank; Q gas,t is the gas purchasing power of the natural gas source at time t; Q chp,t and Q gb,t are the gas power consumed by the cogeneration unit and the gas boiler at time t respectively; 2.2: State space S of the electric-thermal integrated energy system t , which is expressed as follows: Where: price,t is the electricity price purchased from the power grid at time t; is the amount of electricity stored in the battery at time t; is the amount of heat stored in the heat storage tank at time t; t is an hour in a 24-hour day; 2.3: Action space A of the electric-thermal integrated energy system t , which is expressed as follows: A t =[a tcl ,a load ,P chp,t ,P eb,t ,H gb,t ,P t bt ,P t ts ](11); Where: a tcl 4 actions for constant temperature control load; a load 5 actions for price response load; P t bt is the power of the battery at time t; P t ts is the power of the heat storage tank at time t; 2.4: The reward function R of the electric-thermal integrated energy system is expressed as follows: R=C profile -C cost (12); C cost =ρ w P w,t △t+ρ pv P pv,t △t+ρ price,t P grid,t +r gas Q gas,t (14); Where: C profile The benefit obtained from executing the action; C cost is the cost required to perform the action; ρ load is the electricity price of the constant temperature control load; ρ TCL is the electricity price of the constant temperature control load; ρ w The cost price of generating electricity for wind power generation equipment; ρ pv The cost price of electricity generated by photovoltaic power generation equipment; ρ gas is the price of natural gas; Q gas,t is the amount of natural gas purchased from the external network at time t; △t is a certain hour in a 24-hour day.

4. The energy management strategy of the electric-thermal integrated energy system based on the noise network and self-attention mechanism DQN-CE algorithm according to claim 1 is characterized by: In the step 3, based on the noise network and self-attention mechanism DQN-CE algorithm, the current Q network and the target Q network are built; the current Q network is a neural network used to generate an action value function, and the target Q network is used to calculate the target Q value. The target Q network has the same structure as the current Q network, but different parameters; the current Q network and the target Q network both include an input layer, a self-attention mechanism layer, a noise network layer, a noise network layer, and an output layer; First, the current Q network is updated by the improved loss function, where the improved loss function includes mean square error loss and cross entropy loss, and then the parameters of the current Q network are periodically copied to the target Q network.

5. The energy management strategy of the electric-thermal integrated energy system based on the noise network and self-attention mechanism DQN-CE algorithm according to claim 4 is characterized by: In step 3, the current Q network and target Q network structure in the noise network and self-attention mechanism DQN-CE algorithm are constructed. The self-attention mechanism layer is the second layer in the current Q network and the target Q network, and its expression is as follows: Where: Attention(Q,K,V) means mapping a query vector Q, a set of key-value pairs K and V into an output, and the output weight vector is calculated by weighting the values ​​and the softmax function; Q, K and V are the designed state feature vectors; K T is the transpose of the state eigenvector; d k is the dimension of the eigenvector.

6. The energy management strategy of the electric-thermal integrated energy system based on the noise network and self-attention mechanism DQN-CE algorithm according to claim 4 is characterized by: The noise network layer is the third layer in the current Q network and the target Q network. The noise network layer includes: ω is the neural network weight, ω=μ+σ×ε, where μ is the mean of the neural network parameters, σ is the standard deviation, ε is the random noise parameter, ε~N(0,1), N(0,1) represents a normal distribution with a mean of 0 and a standard deviation of 1.

7. The energy management strategy of the electric-thermal integrated energy system based on the noise network and self-attention mechanism DQN-CE algorithm according to claim 4 is characterized by: The improved loss function expression is as follows: a t+1 =softmax(Q(s t+1 ,a,θ))(18); Where: s t+1 is the state at the next moment; a is the action at the next moment; θ' is the parameter of the target Q network; Q' is the target Q network; Q is the current Q network; the softmax(x) function is a normalization function that converts a vector containing unnormalized values ​​into a normalized probability distribution vector; Perform a natural exponential function transformation on each element i of the input vector x; a t+1 is the next action of the current Q network; CrossEntropyLoss is the cross entropy loss function, which is used to measure the difference between the predicted probability distribution and the actual distribution; MSELoss is the mean square error loss function; is the probability value of the target action; m is the total number of samples; a target is the action of the target Q network; L DQN-C E is the improved loss function; is the target Q value of the i-th sample; is the current Q value of the i-th sample; The log probability that the predicted value matches the target value.

8. The energy management strategy of the electric-thermal integrated energy system based on the noise network and self-attention mechanism DQN-CE algorithm according to claim 4 is characterized by: The step 4 comprises the following steps: Step 4.1: Based on steps 1 to 3, the current Q network and the target Q network and the improved loss function are constructed, and the current Q network is trained by interacting with the electric-thermal integrated energy system model; Step 4.2: By loading the trained current Q network model and testing data, a stable and economical operation level of the electric-thermal integrated energy system can be achieved.

Images