Multi-energy system optimization scheduling method based on carbon flow theory and deep reinforcement learning

Abstract

The invention discloses a multi-energy system optimization scheduling method based on a carbon flow theory and deep reinforcement learning. The method comprises the following steps: firstly, carrying out mathematical description on a multi-energy system dynamic optimization problem based on a carbon emission flow theory, and establishing an objective function and constraint conditions of multi-energy system optimization scheduling; then, the scheduling decision problem is expressed as a Markov decision process framework, a reinforcement learning environment is created, and a state space, an action space and a reward function of the multi-energy system are defined; meanwhile, in the depth deterministic strategy gradient algorithm, the training efficiency is improved by introducing an experience replay mechanism based on proportion priority, the overfitting risk in the network training process is reduced by adopting an L2 regularization measure, and the generalization ability of the network in the optimization scheduling decision of the multi-energy system is improved. The method can respond to random fluctuation of a source side and a load side in real time, outputs a coordinated scheduling strategy, ensures low-carbon operation of a multi-energy system, and overcomes the defect that a traditional energy scheduling system cannot quickly adapt to changes.

Description

Technical Field

[0001] The present invention relates to a multi-energy system optimization scheduling method based on carbon flow theory and deep reinforcement learning, belonging to the technical field of smart energy. The application fields of this technology cover energy management, renewable energy integration, scheduling and other fields. Background Art

[0002] Multi-energy systems, with their ability to coordinate and complement each other, are a key technological path to achieving the "dual carbon" goals. A multi-energy system is an integrated multi-energy utilization system formed by coordinating and optimizing the production, transmission, conversion, and consumption of multiple energy sources, such as electricity, gas, and heat, during the planning, construction, and operation stages. These systems comprise energy supply units, conversion units, storage units, and multiple loads. Currently, extensive research has focused on optimizing the operation of multi-energy systems, aiming to achieve rapid supply and demand balance between electricity and heat. However, the lack of reasonable control instructions significantly reduces the economic efficiency and stability of multi-energy systems. Consequently, many scholars have conducted in-depth research on the optimal scheduling of multi-energy systems, aiming to achieve optimal performance, reduce costs, and reduce carbon emissions. Furthermore, the optimal control of multi-energy systems faces numerous uncertainties. The randomness of user loads and the volatility of renewable energy output significantly impact system operation, making the optimal control of multiple energy sources even more difficult.

[0003] Traditional control methods, such as robust optimization, stochastic programming, and model predictive control, rely on distributed knowledge or predictive information about system uncertainties. However, obtaining accurate distributed knowledge or predictive information is often difficult and unrealistic, making these methods difficult to adapt to the dynamic changes in the actual environment. Deep reinforcement learning methods currently demonstrate significant development advantages in multi-energy system scheduling. By combining deep neural networks with reinforcement learning technology, the system can learn optimal decision-making strategies through interaction with the environment. Deep reinforcement learning can cope with complex, nonlinear systems and constantly changing conditions without the need for explicit models. By interacting with the environment, the system optimizes decisions based on reward signals and continuously improves performance. This approach is particularly effective in situations with high real-time requirements and large fluctuations in demand and resources.

[0004] Carbon emission flow (carbon flow) is considered a virtual network flow attached to the system's energy flow. Its purpose is to accurately track carbon elements at all stages of the system and allocate carbon emissions from source to load. Tracking based on carbon flow enables accurate load-side carbon emission measurement, incorporating load-side carbon emissions into optimization targets, and contributing to the system's low-carbon operation.

[0005] In view of this, in order to cope with the renewable energy generation on the supply side and the uncontrollable demand on the load side during the operation of the multi-energy system and to improve the low-carbon nature of the multi-energy system, the present invention is specially proposed. Summary of the Invention

[0006] To address the above technical issues, the present invention discloses a multi-energy system optimization and scheduling method based on carbon flow theory and deep reinforcement learning. This method aims to integrate carbon emission flow theory into the carbon factor analysis and calculation of multi-energy systems, conduct a refined analysis of carbon factors in each link within the system, and integrate carbon flow information into the constraints of the system's optimized operation, thereby unleashing the low-carbon potential of coordinated efforts across all links in the system while achieving precise quantification of carbon emissions from energy consumption on the load side. Secondly, a smart scheduling approach is adopted to reduce manual intervention and address energy market fluctuations and the uncertainty of load energy demand.

[0007] In order to solve the above problems, the present invention adopts the following technical solutions:

[0008] Specifically, the present invention proposes a multi-energy system optimization scheduling method based on carbon flow theory and deep reinforcement learning, which is characterized by comprising the following steps:

[0009] Step (1): Based on the carbon emission flow theory, the objective function and constraints of the multi-energy system dynamic optimization scheduling problem are established to obtain the mathematical model of the multi-energy system dynamic optimization scheduling problem;

[0010] Step (2): Construct a Markov decision process framework for the multi-energy system dynamic optimization scheduling problem and obtain the reinforcement learning environment required for the multi-energy system dynamic optimization scheduling strategy training;

[0011] Step (3): Utilize the interaction between the intelligent agents in the system and the environment to find the optimal scheduling strategy for the multi-energy system, so as to avoid the problem that the traditional energy scheduling method cannot respond to the random changes of the source side and the load side in real time. The priority experience replay mechanism based on proportional priority and L2 regularization are used to improve the deep deterministic policy gradient algorithm.

[0012] Step (4): Use the improved deep deterministic policy gradient algorithm to train the dynamic optimization scheduling strategy of the multi-energy system. After the training is completed, the real-time state of the multi-energy system is input into the deep policy network, and the scheduling action is output to form the coordinated scheduling strategy of the multi-energy system.

[0013] Furthermore, in step (1) of the multi-energy system optimization scheduling method based on carbon flow theory and deep reinforcement learning, the objective function and constraints of the multi-energy system dynamic optimization scheduling problem are established based on the carbon emission flow theory, and the mathematical model of the multi-energy system dynamic optimization scheduling problem is obtained, which specifically includes:

[0014] 1) Establish a source-side carbon flow model for the multi-energy system, including carbon flow models for the upstream power grid and upstream natural gas grid;

[0015] The carbon flow model of the upper-level power grid and the upper-level natural gas grid is specifically as follows:

[0016]

[0017] Where, are the carbon flow rates input into the system from electricity purchased from the upper power grid and gas purchased from the upper natural gas network at time t, respectively, representing the carbon emissions contained in the electricity and natural gas input per unit time; They represent the carbon flow density of input electricity and natural gas at time t, that is, the carbon emissions contained in electricity and natural gas per unit power per unit time; They are the electricity purchased by the upper-level power grid and the gas purchased by the upper-level natural gas grid at time t respectively.

[0018] 2) Establish carbon flow models for energy conversion equipment and energy storage equipment in multi-energy systems, including carbon flow models for cogeneration units, gas boilers, electric boilers, electrical energy storage, and thermal energy storage;

[0019] The carbon flow model of the cogeneration unit is specifically as follows:

[0020]

[0021] Where, are the carbon flux densities at the input, electrical output, and thermal output ends of the cogeneration unit at time t, respectively; and are the gas-to-electricity conversion efficiency and gas-to-heat conversion efficiency of the cogeneration unit, respectively; are the energy flow powers at the input, electrical and thermal output ends of the cogeneration unit at time t respectively.

[0022] The carbon flow model of the gas boiler is specifically as follows:

[0023]

[0024] Where, are the carbon flux densities at the output and input ends of the gas boiler at time t; are the energy flow powers at the output and input ends of the gas boiler at time t; η GB The energy conversion efficiency of the gas boiler.

[0025] The carbon flow model of the electric boiler is specifically as follows:

[0026]

[0027] Where, are the carbon flow densities at the output and input ends of the electric boiler at time t; are the energy flow powers at the output and input ends of the electric boiler at time t; η EB is the energy conversion efficiency of the electric boiler.

[0028] The carbon flow model of the electric energy storage is specifically as follows:

[0029] Charging mode: When the energy storage is charging, it can be equivalent to a load, and the carbon flow inside it depends on the amount of electricity accumulated over time; therefore, during the charging process of the energy storage, the carbon flow charged into the energy storage at time t is:

[0030]

[0031] Where, The carbon flow charged into the energy storage during the charging process of the energy storage during the period t; is the charging power of the energy storage at time t; is the carbon potential of the electric bus at time t; Δt is the scheduling time interval.

[0032] Discharge mode: When the electric energy storage is discharged, it can be equivalent to a generator set, and the carbon flow inside it begins to be released along with the electricity. Therefore, considering the energy conversion rate during charging and discharging, the carbon flow released by the electric energy storage at time t is:

[0033]

[0034] Where, is the carbon flow rate released by the energy storage during the discharge process of the energy storage during period t; is the discharge power of the energy storage at time t; is the carbon flow density at the discharge port of the energy storage at time t; η BES is the energy storage discharge efficiency; is the carbon flow density of the internal electrical energy of the energy storage at t-1.

[0035] Therefore, the carbon flow density of the internal electrical energy of the electric energy storage device at time t can be expressed as:

[0036]

[0037] Where, is the state of charge of the energy storage at time t; C BES The capacity of electrical energy storage.

[0038] The carbon flow model of the thermal energy storage is:

[0039]

[0040] Where, The carbon flow rate charged into the thermal energy storage during the charging process of the thermal energy storage in period t; is the charging power of thermal energy storage at time t; η HES It is the heat storage and heat release efficiency; is the carbon flow density of the internal thermal energy of the thermal energy storage at t-1; is the state of charge of the thermal energy storage at time t; C HES is the capacity of thermal energy storage.

[0041] 3) Establish a carbon flow model for the load side of the multi-energy system, including carbon flow models for both electricity and heat;

[0042] The carbon flow model of the electric load of the multi-energy system is specifically:

[0043]

[0044] Where, is the carbon flux density of the electrical load at time t; and are the power generation power of the photovoltaic unit and its electric energy carbon flow density at time t respectively.

[0045] The carbon flow model of the multi-energy system heat load is specifically:

[0046]

[0047] Where, is the carbon flux density of the heat load at time t.

[0048] 4) Establish the objective function of the multi-energy system optimization scheduling problem, expressed as:

[0049]

[0050] Where, is the energy purchase cost of the multi-energy system at time t; Depreciation costs for charging and discharging energy storage; The carbon emission cost generated by energy consumption on the load side.

[0051] Among them, the energy purchase cost of the multi-energy system at time t is specifically:

[0052]

[0053] Where, ε e,t and ε g，t are the electricity price of the upper-level power grid and the gas price of the upper-level natural gas grid at time t respectively.

[0054] Among them, the depreciation cost of energy storage charging and discharging is specifically:

[0055]

[0056] Where, ω BES and ω HES are the charging and discharging energy depreciation cost coefficients of electric energy storage and thermal energy storage respectively.

[0057] Among them, the carbon emission costs generated by load energy consumption are specifically:

[0058]

[0059] Where, Le，t and L h，t are the electricity load demand and heat load demand at time t respectively; q is the carbon emission cost coefficient.

[0060] 5) Constructing the constraints of the multi-energy system optimization scheduling model, specifically including power balance constraints, upper and lower limits of the system's purchased energy power, equipment operation constraints, and load carbon flow density constraints; the multi-energy system power balance constraints include electric power balance equality constraints, thermal power balance equality constraints, and gas power balance equality constraints, specifically:

[0061]

[0062]

[0063] The upper and lower limits of the system's purchased energy power include two types of inequality constraints: electricity purchased from the upper power grid and gas purchased from the upper natural gas grid. Specifically, they are:

[0064]

[0065] Where, P PG-min and P PG-max are the lower and upper limits of the power purchased from the upper power grid at time t; V S-min and V S -max are the lower limit and upper limit of the gas volume purchased from the superior natural gas network at time t respectively.

[0066] The equipment operation constraints include upper and lower limit constraints for energy conversion equipment and energy storage equipment operation constraints, specifically:

[0067] 1) Energy conversion equipment operation constraints:

[0068]

[0069] Where, P CHP-min and P CHP-max are the lower and upper limits of the output power of the cogeneration unit at time t; H GB-min and H GB-max are the lower and upper limits of the thermal power output of the gas boiler at time t; H EB-min and H EB-max are the lower and upper limits of the thermal power output of the electric boiler at time t respectively.

[0070] 2) Operational constraints of energy storage equipment:

[0071]

[0072] Where, and are the maximum charging power and the maximum discharging power of the energy storage per unit time period; u and u It is a 0-1 variable that represents the charge and discharge state of the energy storage at time t, avoiding simultaneous charging and discharging.

[0073]

[0074] Where, and are the maximum charging power and the maximum releasing power of thermal energy storage per unit time period respectively; u and It is a 0-1 variable that represents the charging and discharging state of the thermal energy storage at time t, avoiding simultaneous charging and discharging.

[0075] The load carbon flow density constraint includes the electric load carbon flow density constraint and the thermal load carbon flow density constraint, specifically:

[0076]

[0077] Where, and The upper limits of carbon flow density for electrical load and thermal load respectively can make the carbon flow reasonably distributed in the system without the situation where a certain energy flow has a high carbon flow density, thus avoiding the emergence of carbon weak links.

[0078] Furthermore, in step (2) of the multi-energy system optimization scheduling method based on carbon flow theory and deep reinforcement learning, a Markov decision process framework for the multi-energy system dynamic optimization scheduling problem is constructed to obtain the reinforcement learning environment required for the multi-energy system dynamic optimization scheduling strategy training, specifically including:

[0079] 1) Define the state space of the Markov decision process:

[0080] Select the uncontrollable electric load demand L e，t , Uncontrollable heat load demand L h，t , photovoltaic power generation Energy storage state of charge Thermal energy storage state of charge Grid electricity sales price ε e,t 、Natural gas network sales price ε g，t , carbon flow density of grid electricity Carbon flow density of natural gas in the gas grid The action at-1 of the previous scheduling period and the scheduling period t are used as state space variables, specifically:

[0081]

[0082] In continuous control problems, the intelligent agent needs to select actions in a continuous action space, which requires the intelligent agent to accurately control its behavior. Taking the action at the previous moment as part of the state space helps the intelligent agent better understand and utilize historical information, helping the intelligent agent to more finely adjust its behavior, thereby making more accurate decisions and achieving more precise control.

[0083] 2) Define the action space of the Markov decision process:

[0084] During the scheduling period t, the system's actions can be represented by the equipment's output, which includes the power purchased from the upper grid, the power generated by the cogeneration unit, the power consumed by the electric boiler, the charging and discharging power of the electric energy storage, and the charging and discharging power of the thermal energy storage. Once the above variables are determined, the output of other parts can be calculated based on the corresponding mathematical model and equality constraints. The action space is specifically:

[0085]

[0086] 3) Define the reward function of the Markov decision process:

[0087] The optimization goal of multi-energy system optimal scheduling is to minimize the total operating cost of the multi-energy system within the scheduling cycle. The goal of the Markov decision process is to maximize the reward within the scheduling cycle. At the same time, a penalty term is set according to the constraints and added to the immediate reward to obtain the final reward function. The reward function of the agent at time t is expressed as:

[0088]

[0089] Where w is the cost value scaling factor; It is a penalty item set to reduce the violation of constraints in the operation of multi-energy systems, including power imbalance penalties and equipment operating condition over-limit penalties.

[0090] 4) Define the state-action value function of the Markov decision process:

[0091] When a state s of the multi-energy system l When it is determined, the multi-energy system dynamically optimizes the scheduling action a l The quality of the state can be measured using the state-action value function Q π (s, a) is used to evaluate the quality of the multi-energy system optimization scheduling action, specifically:

[0092]

[0093] Where γ∈[0,1] is the discount factor, which indicates the influence of the reward at a certain moment in the future on the cumulative reward. The larger γ is, the more importance is attached to the future reward. π (·) is the expectation under policy π.

[0094] The goal of dynamic optimization scheduling of multi-energy systems is to find the optimal strategy π * To maximize the state-action value function, specifically:

[0095]

[0096] Where arg max represents the parameter that makes the function reach its maximum value.

[0097] Furthermore, in step (3) of the multi-energy system optimization scheduling method based on carbon flow theory and deep reinforcement learning, the intelligent agent in the system interacts with the environment to find the optimal scheduling strategy for the multi-energy system, thereby avoiding the problem that the traditional energy scheduling method cannot respond to random changes in the source side and the load side in real time. The priority experience replay mechanism based on proportional priority and L2 regularization are used to improve the deep deterministic policy gradient algorithm, which specifically includes:

[0098] 1) Construct the value network, policy network, target value network and target policy network of the deep deterministic policy gradient algorithm; the value network minimizes the loss function L(θ Q ) to optimize the network parameters, specifically:

[0099] L(θ Q )=E(y t -Q(s t , a t |θ Q )) 2 ;

[0100] Where θ Q is the parameter of the value network; y t is the target Q value, expressed as:

[0101] y t =r t +γQ′(s t+1 , μ′(s t+1 |θ μ′ )|θ Q′ );

[0102] Where s t+1 is the next state that the multi-energy system transfers to after executing the action; θ μ are the parameters of the policy network, θ Q′ and θ μ′ are the parameters of the target value network and the target policy network respectively.

[0103] The policy network is based on gradient information Improve the action, and the sampling policy gradient is expressed as:

[0104]

[0105] Where, J(θ μ ) is the performance objective function of the policy network.

[0106] 2) In action a t Add random noise N t To increase the exploration ability of the network, it is expressed as:

[0107] a t =μ(s t |θ μ )+N t ;

[0108] Where N t Ornstein-Uhlenbeck (OU) noise is a random variable of the OU process, which is often used to simulate a set of time-related noise.

[0109] 3) Introducing the priority experience replay mechanism based on proportional priority into the deep deterministic policy gradient algorithm, first determine the priority p of the sampled experience i i =|δ i |+σ, where the rank of the i-th sample is p i According to the absolute timing differential error |δ i | is sorted, σ is a small positive constant used to ensure that all experiences can be sampled, |δ i Specifically expressed as:

[0110] |δ i |=|r i +γQ′(s i+1 , μ′(s i+1 |θ μ′ )|θ′)-Q(s i ,a i |θ Q )|;

[0111] The probability P of sampling experience i i Expressed as:

[0112]

[0113] Where, Determines the number of priorities used; A value of 0 is equivalent to random sampling.

[0114] Since experiences with high temporal difference errors are replayed more frequently, the proportional priority-based experience replay mechanism may cause the training process to be prone to oscillation. To address this problem, importance sampling weights are used when calculating weight changes and the weights are normalized to ensure that the weights sum to 1:

[0115]

[0116] Where N b is the size of the experience replay pool; max j W j is the maximum weight in the current batch and is used for normalization.

[0117] β is a parameter used to correct the deviation. In order to allow the algorithm to explore more in the early stages of training, β is set to increase gradually over time. The adjustment formula for β is:

[0118]

[0119] Among them, β0 is the initial value, β max is the maximum value, t is the current training step number, and T is the total number of steps.

[0120] Calculate the loss function using weighted loss:

[0121] L′(θ Q )=E(W′ i (y t -Q(s t ,a i |θ Q ))) 2 ;

[0122] 4) In order to reduce the risk of overfitting during network training, L2 regularization is introduced into the deep deterministic policy gradient algorithm to reduce the risk of overfitting during training. L2 regularization is introduced into the performance objective function J(θ μ ), the performance objective function and the corresponding gradient are expressed as:

[0123]

[0124] Where a L is the regularization parameter; N is the mini-batch size.

[0125] 5) Parameters θ of the value network Q The update method is expressed as:

[0126]

[0127] Where μ Q is the learning rate of the network.

[0128] The parameters θ of the policy network μ The update method is expressed as:

[0129]

[0130] Where μ μ is the policy network learning rate.

[0131] 6) Update the parameters of the target value network and the target strategy network, specifically:

[0132] Parameters θ of the target value network and the target policy network Q′ and θ μ′ The update method is expressed as:

[0133] θ Q′ ←τθ Q +(1-τ)θ Q′ ;

[0134] θ μ′ ←τθ μ +(1-τ)θ μ′ ;

[0135] Where τ is the soft update coefficient, τ<<1.

[0136] Furthermore, in step (4) of the multi-energy system optimization scheduling method based on carbon flow theory and deep reinforcement learning, an improved deep deterministic policy gradient algorithm is used to train the multi-energy system dynamic optimization scheduling strategy. After the training is completed, the real-time state of the multi-energy system is input into the deep policy network, and the scheduling action is output to form a multi-energy system coordinated scheduling strategy, which specifically includes:

[0137] 1) Constructing an initial model, wherein the initial model includes a value network, a policy network, a target value network, and a target policy network.

[0138] 2) Inputting each sample state data into the initial model; generating predicted action data corresponding to the sample state data through the policy network, and obtaining the predicted state-action value corresponding to the predicted action data through the value network.

[0139] 3) Based on the predicted action state value and the corresponding target action state value, a loss function calculation result is obtained, and based on the predicted action data and the corresponding predicted action state value, a gradient function calculation result is obtained, and the target action state value is calculated based on the target value network and the target strategy network.

[0140] 4) adjusting the parameters of the value network according to the calculation result of the loss function, and adjusting the parameters of the policy network according to the calculation result of the gradient function.

[0141] 5) The sample state data and the corresponding predicted action data, predicted action state value, and state data at the next moment are stored in the experience replay pool, and the priority corresponding to the sample state data is determined based on the absolute time difference error of the sample state data, so as to select the sample state data from the experience replay pool according to the priority to learn the value network and the policy network.

[0142] 6) Determine the deep policy model based on the trained policy network; input the real-time state of the multi-energy system into the pre-trained deep policy network of the improved deep deterministic policy gradient algorithm, and output the scheduling action.

[0143] The beneficial effects of the present invention are:

[0144] 1) Accurately track and control carbon emissions: Based on carbon flow theory, this technology treats carbon emissions as virtual network flows dependent on energy flows, enabling precise tracking of carbon emissions at every stage of the system. This technology enables load-side carbon emissions to be incorporated into optimization targets during system scheduling, thereby helping to improve the low-carbon operation of multi-energy systems.

[0145] 2) Improving the system's scheduling efficiency under uncertain conditions: Through deep reinforcement learning technology, the present invention can autonomously learn scheduling strategies in the face of uncertainties in renewable energy generation and load-side fluctuations, achieve dynamic optimization of the system, and thus improve the economy and stability of system operation.

[0146] 3) Improving the low-carbon and economic efficiency of multi-energy systems and the utilization of renewable energy: By combining deep reinforcement learning techniques with carbon flow theory, this invention not only achieves load balancing but also effectively optimizes the coordinated and complementary use of various energy sources in multi-energy systems, reducing energy loss and costs. Furthermore, the scheduling strategy of this invention helps to better utilize renewable energy and reduce dependence on fossil fuels, thereby further reducing operating costs and carbon emissions. BRIEF DESCRIPTION OF THE DRAWINGS

[0147] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0148] Figure 1 This is a flow chart of a multi-energy system optimization scheduling method based on carbon flow theory and deep reinforcement learning provided by an embodiment of the present invention;

[0149] Figure 2A schematic diagram of a multi-energy system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0150] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0151] The corresponding multi-energy system in this embodiment is as follows Figure 2 The system's main equipment includes photovoltaic generators, cogeneration units, electric boilers, gas boilers, electric energy storage, and thermal energy storage. The system's energy supply side includes the upstream power grid and natural gas grid. The system's demand side primarily reflects the energy demand of the system's loads.

[0152] In order to achieve low-carbon, economical and efficient operation of multi-energy systems, the present invention proposes a multi-energy system optimization scheduling method based on carbon flow theory and deep reinforcement learning. More specifically, the scheduling optimization training process of the multi-energy system described in this embodiment is as follows: Figure 1 As shown in the figure. Based on the policy-evaluation network structure, the evaluation network implements policy evaluation. The policy network continuously optimizes the agent's actions based on the evaluation network, learning how to select the best actions in different states to maximize rewards and obtain the optimal scheduling strategy.

[0153] The present invention provides a multi-energy system optimization scheduling method based on carbon flow theory and deep reinforcement learning, which specifically includes the following steps:

[0154] Step (1): Based on the carbon emission flow theory, the objective function and constraints of the multi-energy system dynamic optimization scheduling problem are established to obtain the mathematical model of the multi-energy system dynamic optimization scheduling problem;

[0155] Step (2): Construct a Markov decision process framework for the multi-energy system dynamic optimization scheduling problem and obtain the reinforcement learning environment required for the multi-energy system dynamic optimization scheduling strategy training;

[0156] Step (3): Utilize the interaction between the intelligent agents in the system and the environment to find the optimal scheduling strategy for the multi-energy system, so as to avoid the problem that the traditional energy scheduling method cannot respond to the random changes of the source side and the load side in real time. The priority experience replay mechanism based on proportional priority and L2 regularization are used to improve the deep deterministic policy gradient algorithm.

[0157] Step (4): Use the improved deep deterministic policy gradient algorithm to train the dynamic optimization scheduling strategy of the multi-energy system. After the training is completed, the real-time state of the multi-energy system is input into the deep policy network, and the scheduling action is output to form the coordinated scheduling strategy of the multi-energy system.

[0158] Specifically, the mathematical model in step (1) is constructed as follows:

[0159] A source-side carbon flow model for a multi-energy system is established, including carbon flow models for the upstream power grid and the upstream natural gas grid. The carbon flow models for the upstream power grid and the upstream natural gas grid are specifically as follows:

[0160]

[0161] Where, are the carbon flow rates input into the system from electricity purchased from the upper power grid and gas purchased from the upper natural gas network at time t, respectively, representing the carbon emissions contained in the electricity and natural gas input per unit time; They represent the carbon flow density of input electricity and natural gas at time t, that is, the carbon emissions contained in electricity and natural gas per unit power per unit time; They are the electricity purchased by the upper-level power grid and the gas purchased by the upper-level natural gas grid at time t respectively.

[0162] Establish a carbon flow model for energy conversion equipment and energy storage equipment in a multi-energy system, including carbon flow models for cogeneration units, gas boilers, electric boilers, electric energy storage, and thermal energy storage. The carbon flow model for the cogeneration unit is specifically as follows:

[0163]

[0164] Where, are the carbon flux densities at the input, electrical output, and thermal output ends of the cogeneration unit at time t, respectively; and are the gas-to-electricity conversion efficiency and gas-to-heat conversion efficiency of the cogeneration unit, respectively; are the energy flow powers at the input, electrical and thermal output ends of the cogeneration unit at time t respectively.

[0165] The carbon flow model of the gas boiler is specifically as follows:

[0166]

[0167] Where, are the carbon flux densities at the output and input ends of the gas boiler at time t; are the energy flow powers at the output and input ends of the gas boiler at time t; η GB The energy conversion efficiency of the gas boiler.

[0168] The carbon flow model of the electric boiler is specifically as follows:

[0169]

[0170] Where, are the carbon flow densities at the output and input ends of the electric boiler at time t; are the energy flow powers at the output and input ends of the electric boiler at time t; η EB is the energy conversion efficiency of the electric boiler.

[0171] The carbon flow model of the electric energy storage is specifically as follows:

[0172] Charging mode: When the energy storage is charging, it can be equivalent to a load, and the carbon flow inside it depends on the amount of electricity accumulated over time; therefore, during the charging process of the energy storage, the carbon flow charged into the energy storage at time t is:

[0173]

[0174] Where, The carbon flow charged into the energy storage during the charging process of the energy storage during the period t; is the charging power of the energy storage at time t; is the carbon potential of the electric bus at time t; Δt is the scheduling time interval.

[0175] Discharge mode: When the electric energy storage is discharged, it can be equivalent to a generator set, and the carbon flow inside it begins to be released along with the electricity. Therefore, considering the energy conversion rate during charging and discharging, the carbon flow released by the electric energy storage at time t is:

[0176]

[0177] Where, is the carbon flow rate released by the energy storage during the discharge process of the energy storage during period t; is the discharge power of the energy storage at time t; is the carbon flow density at the discharge port of the energy storage at time t; η BES is the energy storage discharge efficiency; is the carbon flow density of the internal electrical energy of the energy storage at t-1.

[0178] Therefore, the carbon flow density of the internal electrical energy of the electric energy storage device at time t can be expressed as:

[0179]

[0180] Where, is the state of charge of the energy storage at time t; C BES The capacity of electrical energy storage.

[0181] The carbon flow model of the thermal energy storage is:

[0182]

[0183] Where, The carbon flow rate charged into the thermal energy storage during the charging process of the thermal energy storage in period t; is the charging power of thermal energy storage at time t; η HES It is the heat storage and heat release efficiency; is the carbon flow density of the internal thermal energy of the thermal energy storage at t-1; is the state of charge of the thermal energy storage at time t; C HES is the capacity of thermal energy storage.

[0184] A carbon flow model for the load side of a multi-energy system is established, including carbon flow models for both electricity and heat. The carbon flow model for the electrical load of the multi-energy system is specifically as follows:

[0185]

[0186] Where, is the carbon flux density of the electrical load at time t; and are the power generation power of the photovoltaic unit and its electric energy carbon flow density at time t respectively.

[0187] The carbon flow model of the multi-energy system heat load is specifically:

[0188]

[0189] Where, is the carbon flux density of the heat load at time t.

[0190] The objective function of the multi-energy system optimization scheduling problem is established and expressed as:

[0191]

[0192] Where, is the energy purchase cost of the multi-energy system at time t; Depreciation costs for charging and discharging energy storage; The carbon emission cost generated by energy consumption on the load side.

[0193] Among them, the energy purchase cost of the multi-energy system at time t is specifically:

[0194]

[0195] Where, ε e,t and ε g，t are the electricity price of the upper-level power grid and the gas price of the upper-level natural gas grid at time t respectively.

[0196] Among them, the depreciation cost of energy storage charging and discharging is specifically:

[0197]

[0198] Where, ω BES and ω HES are the charging and discharging energy depreciation cost coefficients of electric energy storage and thermal energy storage respectively.

[0199] Among them, the carbon emission costs generated by load energy consumption are specifically:

[0200]

[0201] Where, L e,t and L h,t are the electricity load demand and heat load demand at time t respectively; q is the carbon emission cost coefficient.

[0202] The constraints for constructing the multi-energy system optimization scheduling model include power balance constraints, upper and lower limits of the system's purchased energy power, equipment operation constraints, and load carbon flow density constraints. The multi-energy system power balance constraints include electric power balance equation constraints, thermal power balance equation constraints, and gas power balance equation constraints, specifically:

[0203]

[0204] The upper and lower limits of the system's purchased energy power include two types of inequality constraints: electricity purchased from the upper power grid and gas purchased from the upper natural gas grid. Specifically, they are:

[0205]

[0206] Where, P PG-min and P PG-max are the lower and upper limits of the power purchased from the upper power grid at time t; V S-min and V S -max are the lower limit and upper limit of the gas volume purchased from the superior natural gas network at time t respectively.

[0207] The equipment operation constraints include upper and lower limit constraints for energy conversion equipment and energy storage equipment operation constraints, specifically:

[0208] 1) Energy conversion equipment operation constraints:

[0209]

[0210] Where, P CHP-min and P CHP-max are the lower and upper limits of the output power of the cogeneration unit at time t; H GB-min and H GB-max are the lower and upper limits of the thermal power output of the gas boiler at time t; H EB-min and H EB-max are the lower and upper limits of the thermal power output of the electric boiler at time t respectively.

[0211] 2) Operational constraints of energy storage equipment:

[0212]

[0213] Where, and are the maximum charging power and the maximum discharging power of the energy storage per unit time period; u and It is a 0-1 variable that represents the charge and discharge state of the energy storage at time t, avoiding simultaneous charging and discharging.

[0214]

[0215] Where, and are the maximum charging power and the maximum releasing power of thermal energy storage per unit time period respectively; u and It is a 0-1 variable that represents the charging and discharging state of the thermal energy storage at time t, avoiding simultaneous charging and discharging.

[0216] The load carbon flow density constraint includes the electric load carbon flow density constraint and the thermal load carbon flow density constraint, specifically:

[0217]

[0218] Where, and The upper limits of carbon flow density for electrical load and thermal load respectively can make the carbon flow reasonably distributed in the system without the situation where a certain energy flow has a high carbon flow density, thus avoiding the emergence of carbon weak links.

[0219] Specifically, the reinforcement learning framework in step (2) is constructed as follows:

[0220] Define the state space of the Markov decision process:

[0221] Select the uncontrollable electric load demand L e，t , Uncontrollable heat load demand L h，t , photovoltaic power generation Energy storage state of charge Thermal energy storage state of charge Grid electricity sales price ε e,t 、Natural gas network sales price ε g,t , carbon flow density of grid electricity Carbon flow density of natural gas in the gas grid Action a in the previous scheduling period t-1 And the scheduling period t is used as the state space variable, specifically:

[0222]

[0223] Define the action space of the Markov decision process:

[0224] During the scheduling period t, the actions in the system can be represented by the output of the equipment, including the power purchased from the upper power grid, the power generated by the cogeneration unit, the power consumed by the electric boiler, the charging and discharging power of the electric energy storage, and the charging and discharging power of the thermal energy storage. When the above variables are determined, the output of other parts can be obtained based on the corresponding mathematical model and equality constraints. The specific action space is:

[0225]

[0226] Define the reward function of the Markov decision process:

[0227]

[0228] Where w is the cost value scaling factor; It is a penalty item set to reduce the violation of constraints in the operation of multi-energy systems, including power imbalance penalties and equipment operating condition over-limit penalties.

[0229] Define the state-action value function of the Markov decision process:

[0230]

[0231] Where γ∈[0,1] is the discount factor, which indicates the influence of the reward at a certain moment in the future on the cumulative reward. The larger γ is, the more importance is attached to the future reward. π (·) is the expectation under policy π.

[0232] The goal of dynamic optimization scheduling of multi-energy systems is to find the optimal strategy π * To maximize the state-action value function, specifically:

[0233]

[0234] Where arg max represents the parameter that makes the function reach its maximum value.

[0235] Specifically, the improved deep deterministic policy gradient algorithm in step (3) includes the following steps:

[0236] Construct the value network, policy network, target value network and target policy network of the deep deterministic policy gradient algorithm; the value network minimizes the loss function L(θ Q ) to optimize the network parameters, specifically:

[0237] L(θ Q )=E(y t -Q(s t , a t |θ Q )) 2 ;

[0238] Where θ Q is the parameter of the value network; y t is the target Q value, expressed as:

[0239] y t =r t +γQ′(s t+1 , μ′(s t+1 |θ μ′ )|θ Q′ );

[0240] Where s t+1 is the next state that the multi-energy system transfers to after executing the action; θ μ are the parameters of the policy network, θ Q′ and θ μ′ are the parameters of the target value network and the target policy network respectively.

[0241] The policy network is based on gradient information Improve the action, and the sampling policy gradient is expressed as:

[0242]

[0243] Where, J(θ μ ) is the performance objective function of the policy network.

[0244] In action a t Add random noise N t To increase the exploration ability of the network, it is expressed as:

[0245] a t =μ(s t |θ μ )+N t ;

[0246] Where N t is the Ornstein-Uhlenbeck (OU) noise.

[0247] The priority experience replay mechanism based on proportional priority is introduced into the deep deterministic policy gradient algorithm. First, the priority p of the sampled experience i is determined. i =|δ i |+σ, where the rank of the i-th sample is p i According to the absolute timing differential error |δ i | is sorted, σ is a small positive constant used to ensure that all experiences can be sampled, |δ i Specifically expressed as:

[0248] |δ i |=|ri +γQ′(s i+1 , μ′(s i+1 |θ μ′ )|θ′)-Q(s i , a i |θ Q )|;

[0249] The probability P of sampling experience i i Expressed as:

[0250]

[0251] Where, Determines the number of priorities used; A value of 0 is equivalent to random sampling.

[0252] Importance sampling weights are used when calculating weight changes and the weights are normalized:

[0253]

[0254] Where N b is the size of the experience replay pool; max j W j is the maximum weight in the current batch and is used for normalization.

[0255] β is a parameter used to correct the deviation. In order to allow the algorithm to explore more in the early stages of training, β is set to increase gradually over time. The adjustment formula for β is:

[0256]

[0257] Among them, β0 is the initial value, β max is the maximum value, t is the current training step number, and T is the total number of steps.

[0258] Calculate the loss function using weighted loss:

[0259] L′(θ Q )=E(W′ i (y t -Q(s t ,a t |θ Q ))) 2 ;

[0260] Introducing L2 regularization into the performance objective function J(θ μ ), the performance objective function and the corresponding gradient are expressed as:

[0261]

[0262] Where aL is the regularization parameter; N is the mini-batch size.

[0263] The parameter θ of the value network Q The update method is expressed as:

[0264]

[0265] Where μ Q is the learning rate of the network.

[0266] The parameters θ of the policy network μ The update method is expressed as:

[0267]

[0268] Where μ μ is the policy network learning rate.

[0269] Update the parameters of the target value network and the target policy network, specifically:

[0270] Parameters θ of the target value network and the target policy network Q′ and θ μ′ The update method is expressed as:

[0271] θ Q′ ←τθ Q +(1-τ)θ Q′ ;

[0272] θ μ′ ←τθ μ +(1-τ)θ μ′ ;

[0273] Where τ is the soft update coefficient, τ<<1.

[0274] Table 1 System energy conversion and energy storage equipment operating parameter settings

[0275]

[0276] Table 2 Other parameters

[0277]

[0278] Table 3 Improved DDPG training parameter settings

[0279]

[0280] Specifically, in step (4), the real-time state of the multi-energy system is input into the deep strategy network, and the scheduling action is output to form a coordinated scheduling strategy for the multi-energy system, which specifically includes:

[0281] The improved DDPG model training was completed on the AIStation artificial intelligence platform using the Pytorch deep learning framework. A total of 3000 rounds of training were performed, with the maximum training step size set to 24 steps, representing a 24-hour optimization. The final training stop condition was set to the maximum number of training rounds E. max .

[0282] An initial model is constructed, wherein the initial model includes a value network, a policy network, a target value network, and a target policy network.

[0283] Each sample state data is input into the initial model; predicted action data corresponding to the sample state data is generated through the policy network, and the predicted state-action value corresponding to the predicted action data is obtained through the value network.

[0284] Based on the predicted action state value and the corresponding target action state value, a loss function calculation result is obtained, and based on the predicted action data and the corresponding predicted action state value, a gradient function calculation result is obtained, and the target action state value is calculated based on the target value network and the target strategy network.

[0285] The parameters of the value network are adjusted according to the loss function calculation result, and the parameters of the policy network are adjusted according to the gradient function calculation result.

[0286] The sample state data and the corresponding predicted action data, predicted action state value, and state data at the next moment are stored in the experience replay pool, and the priority corresponding to the sample state data is determined based on the absolute temporal difference error of the sample state data, so as to select the sample state data from the experience replay pool according to the priority to learn the value network and the policy network.

[0287] The deep policy model is determined based on the trained policy network; the real-time state of the multi-energy system is input into the deep policy network of the pre-trained improved deep deterministic policy gradient algorithm, and the scheduling action is output.

[0288] The above are exemplary embodiments of the present invention, and the protection scope of the present invention is defined by the claims and their equivalents.

Claims

1. A multi-energy system optimization scheduling method based on carbon flow theory and deep reinforcement learning, characterized in that: The following steps are involved: Step (1): Based on the carbon emission flow theory, the objective function and constraint conditions of the multi-energy system dynamic optimization scheduling problem are established to obtain a mathematical model of the multi-energy system dynamic optimization scheduling problem; Step (2): construct a Markov decision process framework for the multi-energy system dynamic optimization scheduling problem, and obtain the reinforcement learning environment required for the multi-energy system dynamic optimization scheduling strategy training; Step (3): Utilize the interaction between the intelligent agents in the system and the environment to find the optimal scheduling strategy for the multi-energy system, avoid the problem that the traditional energy scheduling method cannot respond to the random changes on the source side and the load side in real time, and adopt the priority experience replay mechanism based on proportional priority and L2 regularization to improve the deep deterministic policy gradient algorithm; Step (4): Use the improved deep deterministic policy gradient algorithm to train the dynamic optimization scheduling strategy of the multi-energy system. After the training is completed, the real-time state of the multi-energy system is input into the deep policy network, and the scheduling action is output to form a coordinated scheduling strategy for the multi-energy system.

2. A multi-energy system optimization scheduling method based on carbon flow theory and deep reinforcement learning as claimed in claim 1, characterized in that: The step (1) specifically comprises the following steps: Step (1-1): Establish a source-side carbon flow model of the multi-energy system, including carbon flow models of the upper power grid and the upper natural gas grid; The carbon flow model of the upper power grid and the upper natural gas grid is specifically as follows: In the formula, are the carbon flow rates input into the system from the power grid and natural gas purchased from the natural gas network at time t, respectively, indicating the carbon emissions contained in the input electricity and natural gas per unit time; They represent the carbon flow density of input electricity and natural gas at time t, that is, the carbon emissions contained in electricity and natural gas per unit power per unit time; They are the electricity purchased by the upper power grid and the gas purchased by the upper natural gas grid at time t respectively; Step (1-2): Establish carbon flow models for energy conversion equipment and energy storage equipment in a multi-energy system, including carbon flow models for cogeneration units, gas boilers, electric boilers, electric energy storage, and thermal energy storage; The carbon flow model of the cogeneration unit is specifically: In the formula, are the carbon flow densities at the input end, electrical output end, and thermal output end of the cogeneration unit at time t, respectively; and They are the gas-to-electricity conversion efficiency and gas-to-heat conversion efficiency of the cogeneration unit, respectively; are the energy flow powers at the input end, electrical output end, and thermal output end of the cogeneration unit at time t respectively; The carbon flow model of the gas boiler is specifically: In the formula, are the carbon flow densities at the output and input ends of the gas boiler at time t respectively; are the energy flow power at the output and input ends of the gas boiler at time t; η GB The energy conversion efficiency of the gas boiler; The carbon flow model of the electric boiler is specifically: In the formula, are the carbon flow densities at the output and input ends of the electric boiler at time t respectively; are the energy flow powers at the output and input ends of the electric boiler at time t; η EB is the energy conversion efficiency of the electric boiler; The carbon flow model of the electric energy storage is specifically: 1) Charging mode: When the electric energy storage is charged, it can be equivalent to a load, and the carbon flow inside it depends on the accumulation of electricity over time; therefore, during the charging process of the electric energy storage, the carbon flow charged into the electric energy storage at time t is: In the formula, The carbon flow charged into the energy storage during the charging process of the energy storage during the period t; is the charging power of the electric energy storage at time t; is the carbon potential of the electric bus at time t; Δt is the scheduling time interval; 2) Discharge mode: When the electric energy storage is discharged, it can be equivalent to a generator set, and the carbon flow inside it begins to be released along with the electricity. Therefore, considering the energy conversion rate during charging and discharging, the carbon flow released by the electric energy storage at time t is: In the formula, is the carbon flow rate released by the energy storage during the discharge process of the energy storage during period t; is the discharge power of the energy storage at time t; is the carbon flow density at the discharge port of the energy storage at time t; η BES is the discharge efficiency of electrical energy storage; is the carbon flow density of the internal electric energy of the electric energy storage at t-1; Therefore, the carbon flow density of the internal electric energy of the electric energy storage device at time t can be expressed as: In the formula, is the state of charge of the energy storage at time t; C BES The capacity of electrical energy storage; The carbon flow model of the thermal energy storage is: In the formula, is the carbon flow charged into the thermal energy storage during the charging process, during period t; is the charging power of thermal energy storage at time t; η HES It is the heat release efficiency of thermal energy storage; is the carbon flow density of the internal thermal energy of the thermal energy storage at t-1; is the charge state of the thermal energy storage at time t; C HES is the capacity of thermal energy storage; Step (1-3): Establish a carbon flow model on the load side of the multi-energy system, including carbon flow models of electricity and heat; The carbon flow model of the electric load of the multi-energy system is specifically: In the formula, is the carbon flow density of the electrical load at time t; and are the power generation power of the photovoltaic unit and its electric energy carbon flow density at time t respectively; The carbon flow model of the heat load of the multi-energy system is specifically: In the formula, is the carbon flux density of the heat load at time t; Step (1-4): Establish the objective function of the multi-energy system optimization scheduling problem, expressed as: In the formula, is the energy purchase cost of the multi-energy system at time t; Depreciation costs for charging and discharging energy storage; The carbon emission cost generated by energy consumption on the load side; Among them, the energy purchase cost of the multi-energy system at time t is specifically: In the formula, ε e,t and ε g，t are the electricity price of the upper power grid and the gas price of the upper natural gas grid at time t respectively; Among them, the depreciation cost of energy storage charging and discharging is specifically: In the formula, ω BES and ω HES are the depreciation cost coefficients of charging and discharging energy for electric energy storage and thermal energy storage respectively; Among them, the carbon emission costs generated by load energy consumption are specifically: Where, L e,t and L h，t are the electricity load demand and heat load demand at time t respectively; q is the carbon emission cost coefficient; Steps (1-5): Construct the constraints of the multi-energy system optimization scheduling model, including power balance constraints, upper and lower limits of the system's purchased energy power, equipment operation constraints, and load carbon flow density constraints; The multi-energy system power balance constraint includes an electric power balance equation constraint, a thermal power balance equation constraint and a gas power balance equation constraint, specifically: The upper and lower limits of the system's purchased energy power include two types of inequality constraints: purchasing electricity from the upper power grid and purchasing gas from the upper natural gas grid, specifically: Where P PG-min and P PG-max are the lower and upper limits of the power purchased from the upper power grid at time t; V S-min and V S-max are the lower and upper limits of the gas volume purchased from the upper-level natural gas network at time t, respectively; The equipment operation constraints include upper and lower limit constraints for energy conversion equipment operation and energy storage equipment operation constraints, specifically: 1) Energy conversion equipment operation constraints: Where P CHP-min and P CHP-max are the lower and upper limits of the output power of the cogeneration unit at time t; H GB-min and H GB -max are the lower and upper limits of the thermal power output of the gas boiler at time t; H EB-min and H EB-max are the lower and upper limits of the thermal power output of the electric boiler at time t respectively; 2) Operation constraints of energy storage equipment: In the formula, and are the maximum charging power and the maximum discharging power of the energy storage per unit time period respectively; u and It is a 0-1 variable, which represents the charging and discharging state of the energy storage at time t, and avoids simultaneous charging and discharging; In the formula, and are the maximum value of the thermal energy storage charging power and the maximum value of the thermal energy dissipation power per unit time period; u and It is a 0-1 variable, which represents the charging and discharging state of the thermal energy storage at time t, and avoids charging and discharging at the same time; The load carbon flow density constraint includes the electric load carbon flow density constraint and the thermal load carbon flow density constraint, specifically: In the formula, and The upper limits of carbon flow density for electrical load and thermal load respectively can make the carbon flow reasonably distributed in the system without the situation that a certain energy flow has a high carbon flow density, thus avoiding the emergence of carbon weak links.

3. The multi-energy system optimization scheduling method based on carbon flow theory and deep reinforcement learning as claimed in claim 1, characterized in that: The step (2) specifically comprises the following steps: The basic components of reinforcement learning include the state set S that represents the environment, the action set A that represents the agent's actions, and the reward r for the agent. The multi-energy system is the agent's environment, and the agent makes optimal scheduling decisions by adjusting the output of the equipment in the system. At time t, the environment provides the agent with the observed system state s t ∈S, the agent is based on the strategy π (strategy π is a function that maps state s to action a), that is, π; S→A→a=π(s) and the multi-energy system state s t Generate dynamic action t ; Step (2-1): Define the state space of the Markov decision process: Select the uncontrollable load demand L e，t , Uncontrollable heat load demand L h,t , Photovoltaic power generation Energy storage state of charge Thermal energy storage state of charge Grid electricity sales price ε e,t 、Natural gas network priceε g，t , Carbon flow density of grid electricity Carbon flow density of natural gas in the gas grid Action a in the previous scheduling period t-1 And the scheduling period t is used as the state space variable, specifically: In continuous control problems, the agent needs to select actions in a continuous action space, which requires the agent to accurately control its behavior. Taking the action at the previous moment as part of the state space helps the agent to better understand and utilize historical information, and helps the agent to more finely adjust its behavior, thereby making more accurate decisions and achieving more precise control. Step (2-2): Define the action space of the Markov decision process: In the dispatching period t, the action in the system can be represented by the output of the equipment, including the power purchased from the upper power grid, the power generation power of the cogeneration unit, the power consumption of the electric boiler, the charging and discharging power of the electric energy storage, and the charging and discharging power of the thermal energy storage. When the above variables are determined, the output of other parts can be obtained according to the corresponding mathematical model and equation constraints. The action space is specifically: Step (2-3): Define the reward function of the Markov decision process: The optimization goal of multi-energy system optimization scheduling is to minimize the total operating cost in steps (1-4). The goal of the Markov decision process is to maximize the reward within the scheduling cycle. At the same time, the penalty term is set according to the constraints and added to the immediate reward to obtain the final reward function. The reward function of the agent at time t is expressed as: Where w is the cost value scaling factor; It is a penalty item set to reduce the violation of constraints in the operation of multi-energy systems, including power imbalance penalty and equipment operating condition over-limit penalty; Step (2-4): Define the state-action value function of the Markov decision process: When a state s of the multi-energy system l When determined, the multi-energy system dynamically optimizes the scheduling action a l The quality of the state can be expressed using the state-action value function Q π (s, a) is used to evaluate the quality of the multi-energy system optimization scheduling action, specifically: In the formula, γ∈[0,1] is the discount factor, which indicates the influence ratio of the reward at a certain moment in the future on the cumulative reward. The larger γ is, the more importance is attached to the future reward. π (·) is the expectation under strategy π; The goal of dynamic optimization scheduling of multi-energy systems is to find the optimal strategy π* to maximize the state-action value function, specifically: In the formula, arg max represents the parameter that makes the function reach the maximum value.

4. The multi-energy system optimization scheduling method based on carbon flow theory and deep reinforcement learning as claimed in claim 1, characterized in that: The deep reinforcement learning algorithm selected for solving the Markov decision process is a deep deterministic policy gradient algorithm, and the deep deterministic policy gradient algorithm is improved by using a priority experience replay mechanism based on proportional priority and L2 regularization. The step (3) specifically includes the following steps: Step (3-1): Construct the value network, policy network, target value network and target policy network of the deep deterministic policy gradient algorithm; the value network minimizes the loss function L(θ Q ) to optimize the network parameters, specifically: L(θ Q )=E(y t -Q(s t ,a t |θ Q )) 2 ; In the formula, θ Q is the parameter of the value network; y t is the target Q value, expressed as: y t =r t +γQ′(s t+1 ,μ′(s t+1 |θ μ′ )|θ Q′ ); In the formula, s t+1 is the next state that the multi-energy system transfers to after executing the action; θ μ are the parameters of the policy network, θ Q′ and θ μ′ are the parameters of the target value network and the target policy network respectively; The policy network is based on gradient information Improve the action, the sampling policy gradient is expressed as: In the formula, J(θ μ ) is the performance objective function of the policy network; Step (3-2): In action a t Add random noise N t To increase the exploration ability of the network, it is expressed as: a t =μ(s t |θ μ )+N t ; Where N t Ornstein-Uhlenbeck (OU) noise is a random variable of the OU process, which is often used to simulate a set of noise associated with time. Step (3-3): Introduce the priority experience replay mechanism based on proportional priority into the deep deterministic policy gradient algorithm. First, determine the priority p of the sampled experience i. i =|δ i |+σ, where the rank of the i-th sample is p i According to the absolute timing difference error |δ i | is sorted, σ is a small positive constant used to ensure that all experiences can be sampled, |δ i |Specifically expressed as: |d i |=|r i +γQ′(s i+1 ,μ′(s i+1 |θ μ′ )|θ′)-Q(s i ,a i |θ Q )|; The probability P of sampling experience i i It is expressed as: In the formula, Determines the number of priorities used; If it is equal to 0, it is equivalent to random sampling; Since experiences with high temporal difference errors are replayed more frequently, the proportional priority-based experience replay mechanism may cause the training process to be prone to oscillation. To address this problem, importance sampling weights are used when calculating weight changes and the weights are normalized to ensure that the weights sum to 1: Where N b is the size of the experience replay pool; max j W j is the maximum weight in the current batch, used for normalization; β is a parameter used to correct the deviation. In order to allow the algorithm to explore more in the early stages of training, β is set to increase gradually over time. The adjustment formula for β is: Where β0 is the initial value, β max is the maximum value, t is the current training step number, and T is the total step number; The loss function is calculated using weighted loss: L′(θ Q )=E(W′ i (y t -Q(s t ,a t |θ Q ))) 2 ; Step (3-4): In order to reduce the risk of overfitting during network training, L2 regularization is introduced into the deep deterministic policy gradient algorithm to reduce the risk of overfitting during training. L2 regularization is introduced into the performance objective function J(θ μ ), the performance objective function and the corresponding gradient are expressed as: In the formula, α L is the regularization parameter; N is the mini-batch size; Step (3-5): Value network parameters θ Q The update method is expressed as: In the formula, μ Q is the learning rate of the value network; The parameters of the policy network are μ The update method is expressed as: Where: μ μ is the policy network learning rate; Step (3-6): Update the parameters of the target value network and the target strategy network, specifically: The parameters θ of the target value network and the target policy network Q′ and θ μ′ The update method is expressed as: i Q′ ←tth Q +(1-τ)θ Q′ ; i μ′ ←tth μ +(1-τ)θ μ′ ; Wherein, τ is the soft update coefficient, τ<<1.

5. The multi-energy system optimization scheduling method based on carbon flow theory and deep reinforcement learning as claimed in claim 1, characterized in that: The step (4) specifically comprises the following steps: Step (4-1): constructing an initial model, wherein the initial model includes a value network, a policy network, a target value network, and a target policy network; Step (4-2): input each sample state data into the initial model; generate predicted action data corresponding to the sample state data through the policy network, and obtain the predicted state-action value corresponding to the predicted action data through the value network; Step (4-3): based on the predicted action state value and the corresponding target action state value, a loss function calculation result is obtained, and based on the predicted action data and the corresponding predicted action state value, a gradient function calculation result is obtained, and the target action state value is calculated based on the target value network and the target strategy network; Step (4-4): adjusting the parameters of the value network according to the calculation result of the loss function, and adjusting the parameters of the policy network according to the calculation result of the gradient function; Step (4-5): storing the sample state data and the corresponding predicted action data, predicted action state value, and state data at the next moment in the experience replay pool, and determining the priority corresponding to the sample state data based on the absolute time difference error of the sample state data, so as to select the sample state data from the experience replay pool according to the priority to learn the value network and the policy network; Step (4-6): Determine the deep policy model based on the trained policy network; input the real-time state of the multi-energy system into the pre-trained deep policy network of the improved deep deterministic policy gradient algorithm, and output the scheduling action.

Images