A comprehensive energy system optimization method and system under multiple uncertain environments

Abstract

The present application relates to the technical field of integrated energy system, and provides an integrated energy system optimization method and system under multiple uncertain environments, comprising: obtaining the state of the integrated energy system, and obtaining the action of multi-energy flow equipment through an actor network; wherein, embedding the MPC actor into a double-delay deep deterministic policy gradient architecture, storing the transition tuples of the actor network and the MPC actor into an agent experience replay pool and an expert experience replay pool respectively, according to the time series difference error of the samples in the two experience replay pools, giving each sample a priority, determining a mixing ratio according to the time step of training, combining the mixing ratio and the priority, calculating the sampling probability of the sample, and then controlling the update of the critic network to have a high degree of dependence on the expert experience replay pool in the early training stage and a high degree of dependence on the agent experience replay pool in the later training stage. The performance is better in the IES environment coupled with multiple uncertainties.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of integrated energy systems, and particularly relates to an integrated energy system optimization method and system under multiple uncertain environments. BACKGROUND

[0002] The statements in this section merely provide background information related to the application and do not necessarily constitute prior art.

[0003] Compared with the traditional independent energy supply system, the integrated energy system (IES) can break through the operation barriers between different energy categories and improve the renewable energy consumption and system comprehensive energy efficiency due to the basic connotation of multi-energy complementation and coordinated optimization. However, the multi-physical field coupling characteristics of the IES also pose a huge hidden danger for the cross-domain transmission of multiple uncertainties when realizing multi-energy complementation, which directly threatens the system operation safety.

[0004] Existing research usually decouples the external uncertainty of the IES into three categories: renewable energy such as wind and light, multi-element load such as cold, heat and electricity, and energy price. Traditional scheduling methods mainly focus on a single or local uncertainty source. Based on the traditional mechanism methods such as robust optimization, stochastic optimization and information gap decision, pseudo joint modeling is realized through simple linear superposition or probabilistic scenario enumeration of constraint boundaries, which is difficult to depict the internal correlation of multiple uncertainties. The direct consequence is that a large number of boundary constraints and redundant scenarios need to be introduced to cover all possible scenarios, resulting in an overly conservative operation strategy and a large amount of resource waste in response to low-probability or even unachievable scenario combinations. In addition, the above methods usually require mathematical models or prediction information, and the reliability of the final decision depends largely on the accuracy of the system model and the prediction method, which is difficult to adapt to the high-dimensional non-convex uncertainty space of wind-light-cold-heat-electricity-price.

[0005] Artificial intelligence is reconfiguring the solution approach to energy problems. As a typical representative, deep reinforcement learning (DRL) can break through the curse of dimensionality of traditional mechanism modeling and mine the implicit correlation rules in the high-dimensional non-convex uncertainty space in a data-driven manner due to its end-to-end model-free learning characteristics. However, although DRL has significant advantages in theory, there are still many defects in practical application:

[0006] Firstly, representative algorithms such as double-delay policy gradient (TD3) are extremely sensitive to hyperparameters, have high difficulty in parameter tuning, weak generalization ability, and low lower limit although they have high upper limit, and are not stable enough;

[0007] Secondly, the reward function design is highly complex, and if the traditional operation cost is directly used as the reward, the reward converges, but the action of the agent still violates the system hard constraint. Even if the constraint condition is expressed by introducing a penalty term, it is still difficult to effectively guarantee the constraint feasibility;

[0008] Thirdly, the source-load-price environmental parameters in the IES have partial observability and cannot fully satisfy the Markov decision assumption, which limits the learning of the agent and causes unstable performance.

[0009] Finally, the IES has a high-dimensional joint state-action space, which leads to highly sparse rewards in the early training stage, and the agent often converges the action value to the boundary region without prior conditions, cannot fully explore the optimal strategy, and the training process converges slowly and has low efficiency.

[0010] However, this does not mean that DRL has fundamental limitations in the scheduling field. Its strong adaptive environmental representation capability provides an irreplaceable advantage for coping with cross-domain multiple uncertainties. The core challenge is how to improve the interaction between the DRL agent and the environment, while retaining the advantages of artificial intelligence and fitting the essence of the IES optimization problem. SUMMARY

[0011] To solve the technical problems in the background art, the present application provides a comprehensive energy system optimization method and system in a multiple uncertainty environment, proposes a twin delayed deep deterministic policy gradient (TD3) and model predictive control (MPC) collaborative optimization framework, and on this basis, designs a hybrid priority experience replay mechanism to balance the coordination of expert experience and autonomous exploration, dynamically adjusts the sampling weight through double buffering, prioritizes learning the demonstration experience generated by the MPC actor in the early training stage to avoid the blindness of the random policy, gradually increases the autonomous sampling weight as the interaction data accumulates, encourages the agent to explore autonomously, and adapts to the source-load-price cross-domain uncertainty correlation rules, and has better performance in the IES environment with multiple uncertainty coupling.

[0012] To achieve the above purpose, the present application adopts the following technical solutions:

[0013] The first aspect of the present application provides a comprehensive energy system optimization method in a multiple uncertainty environment, which comprises:

[0014] Obtaining the state of the comprehensive energy system;

[0015] Based on the state, obtaining the multi-energy flow device action through the actor network;

[0016] In the training process of the actor network, the MPC actor is embedded in a double-delay deep deterministic policy gradient architecture, the transition tuples of the actor network and the MPC actor are respectively stored in an agent experience replay pool and an expert experience replay pool, a priority is assigned to each sample in the two experience replay pools according to a time difference error of the sample, a mixing ratio is determined according to a time step of training, a sampling probability of the sample is calculated by combining the mixing ratio and the priority, and then the update of the critic network is controlled to have a high degree of dependence on the expert experience replay pool in the early stage of training and a high degree of dependence on the agent experience replay pool in the later stage of training, and the value gradient of the action of the critic network is used to guide the optimization of the output action of the actor network.

[0017] Further, the state includes wind power, light power, heat load, electricity load, gas load, electricity price, output electric power of the combined heat and power device, input electric power of the electric-to-gas device, and state of charge of the energy storage in a time segment with the current time period as a midpoint.

[0018] Further, the multi-energy flow device action includes a change amount of charging power, a change amount of discharging power, a change amount of electricity purchasing power, a change amount of input gas power of the combined heat and power device, a change amount of input electric power of the electric-to-gas device, a change amount of input electric power of the gas boiler, and a change amount of gas purchasing power.

[0019] Further, in the training process of the actor network, for each time step, the following steps are performed:

[0020] The MPC actor selects an action in a current time step state, interacts with the environment to obtain a reward and a next time step state, stores the transition tuple of the MPC actor in an expert experience replay pool, and sets a priority of each sample in the expert experience replay pool;

[0021] The actor network selects an action in a current time step state, interacts with the environment to obtain a reward and a next time step state, stores the transition tuple of the actor network in an agent experience replay pool, and sets a priority of each sample in the agent experience replay pool;

[0022] If the current time step is an integer multiple of an experience replay interval, the critic network is subjected to multiple mini-batch update operations based on the expert experience replay pool and the agent experience replay pool in combination with a target network, and the priority of the sample in the experience replay pool is updated;

[0023] If the current time step is an integer multiple of a set interval, the actor network and the target network are updated.

[0024] Further, the reward is the negative of the sum of a comprehensive energy system operation cost, a wind and light curtailment penalty term, and a device over-limit penalty term.

[0025] Further, the mini-batch update operation includes:

[0026] sample transition data from the experience replay pool according to a sampling probability of the sample;

[0027] for the sampled transition data, calculate an importance sampling weight based on the sampling probability;

[0028] in the current time step state, generate a target action by using a target actor network, add Gaussian noise to the target action, calculate a time difference target according to a target critic network, calculate a difference between the time difference target and a result of the critic network to obtain a time difference error;

[0029] update the priority of the sample in the experience replay pool according to the time difference error;

[0030] calculate a loss of the critic network based on the time difference error and the importance sampling weight, and update a parameter of the critic network.

[0031] Further, the sampling probability is:

[0032] ;

[0033] wherein P j is a sampling probability of the jth sample; a linear weight decay coefficient , and are initial and final decay coefficients respectively, T is a total training step number, l represents a current time step; type j = MPC represents that the jth sample is from an expert experience replay pool, type j = Agen t represents that the jth sample is from an agent experience replay pool; D agent and D mpc are the number of samples in the agent experience replay pool and the expert experience replay pool respectively; β is a priority intensity coefficient; p j is a priority of the jth sample, p z and p d represent the priority of the sample from the agent experience replay pool and the expert experience replay pool respectively.

[0034] A second aspect of the present application provides a comprehensive energy system optimization system in a multiple uncertain environment, comprising:

[0035] a data acquisition module configured to acquire a state of the comprehensive energy system;

[0036] an optimization module configured to obtain the multi-energy flow device action through the actor network based on the state;

[0037] In the training process of the actor network, the MPC actor is embedded into a double-delay deep deterministic policy gradient architecture, the transition tuples of the actor network and the MPC actor are respectively stored in an agent experience replay pool and an expert experience replay pool, a priority is given to each sample according to the time difference error of the samples in the two experience replay pools, a mixing ratio is determined according to the time step of training, the sampling probability of the sample is calculated by combining the mixing ratio and the priority, and then the update of the critic network is controlled to have a high dependence on the expert experience replay pool in the early training stage and a high dependence on the agent experience replay pool in the later training stage, and the value gradient of the action is guided by the critic network to guide the optimization of the output action of the actor network.

[0038] Further, the state includes wind power, light power, heat load, electricity load, gas load, electricity price, output electric power of the combined heat and power device, input electric power of the electric-to-gas device and state of charge of the energy storage in a time segment with the current time period as the midpoint.

[0039] Further, the multi-energy flow device action includes a change amount of charging power, a change amount of discharging power, a change amount of electricity purchasing power, a change amount of input gas power of the combined heat and power device, a change amount of input electric power of the electric-to-gas device, a change amount of input electric power of the gas boiler and a change amount of gas purchasing power.

[0040] Compared with the prior art, the present application has the following beneficial effects:

[0041] The present application proposes a TD3 and MPC collaborative optimization framework, and on this basis, a mixed priority experience replay mechanism is designed to balance the balance of expert experience and autonomous exploration, and through dynamic adjustment of the sampling weight of the double buffer, the demonstration experience generated by the MPC actor is preferentially learned in the early training stage to avoid the blindness of the random strategy, and the autonomous sampling weight is gradually improved with the accumulation of interaction data to encourage the autonomous exploration of the agent and adapt to the source-load-price cross-domain uncertainty correlation rule.

[0042] The TD3 and MPC collaborative optimization framework proposed in the present application inherits the ability of DRL to learn and process uncertainty from the environment and the ability of MPC to provide basic performance using a physical model, and compared with the traditional TD3, the TD3MPC reward function design is simple, the convergence speed is faster, the result is better and the parameter adjustment difficulty is significantly reduced.

[0043] The present application expands the single time state into an extended state space covering historical and future information, and models the state transition characteristics of the Markov decision process based on the probability scenario technology to enhance the generalization of TD3 to complex energy problems. BRIEF DESCRIPTION OF DRAWINGS

[0044] The accompanying drawings, which form a part of this specification, are included to provide a further understanding of the application, and are incorporated into and constitute a part of this specification. The embodiments of these drawings are shown to explain the present application and are not intended to limit the present application in any manner.

[0045] Figure 1 is a TD3MPC architecture diagram of embodiment one of the present application;

[0046] Figure 2 is an IES structure diagram of embodiment one of the present application;

[0047] Figure 3 is a TD3MPC convergence result of embodiment one of the present application;

[0048] Figure 4 is a convergence comparison result of embodiment one of the present application;

[0049] Figure 5 is a multi-scenario scheduling result of embodiment one of the present application. DETAILED DESCRIPTION

[0050] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application.

[0051] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those of ordinary skill in the art to which the present application belongs.

[0052] Embodiment one

[0053] The embodiment provides a comprehensive energy system optimization method in a multiple-uncertainty environment.

[0054] The core of solving the problems described in the background art is to compress the exploration space of the agent to the physically feasible domain and the high-value decision area, thereby improving the reward density. A feasible method is to introduce an expert guidance mechanism for DRL. Model predictive control (MPC) becomes the first choice of experts due to its open architecture design and mature theoretical system. In fact, MPC and DRL have their own advantages and disadvantages and complement each other. The cooperation of the two not only solves the problems of DRL such as super parameter sensitivity, sparse reward and poor interpretability through physical guidance, but also makes up for the inherent defects of MPC itself such as model dependence and short-sightedness. In view of the multiple uncertainties faced by IES, the cooperation of MPC and DRL is expected to play an important role.

[0055] The embodiment provides a comprehensive energy system optimization method in a multiple uncertain environment, proposes a TD3 and MPC collaborative optimization framework, in the framework, a model prediction control (MPC) is embedded into a twin delayed deep deterministic policy gradient (TD3) architecture as an expert actor, and a safe baseline action sequence conforming to physical constraints is provided through rolling horizon optimization. TD3 is further adaptively explored in a neighborhood based on data driving, compensates for model mismatch and adapts to the multiple uncertain environment. A hybrid priority experience replay mechanism is designed to coordinate the balance between expert experience and autonomous exploration. On this basis, a single time state is expanded into an extended state space covering historical and future information, and the state transition characteristics of a Markov decision process are modeled based on a probability scenario technology, so as to strengthen the generalization of TD3 to complex energy problems.

[0056] The comprehensive energy system optimization method in the multiple uncertain environment provided by the embodiment comprises the following steps:

[0057] Step 1, obtaining historical source-load-price data of the comprehensive energy system, establishing a park physical model, including an objective function (reward function), an equality constraint, an inequality constraint, and converting the same into a Markov decision process.

[0058] As shown in Figure 2 The studied northern IES demonstration park contains three kinds of heterogeneous energy: electricity, heat and gas, which are managed by a unified operator in a centralized manner. The park obtains energy from the upper-level power grid, gas network, wind power station and photovoltaic power station, and is interconnected through coupling devices such as combined heat and power (CHP), gas boiler (GB) and power-to-gas (P2G), forming a multi-energy flow coordination network to meet the multi-type energy demand of terminal users. The random fluctuations of renewable energy, multi-energy load and price signal are transmitted through physical coupling and market mechanism, which seriously affects the stable operation of the IES. It should be noted that the studied comprehensive energy system is a typical low-voltage distribution network structure, and the topological structure of the electric / thermal / gas network is compact and the transmission distance is short, and the influence of network power flow distribution by line parameters and pipeline pressure drop can be ignored. The comprehensive energy system is modeled according to the comprehensive energy system structure diagram.

[0059] (1) Energy conversion equipment group.

[0060] The energy conversion equipment in the comprehensive energy system includes CHP, P2G and GB, and the input and output power balance equations are as follows:

[0061] (1);

[0062] wherein, , They are respectively l CHP outputs electrical and thermal power during the time period; , These are the power generation and gas production efficiencies of CHP, respectively. for l CHP input gas power during the time period; , They are respectively l The output gas power and input electrical power of the P2G during the time period For electro-gas conversion efficiency; , They are respectively l The output thermal power and input electrical power of the EB during the time period The heat production efficiency of the electric boiler (EB); , They are respectively l Output thermal power and input gas power of GB during the time period The heat generation efficiency is GB.

[0063] (2) Energy storage system.

[0064] The integrated energy system is equipped with an energy storage system, and the energy charging and discharging balance model is as follows:

[0065] (2);

[0066] in, for l Energy storage over a period of time The self-loss coefficient, and They are respectively l Charge and discharge power during the period and These are the charge / discharge efficiencies, E bat The total capacity of the energy storage device. and These are 0-1 variables, where 1 represents the working state and 0 represents the non-working state. △ l Indicates a time interval.

[0067] (3) Rated power and ramping power constraints.

[0068] (3);

[0069] in, , , They are respectively l Time-of-use equipment i The output power and the upper and lower limits of the output power; For equipment power variation; , These are the upper and lower limits of climbing power, respectively.

[0070] (4) Power balance constraint.

[0071] (4);

[0072] In the formula, , , They are respectively l Changes in electricity, heat, and gas demand over a period of time. , They are respectively l Changes in electricity and gas purchases over time periods. for l The change in input electrical power of P2G over time period, for l Changes in photovoltaic power generation over a period of time for l Changes in wind power generation over a period of time , They are respectively l The changes in electrical and thermal power output of CHP over time period. , They are respectively l The changes in output thermal power and input gas power of GB during the time period. for l Changes in CHP input gas power over time period for l The change in the output gas power of P2G over a period of time.

[0073] Step 2: Adaptively improve the traditional TD3 state space by designing an extended state space that includes historical and future information to ensure the consistency of the Markov decision process; model the transition probabilities of random state variables based on probabilistic scenario generation technology to avoid interference from abnormal probability scenarios on the overall policy; establish an MPC Actor based on MPC on the basis of the traditional Actor network, explicitly consider the physical constraints of the state and input, generate a baseline control action sequence with safety and economy, and store it together with the state-reward signal in the expert experience replay pool (Demo buffer). Its objective function is co-designed with the improved TD3 reward function to ensure consistency of the optimization objective; the improved TD3 agent further explores the globally optimal policy in the neighborhood of the baseline control action taken from the Demo buffer, and stores the exploration experience in the agent experience replay pool (Agentbuffer).

[0074] The core advantage of MPC is that it can generate optimal control sequences by solving constrained optimization problems online with dynamic updated system states and environment information. It is worth noting that MPC and DRL have significant isomorphism in mathematical form, both of which are based on state variables, control variables and objective functions. The formal similarity is one of the reasons for choosing MPC for physical guidance. MPC Actor inherits the standard MPC framework, uses a state-space model to describe the system dynamics, and generates a baseline control strategy in a deterministic scenario. Unlike traditional MPC, which only uses the first-step control amount, MPC Actor stores the optimal control sequence and its corresponding state-reward information in the Demo buffer in the entire time domain, and constructs a training sample set that matches the extended state space. Each rolling horizon solves the following optimization problem:

[0075] (5);

[0076] where, z is the predicted time-domain output; z max and z min max and min represent the maximum and minimum values of the predicted time-domain output, respectively; w is the reference trajectory, which is obtained by one-day optimization once, and the optimization objective function is consistent with the TD3 reward function; and are weight factors; is the state variable matrix of the multi-energy flow coupling device; is the coefficient matrix; is the future control increment matrix of the multi-energy flow coupling device, △ u ( l ) represents the control increment in the l time period; H is a lower triangular matrix, and its non-zero elements are defined as ; is the future disturbance vector, including wind, light new energy, electricity, heat, gas load and electricity price; d ( l ) represents the disturbance variable in the l time period; In MPC Actor, the source-load-price random variable is assumed to be deterministic, obtained from the prediction model; N p and H d are the lengths of the prediction time domain and the control time domain, respectively, and the selection rules are introduced later; H d is a lower triangular matrix, and its non-empty elements are defined as ; I is an identity matrix; Tis a lower triangular matrix with all its non-zero elements being identity matrices; u min are the minimum and maximum values of the control variable respectively; u max are the minimum and maximum values of the control variable respectively; △ u min are the minimum and maximum values of the control variable respectively; u max are the minimum and maximum values of the control variable respectively; y min are the minimum and maximum values of the control variable respectively; y max are the minimum and maximum values of the system predicted output respectively; Q , N , N d are the coefficient matrices of the single-step state-space model respectively. M The single-step state-space model can be expressed as:

[0077]

[0078] (6). wherein,

[0079] , A , B B d , C is the coefficient matrix, which can be given according to the multi-energy flow coupling device constraints and power balance equations; the embodiment only gives the state-space model of the power system; the state variable matrix wherein, is the electricity purchase amount of the time period; the control variable matrix l ; ; the disturbance variable matrix ; according to the energy storage SOC constraint, the power balance constraint and other constraints, the following can be obtained: ; ; ; ; wherein, , .

[0080] TD3 algorithm is an improved Actor-Critic framework for high-dimensional continuous action space control problems. Compared with the traditional deep deterministic policy gradient, it has three technical innovations of double Critic network, delayed policy update and target policy smoothing, which significantly improves the stability and convergence efficiency of the policy. The IES is modeled as a partially observable Markov decision process, which can be represented by a five-tuple , in addition to multiple random variables, the state transition probability P will be implicitly defined by the dynamic equation; ​The reward discount factor is set to a fixed value of 0.9; the state space S、 The action space A and the reward function R are defined as follows:

[0081] The state space S DRL usually assumes that the system satisfies the Markov decision principle, i.e., the future state only depends on the current state and is independent of the previous state. However, the source-load-price random variables in IES often exhibit periodic or inertial characteristics. The Markovian assumption of the traditional TD3 algorithm is violated, and the policy that only depends on the current state of time is prone to suboptimal actions. Therefore, the embodiment constructs an extended state space, uses a time-series state segment with past and future information to replace a single state variable to represent the system state, and improves the traditional one-to-one state-action mapping to a many-to-one mapping to improve the completeness of state information. l The time period extended state space includes wind and solar power generation, thermal, electrical and gas load, electricity price and multi-energy flow coupling device operating state, and can be expressed as: ; wherein, is the state i of the time period l - n to l + n , specifically: represents the power of wind power in l-n to l+n , represents the power of photovoltaic in l-n to l+n , represents the electricity price in l-n to l+n , represents the electrical load in l-n to l+n , represents the thermal load in l-n to l+n , represents the gas load in l-n to l+n , represents the output electric power of CHP in l-n to l+ n , represents the input electric power of P2G in l-n to l+n , represents the SOC (state of charge) of energy storage in l-n to l+n ; l - n tol -1 period of state variables are true values; l to l + n Period of source-load-price random state variables depend on external point prediction model; n Can be determined according to the autocorrelation analysis result; The state variables of the coupling device are determined by the MPC optimization result.

[0082] Action space A : Action means adjustable variable. In this embodiment, the internal power balance of the integrated energy system is ensured by the action of the multi-energy flow device. The TD3 action space is consistent with the action sequence output by the MPC Actor, and can be described as: , wherein, represents the change amount of charging power in l-n to l+n period, represents the change amount of discharging power in l-n to l+n period, represents the change amount of purchased power in l-n to l+n period, represents the change amount of CHP input gas power in l-n to l+n period, represents the change amount of P2G input electric power in l-n to l+n period, represents the change amount of GB input gas power in l-n to l+n period, represents the change amount of purchased gas power in l-n to l+n period; All actions in the state space are obtained during optimization, but only the first action is executed.

[0083] State transition probability P : The traditional TD3 algorithm is based on the assumption of Markov decision process, and only models the state transition through a single sample trajectory, which is difficult to describe the randomness of state transition caused by the multiple uncertainties of source-load-price in IES. Specifically, after l action a l , l the random state variables in the +1 period cannot be accurately obtained. If only relying on the deterministic prediction value will result in insufficient adaptability of the policy network to the random environment.

[0084] This embodiment simulates the uncertainty in the Markov decision-making process based on probabilistic scenario generation technology, and probabilistically models the transition characteristics of the random state space. The transition characteristics of the remaining states can be obtained from the system dynamics of the MPC Actor. The TD objective is updated using the expected value of the action value function of the random state. The update process can be expressed as:

[0085] (7);

[0086] in, y l Indicates TD target, express i A target critic network, These are the parameters of the i-th target critic network. As a discount factor, r l for l Momentary rewards; E ( ) represents the expectation function. Clustering is performed on five random scenarios representing future wind, solar, cooling, heating, and power generation. Since the clustering results in a discretized set of scenarios... It can be represented as:

[0087] (8);

[0088] Among them, s k,l+1 for l The k-th possible scenario in time interval +1 p ( s k,l+1 | s l , a l ) for scene s k,l+1 The probability of occurrence M For the number of clustering scenarios, For the k-th scenario l Actions during time interval +1; for ease of analysis, a random state space is defined. ;exist l time, s l+1 middle l - n to l The random state space at time -1 w l+1 ( l - n , l -1) is known, while w l+1 ( l , l+n+1) there are multiple possibilities; in order to guarantee the matching of source-load-price multiple scenarios and w l+1 ( l - n , l -1) and w l+1 ( l , l+n+ 1) the matching of scenarios, based on Latin hypercube sampling, superimposes error generation on the basis of point prediction results l to l+n+1 the random state variable scenario set at time W ; then based on K-means clustering, the initial scenarios are reduced, thereby obtaining p( w k , l+1 | w l , a l ) and w k,l+1 ; since the remaining states are deterministic, p( w k , l+1 | w l , a l ) is , w k,l+1 and the combination of the determined state is The natural matching of the multi-element prediction result can ensure that the generated scenario set meets the consistency requirements in the time and space dimensions; if the prediction model is lacking, the initial scenario can be obtained by sampling the fitted probability distribution during scenario generation.

[0089] Reward function R : DRL realizes long-term reward maximization through continuous trial and error-feedback-evaluation, and sets the opposite number of the optimization goal as the immediate reward r l , which can be expressed as:

[0090] (9) ;

[0091] F 1 is the comprehensive energy system operation cost, which can be expressed as:

[0092] (10) ;

[0093] Among them, , are the electricity price and gas price at time l , respectively , respectively are the power purchased at time t and g t respectively; l , respectively are the power purchased at time t and g t respectively; l i respectively are the operation and maintenance cost and output power of the i-th device at time t.

[0094] F 2 is the penalty term of abandoned wind and light, which can be expressed as:

[0095] (11);

[0096] wherein, , respectively are the abandoned wind and light cost coefficients at time t; l , respectively are the abandoned wind and light amounts.

[0097] 3 is the penalty term of device over-limit, which is added to the immediate reward to obtain the final reward function according to the penalty mechanism of the constraint condition. In order to reduce the fitting difficulty and improve the accuracy and convergence stability of the result, the linear penalty is used instead of the step penalty form in the embodiment. For any constraint F ( i ) ≥ 0, the penalty term is designed as follows: g x i ) ≥ 0, the penalty term is designed as follows:

[0098] (12);

[0099] wherein, β i is a penalty coefficient, and a constant coefficient is set according to different over-limit penalties; g ( x i ) is the over-limit penalty containing the output of each device and the climbing.

[0100] Step 4, in order to coordinate the balance between expert experience and autonomous exploration, a hybrid priority experience replay mechanism is designed, and the sampling weight is dynamically adjusted through double buffering. In the early stage of training, the demonstration experience generated by the MPC Actor is preferentially learned to avoid the blindness of the random strategy. With the accumulation of interaction data, the autonomous sampling weight is gradually improved to encourage the agent to explore autonomously and adapt to the source-load-price cross-domain uncertainty association rules.

[0101] ​Experience replay mechanism: Prioritized experience replay improves the traditional experience replay by introducing a priority and importance weighting mechanism, prioritizing samples with high TD errors to improve the efficiency of policy improvement. Considering that the experience replay pool of the proposed framework consists of two parts, Agent buffer and Demo buffer, this embodiment designs a new hybrid prioritized experience replay mechanism to balance the conflict between policy exploration and knowledge transfer. It should be pointed out that the prioritized experience replay mechanism is not necessary when the proposed framework is migrated and applied. If the reward signal of the target problem is dense and the environment is stable and dynamic, uniform replay can be directly used to avoid additional computational overhead, thereby reducing the training complexity.

[0102] According to the TD error, a priority is given to each sample:

[0103] (13);

[0104] wherein, is an expert experience priority decay coefficient, which is used to reduce the overfitting risk of expert samples; is a very small constant to avoid zero priority; Q (*) represents a critic network, and are the state and action of the jth sample, respectively; is the parameter of the critic network, i =1 or 2, i.e., there are two critic networks; is the time difference (TD) target of the jth sample.

[0105] Define the linear weight decay coefficient , which controls the mixing ratio of expert experience and autonomous exploration experience:

[0106] (14);

[0107] wherein, and are the initial and final decay coefficients, respectively; the initial one depends entirely on expert experience, and the final one is mainly autonomous exploration; T is the total number of training steps; l represents the current time step.

[0108] Combined with the mixing ratio and the priority, the sampling probability of the sample j is defined as:

[0109] (15);

[0110] wherein, type j = MPC represents that the jth sample comes from the Demo buffer,p j is the priority of the jth sample, p z and p d represent the priority of the samples from Agent buffer (agent experience replay pool) and Demo buffer (expert experience replay pool), respectively, D agent and D mpc denote the number of samples in the two experience replay pools Agent buffer and Demo buffer, respectively; is an indicator function, which equals to 1 if the sample is from the same source, otherwise 0; β is the priority intensity coefficient, which equals to 0 means uniform sampling; is to eliminate the bias introduced by priority sampling, and the importance sampling weight is calculated as:

[0111] (16);

[0112] where, denotes the total number of samples in the two experience replay pools, P j denotes the sampling probability of sample j, α is a hyperparameter to determine the impact of counteracting priority experience replay on the convergence result.

[0113] corresponds to:

[0114] (17);

[0115] where, denotes the loss function with importance weighting, N is the number of samples extracted at a time.

[0116] So far, the collaborative optimization framework based on TD3MPC has been established, and the overall framework is shown in Figure 1 , which includes:

[0117] (1) Initialize parameters: small batch data size k , experience replay interval K , total training step number T ;

[0118] (2) Initialize network: use random parameters , and to initialize the two Critic networks (critic network) Q and Actor network (actor network) π, respectively.

[0119] (3) Initialize target networks: initialize the target networks (target critic network 1, target critic network 2 and target actor network) parameters as , and .

[0120] (4) Initialize experience replay pools: create two types of experience replay pools D MPC and D Agent .

[0121] (5) Start the training loop, for each time period (time step) l :

[0122] (501) Interact based on MPC policy and store experience: obtain the comprehensive energy system l time period state s l , select action s l under the state a l,1 , that is a MPC , interact with the environment (through formula (9)) to get reward r l,1 and next time period state s l+1,1 ; store the transition tuple of the MPC actor s l , a l,1 , r l,1 , s l+1,1 into D MPC , and set the priority p l,1 to .

[0123] (502) Interact based on regular policy and store experience: select action s l under the state a l,2 according to the actor (actor network), interact with the environment to get reward r l,2 and next time period state s l+1,2 ; store the transition tuple of the actor network s l , a l,2 , r l,2 , s l+1,2) into D Agent , priority p l,2 set to .

[0124] (503) Determine whether to trigger update: if the current step number l is an integer multiple of the experience replay interval K, if yes, execute:

[0125] (A) Mini-batch update loop: perform k times of mini-batch update operation, that is, for j = 1 ~ k, execute:

[0126] (a) Sample experience: sample transition data from the experience replay according to the probability distribution P j defined by formula (15);

[0127] (b) Calculate importance sampling weight: for the sampled transition data, calculate the importance sampling weight w j according to formula (16) based on the probability distribution P j , used to correct the bias caused by priority sampling;

[0128] (c) Calculate the noisy target action: in the current time step state, use the target Actor network to generate the target action, add noise to the target action, calculate the TD target (target value, including target 1 and target 2) according to the target critic network and according to formula (7) y l ;

[0129] When calculating y l , add Gaussian noise to the action to prevent the policy from overfitting to extreme actions, which can make the target Q value smoother and enhance the robustness of the policy:

[0130] ;

[0131] wherein, is the parameter of the target Actor network, is a Gaussian distribution with mean 0 and standard deviation , and c takes 0.5.

[0132] (d) Update experience priority: calculate the difference between the TD target and the critic network result, calculate the time difference (TD) error , which measures the difference between the predicted value and the target value; according to formula (13), update the priority of the corresponding experience in the replay buffer according to the calculated TD error p j (including p j,1and p j,2 , let the subsequent sampling pay more attention to high-value experience;

[0133] (e) Calculate Critic network loss: based on TD target y l and importance sampling weight w j , according to formula (17), calculate the loss L(θ i ) of Critic network (including L1 and L2), that is, let the current Q network prediction value approach the target value calculated by reward, discount factor and target network, and the error is squared expectation;

[0134] (f) Update Critic network parameters: use the gradient of loss function L(θ i ) to update Critic network (including first Critic network, referred to as Critic1, and second Critic network, referred to as Critic2) parameters θ i , so that the Critic network can more accurately evaluate the value of action;

[0135] (g) End of mini-batch update cycle: complete k times of mini-batch update.

[0136] (B) Determine whether to update Actor and target network: if the current step l is an integer multiple of the set interval d, execute:

[0137] (a) Update Actor network parameters: update Actor network parameters with deterministic policy gradient, the formula is , with the help of the value gradient of action of Critic network, guide the Actor to output better action;

[0138] (b) Update target network parameters: update target network parameters in soft update mode, the formula is:

[0139] ;

[0140] ;

[0141] Let the target network slowly follow the current network to improve the stability of training; wherein, is the weight coefficient;

[0142] (c) End of Actor and target network update judgment: complete the Actor and target network update related operations.

[0143] (C) End of trigger update judgment: complete the update operation.

[0144] (504) Prepare the next state: add exploration noise to select the next action, and update the current state s l to s l+1 , and enter the next round of training cycle.

[0145] (6) When l the total number of training steps T is reached, the training cycle is ended.

[0146] After training, the actual actor network is used, and other networks are used during training. In actual application, the state is input to obtain the device action. That is, the state of the integrated energy system is obtained, and the multi-energy flow device action is obtained through the trained actor network.

[0147] To verify the effectiveness of the proposed method, a certain IES demonstration park in the north is selected for simulation verification. The hardware environment of the example simulation is an Intel(R) Core(TM) i9-14900HX CPU, 32GB memory notebook computer. Model construction and training are carried out under the PyTorch deep learning framework.

[0148] The convergence of the agent during the training process is shown in Figure 3 . In the early stage of training, the algorithm is in the exploration stage, and since the initial action selection is based on the deterministic environment, the agent obtains a small reward value after decision-making. After about 3000 rounds of training, the agent begins to gradually converge to a stable reward interval, indicating that it has learned the optimal scheduling strategy. Due to the multiple uncertainties of the source-load-price environment and the noise mechanism of the algorithm, the reward value of the agent will inevitably oscillate during the training process.

[0149] To further highlight the convergence performance of TD3MPC, Figure 4The average reward convergence of different algorithms under 10 random seeds is presented. The shaded area in the figure represents the maximum and minimum value range of the training results, and the dark broken line represents the average value of the 10 training results after smoothing. To clarify the characteristics of the TD3 algorithm itself and the advantages of state space adaptive improvement, the improved TD3, which introduces state probability transitions and temporal segment representation, is compared with the traditional TD3. From the convergence results, the improved TD3 stabilizes at 9000 steps, while the traditional TD3 gradually stabilizes at 14000 steps, and the final convergence reward of the improved TD3 is slightly higher. This is because the improved TD3 effectively avoids unreasonable state transitions by modeling the random transition characteristics of the state space, thereby obtaining a more accurate Q-value estimate and ensuring the direction of algorithm updates. Moreover, compared with the traditional single-time state variables, the expanded state space provides richer temporal information for the mapping from environment to policy. To demonstrate the positive gain of MPC for DRL, the improved TD3 is further compared with TD3MPC. TD3MPC, which incorporates MPC expert experience guidance, converges rapidly after about 4000 rounds. Compared to the initial completely random exploration mode, the introduction of MPC Actors significantly improves the starting point of exploration, substantially increases the initial reward, avoids blind trial and error in ineffective action domains, and accelerates the rate of increase in accumulated rewards in the early stages. In the later stages, the agent continuously learns through hybrid priority experience replay and interaction with the environment, further exploring based on expert experience. This allows it to break out of the sample space with small fluctuation ranges when the reward function tends to a local optimum, and explore towards the optimal solution, resulting in a significant improvement in convergence rewards in the later stages.

[0150] Because the test site has abundant wind and solar resources, the load is largely met by renewable energy sources. This leads to a significant impact on power supply stability due to the volatility of wind and solar power. Furthermore, the uncertainty of electricity prices and diverse loads makes power balancing extremely difficult. To verify the effectiveness of the proposed algorithm, one day was randomly selected from winter, summer, and the transition season as a test set, and the scheduling results were analyzed. Figure 5 It can be seen that the proposed strategy can basically maintain the power supply and demand balance. Since there is no heat load demand in summer, the energy purchase cost and operation and maintenance cost are the lowest. The randomly selected transition season test day is close to winter, so the costs of both are similar.

[0151] From the equipment output, most of the electricity load gap is supplied by CHP due to the low-cost effect of combined heat and power, and the grid acts as a backup resource to output at a lower electricity price. The energy storage system operating mode is similar in winter, summer, and autumn, charging when there is abundant wind and solar resources, and discharging when the electricity price is high. It is worth noting that the agent of TD3MPC does not blindly choose energy storage charging when renewable energy is abundant. It can comprehensively evaluate the future trend of electricity, gas prices and multi-load changes through real-time information, and cooperatively control P2G and energy storage system to consume renewable energy, which can ensure that the battery completes energy storage before the demand peak, and is strong in the whole. Due to the high cost of P2G, in the period without pressure of renewable energy consumption, the gas demand gap is supplemented by the gas network. In autumn and winter, the heat load is mainly supplied by CHP. Since its operation is affected by the multi-energy flow of electricity, gas, and heat, the agent flexibly controls its operation in the heat-determining electricity or electricity-determining heat mode, and the remaining heat load gap is met by the GB.

[0152] Table 1, optimal scheduling results of different schemes

[0153]

[0154] From Table 1, TD3MPC is superior to other models in all indicators, closest to the ideal situation. Compared with TD3, the cost of abandoning wind and light, the cost of abandoning load and the operation cost of Improved TD3 are reduced by 53.93%, 42.86% and 4.02% respectively. This means that TD3 tends to adopt a more conservative operation strategy and invest more controllable devices to support the load. However, the increase in cost does not improve the situation of abandoning wind and light and load. This is because unreasonable state transition is almost impossible to occur in actual operation, and excessive attention to it leads to meaningless increase in cost. And IES has strong state parameter time sequence correlation, which violates the Markov principle and leads to insufficient decision reference. Improved TD3 solves the above problems through improvement of state space, significantly improving the adaptability of the agent to the source-load-price multi-uncertain environment. Compared with Improved TD3, the cost of abandoning wind and light, the cost of abandoning load and the total cost of TD3MPC are further reduced by 17.28%, 23.74% and 3.97%, which shows that the MPC actor can guide the Improved TD3 to converge to the optimal by giving a baseline strategy that meets the physical constraints. In fact, in the pre-research stage of this study, RMPC and stochastic MPC were tried as expert strategies to guide DRL. However, due to the significant performance degradation of such MPC variants considering environmental uncertainty in the IES multi-dimensional uncertain environment, it is impossible to provide a stable and correct exploration starting point. In contrast, deterministic MPC is more efficient due to its stability. Without post-adjustment of Improved TD3, only MPC is difficult to obtain a reliable operation strategy, although its operation cost is low, but due to shortsightedness and environmental noise, it cannot avoid the risk of multiple uncertainties, and the penalty for abandoning wind, light and load is extremely high. In summary, the TD3MPC framework that cooperates Improved TD3 and MPC has better performance in the IES environment with multiple uncertain couplings.

[0155] The RMPC and RO based on the worst-case scenario have lower amounts of abandoned wind, light and load, which is similar when facing local uncertainty. However, in the source-load-price multi-uncertain environment, multiple worst-case scenarios need to be modeled, and the increase in cost becomes unacceptable. Due to the difficulty in depicting the joint probability distribution of multiple random variables, RO and CCP, although they pay a higher cost, still cannot obtain a reliable operation strategy. Compared with traditional optimization methods, DRL has stronger adaptability to multi-uncertain environment with the help of deep learning's powerful representation ability, and TD3MPC further enhances this feature.

[0156] In this embodiment, MPC and DRL present significant complementary characteristics in IES scheduling. The TD3MPC framework inherits the ability of DRL to learn to deal with uncertainty from the environment and the ability of MPC to provide basic performance with a physical model. Compared with traditional TD3, the TD3MPC reward function design is simple, the convergence speed is faster, the result is better, and the parameter tuning difficulty is significantly reduced.

[0157] In this embodiment, the hybrid priority experience replay mechanism based on the priority experience replay mechanism balances the efficiency of expert experience utilization and exploration diversity through a double-weight priority design, which is more suitable for the training of TD3MPC, and the training time is shortened by 20.87% and the result is better.

[0158] In this embodiment, the strong time sequence correlation of the IES state variable causes the Markov decision assumption of the traditional DRL to fail. Expanding the state space to replace the single-time random variable with an ACF greater than 0.7 state variable sequence can effectively eliminate the pseudo-Markov interference. On this basis, combined with the probability scenario generation technology, the extended state space transition characteristics are explicitly modeled, which significantly improves the agent's estimation ability of uncertainty.

[0159] Embodiment two

[0160] The embodiment provides a comprehensive energy system optimization system in a multiple uncertainty environment, which specifically comprises:

[0161] A data acquisition module configured to acquire a state of the comprehensive energy system;

[0162] An optimization module configured to obtain a multi-energy flow device action through an actor network based on the state;

[0163] In the training process of the actor network, the MPC actor is embedded in the double-delay deep deterministic policy gradient architecture, the transition tuples of the actor network and the MPC actor are respectively stored in the agent experience replay pool and the expert experience replay pool, the priority of each sample is determined according to the time sequence difference error of the samples in the two experience replay pools, the mixing ratio is determined according to the time step of training, the sampling probability of the sample is calculated by combining the mixing ratio and the priority, and then the update of the critic network is controlled. The dependence on the expert experience replay pool is high in the early training stage, and the dependence on the agent experience replay pool is high in the later training stage, and the value gradient of the action is guided by the critic network to guide the actor network to optimize the output action.

[0164] Further, the state includes wind power, light power, heat load, electricity load, gas load, electricity price, output electric power of the combined heat and power device, input electric power of the electric-to-gas device, and state of charge of the energy storage in a time segment with the current period as the midpoint.

[0165] Further, the multi-energy flow device action includes a variation amount of charging power, a variation amount of discharging power, a variation amount of power purchase, a variation amount of input gas power of a combined heat and power device, a variation amount of input electric power of an electric-to-gas device, a variation amount of input electric power of a gas boiler, and a variation amount of gas purchase.

[0166] It should be noted that the modules in the embodiment correspond to the steps in the first embodiment one by one, and the specific implementation process is the same, which will not be repeated here.

[0167] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for integrated energy system optimization under multiple uncertainties, characterized in that, The method comprises the following steps: acquiring historical source-load-price data of the integrated energy system, converting the data into a Markov decision process to obtain a state of the integrated energy system; based on the state, obtaining a multi-energy flow device action through an actor network; during the training process of the actor network, embedding an MPC actor into a double-delay deep deterministic policy gradient architecture, storing transition tuples of the actor network and the MPC actor into an agent experience replay pool and an expert experience replay pool respectively, assigning a priority to each sample according to a time difference error of the sample in the two experience replay pools, determining a mixing ratio according to a time step of the training, combining the mixing ratio and the priority to calculate a sampling probability of the sample, and then controlling the update of a critic network to have a high degree of dependence on the expert experience replay pool in the early stage of the training and a high degree of dependence on the agent experience replay pool in the later stage of the training, and guiding the actor network to optimize the output action by means of the value gradient of the action of the critic network; during the training process of the actor network, for each time step, the following steps are performed: the MPC actor selects an action under the current time step state, interacts with the environment to obtain a reward and a next time step state, stores the transition tuple of the MPC actor into the expert experience replay pool, and sets the priority of each sample in the expert experience replay pool; the actor network selects an action under the current time step state, interacts with the environment to obtain a reward and a next time step state, stores the transition tuple of the actor network into the agent experience replay pool, and sets the priority of each sample in the agent experience replay pool; if the current time step is an integer multiple of an experience replay interval, the critic network is subjected to multiple small batch update operations based on the expert experience replay pool and the agent experience replay pool in combination with a target network, and the priority of the samples in the experience replay pool is updated; if the current time step is an integer multiple of a set interval, the actor network and the target network are updated; the small batch update operation comprises: sampling transition data from the experience replay pool according to the sampling probability; for the sampled transition data, calculating an importance sampling weight based on the sampling probability; under the current time step state, generating a target action by using a target actor network, adding Gaussian noise to the target action, calculating a time difference target according to a target critic network, calculating a difference between the time difference target and a result of the critic network to obtain a time difference error; updating the priority of the samples in the experience replay pool according to the time difference error; calculating a loss of the critic network based on the time difference error and the importance sampling weight, and updating a parameter of the critic network; the sampling probability is: ； where P j is the sampling probability of the jth sample; linear weight decay coefficient , and are the initial and final decay coefficients, respectively, T is the total number of training steps, l denotes the current time step; type j = MPC represents that the jth sample comes from the expert experience replay pool, type j = Agen t represents that the jth sample comes from the agent experience replay pool; D agent and D mpc are the number of samples in the agent experience replay pool and the expert experience replay pool, respectively; β is the priority intensity coefficient; p j is the priority of the jth sample, p z and p d represent the priority of the samples from the agent experience replay pool and the expert experience replay pool, respectively.

2. The method of claim 1, wherein, the state comprises wind power, light power, heat load, electricity load, gas load, electricity price, output electric power of a combined heat and power device, input electric power of an electric-to-gas device, and state of charge of energy storage in a time segment with a current time period as a midpoint.

3. The method of claim 1, wherein, the multi-energy flow device action comprises a change amount of charging power, a change amount of discharging power, a change amount of purchased electricity, a change amount of input gas power of the combined heat and power device, a change amount of input electric power of the electric-to-gas device, a change amount of input electric power of a gas boiler, and a change amount of purchased gas.

4. The method of claim 1, wherein, The reward is the opposite of the sum of the integrated energy system operation cost, wind and light abandonment penalty term, and equipment over-limit penalty term.

5. A comprehensive energy system optimization system under multiple uncertain environments, characterized in that, The integrated energy system optimization method in a multiple uncertain environment according to any one of claims 1-4 comprises: a data acquisition module configured to acquire a state of the integrated energy system; an optimization module configured to obtain a multi-energy flow device action through an actor network based on the state; In the training process of the actor network, the MPC actor is embedded into a double-delay deep deterministic policy gradient architecture, the transition tuples of the actor network and the MPC actor are respectively stored in an agent experience replay pool and an expert experience replay pool, a priority is assigned to each sample according to the time difference error of the samples in the two experience replay pools, a mixing ratio is determined according to the time step of the training, the sampling probability of the sample is calculated by combining the mixing ratio and the priority, and then the update of the critic network is controlled to have a high degree of dependence on the expert experience replay pool in the early training stage and a high degree of dependence on the agent experience replay pool in the later training stage, and the critic network guides the optimization of the output action of the actor network by means of the value gradient of the action.

6. The integrated energy system optimization system under multiple uncertain environments of claim 5, wherein, The state includes wind power, light power, heat load, electricity load, gas load, electricity price, output electric power of the combined heat and power device, input electric power of the electric-to-gas device, and state of charge of the energy storage in a time segment with the current time period as the midpoint.

7. The integrated energy system optimization system under multiple uncertain environments of claim 5, wherein, The multi-energy flow device action includes a change amount of charging power, a change amount of discharging power, a change amount of purchased electricity, a change amount of input gas power of the combined heat and power device, a change amount of input electric power of the electric-to-gas device, a change amount of input electric power of the gas boiler, and a change amount of purchased gas power.

Images