A near-end strategy optimization scheduling method considering weak links of integrated energy system

Abstract

The application provides a near-end strategy optimization scheduling method considering weak links of a comprehensive energy system. The method adopts a near-end strategy optimization method to alternately perform interaction between sampling data and optimization of a clipped agent objective function using a stochastic gradient descent, evaluate the weak links, and schedule the comprehensive energy system, so that the weak links have greater safety, and the stability of operation of the comprehensive energy system is improved.

Description

Technical Field

[0001] This invention belongs to the fields of integrated energy systems, energy, and artificial intelligence, and relates to a scheduling method based on deep reinforcement learning control methods that considers the weak links of integrated energy systems. Background Technology

[0002] Existing integrated energy system scheduling methods do not take into account the weak links in the integrated energy system and the real-time changes that occur in these weak links.

[0003] Existing methods for assessing weaknesses are extremely difficult to correct in existing integrated energy systems, and cannot completely correct weaknesses in integrated energy systems, especially those dynamically identified when parameters and structures change.

[0004] Therefore, a near-end strategy optimization scheduling method that considers the weak links in the integrated energy system is proposed to avoid the problem of weak links in the integrated energy system, while providing scheduling instructions. Summary of the Invention

[0005] A near-end strategy optimization scheduling method considering the weak links in an integrated energy system is proposed. This method employs near-end strategy optimization scheduling to schedule the weak links in the integrated energy system, exhibiting real-time and deterministic characteristics. The steps in its application are as follows:

[0006] Step (1) Based on the characteristics of the integrated energy system, establish an integrated energy system model to describe the connection methods of each link, the coupling between links, and the energy conversion of links;

[0007] An integrated energy system includes the energy supply network, energy exchange, and energy storage components.

[0008] The energy supply network provides energy and inputs it into the energy exchange and energy storage links;

[0009] The energy exchange process enables the exchange of energy between the energy supply network and the energy storage process, thereby achieving coupling between the two processes.

[0010] The energy storage process utilizes energy storage devices to store electrical, thermal, and gaseous energy.

[0011] Establish models for the energy supply network, energy exchange, and energy storage links;

[0012] Step (2) Initialize the parameters of the near-end policy optimization scheduling method;

[0013] The proximal policy optimization scheduling method is a model-free, online, policy-compliant policy gradient reinforcement learning method. It addresses the problem of weak links by alternately using stochastic gradient descent as an alternative objective function between environmental interaction sampling data and optimized pruned data.

[0014] The parameters for the initialization of the near-end policy optimization scheduling method are: Initialization of the near-end policy optimization scheduling method Current network random parameter values Initialize the near-end policy optimization scheduling method Current network random parameter values ;

[0015] Step (3): Obtain the status of the integrated energy system ;

[0016] (1)

[0017] in,

[0018] ×100% (2)

[0019] (3)

[0020] (4)

[0021] (5)

[0022] (6)

[0023] (7)

[0024] In the formula, Indicates the photovoltaic energy absorption rate; This indicates the maximum output power of photovoltaic theory; Indicates the actual amount of photovoltaic power absorbed; Indicates the quantity of photovoltaic power; Indicates carbon dioxide emissions; express The electrical power of photovoltaic power at any given moment; express The actual electrical power of photovoltaic power at any given time; This indicates the scheduling time, taken as 1 hour; and These are respectively expressed as the annual carbon dioxide emissions corresponding to electricity generation and heat production, and the annual emissions corresponding to the purchase and consumption of electricity from the grid; This indicates the carbon dioxide emission factor corresponding to the use of natural gas; This represents the carbon dioxide emission coefficient corresponding to the electricity consumption; This indicates the amount of natural gas energy consumed; Indicates the amount of electricity purchased;

[0025] (8)

[0026] (9)

[0027] (10)

[0028] (11)

[0029] (12)

[0030] In the formula, This indicates the total cost of expanding the integrated energy system; This indicates the investment cost for expanding the equipment capacity of the integrated energy system; Indicates annual operating costs; This indicates the annual maintenance cost; This indicates the total number of units required for expansion of the integrated energy system equipment. Indicates device Annual value coefficient; Indicates device Investment cost per unit capacity; Indicates device Increase capacity; Indicates the discount rate; Indicates device Lifespan; express Electricity price at any time; Indicates the price of natural gas; and They are respectively represented as The power purchased from the power grid at all times and the power input from the combined heat and power plant;

[0031] (13)

[0032] (14)

[0033] (15)

[0034] (16)

[0035] (17)

[0036] In the formula, Indicates photovoltaic; Indicates combined heat and power (CHP); Indicates a heat pump; Indicates electrical energy storage; Indicates thermal energy storage; , , , and These are respectively represented as maintenance costs for photovoltaic, combined heat and power, heat pump, electric energy storage, and thermal energy storage; , , , and The maintenance costs per unit power for photovoltaic, combined heat and power, heat pump, electric energy storage, and thermal energy storage are respectively expressed as follows: express The electrical power output of the cogeneration at all times; express The thermal power output of cogeneration at all times; express The heat power of the heat pump at all times; and They are respectively represented as The absolute values ​​of the power of electrical energy storage and thermal energy storage, and the absolute values ​​of the power of energy release at any given moment;

[0037] Integrated energy system equipment The operational constraints are:

[0038] (18)

[0039] The power balance constraints that the integrated energy system must meet are:

[0040] (19)

[0041] The integrated energy system must satisfy the following heat and power balance constraints:

[0042] (20)

[0043] In the formula, Represented as Time device The power value; Indicates the modified equipment The upper limit of output; Indicates the equipment before modification. The upper limit of output; and express The charging and discharging power of the energy storage device at all times; express The electrical power of the heat pump at all times; Represented as Constant electrical load power; and Represented as The charging and discharging power of the electrical energy storage at any given time; Represented as Constant heat load power; Represented as The charging power of thermal energy storage at all times;

[0044] Step (4) Optimize scheduling method according to near-end strategy - Current network generation One experience;

[0045] The order of experience is , , , , ..., , , , ;

[0046] in, For state observation, This is the action performed from that state. The next state; From arrive Rewards received; while in the state At that time, the near-end strategy optimization method uses computation. Execute each action in the operating space. The probability and based on the probability distribution; For the present The initial time step of each experience set; at the beginning of the training set... =1, for each subsequent group in the same training set One experience For each empirical sequence that does not contain terminal states, Equal to the experience horizon option value, otherwise The state is terminated;

[0047] Step (5) Calculate the return function and dominance function ;

[0048] Return function for:

[0049] (twenty one)

[0050] (twenty two)

[0051] in, For the reward function; , , , It is a scaling factor; This is for frequency deviation; Cost of energy used; Carbon dioxide emissions; The cost of a failure in the weakest link of a comprehensive energy system; if If it is a terminated state, then If it is 0, otherwise it is 1; if If it is not in a terminated state, then the future rewards of the discount include the near-end policy optimization scheduling method. network The discounted state-value function is calculated using a near-end policy optimization scheduling method. network Calculate the advantage function :

[0052] (twenty three)

[0053] To calculate the advantage function, which is the discounted sum of time difference errors:

[0054] (twenty four)

[0055] in, As a smoothing factor, Discount factor;

[0056] Step (6) From Learn from small-batch experience in different periods;

[0057] For each learning period, from the current set of experiences, the size is... The data is sampled from a random mini-batch dataset, where each element of the mini-batch contains the current experience and the corresponding return and advantage function values; the loss is updated by minimizing the loss of all sampled mini-batch data. parameter;

[0058] (25)

[0059] For the first mini-batch dataset The return value of each element;

[0060] Based on the recent denormalized dominance values, the dominance value is compared with the dominance value. Perform normalization; if the normalization advantage method option is none, no normalization advantage value is needed:

[0061] (26)

[0062] If the normalization dominance method option is set to current, the dominance value is normalized based on the non-normalized dominance in the current mini-batch:

[0063] (27)

[0064] To standardize the dominant value; It is a function of average value; It is a function of standard deviation;

[0065] Step (7) minimizes the execution strategy loss function of all sampled mini-batch data. To update the execution strategy parameters;

[0066] (28)

[0067] in, and The first batch of small batches The advantage function and return value of each element; It is a minimum value function; For entropy loss; For a given updated policy parameter At that time, in the state Execute the following operation The probability of; When in a state Take action in time The probability, given the previous policy parameters before the current learning period. ; The clipping factor; Entropy loss weighting factor;

[0068] Step (8) Repeat steps (4) to (7) until training reaches the termination state;

[0069] Step (9) Output the dispatch output of each energy source in the weak link of the integrated energy system, thereby avoiding problems in the weak link of the integrated energy system.

[0070] The present invention has the following advantages and effects compared with the prior art:

[0071] (1) A near-end strategy optimization scheduling method considering the weak links of a comprehensive energy system uses deep reinforcement learning in artificial intelligence for dynamic scheduling, which has real-time and fast performance and can update the scheduling strategy online in real time. This invention patent is the first to apply the control method based on deep reinforcement learning to the scheduling and assessment of weak links of a comprehensive energy system, and can use the control method to solve the scheduling and assessment problems simultaneously with the dynamic idea.

[0072] (2) A near-end strategy optimization scheduling method that considers the weak links of the integrated energy system can give the weak links greater security.

[0073] (3) A near-end strategy optimization scheduling method that considers the weak links of the integrated energy system adopts an intelligent scheduling method based on deep reinforcement learning to avoid problems in the weak links. Attached Figure Description

[0074] Figure 1 This is a framework diagram of the method of the present invention. Detailed Implementation

[0075] The present invention proposes a near-end strategy optimization scheduling method that considers the weak links of a comprehensive energy system, which is described in detail below with reference to the accompanying drawings:

[0076] Figure 1 This is a framework diagram of the method of the present invention. The first step is to establish a comprehensive energy system model based on the characteristics of the comprehensive energy system; the second step is to initialize the parameters of the near-end strategy optimization scheduling method; the third step is to obtain the state of the comprehensive energy system. The fourth step is to optimize the scheduling method according to the near-end strategy. - Current network generation One experience; the fifth step is to calculate the return function. and dominance function Step 6, from The seventh step involves learning from small-batch experiences over several periods; and minimizing the execution strategy loss function across all sampled small-batch data. To update the execution strategy parameters; Step 8, repeat steps (4) to (7) until the training reaches the termination state; Step 9, output the scheduling output of each energy source in the weak link of the integrated energy system.

[0077] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1.A method for near-end strategy optimization scheduling considering weak links of an integrated energy system, characterized in that, The steps in use are: Step (1) establishing a comprehensive energy system model according to the characteristics of the comprehensive energy system, describing the connection mode of each link, the coupling between links and the energy conversion of each link; The comprehensive energy system comprises a power supply network link, an energy exchange link and an energy storage link; The power supply network link provides energy and inputs into the energy exchange link and the energy storage link; The energy exchange link exchanges energy between the power supply network link and the energy storage link to achieve the coupling between the links; The energy storage link stores electrical energy, thermal energy and gas energy by using energy storage devices; The model of the power supply network link, the energy exchange link and the energy storage link is established; Step (2) initializing the proximal policy optimization scheduling method parameters; The proximal policy optimization scheduling method is a model-free, online and policy-compliant policy gradient reinforcement learning method, which solves the problem of weak links by alternately using random gradient descent between environment interaction sampling data and optimized clipping data to solve the problem of weak links. parameters of the to-be-initialized proximal policy optimization scheduling method are: initializing the proximal policy optimization scheduling method current network random parameter value of , initializing the proximal policy optimization scheduling method current network random parameter value of ; Step (3): Obtain the state of the integrated energy system ; (1) Wherein, ×100％ (2) (3) (4) (5) (6) (7) In the formula, represents the photovoltaic energy consumption rate; represents the photovoltaic theoretical maximum power output; represents the photovoltaic actual consumption amount; represents the photovoltaic quantity; represents the carbon dioxide emission amount; represents represents the photovoltaic electric power at the moment; represents represents the photovoltaic actual electric power at the moment; represents the dispatch time, taking 1 hour; and respectively represent the carbon dioxide annual emission amounts corresponding to the power generation and heat production and the purchase and consumption of electric energy from the power grid; represents the carbon dioxide emission coefficient corresponding to the use of natural gas; represents the carbon dioxide emission coefficient corresponding to the use of electric energy; represents the consumed natural gas energy; represents the purchased electric energy; (8) (9) (10) (11) (12) In the formula, represents the total cost of expanding the comprehensive energy system; represents the investment cost of expanding the equipment of the comprehensive energy system; represents the annual operation cost; represents the annual maintenance cost; represents the total number of equipment expansion of the comprehensive energy system; represents the equipment annual value coefficient; represents the equipment unit capacity investment cost; represents the equipment expansion capacity; represents the discount rate; represents the life of the equipment ; represents time electricity price; represents the natural gas price; and respectively represent the power purchased from the power grid and the power input at the combined heat and power input end at time. (13) (14) (15) (16) (17) wherein represents the photovoltaic; represents the cogeneration; represents the heat pump; represents the electrical energy storage; represents the thermal energy storage; , , , and represent the maintenance costs for photovoltaic, cogeneration, heat pump, electrical energy storage and thermal energy storage, respectively; , , , and represent the maintenance costs per power for photovoltaic, cogeneration, heat pump, electrical energy storage and thermal energy storage, respectively; represents the electrical power of the cogeneration at time t; represents the thermal power of the cogeneration at time t; represents the thermal power of the heat pump at time t; and represent the absolute value of the electrical and thermal energy storage power and the absolute value of the discharging power at time , respectively; Integrated energy system device The operating constraints of the integrated energy system device are: (18) The electrical power balance constraint that the comprehensive energy system needs to meet is: (19) The thermal power balance constraint that the comprehensive energy system needs to meet is: (20) In the formula, Represented as Time device The power value; Indicates the modified equipment The upper limit of output; Indicates the equipment before modification. The upper limit of output; and express The charging and discharging power of the energy storage device at all times; express The electrical power of the heat pump at all times; Represented as Constant electrical load power; and Represented as The charging and discharging power of the electrical energy storage at any given time; Represented as Constant heat load power; Represented as The charging power of thermal energy storage at all times; Step (4) Optimizing the scheduling method according to the near-end strategy - Current network generation Individual experience; The order of experience is , , , ,..., , , , ; wherein, is the state observation, is the action performed from the state, is the next state; is the reward obtained from to ; is the start time step of the current experience set; at the beginning of the training set, = 1 for each subsequent experience in the same training set, ; for each sequence of experiences that does not contain the terminal state, is equal to the experience horizon option value, otherwise is the terminal state; Step (5) Calculate return function and advantage function ; Return function is: (21) (22) wherein is a reward function; , , , is a scaling factor; is a frequency deviation; is a cost of energy used; is a carbon dioxide emission; is a cost of failure of the weakest link of the integrated energy system; if is a terminal state, then is 0, otherwise 1; if is not a terminal state, then the discounted future reward includes a near-optimal policy optimization dispatch method network computes a discounted state value function, using a near-optimal policy optimization dispatch method network computes a advantage function : (23) To compute the advantage function, which is the discounting sum of the time-difference error: (24) wherein, is a smoothing factor, is a discount factor; Step (6) from learning from small batches of experience over epochs; For each learning epoch, sample a mini-batch of size from the current experience set, each element of the mini-batch containing the current experience as well as the corresponding return and advantage function values; update parameters by minimizing the loss over all sampled mini-batches; (25) the return value for the first element of the small batch dataset; Normalizing the recent non-normalized advantage values to advantage values performed; if the normalize advantage method option is none: (26) If the normalized advantage method option is the current, the advantage value is normalized according to the non-normalized advantage in the current small batch: (27) to normalize the advantage values; to average the function values; to standardize the function values; Step (7) updating the execution policy parameters by minimizing the execution policy loss function for all sampled mini-batch data ; (28) where, and are the advantage function and return value for the th element of the mini-batch, respectively; is the minimum value function; is the entropy loss; is the probability of performing action at state given the policy parameters ; is the probability of taking action when in state given the previous policy parameters before the current learning epoch; is the clipping factor; is the entropy loss weight factor; Step (8) repeating steps (4) to (7) until the training reaches a termination state; Step (9) outputting the scheduling output of each energy in the weak link of the scheduling comprehensive energy system, so as to avoid problems in the weak link of the comprehensive energy system.

Images