Comprehensive energy system Pareto optimal scheduling method based on variable multi-target weight

Abstract

The invention discloses an integrated energy system Pareto optimal scheduling method based on multi-target variable weights. According to the method, the Markov decision is used for describing the low-carbon economic dispatching problem of the comprehensive energy system, and an Actor-Critic network is built and comprises an Actor network and two Critic networks which focus on the operation cost and the carbon emission cost respectively. And inputting the state of the comprehensive energy system environment, outputting an optimal action through an Actor network, obtaining a reward vector and a system state at the next moment after execution, and storing a quintuple training sample to an experience storage area. And respectively carrying out weighted fusion on the dominant functions of the two Critic networks by using K groups of weights to obtain dominant function values of the networks under different weights, updating network parameters through a near-end strategy optimization algorithm, and optimizing a scheduling strategy. And after a predetermined round of near-end strategy optimization is completed, triggering adaptive updating of the dominant function weight, so that the intelligent agent learns a Pareto optimal decision.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of new energy, and relates to scheduling optimization of a multi-energy system, in particular to a Pareto optimal scheduling method for an integrated energy system based on variable multi-objective weights. BACKGROUND

[0002] An integrated energy system (IES) realizes multi-energy fusion and complementary utilization of renewable energy, energy storage devices and traditional energy through optimal design of a multi-energy system, and provides an innovative solution for optimizing overall operation efficiency of an energy system and reducing comprehensive energy consumption of the system. The IES accesses wind and light resources, deploys distributed energy storage systems and heating systems in a matching manner, and constructs a green, elastic, self-consistent and sustainable energy system.

[0003] However, an optimal scheduling strategy for the integrated energy system has strong dependence on prediction accuracy of renewable energy output and load demand. A deep reinforcement learning (DRL) method learns a strategy through interaction between an agent and an environment, does not need to predict or model renewable energy output and load demand, and can find an action that can bring the maximum reward value, and is regarded as a solution to an IES scheduling problem. However, a previous reinforcement learning algorithm still has significant technical bottlenecks when processing a multi-objective scheduling task. On the one hand, there is strong coupling between multiple scheduling objectives, and when conflicts occur between multiple objectives, it is difficult to accurately locate the root cause of the conflicts, and even a gradient disappearance phenomenon can be caused. On the other hand, objective weights are usually fixed, which seriously restricts flexibility of a scheduling scheme, and finally causes a scheduling result of the integrated energy system to deviate from an expected optimal objective. SUMMARY

[0004] In view of the deficiencies of the prior art, the application provides a Pareto optimal scheduling method for an integrated energy system based on variable multi-objective weights. On the basis of a deep reinforcement learning algorithm, the economic efficiency and low carbon efficiency of the integrated energy system are considered, an improved PPO algorithm is used to improve accuracy and flexibility in network updating, and then convergence results of an agent are improved, and operation cost and carbon emission cost of the integrated energy system are significantly reduced.

[0005] The Pareto optimal scheduling method for the integrated energy system based on variable multi-objective weights specifically includes the following steps.

[0006] Step (1), according to electricity price, gas price and operation constraints of each unit in the integrated energy system, considering economic and low carbon efficiency, a target function y1 based on operation cost and a target function y2 based on carbon emission cost are constructed.

[0007] Step (2): Use Markov decision to describe the low-carbon economic dispatch problem of the integrated energy system, build an Actor-Critic network, including an Actor network and two Critic1 and Critic2 networks that focus on operating costs and carbon emission costs respectively, and initialize the network parameters using an orthogonal initialization method.

[0008] Step (3): Using random uniform sampling, generate The initial advantage function weights ω = [ω(0,k)1 ω(0,k)2] are given, where ω(0,k)1 and ω(0,k)2 represent the initial weights of the k-th advantage function of the Critic1 and Critic2 networks, respectively, satisfying ω(0,k)1 + ω(0,k)2 = 1 and ω(0,k)1 and ω(0,k)2 ≥ 0.

[0009] Step (4): Each weight ω starts an independent thread to interact with the IES environment, and inputs photovoltaic power generation, electrical load, thermal load, electricity price, time, energy storage, and heat storage as the current state s. t The Actor network, for each set of weights, outputs the optimal action 'a' that matches that preference from a Gaussian distribution. t The IES environment is based on action a t Determine the unit's output and obtain the reward vector r. t =[-y1 -y2] and the state quantity s at the next time step t+1 Storing training samples (s) t ,a t ,r t ,s t+1 ,ω) to the experience storage area.

[0010] Step (5): When the number of training samples stored in the experience storage area reaches a set threshold, n training samples are randomly sampled from the experience storage area. The Critic1 network is rewarded r(1) based on the operating cost. t Calculate the dominance function A(1) t The Critic2 network rewards r(2) based on carbon emission costs. t Calculate the dominance function A(2) t , r(1) t r(2) t Representing the reward vector r respectively t The first and second elements in.

[0011] Then, the dominance function values ​​of the two Critic networks are weighted according to their respective weights to obtain the dominance function value A of the Actor-Critic network under weight ω. t .

[0012] Step (6), update the parameters of Actor network and two Critic networks according to proximal policy optimization algorithm.

[0013] Step (7), initialize an empty set as the Pareto archive After completing the PPO update of the scheduled round, trigger the adaptive update of the advantage function weight:

[0014] s7.1, evaluate the dual-objective performance of the Actor-Critic network under the current weight:

[0015] Under the multi-objective optimization framework, for the kth weight, first calculate its average operating cost and average carbon emission cost in the scheduling period, get the corresponding cost solution vector , and According to the average operating cost in ascending order, add the Pareto archive . Based on the definition of dominated and non-dominated solutions in multi-objective optimization, delete the dominated solutions in the Pareto archive , keep the non-dominated solutions, and calculate the total hyper-volume of the updated Pareto archive :

[0016]

[0017] Wherein, is the set reference point, M is the size of the Pareto archive , and is the minimum carbon emission value of the solution ranked i and after.

[0018] Then calculate the hyper-volume contribution ratio C of the solution generated by the action output by the Actor network under the target preference corresponding to each weight k , divide the target space into K grid regions according to the number K of initial weights, and then convert C k into the contribution ratio of the kth region:

[0019]

[0020] Wherein, is the non-dominated solution in the Pareto archive that falls in the kth region, In order to avoid the constant with denominator 0.

[0021] s7.2, calculate the contribution deviation of each weight:

[0022]

[0023] Wherein, The proportion of the average contribution of each weight, ; According to the contribution deviation D of each weight k Update the weight:

[0024]

[0025] Wherein, ω(l,k) m Indicate the weight of the kth group of advantage function of the Critic m network after the lth update, m=1, 2, Indicate the weight learning rate, | | indicates taking absolute value. The updated weight value Is normalized.

[0026] Step (8), repeat steps (4)-(7), constantly update the Actor network, Critic1 network, Critic2 network and weight, so that the agent learns the Pareto optimal decision, when the training times reach the preset round, stop training, save the Actor-Critic network model and all weights. After training, the saved model has learned the general logic of the Pareto optimal decision. In subsequent applications, you can directly set any new weight that meets the constraints, and the model will automatically generate a decision that matches the preference based on the trade-off rules learned during training. Without retraining, it can adapt to diverse scene requirements.

[0027] The present application has the following beneficial effects:

[0028] 1. A multi-Critic network architecture is proposed, each Critic focuses on a single target, avoiding gradient interference between targets, and the Actor only obtains target gradients from the corresponding Critic according to the current weight distribution, so that the update direction is more clear; each Critic outputs an independent value estimate of the corresponding target, which facilitates the analysis of strategy tendency. This architecture effectively solves the problem that a single Critic needs to integrate multiple target rewards, the output comprehensive value function is difficult to analyze the contribution of each target, and is prone to gradient conflict.

[0029] 2. A dynamic weight adaptive method is proposed to achieve accurate multi-objective trade-off. Based on the multi-Critic network, the advantage function of different targets is evaluated respectively, and an online weight adaptive mechanism is combined to dynamically adjust the target weight. This method allows any weight in the test phase, and flexibly responds to the dynamic needs of the management layer.

[0030] 3. In the training phase, the initial weight is optimized in parallel to cover the multi-objective space, ensuring the universality and representativeness of the solution set, and then the non-dominated solution is screened based on hyper volume optimization to generate high-quality Pareto frontier, which effectively solves the problem that a single strategy cannot provide trade-off relationship between multiple objectives. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 This is a schematic diagram of the integrated energy system.

[0032] Figure 2 A flowchart of the Pareto optimal scheduling method for a multi-objective, weighted integrated energy system;

[0033] Figure 3 Flowchart for weight update. Detailed Implementation

[0034] The present invention will be further explained below with reference to the accompanying drawings;

[0035] This embodiment is for the following: Figure 1 The integrated energy system shown provides a Pareto optimal scheduling method for unit output based on multi-objective weighted variable integrated energy system, and sets the scheduling time step. The optimization period is 60 minutes, meaning the number of optimization cycles T per day is 24. The specific steps are as follows:

[0036] Step (1): Based on electricity prices, gas prices, and the operational constraints of photovoltaic (PV) generator units, combined heat and power (CHP) units, electric boilers (EB), gas boilers (GB), electric energy storage (ESS), and thermal energy storage (TSS) units in the integrated energy system, and considering economic and low-carbon aspects, construct an objective function y1 based on operating costs and an objective function y2 based on carbon emission costs:

[0037]

[0038]

[0039] Among them, C E (t), C G (t), C CHP (t), C ESS (t) represent the costs of purchasing electricity from the main grid, gas from the gas grid, operating costs of the combined heat and power unit, and operating costs of the energy storage unit during time period t, respectively. F(t) represents the carbon price at the current moment, F(t) represents the carbon factor at the current moment, and P(t) represents the electricity consumption excluding renewable energy sources. G G represents the average carbon density of natural gas at the current moment. CHP (t) and G GB (t) represents the amount of natural gas consumed by the combined heat and power unit and the gas-fired boiler.

[0040] Step (2), the low-carbon economic dispatch problem of the integrated energy system is described by Markov decision, and an Actor-Critic network is built, including an Actor network and two Critic1 networks and Critic2 networks focusing on operation cost and carbon emission cost respectively. The orthogonal initialization method is used to initialize the network parameters.

[0041] Step (3), as shown in Figure 2 , a random uniform sampling method is used to generate K sets of initial advantage function weights ω=[ω(0,k)1 ω(0,k)2]:

[0042] ω(0,k)1=u1 / (u1+u2)

[0043] ω(0,k)2=1-ω(0,k)1

[0044] Wherein, ω(0,k)1 is the weight of the advantage function corresponding to the Critic1 network, ω(0,k)2 is the weight of the advantage function corresponding to the Critic2 network, u1 and u2 are random numbers independently drawn from the uniform distribution U(0,1). ω(0,k)1+ω(0,k)2=1 and ω(0,k)1, ω(0,k)2≥0.

[0045] Step (4), each set of weights ω starts an independent thread to interact with the integrated energy system environment, inputs the current state s t , the Actor network outputs the optimal action a t that matches the preference of each set of weights from the Gaussian distribution, obtains the reward vector r t =[-y1 -y2] and the next time state s t+1 from the environment, and stores the training samples (s t , a t , r t , s t+1 , ω) to the experience storage area.

[0046] The state s t includes photovoltaic power generation, electrical load, thermal load, electricity price, time, electricity storage and heat storage. The action a t includes the output of combined heat and power units, electric boilers, electricity storage systems and heat storage systems.

[0047] Step (5), when the number of training samples stored in the experience storage area reaches the set threshold, n training samples are randomly sampled from the experience storage area, and the advantage function value A(m) t of each training sample corresponding to the Critic m network is calculated first:

[0048]

[0049] where m represents the Critic network index, 1 represents the Critic1 network focusing on the running cost, and 2 represents the Critic2 network focusing on the carbon emission cost. 、 is a weight parameter, and n represents the prediction time; V m (s t+n ) is the output value of the mth Critic network at the t+n moment. r(1) t , r(2) t respectively represent the first and second elements in the reward vector r t .

[0050] Then, the advantage functions corresponding to the two Critic networks are weighted to obtain the final advantage function value A t :

[0051] A t = ω(0, k)1×A(1) t + ω(0, k)2×A(2) t

[0052] The variance of the advantage function values of the n training samples is calculated, and the training samples with a variance greater than 0.8 are removed to optimize the quality of the training samples.

[0053] Step (6), according to the proximal policy optimization algorithm, a loss function L(θ1) of the Actor network is constructed, and the network parameter θ1 of the Actor network is updated:

[0054]

[0055]

[0056]

[0057] where E represents the expectation, represents the ratio of the new and old policy probabilities, clip FU represents the clipping function, is the weight coefficient of the policy entropy , is the clipping coefficient, represents the discount coefficient of the advantage function value A t , represents the absolute value. represents the probability of selecting the action a t in the state s t .

[0058] Step (7), a loss function L(θ 2,m), the parameter of Critic network is updated by the difference between advantage function value and network output value 2,m :

[0059]

[0060] Step (8), as shown in Figure 3 , every 50 rounds of PPO update is completed, the adaptive update of advantage function weight is triggered:

[0061] s8.1, evaluate the double target performance of Actor-Critic network under the current weight:

[0062] Under the multi-objective optimization framework, for the kth group of weights, first calculate the average running cost and the average carbon emission cost in the scheduling period, get the corresponding cost solution vector , arrange according to the average running cost in ascending order, and add to the Pareto archive . In multi-objective optimization, if all target function values of solution a are not worse than solution b, and at least one target function value is strictly better than solution b, then solution a dominates solution b. According to the definition of dominated solution and non-dominated solution, delete the dominated solution in the Pareto archive , keep the non-dominated solution, and calculate the total hyper volume of the updated Pareto archive :

[0063]

[0064] Wherein, is the reference point, which is set to = in this embodiment; M is the size of the Pareto front , and is the minimum carbon emission value of the solution ranked i and after.

[0065] Then calculate the hyper volume contribution ratio C of each weight corresponding strategy k , divide the target space into K grid regions according to the number of initial weights , so as to convert C k into the contribution ratio of the kth region:

[0066]

[0067] Wherein, is the non-dominated solution in the Pareto archive falling in the kth region, , which is used to avoid the value of denominator part being 0.

[0068] s8.2, calculate the contribution deviation of each weight:

[0069]

[0070] wherein, is the average contribution ratio of each weight, .

[0071] When D k > 0, it indicates that the area needs to be strengthened; D k ≤ 0 indicates that the exploration is sufficient, and the weight needs to be weakened. According to the contribution deviation D k of each weight, the weight is updated as follows:

[0072]

[0073] wherein, in the embodiment, the weight learning rate is set to 0.001. In order to meet the weight constraint condition, the updated weight value ω(l+1,k) m needs to be normalized.

[0074] Step (9), repeat steps (4) to (8) to continuously update the Actor network, Critic1 network, Critic2 network and weight, so that the intelligent agent learns the Pareto optimal decision. When the training times reach the preset number of rounds, stop training, and save the Actor-Critic network model and all weights.

[0075] The weight combination of the operation cost and the carbon emission cost is set to [0.9, 0.1], [0.6, 0.4], [0.4, 0.6] and [0.1, 0.9] respectively, and the trained intelligent agent is used to schedule the integrated energy system for 10 days.

[0076] At the same time, in order to prove the beneficial effect of the method, 4 intelligent agents are trained using the PPO algorithm, and the weight combination of the operation cost and the carbon emission cost is set to [0.9, 0.1], [0.6, 0.4], [0.4, 0.6] and [0.1, 0.9] respectively during the training process. Then, the 4 intelligent agents trained by the PPO algorithm are used to schedule the integrated energy system for 10 days, and the costs of the PPO algorithm and the method are compared. The results are shown in Tables 1 to 4:

[0077] Table 1

[0078]

[0079] Table 2

[0080]

[0081] Table 3

[0082]

[0083] Table 4

[0084]

[0085] According to the table content, the model saved by the method can automatically generate decisions matching different preference weights based on the trade-off rules learned during training, without the need for retraining, thereby adapting to diversified scene requirements. Moreover, compared with the agent obtained by the PPO algorithm, the running cost and carbon emission cost are both significantly reduced.

Claims

1. A Pareto optimal scheduling method for integrated energy systems with variable weights based on multiple objectives uses Markov decision-making to describe the low-carbon economic scheduling problem of integrated energy systems. An Actor-Critic network is constructed, and the unit output actions are output according to the state of the integrated energy system. Rewards are obtained after executing the actions. The Actor-Critic network parameters are updated through a proximal policy optimization algorithm to optimize the scheduling strategy. Its key features are: The Actor-Critic network calculates the dominance function under operating cost and carbon emission cost using two Critic networks respectively; initializes K sets of dominance function weights, and uses the K sets of weights to weighted fuse the dominance functions of the two Critic networks to obtain the dominance function values ​​of the Actor-Critic network under different weights, and updates the network parameters accordingly. After completing the pre-defined rounds of proximal policy optimization, the adaptive update of the advantage function weights is triggered. First, non-dominated solutions are screened through Pareto dominance, and then the performance contribution of each non-dominated solution is evaluated based on the hypervolume index. In this way, the weight distribution is dynamically adjusted so that the agent learns the Pareto optimal decision. After training, the weights of operating cost and carbon emission cost are adaptively set according to scheduling preferences, and the scheduling scheme is output using the Actor-Critic network.

2. The Pareto optimal scheduling method for integrated energy systems based on multi-objective variable weights as described in claim 1, characterized in that: The Actor-Critic network includes an Actor network, a Critic1 network, and a Critic2 network, and the network parameters are initialized using an orthogonal initialization method. The unit status, load demand, time, and price information of the integrated energy system environment are input as the status of the Actor-Critic network. t The Actor network outputs the optimal action 'a' from a Gaussian distribution. t The integrated energy system environment is based on action a t Determine the output of each unit and obtain the reward vector r. t =[-y1 -y2] and the state quantity s at the next time step t+1 Storing training samples (s) t ,a t ,r t ,s t+1 ,ω) to the experience storage area; where y1 and y2 represent the objective function based on operating cost and the objective function based on carbon emission cost, respectively, and ω represents the weight of the advantage function.

3. The Pareto optimal scheduling method for integrated energy systems based on multi-objective variable weights as described in claim 1 or 2, characterized in that: Based on electricity prices, gas prices, and the operational constraints of each component in the integrated energy system, we construct an objective function y1 based on operating costs and an objective function y2 based on carbon emission costs. The objective function y1 based on operating costs is the minimum sum of electricity purchase cost, gas purchase cost, and operating cost of each unit within the scheduling period T; The objective function y2 based on carbon emission costs is: ； in, F(t) represents the carbon price at the current moment, F(t) represents the carbon factor at the current moment, and P(t) represents the electricity consumption excluding renewable energy sources. G G represents the average carbon density of natural gas at the current moment. CHP (t) and G GB (t) represents the amount of natural gas consumed by the combined heat and power unit and the gas-fired boiler.

4. The Pareto optimal scheduling method for integrated energy systems based on multi-objective variable weights as described in claim 1 or 2, characterized in that: When the number of training samples stored in the experience storage area reaches a set threshold, n training samples are randomly sampled from the experience storage area. The advantage function value A(1) for each training sample corresponding to the Critic1 network and the Critic2 network is calculated respectively. t A(2) t Then, the advantage functions corresponding to the two Critic networks are weighted to obtain the final advantage function value A. t : A t =ω(l,k)1×A(1) t +ω(l,k)2×A(2) t ; Where ω(l,k)1 and ω(l,k)2 represent the weights of the k-th group of advantage functions of the Critic1 network and Critic2 network in the l-th adaptive update, respectively.

5. The Pareto optimal scheduling method for integrated energy systems based on multi-objective variable weights as described in claim 4, characterized in that: Calculate the variance of the dominance function values ​​of n training samples, remove training samples with variance greater than 0.8, and optimize the quality of training samples.

6. The Pareto optimal scheduling method for integrated energy systems based on multi-objective variable weights as described in claim 4, characterized in that: The initial advantage function weights are generated using a random uniform sampling method: ω(0,k)1=u1 / (u1+u2); ω(0,k)2=1-ω(0,k)1; Where ω(0,k)1 is the weight of the dominance function corresponding to Critic1 network, ω(0,k)2 is the weight of the dominance function corresponding to Critic2 network, and u1 and u2 are random numbers independently drawn from the uniform distribution U(0,1); ω(0,k)1+ω(0,k)2=1 and ω(0,k)1, ω(0,k)2≥0.

7. The Pareto optimal scheduling method for integrated energy systems based on multi-objective variable weights as described in claim 1, characterized in that: The adaptive update method for the advantage function weights is as follows: Calculate the average operating cost and average carbon emission cost of each weight group within the scheduling period to obtain the corresponding cost solution vector. Sort the vectors in ascending order of average operating cost and add them to the Pareto archive. Retain the non-dominated solutions in the Pareto archive and calculate the total hypervolume of the Pareto archive. Then calculate the hypervolume contribution C of the strategy corresponding to each weight. k And the contribution deviation D of each weight k Based on the contribution deviation D of each weight k Perform adaptive updates.

8. The Pareto optimal scheduling method for integrated energy systems based on multi-objective variable weights as described in claim 7, characterized in that: When D k When D > 0, the weights need to be increased; k When the value is ≤0, the weight needs to be reduced.

9. The Pareto optimal scheduling method for integrated energy systems based on multi-objective variable weights as described in claim 7 or 8, characterized in that: ； in, Let ω(l,k) represent the weight learning rate. m Let represent the weights of the k-th group of advantage functions in the Critic m network after the l-th update, where m = 1, 2.

10. The Pareto optimal scheduling method for integrated energy systems based on multi-objective variable weights as described in claim 9, characterized in that: Set weight learning rate After every 50 rounds of near-end strategy optimization, an adaptive update of the advantage function weights is triggered.

Images