Multi-objective weight variable based pareto optimal scheduling method for integrated energy system

Abstract

The application discloses a comprehensive energy system Pareto optimal scheduling method based on multi-target weight variability. The method uses Markov decision to describe the low-carbon economic scheduling problem of the comprehensive energy system, builds an Actor-Critic network, including an Actor network and two Critic networks which are respectively focused on operation cost and carbon emission cost. The state of the environment of the comprehensive energy system is input, the optimal action is output through the Actor network, the reward vector and the next time system state are obtained after execution, and the five-tuple training sample is stored in the experience storage area. The advantage function of the two Critic networks is weighted and fused by using K sets of weights, the advantage function values of the networks under different weights are obtained, the network parameters are updated through the proximal policy optimization algorithm, and the optimal scheduling strategy is obtained. After completing the scheduled rounds of proximal policy optimization, the adaptive update of the advantage function weight is triggered, so that the intelligent agent learns the Pareto optimal decision.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of new energy, and relates to scheduling optimization of a multi-energy system, in particular to a Pareto optimal scheduling method for an integrated energy system based on variable multi-objective weights. BACKGROUND

[0002] An integrated energy system (IES) realizes multi-energy fusion and complementary utilization of renewable energy, energy storage devices and traditional energy through optimal design of a multi-energy system, and provides an innovative solution for optimizing overall operation efficiency of an energy system and reducing comprehensive energy consumption of the system. The IES accesses wind and light resources, deploys distributed energy storage systems and heating systems in a matching manner, and constructs a green, elastic, self-consistent and sustainable energy system.

[0003] However, an optimal scheduling strategy for the integrated energy system has strong dependence on prediction accuracy of renewable energy output and load demand. A deep reinforcement learning (DRL) method learns a strategy through interaction between an agent and an environment, does not need to predict or model renewable energy output and load demand, and can find an action that can bring the maximum reward value, and is regarded as a solution to an IES scheduling problem. However, a previous reinforcement learning algorithm still has significant technical bottlenecks when processing a multi-objective scheduling task. On the one hand, there is strong coupling between multiple scheduling objectives, and when conflicts occur between multiple objectives, it is difficult to accurately locate the root cause of the conflicts, and even a gradient disappearance phenomenon can be caused. On the other hand, objective weights are usually fixed, which seriously restricts flexibility of a scheduling scheme, and finally causes a scheduling result of the integrated energy system to deviate from an expected optimal objective. SUMMARY

[0004] In view of the deficiencies of the prior art, the application provides a Pareto optimal scheduling method for an integrated energy system based on variable multi-objective weights. On the basis of a deep reinforcement learning algorithm, the economic efficiency and low carbon efficiency of the integrated energy system are considered, an improved PPO algorithm is used to improve accuracy and flexibility in network updating, and then convergence results of an agent are improved, and operation cost and carbon emission cost of the integrated energy system are significantly reduced.

[0005] The Pareto optimal scheduling method for the integrated energy system based on variable multi-objective weights specifically includes the following steps.

[0006] Step (1), according to electricity price, gas price and operation constraints of each unit in the integrated energy system, considering economic and low carbon efficiency, a target function y1 based on operation cost and a target function y2 based on carbon emission cost are constructed.

[0007] Step (2): Use Markov decision to describe the low-carbon economic dispatch problem of the integrated energy system, build an Actor-Critic network, including an Actor network and two Critic1 and Critic2 networks that focus on operating costs and carbon emission costs respectively, and initialize the network parameters using an orthogonal initialization method.

[0008] Step (3): Using random uniform sampling, generate The initial advantage function weights ω = [ω(0,k)1 ω(0,k)2] are given, where ω(0,k)1 and ω(0,k)2 represent the initial weights of the k-th advantage function of the Critic1 and Critic2 networks, respectively, satisfying ω(0,k)1 + ω(0,k)2 = 1 and ω(0,k)1 and ω(0,k)2 ≥ 0.

[0009] Step (4): Each weight ω starts an independent thread to interact with the IES environment, and inputs photovoltaic power generation, electrical load, thermal load, electricity price, time, energy storage, and heat storage as the current state s. t The Actor network, for each set of weights, outputs the optimal action 'a' that matches that preference from a Gaussian distribution. t The IES environment is based on action a t Determine the unit's output and obtain the reward vector r. t =[-y1 -y2] and the state quantity s at the next time step t+1 Storing training samples (s) t ,a t ,r t ,s t+1 ,ω) to the experience storage area.

[0010] Step (5): When the number of training samples stored in the experience storage area reaches a set threshold, n training samples are randomly sampled from the experience storage area. The Critic1 network is rewarded r(1) based on the operating cost. t Calculate the dominance function A(1) t The Critic2 network rewards r(2) based on carbon emission costs. t Calculate the dominance function A(2) t , r(1) t r(2) t Representing the reward vector r respectively t The first and second elements in.

[0011] Then, the dominance function values ​​of the two Critic networks are weighted according to their respective weights to obtain the dominance function value A of the Actor-Critic network under weight ω. t .

[0012] Step (6), update the parameters of Actor network and two Critic networks according to proximal policy optimization algorithm.

[0013] Step (7), initialize an empty set as the Pareto archive After completing the PPO update of the scheduled round, trigger the adaptive update of the advantage function weight:

[0014] s7.1, evaluate the dual-objective performance of the Actor-Critic network under the current weight:

[0015] Under the multi-objective optimization framework, for the kth weight, first calculate its average operating cost and average carbon emission cost in the scheduling period, get the corresponding cost solution vector , and According to the average operating cost in ascending order, add the Pareto archive . Based on the definition of dominated and non-dominated solutions in multi-objective optimization, delete the dominated solutions in the Pareto archive , keep the non-dominated solutions, and calculate the total hyper-volume of the updated Pareto archive :

[0016]

[0017] Wherein, is the set reference point, M is the size of the Pareto archive , and is the minimum carbon emission value of the solution ranked i and after.

[0018] Then calculate the hyper-volume contribution ratio C of the solution generated by the action output by the Actor network under the target preference corresponding to each weight k , divide the target space into K grid regions according to the number K of initial weights, and then convert C k into the contribution ratio of the kth region:

[0019]

[0020] Wherein, is the non-dominated solution in the Pareto archive that falls in the kth region, In order to avoid the constant with denominator 0.

[0021] s7.2, calculate the contribution deviation of each weight:

[0022]

[0023] Wherein, The proportion of the average contribution of each weight, ; According to the contribution deviation D of each weight k Update the weight:

[0024]

[0025] Wherein, ω(l,k) m Indicate the weight of the kth group of advantage function of the Critic m network after the lth update, m=1, 2, Indicate the weight learning rate, | | indicates taking absolute value. The updated weight value Is normalized.

[0026] Step (8), repeat steps (4)-(7), constantly update the Actor network, Critic1 network, Critic2 network and weight, so that the agent learns the Pareto optimal decision, when the training times reach the preset round, stop training, save the Actor-Critic network model and all weights. After training, the saved model has learned the general logic of the Pareto optimal decision. In subsequent applications, you can directly set any new weight that meets the constraints, and the model will automatically generate a decision that matches the preference based on the trade-off rules learned during training. Without retraining, it can adapt to diverse scene requirements.

[0027] The present application has the following beneficial effects:

[0028] 1. A multi-Critic network architecture is proposed, each Critic focuses on a single target, avoiding gradient interference between targets, and the Actor only obtains target gradients from the corresponding Critic according to the current weight distribution, so that the update direction is more clear; each Critic outputs an independent value estimate of the corresponding target, which facilitates the analysis of strategy tendency. This architecture effectively solves the problem that a single Critic needs to integrate multiple target rewards, the output comprehensive value function is difficult to analyze the contribution of each target, and is prone to gradient conflict.

[0029] 2. A dynamic weight adaptive method is proposed to achieve accurate multi-objective trade-off. Based on the multi-Critic network, the advantage function of different targets is evaluated respectively, and an online weight adaptive mechanism is combined to dynamically adjust the target weight. This method allows any weight in the test phase, and flexibly responds to the dynamic needs of the management layer.

[0030] 3. In the training phase, the initial weight is optimized in parallel to cover the multi-objective space, ensuring the universality and representativeness of the solution set, and then the non-dominated solution is screened based on hyper volume optimization to generate high-quality Pareto frontier, which effectively solves the problem that a single strategy cannot provide trade-off relationship between multiple objectives. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 This is a schematic diagram of the integrated energy system.

[0032] Figure 2 A flowchart of the Pareto optimal scheduling method for a multi-objective, weighted integrated energy system;

[0033] Figure 3 Flowchart for weight update. Detailed Implementation

[0034] The present invention will be further explained below with reference to the accompanying drawings;

[0035] This embodiment is for the following: Figure 1 The integrated energy system shown provides a Pareto optimal scheduling method for unit output based on multi-objective weighted variable integrated energy system, and sets the scheduling time step. The optimization period is 60 minutes, meaning the number of optimization cycles T per day is 24. The specific steps are as follows:

[0036] Step (1): Based on electricity prices, gas prices, and the operational constraints of photovoltaic (PV) generator units, combined heat and power (CHP) units, electric boilers (EB), gas boilers (GB), electric energy storage (ESS), and thermal energy storage (TSS) units in the integrated energy system, and considering economic and low-carbon aspects, construct an objective function y1 based on operating costs and an objective function y2 based on carbon emission costs:

[0037]

[0038]

[0039] Among them, C E (t), C G (t), C CHP (t), C ESS (t) represent the costs of purchasing electricity from the main grid, gas from the gas grid, operating costs of the combined heat and power unit, and operating costs of the energy storage unit during time period t, respectively. F(t) represents the carbon price at the current moment, F(t) represents the carbon factor at the current moment, and P(t) represents the electricity consumption excluding renewable energy sources. G G represents the average carbon density of natural gas at the current moment. CHP (t) and G GB (t) represents the amount of natural gas consumed by the combined heat and power unit and the gas-fired boiler.

[0040] Step (2), the low-carbon economic dispatch problem of the integrated energy system is described by Markov decision, and an Actor-Critic network is built, including an Actor network and two Critic1 networks and Critic2 networks focusing on operation cost and carbon emission cost respectively. The orthogonal initialization method is used to initialize the network parameters.

[0041] Step (3), as shown in Figure 2 , a random uniform sampling method is used to generate K sets of initial advantage function weights ω=[ω(0,k)1 ω(0,k)2]:

[0042] ω(0,k)1=u1 / (u1+u2)

[0043] ω(0,k)2=1-ω(0,k)1

[0044] Wherein, ω(0,k)1 is the weight of the advantage function corresponding to the Critic1 network, ω(0,k)2 is the weight of the advantage function corresponding to the Critic2 network, u1 and u2 are random numbers independently drawn from the uniform distribution U(0,1). ω(0,k)1+ω(0,k)2=1 and ω(0,k)1, ω(0,k)2≥0.

[0045] Step (4), each set of weights ω starts an independent thread to interact with the integrated energy system environment, inputs the current state s t , the Actor network outputs the optimal action a t that matches the preference of each set of weights from the Gaussian distribution, obtains the reward vector r t =[-y1 -y2] and the next time state s t+1 from the environment, and stores the training samples (s t , a t , r t , s t+1 , ω) to the experience storage area.

[0046] The state s t includes photovoltaic power generation, electrical load, thermal load, electricity price, time, electricity storage and heat storage. The action a t includes the output of combined heat and power units, electric boilers, electricity storage systems and heat storage systems.

[0047] Step (5), when the number of training samples stored in the experience storage area reaches the set threshold, n training samples are randomly sampled from the experience storage area, and the advantage function value A(m) t of each training sample corresponding to the Critic m network is calculated first:

[0048]

[0049] where m represents the Critic network index, 1 represents the Critic1 network focusing on the running cost, and 2 represents the Critic2 network focusing on the carbon emission cost. 、 is a weight parameter, and n represents the prediction time; V m (s t+n ) is the output value of the mth Critic network at the t+n moment. r(1) t , r(2) t respectively represent the first and second elements in the reward vector r t .

[0050] Then the advantage functions corresponding to the two Critic networks are weighted to obtain the final advantage function value A t :

[0051] A t = ω(0, k)1×A(1) t + ω(0, k)2×A(2) t

[0052] The variance of the advantage function values of the n training samples is calculated, and the training samples with a variance greater than 0.8 are removed to optimize the quality of the training samples.

[0053] Step (6), according to the proximal policy optimization algorithm, a loss function L(θ1) of the Actor network is constructed, and the network parameter θ1 of the Actor network is updated:

[0054]

[0055]

[0056]

[0057] where E represents the expectation, represents the ratio of the new and old policy probabilities, clip FU represents the clipping function, is the weight coefficient of the policy entropy , is the clipping coefficient, represents the discount coefficient of the advantage function value A t , represents the absolute value. represents the probability of selecting action a t in state s t .

[0058] Step (7), a loss function L(θ 2,m), the parameter of Critic network is updated by the difference between the advantage function value and the network output value 2,m :

[0059]

[0060] Step (8), as shown in Figure 3 , every 50 rounds of PPO update is completed, the adaptive update of the advantage function weight is triggered:

[0061] s8.1, evaluate the dual-objective performance of the Actor-Critic network under the current weight:

[0062] Under the multi-objective optimization framework, for the kth group of weights, first calculate the average running cost and the average carbon emission cost in the scheduling period, get the corresponding cost solution vector , and arrange in ascending order of average running cost , and add to the Pareto archive . In multi-objective optimization, if all objective function values of solution a are not worse than solution b, and at least one objective function value is strictly better than solution b, then solution a dominates solution b. According to the definition of dominated solution and non-dominated solution, delete the dominated solution in the Pareto archive , and keep the non-dominated solution. Calculate the total hyper-volume of the updated Pareto archive :

[0063]

[0064] wherein is the reference point, which is set to = in this embodiment; M is the size of the Pareto front , and is the minimum carbon emission value of the solutions ranked i and after.

[0065] Then calculate the hyper-volume contribution ratio C of each weight corresponding strategy k Divide the target space into K grid regions according to the number of initial weights , so as to convert C k into the contribution ratio of the kth region:

[0066]

[0067] wherein is the non-dominated solution in the Pareto archive falling in the kth region, , which is used to avoid the value of the denominator part being 0.

[0068] s8.2, calculate the contribution deviation of each weight:

[0069]

[0070] wherein, is the average contribution ratio of each weight, .

[0071] When D k > 0, it indicates that the area exploration is insufficient, and the weight needs to be enhanced; D k ≤ 0 indicates that the exploration is sufficient, and the weight needs to be weakened, and the weight is updated according to the contribution deviation D k of each weight:

[0072]

[0073] wherein, in the embodiment, the weight learning rate In order to meet the weight constraint condition, the updated weight value ω(l+1,k) m needs to be normalized.

[0074] Step (9), repeat steps (4) to (8), continuously update the Actor network, Critic1 network, Critic2 network and weight, so that the agent learns the Pareto optimal decision, when the training times reach the preset rounds, stop training, save the Actor-Critic network model and all weights.

[0075] The weight combination of the operation cost and the carbon emission cost is set to [0.9, 0.1], [0.6, 0.4], [0.4, 0.6], [0.1, 0.9] respectively, and the trained agent is used to schedule the integrated energy system for 10 days.

[0076] At the same time, in order to prove the beneficial effect of the method, 4 agents are trained using the PPO algorithm, and the weight combination of the operation cost and the carbon emission cost is set to [0.9, 0.1], [0.6, 0.4], [0.4, 0.6], [0.1, 0.9] respectively in the training process, and then the 4 agents trained by the PPO algorithm are used to schedule the integrated energy system for 10 days, and the costs of the PPO algorithm and the method are compared, and the results are shown in Tables 1 to 4:

[0077] Table 1

[0078]

[0079] Table 2

[0080]

[0081] Table 3

[0082]

[0083] Table 4

[0084]

[0085] According to the table content, the model saved by the method can automatically generate decisions matching different preference weights based on the trade-off rules learned during training, without the need for retraining, thereby adapting to diversified scene requirements. Moreover, compared with the agent obtained by the PPO algorithm, the running cost and carbon emission cost are both significantly reduced.

Claims

1. A Pareto optimal scheduling method for integrated energy systems with variable weights based on multiple objectives uses Markov decision-making to describe the low-carbon economic scheduling problem of integrated energy systems. An Actor-Critic network is constructed, and the unit output actions are output according to the state of the integrated energy system. Rewards are obtained after executing the actions. The Actor-Critic network parameters are updated through a proximal policy optimization algorithm to optimize the scheduling strategy. Its key features are: The Actor-Critic network calculates the advantage functions under the running cost and the carbon emission cost through two Critic networks respectively; K sets of advantage function weights are initialized, and the advantage functions of the two Critic networks are weighted and fused respectively by using the K sets of weights to obtain the advantage function values of the Actor-Critic network under different weights, and the network parameter updating is performed; After completing the predetermined round of proximal policy optimization, the adaptive updating of the advantage function weights is triggered, the non-dominated solutions are first screened through the Pareto dominance relationship, and then the performance contribution of each non-dominated solution is evaluated based on the hyper volume index, so as to dynamically adjust the weight distribution, so that the agent learns the Pareto optimal decision, and after the training is completed, the weights of the running cost and the carbon emission cost are adaptively set according to the scheduling preference, and the scheduling scheme is output by using the Actor-Critic network. 2.The method of claim 1, wherein the method is characterized in that: The Actor-Critic network comprises an Actor network, a Critic1 network and a Critic2 network, and the network parameters are initialized by using an orthogonal initialization method; Input the unit state, load demand, time and price information of the comprehensive energy system environment as the state s of the Actor-Critic network t , the Actor network outputs the optimal action a from the Gaussian distribution t , the comprehensive energy system environment determines the output of each unit according to the action a t , obtains the reward vector r t =[-y1 -y2] and the next time state s t+1 , stores the training sample (s t , a t , r t , s t+1 , ω) to the experience storage area; wherein y1 and y2 respectively represent the target function based on the operation cost and the target function based on the carbon emission cost, and ω represents the advantage function weight. 3.The method of claim 1 or 2, wherein: According to the electricity price, the gas price and the operation constraints in the comprehensive energy system, a target function y1 based on the running cost and a target function y2 based on the carbon emission cost are constructed; The target function y1 based on the running cost is the minimum value of the sum of the electricity purchase cost, the gas purchase cost and the operation cost of each unit in the scheduling period T. The target function y2 based on the carbon emission cost is: wherein, F(t) represents the carbon price at the current time, F(t) represents the electric carbon factor at the current time, P(t) is the amount of electricity used other than renewable energy, F G G(t) represents the average carbon density of natural gas at the current time, G chp (t) and G gb (t) is the amount of natural gas consumed by the combined heat and power unit and the gas boiler. 4.The method of claim 1 or 2, wherein: When the number of training samples stored in the experience storage reaches a set threshold, n training samples are randomly sampled from the experience storage, and the advantage function values A(1) t , A(2) t of each training sample corresponding to Critic1 network and Critic2 network are calculated respectively, then the advantage functions corresponding to the two Critic networks are weighted to obtain the final advantage function value A t : A t = ω(l, k)1× A(1) t + ω(l, k)2× A(2) t Wherein, ω(l,k)1 and ω(l,k)2 respectively represent the weight of the kth set of advantage function of the Critic1 network and the Critic2 network in the lth adaptive updating. 5.The method of claim 4, wherein: The variance of the advantage function values of n training samples is calculated, and the training samples with the variance greater than 0.8 are removed to optimize the quality of the training samples. 6.The method of claim 4, wherein: The initial advantage function weights are generated by using a random uniform sampling method: ω(0,k)1=u1 / (u1+u2) ω(0,k)2=1-ω(0,k)1 wherein ω(0, k)1is the weight of the advantage function corresponding to the Critic1 network, ω(0, k)2is the weight of the advantage function corresponding to the Critic2 network, u1, u2are random numbers independently drawn from a uniform distribution U(0, 1); ω(0, k)1+ ω(0, k)2= 1 and ω(0, k)1, ω(0, k)2≥ 0. 7.The method of claim 1, wherein: The adaptive updating method of the advantage function weights is: The average running cost and the average carbon emission cost of each set of weights in the scheduling period are calculated to obtain the corresponding cost solution vector, which is arranged in ascending order according to the average running cost, and added to the Pareto archive; the non-dominated solutions in the Pareto archive are retained, and the total hyper volume of the Pareto archive is calculated. Recalculate the contribution ratio C of the hyper-volume of each weight corresponding strategy k and the contribution deviation D of each weight k , according to the contribution deviation D of each weight k Adaptive update. 8.The method of claim 7, wherein: When D k > 0, the weight needs to be enhanced; D k ≤ 0, the weight needs to be weakened. 9.The method of claim 7 or 8, wherein: ； wherein, denotes the weight learning rate, ω(l, k) m denotes the weight of the kth set of advantage functions of the Critic m network after the lth update, m = 1, 2. 10.The method of claim 9, wherein: Set weight learning rate After every 50 completed episodes of proximal policy optimization, trigger an adaptive update of the advantage function weights.

Images