Flexible virtual power plant optimization scheduling method based on deep reinforcement learning

Abstract

The invention is suitable for the technical field of power plant optimization scheduling, and provides a flexible virtual power plant optimization scheduling method based on deep reinforcement learning, comprising the following steps: establishing a flexible resource model including a storage battery energy storage equipment model, an air conditioning load model and a demand response load model; flexible virtual power plant optimization scheduling is carried out based on deep reinforcement learning, and the method specifically comprises the steps that importance sampling and an experience pool are set through a near-end strategy optimization algorithm, so that an agent is updated multiple times after interacting with the environment once, and the efficiency of the algorithm is improved; and constructing a deep reinforcement learning optimization model based on the state space, the action space and the reward function. According to the method, a neural network structure and a deep reinforcement learning algorithm are effectively combined to cope with a complex and high-dimensional state space. And a supervised learning pre-training method is used to train the initial strategy network in advance, and the optimization result of the strategy network learning model driving method is promoted.

Description

Technical Field

[0001] The present invention relates to the technical field of power plant optimization scheduling, and in particular to a flexible virtual power plant optimization scheduling method based on deep reinforcement learning. Background Art

[0002] The construction of new power systems in my country is progressing steadily, the speed of electricity substitution on the energy consumption side is accelerating, and massive, dispersed distributed resources are constantly emerging. It is undoubtedly one of the key steps to build a new power system to aggregate and efficiently allocate these flexible resources on a large scale, which will provide an important solution for the supply and demand matching and flexible operation of the new power system. Flexible resource virtual power plants (VPPs) can participate in power system scheduling by aggregating flexible resources, improve power system flexibility, reduce the cost of expanding adjustable resources, and promote the flexible consumption of clean energy. However, current research on the optimal scheduling of VPPs is mainly focused on supply-side resources, while there are few studies on the participation of flexible resources in VPP scheduling, resulting in the potential of distributed flexible resources not being fully explored and utilized. Traditional reinforcement learning algorithms are difficult to deal with power systems with many devices and complex structures. Therefore, it is necessary to provide a flexible virtual power plant optimization scheduling method based on deep reinforcement learning to solve the above problems. Summary of the invention

[0003] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a flexible virtual power plant optimization scheduling method based on deep reinforcement learning to solve the problems existing in the above-mentioned background technology.

[0004] The present invention is implemented as follows: a flexible virtual power plant optimization scheduling method based on deep reinforcement learning, the method comprising the following steps:

[0005] Establishing a flexibility resource model, wherein the flexibility resource model includes a battery energy storage device model, an air conditioning load model, and a demand response load model;

[0006] Flexible virtual power plant optimization scheduling is performed based on deep reinforcement learning, specifically including: setting importance sampling and experience pool through proximal policy optimization algorithm so that the intelligent agent performs multiple updates after one interaction with the environment to improve the efficiency of the algorithm; building a deep reinforcement learning optimization model based on state space, action space and reward function; the experience pool collects and saves scheduling sample data, and based on the scheduling sample data, the initial policy network is pre-trained using supervised learning, so that the policy network has basic scheduling capabilities at the initial stage of training, thereby guiding the intelligent agent to perform further reinforcement learning training.

[0007] As a further solution of the present invention: the battery energy storage device model is used to ensure that the state of charge of the energy storage device is within a preset range. The battery energy storage device model is as follows:

[0008]

[0009] s min ≤s(t)≤s max ;

[0010] In the formula, s(t) is the current SOC of the battery energy storage device; s(t-1) is the SOC of the battery energy storage device in the previous period; P(t) is the charge and discharge power of the battery energy storage device at time t; Δt is the unit time length; D max is the maximum capacity of the battery energy storage device; min 、s max They are the lower limit and upper limit of SOC respectively. SOC represents the state of charge of the battery energy storage device.

[0011] As a further solution of the present invention: the air conditioning load model is as follows:

[0012]

[0013] T in,min ≤T in (t)≤T in,max ;

[0014] Where: T in , T out , T k are indoor air temperature, outdoor air temperature, and indoor building material temperature; C in , C k are the thermal mass of indoor air and the thermal mass of indoor building materials respectively; H in , H k are the thermal conductivity of indoor air and building materials respectively; P A is the power of the air conditioner. When the air conditioner is turned off, the power is set to 0. A The energy released per unit power consumption; T in,min , T in,max The upper and lower limits set for the indoor air temperature respectively.

[0015] As a further solution of the present invention: the demand response load model is as follows:

[0016] L d (t) = L base (t)-L shift (t)+L due (t);

[0017] Lshift (t) = L base (t)bl t ;

[0018]

[0019] Where: L d (t), L base (t), L shift (t), L due (t) are load power consumption, basic load power consumption, transferable load power consumption and transfer load power consumption; l is the electricity price level coefficient; b is the price sensitivity coefficient; y i,t =1 means transfer, y i,t =0 means no transfer.

[0020] As a further solution of the present invention: in the proximal strategy optimization algorithm, two Actor networks are set: the Old-Actor network and the New-Actor network. The Old-Actor network is used to interact with the environment and perform sampling to obtain sampled data. The Critic network uses the sampled data to train the New-Actor network. Every several rounds, the New-Actor network will copy its parameters to the Old-Actor network to update the agent strategy.

[0021] As a further solution of the present invention: the state space S represents the set of all possible states that the system is currently in, each state contains all the information of the system at a certain moment, and the state space includes the output of distributed energy, the planned output value of the virtual power plant, the battery SOC, the demand response load and the average room temperature of the air conditioning load users;

[0022]

[0023] Where: G i,der (t) is the output value of the ith distributed energy in period t; P out (t) is the planned output value of the flexible virtual power plant in period t; L j,d (t) is the power of the jth demand response load in period t; L k,a (t) is the load required by the kth air conditioning load in the t period; T a (t) is the average room temperature of air-conditioning load users.

[0024] As a further solution of the present invention: the action space A represents the set of all possible actions performed by the agent in each state, and each action represents the operation taken by the agent in a specific state. For the power generation equipment that can be directly controlled by the flexible virtual power plant, its action value is the output G of the equipment i,der(t); For energy storage equipment, its action value is the battery charging and discharging power P C (t); For the virtual power plant, which indirectly controls the demand response load and air conditioning load regulated by electricity price and air conditioning temperature, the corresponding actions are the electricity price adjustment range Z(t) and the air conditioning set temperature T a (t);

[0025] A=[G 1,der (t),G 2,der (t),…,G n,der (t),P C (t),Z(t),T a (t)].

[0026] As a further solution of the present invention: the reward function R represents the reward obtained by the agent after performing a certain action, and the reward function enables the agent to select an action that can maximize long-term benefits; the reward function is the sum of the operating cost and penalty term of the flexible virtual power plant;

[0027] R=-(Q cost +Q pu )Q pu =Q pu1 +Q pu2 +Q pu3 ;

[0028] Where: Q cost is the operating cost; Q pu is the penalty term; Q pu1 , Q pu2 and Q pu3 They are the penalty items for the flexible virtual power plant failing to meet the planned output constraint conditions, the room temperature conditions of air-conditioning users, and the overcharging and over-discharging of energy storage equipment;

[0029]

[0030] Q pu3 (t) = q pu3 [max(0,s min -s(t))+max(0,s(t)-s max )]D max ;

[0031] Where: q pu1 ,q pu2 ,q pu3 They are the planned processing penalty coefficient, the air conditioning load room temperature over-limit penalty coefficient, and the battery overcharge and over-discharge penalty coefficient; T k,a (t) is the room temperature of the kth air-conditioning user in period t.

[0032] As a further solution of the present invention: during the pre-training process of the initial policy network, a neural network is constructed as the initial policy network, and the parameter setting of the initial policy network is consistent with the Actor network in the intelligent agent.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The present invention establishes a flexible resource virtual power plant model including energy storage equipment, air-conditioning load and demand response load, and proposes an optimization scheduling method of deep reinforcement learning combined with neural network technology based on the proximal policy optimization (PPO) algorithm, which includes state, action space and reward function. It can predict load demand, temperature and other environmental factors according to the output of wind power and photovoltaic power, and realize the regulation of flexible resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Flowchart of a flexible virtual power plant optimization scheduling method based on deep reinforcement learning. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0037] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments.

[0038] like Figure 1 As shown, an embodiment of the present invention provides a flexible virtual power plant optimization scheduling method based on deep reinforcement learning, and the method includes the following steps:

[0039] S100, establishing a flexibility resource model, wherein the flexibility resource model includes a battery energy storage device model, an air conditioning load model, and a demand response load model;

[0040] S200, optimizes the scheduling of flexible virtual power plants based on deep reinforcement learning, specifically including: setting importance sampling and experience pool through the proximal strategy optimization algorithm so that the intelligent agent performs multiple updates after one interaction with the environment to improve the efficiency of the algorithm; constructing a deep reinforcement learning optimization model based on three key elements: state space, action space and reward function; the experience pool collects and saves scheduling sample data, and based on the scheduling sample data, uses supervised learning to pre-train the initial policy network, so that the policy network has basic scheduling capabilities at the beginning of training, thereby guiding the intelligent agent to perform further reinforcement learning training.

[0041] In the embodiment of the present invention, a flexibility resource model is first established, wherein the battery energy storage device model is used to ensure that the state of charge of the energy storage device is within a preset appropriate range. The battery energy storage device model is as follows:

[0042]

[0043] s min ≤s(t)≤s max ;

[0044] In the formula, s(t) is the current SOC of the battery energy storage device; s(t-1) is the SOC of the battery energy storage device in the previous period; P(t) is the charge and discharge power of the battery energy storage device at time t; Δt is the unit time length; D max is the maximum capacity of the battery energy storage device; min 、s max They are the lower limit and upper limit of SOC respectively. SOC represents the state of charge of the battery energy storage device.

[0045] In the embodiment of the present invention, since the air conditioning load is greatly affected by weather factors, the air conditioning load accounts for a high proportion in summer and has a large adjustment potential. Therefore, the establishment of the air conditioning load model will simultaneously consider the impact of indoor and outdoor temperatures on the load demand. At the same time, the air conditioning equipment also provides users with a comfortable temperature. The air conditioning load model is as follows:

[0046]

[0047] T in,min ≤T in (t)≤T in,max ;

[0048] Where: T in , T out , T k are indoor air temperature, outdoor air temperature, and indoor building material temperature; C in , Ck are the thermal mass of indoor air and the thermal mass of indoor building materials respectively; H in , H k are the thermal conductivity of indoor air and building materials respectively; P A is the power of the air conditioner. When the air conditioner is turned off, the power is set to 0. A The energy released per unit power consumption; T in,min , T in,max The upper and lower limits set for the indoor air temperature respectively.

[0049] In the embodiment of the present invention, the demand response load is mainly a transferable load affected by price, and the demand response load model is as follows:

[0050] L d (t) = L base (t)-L shift (t)+L due (t);

[0051] L shift (t) = L base (t)bl t ;

[0052]

[0053] Where: L d (t), L base (t), L shift (t), L due (t) are load power consumption, basic load power consumption, transferable load power consumption and transfer load power consumption; l is the electricity price level coefficient; b is the price sensitivity coefficient; y i,t =1 means transfer, y i,t =0 means no transfer.

[0054] In an embodiment of the present invention, in the proximal policy optimization algorithm, two Actor networks are set: the Old-Actor network and the New-Actor network. The Old-Actor network is used to interact with the environment and perform sampling to obtain sampled data. The Critic network uses the sampled data to train the New-Actor network. Every several rounds, the New-Actor network will copy its parameters to the Old-Actor network to update the agent strategy.

[0055] In the embodiment of the present invention, the state space, action space and reward function jointly define the problem and guide the learning and strategy optimization process of the intelligent agent. These three components work together to determine how the intelligent agent adjusts its strategy to achieve the goal. Among them, the state space S represents the set of all possible states that the system is currently in, and each state contains all the information of the system at a certain moment. In this solution, due to the existence of flexible resources, the state space includes distributed energy output, virtual power plant planned output value, battery SOC, demand response load and air conditioning load user average room temperature;

[0056]

[0057] Where: G i,der (t) is the output value of the ith distributed energy in period t; P out (t) is the planned output value of the flexible virtual power plant in period t; L j,d (t) is the power of the jth demand response load in period t; L k,a (t) is the load required by the kth air conditioning load in the t period; T a (t) is the average room temperature of air-conditioning load users.

[0058] In the embodiment of the present invention, the action space A represents the set of all possible actions performed by the agent in each state. Each action represents the operation taken by the agent in a specific state. For the power generation equipment that can be directly controlled by the flexible virtual power plant, its action value is the output G of the equipment. i,der (t); For energy storage equipment, its action value is the battery charging and discharging power P C (t); For the virtual power plant, which indirectly controls the demand response load and air conditioning load regulated by electricity price and air conditioning temperature, the corresponding actions are the electricity price adjustment range Z(t) and the air conditioning set temperature T a (t);

[0059] A=[G 1,der (t),G 2,der (t),…,G n,der (t),P C (t),Z(t),T a (t)].

[0060] In the embodiment of the present invention, the reward function R represents the reward obtained by the agent after performing a certain action, and the reward function enables the agent to select an action that can maximize long-term benefits; the reward function is the sum of the operating cost and penalty term of the flexible virtual power plant;

[0061] R=-(Qcost +Q pu )Q pu =Q pu1 +Q pu2 +Q pu3 ;

[0062] Where: Q cost is the operating cost; Q pu is the penalty term; Q pu1 , Q pu2 and Q pu3 They are the penalty items for the flexible virtual power plant failing to meet the planned output constraint conditions, the room temperature conditions of air-conditioning users, and the overcharging and over-discharging of energy storage equipment;

[0063]

[0064] Q pu3 (t) = q pu3 [max(0,s min -s(t))+max(0,s(t)-s max )]D max ;

[0065] Where: q pu1 ,q pu2 ,q pu3 They are the planned processing penalty coefficient, the air conditioning load room temperature over-limit penalty coefficient, and the battery overcharge and over-discharge penalty coefficient; T k,a (t) is the room temperature of the kth air-conditioning user in period t.

[0066] In the embodiment of the present invention, in order to ensure that the intelligent agent can be trained in an orderly manner in the early stage of interaction with the environment, the method collects and saves high-quality scheduling sample data in the experience pool. Based on these sample data, supervised learning is used to pre-train the initial policy network. In this way, the policy network can have a good basic scheduling ability at the beginning of training, and then effectively guide the intelligent agent to perform further reinforcement learning training. During the pre-training of the initial policy network, a neural network is constructed as the initial policy network, and the parameter settings of the initial policy network are consistent with the Actor network in the intelligent agent. Given the system state s t , the solution a can be obtained through the initial strategy network t , a t =π θ (s t ).

[0067] In order to evaluate the effect of the initial strategy network learning experience replay pool unit scheduling experience, this paper defines the loss value of the tth stage in the trajectory τ:

[0068]

[0069] Where: represents the actual output value of the ith distributed energy in the experience playback pool during period t, Represented by the neural network π θ Predicted output value. Loss function L t The goal of (θ) is to minimize the error between the predicted output and the actual output, ensuring that the pre-trained model can effectively capture the scheduling characteristics. In the conventional deep reinforcement learning training process, the state value function V(S t ) is defined as the expected return from time t to the future, and its calculation method is:

[0070]

[0071] Where: discount factor γ∈[0,1], r k is the reward obtained at time k, γ represents the importance of the reward at the future moment to the current reward. The larger its value is, the greater the impact of the reward at the future moment on the transition of the current state, and vice versa. Similarly, the cumulative loss function L(S0) is used to represent the execution of strategy π from the initial state S0. θ Until the end state S T Total loss:

[0072]

[0073] In the formula, C t is the loss value at time t, including penalty items such as the virtual power plant failing to meet the planned output, air conditioning temperature exceeding the limit, overcharging and over-discharging of energy storage equipment, etc. By minimizing the total loss L(S0), the intelligent agent can optimize the scheduling strategy and improve the operating efficiency of the flexible virtual power plant.

[0074] In summary, the embodiments of the present invention provide a solution support for flexible resources to participate in virtual power plant scheduling. At the same time, the neural network structure and deep reinforcement learning algorithm are effectively combined to deal with complex, high-dimensional state spaces. The initial policy network is trained in advance using a supervised learning pre-training method to promote the optimization results of the policy network learning model-driven method.

[0075] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the disclosure in the specification and examples. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. A flexible virtual power plant optimization scheduling method based on deep reinforcement learning, characterized in that: The method comprises the following steps: Establishing a flexibility resource model, wherein the flexibility resource model includes a battery energy storage device model, an air conditioning load model, and a demand response load model; Flexible virtual power plant optimization scheduling is performed based on deep reinforcement learning, specifically including: setting importance sampling and experience pool through proximal policy optimization algorithm so that the intelligent agent performs multiple updates after one interaction with the environment to improve the efficiency of the algorithm; building a deep reinforcement learning optimization model based on state space, action space and reward function; the experience pool collects and saves scheduling sample data, and based on the scheduling sample data, the initial policy network is pre-trained using supervised learning, so that the policy network has basic scheduling capabilities at the initial stage of training, thereby guiding the intelligent agent to perform further reinforcement learning training.

2. The flexible virtual power plant optimization scheduling method based on deep reinforcement learning according to claim 1 is characterized in that: The battery energy storage device model is used to ensure that the state of charge of the energy storage device is within a preset range. The battery energy storage device model is as follows: s min ≤s(t)≤s max ; In the formula, s(t) is the current SOC of the battery energy storage device; s(t-1) is the SOC of the battery energy storage device in the previous period; P(t) is the charge and discharge power of the battery energy storage device at time t; Δt is the unit time length; D max is the maximum capacity of the battery energy storage device; min 、s max They are the lower limit and upper limit of SOC respectively. SOC represents the state of charge of the battery energy storage device.

3. The flexible virtual power plant optimization scheduling method based on deep reinforcement learning according to claim 2 is characterized in that: The air conditioning load model is as follows: T in,min ≤T in (t)≤T in,max ; Where: T in , T out , T k are indoor air temperature, outdoor air temperature, and indoor building material temperature; C in , C k are the thermal mass of indoor air and the thermal mass of indoor building materials respectively; H in , H k are the thermal conductivity of indoor air and building materials respectively; P A is the power of the air conditioner. When the air conditioner is turned off, the power is set to 0. A The energy released per unit power consumption; T in,min , T in,max The upper and lower limits set for the indoor air temperature respectively.

4. The flexible virtual power plant optimization scheduling method based on deep reinforcement learning according to claim 3 is characterized in that: The demand response load model is as follows: L d (t)=L base (t)-L shift (t)+L due (t); L shift (t)=L base (t)bl t ; Where: L d (t), L base (t), L shift (t), L due (t) are load power consumption, basic load power consumption, transferable load power consumption and transfer load power consumption; l is the electricity price level coefficient; b is the price sensitivity coefficient; y i,t =1 means transfer, y i,t =0 means no transfer.

5. The flexible virtual power plant optimization scheduling method based on deep reinforcement learning according to claim 1 is characterized in that: In the proximal policy optimization algorithm, two Actor networks are set up: the Old-Actor network and the New-Actor network. The Old-Actor network is used to interact with the environment and sample to obtain sampled data. The Critic network uses the sampled data to train the New-Actor network. Every several rounds, the New-Actor network will copy its parameters to the Old-Actor network to update the agent strategy.

6. The flexible virtual power plant optimization scheduling method based on deep reinforcement learning according to claim 5 is characterized in that: The state space S represents the set of all possible states that the system is currently in. Each state contains all the information of the system at a certain moment. The state space includes distributed energy output, virtual power plant planned output value, battery SOC, demand response load and average room temperature of air conditioning load users; Where: G i,der (t) is the output value of the ith distributed energy in period t; P out (t) is the planned output value of the flexible virtual power plant in period t; L j,d (t) is the power of the jth demand response load in period t; L k,a (t) is the load required by the kth air conditioning load in the t period; T a (t) is the average room temperature of air-conditioning load users.

7. The method for optimizing and dispatching a flexible virtual power plant based on deep reinforcement learning according to claim 6 is characterized in that: The action space A represents the set of all possible actions performed by the agent in each state. Each action represents the operation taken by the agent in a specific state. For the power generation equipment that can be directly controlled by the flexible virtual power plant, its action value is the output G of the equipment. i,der (t); For energy storage equipment, its action value is the battery charging and discharging power P C (t); For the virtual power plant, which indirectly controls the demand response load and air conditioning load regulated by electricity price and air conditioning temperature, the corresponding actions are the electricity price adjustment range Z(t) and the air conditioning set temperature T a (t); 8. The method for optimizing and dispatching a flexible virtual power plant based on deep reinforcement learning according to claim 7 is characterized in that: The reward function R represents the reward obtained by the agent after performing a certain action, and the reward function enables the agent to select an action that can maximize long-term benefits; the reward function is the sum of the operating cost and penalty term of the flexible virtual power plant; R=-(Q cost +Q pu ) Q pu =Q pu1 +Q pu2 +Q pu3 ; Where: Q cost is the operating cost; Q pu is the penalty term; Q pu1 , Q pu2 and Q pu3 They are the penalty items for the flexible virtual power plant failing to meet the planned output constraint conditions, the room temperature conditions of air-conditioning users, and the overcharging and over-discharging of energy storage equipment; Where: q pu1 ,q pu2 ,q pu3 They are the planned processing penalty coefficient, the air conditioning load room temperature over-limit penalty coefficient, and the battery overcharge and over-discharge penalty coefficient; T k,a (t) is the room temperature of the kth air-conditioning user in period t.

9. The method for optimizing and dispatching a flexible virtual power plant based on deep reinforcement learning according to claim 8 is characterized in that: During the pre-training of the initial policy network, a neural network is constructed as the initial policy network, and the parameter settings of the initial policy network are consistent with the Actor network in the agent.

Images