Public building energy supply system scheduling method based on improved reinforcement learning algorithm

Abstract

The invention discloses a public building energy supply system scheduling method based on an improved reinforcement learning algorithm, which solves the defects in the prior art, and comprises the following steps: step 1, defining an energy management system of a public building energy supply system as an intelligent agent, and forming an action space of the intelligent agent by action output of a controllable unit in the public building energy supply system; 2, dividing an action space into an inner layer and an outer layer, directly outputting an outer-layer action through a reinforcement learning algorithm model, and calculating an inner-layer action through a quadratic programming algorithm model; and step 3, combining the outer-layer action and the inner-layer action as the action of the intelligent agent at the moment t, and performing energy scheduling on each subsystem in the public building energy supply system by the energy management system according to the action.

Description

Technical Field

[0001] The present invention relates to the technical field of integrated energy systems, and in particular to a method for scheduling a public building energy supply system based on an improved reinforcement learning algorithm. Background Art

[0002] With the rapid development of distributed renewable energy power generation technology, public building energy supply systems are widely equipped with distributed renewable energy power generation devices. This not only improves the absorption rate of new energy, but also transforms the public building energy supply system from a traditional energy consumer to an energy producer and consumer. The two-way flow of energy can be achieved between the public building energy supply system and the main power grid. It can not only purchase electricity from the main power grid, but also actively participate in the electricity market to sell electricity and obtain benefits.

[0003] The grid-connected public building energy supply system contains a variety of controllable equipment such as wind power, photovoltaics, diesel generators, energy storage systems, electric vehicles, etc. Due to the diversification and complexity of controllable equipment, the output actions of reinforcement learning in the scheduling method of the grid-connected public building energy supply system are increasing, and the dimension of the action space is also increasing. Reinforcement learning is difficult to train, resulting in a decrease in the efficiency of the scheduling method of the grid-connected public building energy supply system and a decrease in economic benefits. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the scheduling method of the grid-connected public building energy supply system in the prior art, that is, the output actions of reinforcement learning are increasing, the dimension of the action space is also increasing, the reinforcement learning is difficult to train, and the efficiency of the scheduling method of the grid-connected public building energy supply system is reduced, and a scheduling method for the public building energy supply system based on an improved reinforcement learning algorithm is provided.

[0005] The purpose of the present invention is achieved through the following technical solutions: The public building energy supply system scheduling method based on the improved reinforcement learning algorithm includes the following steps: Step 1, define the energy management system of the public building energy supply system as an intelligent agent, and the action outputs of the controllable units in the public building energy supply system constitute the action space of the intelligent agent; Step 2: Divide the action space into two layers, the outer layer action is directly output through the reinforcement learning algorithm model, and the inner layer action is calculated through the quadratic programming algorithm model; Step 3: Combine the outer action and the inner action as the action of the agent at time t, and the energy management system dispatches energy for each subsystem in the public building energy supply system according to the action.

[0006] The solution of the present invention transforms the problem of public building energy supply system scheduling into the form required by Markov decision making. The basic elements of the Markov decision model are the environmental observation state s t, the agent outputs action a t , the reward r obtained by the agent t , and the state transition probability p. Define the state space of the economic dispatch problem of the public building energy supply system as S:s t ∈S, the action space is A:a t ∈A, the reward space is R:r t ∈R. This scheme takes the energy management system of the public building energy supply system as an intelligent agent. At time t, the environmental state obtained by the intelligent agent includes the power supply of renewable energy, load power, charge state of the energy storage system, charging status of electric vehicles, and electricity purchase price of the public building energy supply system. The state space can be expressed as follows:

[0007] The action space mainly includes the action of the energy storage system, the action of electric vehicles, the action of diesel generators, and the power exchange between the public building energy supply system and the main grid. According to the set state space, some states, such as wind power and photovoltaic power generation, are related to the external environment and are not affected by the action of the energy management system, while the state of charge of the energy storage system and the charging status of electric vehicles are affected by the action of the system at the previous moment. At the same time, according to the previous discussion, the energy storage system and electric vehicles will also directly affect the production and consumption capacity of the public building energy supply system. This shows that different actions have different environmental impact effects. According to the different characteristics of the action space, the action space can be divided into two layers: the outer layer is the action of the energy storage system at the moment and the charging power of the electric vehicle; the inner layer is the exchange power between the public building energy supply system and the main grid and the diesel generator power.

[0008] The reinforcement learning algorithm is basically based on the value iteration algorithm. The value iteration algorithm has the advantages of stability and easy training. However, since the action form output by the relevant algorithm is discrete, in more complex scenarios, it will lead to a large action space for the intelligent agent, which will cost more computing resources and affect the training efficiency. According to the different characteristics of the action space, this scheme introduces a quadratic programming algorithm to reduce the dimension of the action space. Since the outer layer action has a direct impact on the subsequent state, the outer layer action is directly output by the reinforcement learning algorithm, and the generalized production and consumption capacity of the public building energy supply system can be directly determined according to the outer layer action. And the production and consumption capacity limits the transaction capacity between the public building energy supply system and the main power grid, so when the outer layer action is determined, the inner layer action can be calculated by the quadratic programming algorithm.

[0009] Preferably, the action space is expressed as follows: A={a out , a in} a out = {P tESS , P t EV} a in = {P t diesel , P t sell ,P t buy}; Among them, A represents the action space, a out Indicates outer action, a in Indicates inner action, P t ESS represents the action of the energy storage system at time t, P t EV Represents the charging power of electric vehicles, P t diesel Indicates diesel generator power, P t sell and P t buy Represents the exchange power between the public building energy supply system and the main power grid.

[0010] Preferably, in the outer layer action, the action of the energy storage system at time t is a discrete action, which is as follows: P t ESS ={-1.0, -0.5, 0.0, 0.5, 1.0}; The charging power of electric vehicles works as follows: P t EV ={0.0, 0.5, 1.0}; When the action value is 1 / -1, it means that the energy storage system and electric vehicles are charged / discharged at maximum power; when the action value is 0.5 / -0.5, it means that the charging power / discharging power is half of the rated power; when the action value is 0, it means that no action is taken.

[0011] Preferably, the quadratic programming algorithm model is as follows: 0≤P t buy , P t sell ≤|P t GPCC | P t GPCC +P t diesel +P t buy -Pt sell =0; Among them, ρ t buy Indicates the electricity purchase price, represents the electricity price, α and β are custom coefficients, P t GPCC Represents the broad production and sales capacity of public building energy supply systems.

[0012] Preferably, the instant reward of the agent is set as follows: Among them, r t For instant rewards, the cumulative rewards are obtained by accumulating all instant rewards, and the parameters of the reinforcement learning algorithm model are adjusted with the highest cumulative reward as the goal, thereby optimizing the quadratic programming algorithm model.

[0013] Preferably, the instant reward of the intelligent agent also introduces a penalty item to punish the electric vehicle for not meeting the charging requirement, specifically: in, is the penalty coefficient, Is the charging demand. Since the public building energy supply system contains electric vehicle units, the charging demand of electric vehicles needs to be met, so a penalty item needs to be introduced in the reinforcement learning reward to punish the behavior of not meeting the charging demand.

[0014] The beneficial effects of the present invention are as follows: the scheme of the present invention takes into account the coordinated optimization of multiple controllable units in the public building energy supply system, including energy storage systems, electric vehicles, diesel power generation and the main power grid. According to the dual characteristics of the public building energy supply system having production and consumption capabilities, the generalized production and consumption capabilities of the public building energy supply system are defined. At the same time, combined with the generalized production and consumption capabilities, the traditional reinforcement learning is improved, the characteristics of the actions of different units in the public building energy supply system are analyzed, the action space of the energy management system based on reinforcement learning is divided into a two-layer structure, and quadratic programming is used to reduce the dimension of the action space. The algorithm proposed in the scheme of the present invention has better convergence characteristics and can obtain higher rewards in the scheduling of the public building energy supply system. The scheme of the present invention has strong robustness, and its training effect is close to the results in the deterministic scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a flow chart of the present invention; Figure 2 It is a schematic diagram of the cumulative rewards of different algorithms in the training process of the present invention. DETAILED DESCRIPTION

[0016] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0017] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0018] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0019] Example: A method for dispatching energy supply systems for public buildings based on improved reinforcement learning algorithms, such as Figure 1 As shown, the following steps are included: Step 1, define the energy management system of the public building energy supply system as an intelligent agent, and the action outputs of the controllable units in the public building energy supply system constitute the action space of the intelligent agent; Step 2: Divide the action space into two layers, the outer layer action is directly output through the reinforcement learning algorithm model, and the inner layer action is calculated through the quadratic programming algorithm model; Step 3: Combine the outer action and the inner action as the action of the agent at time t, and the energy management system dispatches energy for each subsystem in the public building energy supply system according to the action.

[0020] The solution of the present invention transforms the problem of public building energy supply system scheduling into the form required by Markov decision making. The basic elements of the Markov decision model are the environmental observation state s t , the agent outputs action a t , the reward r obtained by the agent t , and the state transition probability p. Define the state space of the economic dispatch problem of the public building energy supply system as S:s t ∈S, the action space is A:a t ∈A, the reward space is R:r t∈R. This scheme takes the energy management system of the public building energy supply system as an intelligent agent. At time t, the environmental state obtained by the intelligent agent includes the power supply of renewable energy, load power, charge state of the energy storage system, charging status of electric vehicles, and electricity purchase price of the public building energy supply system. The state space can be expressed as follows:

[0021] The action space mainly includes the action of the energy storage system, the action of electric vehicles, the action of diesel generators, and the power exchange between the public building energy supply system and the main grid. According to the set state space, some states, such as wind power and photovoltaic power generation, are related to the external environment and are not affected by the action of the energy management system, while the state of charge of the energy storage system and the charging status of electric vehicles are affected by the action of the system at the previous moment. At the same time, according to the previous discussion, the energy storage system and electric vehicles will also directly affect the production and consumption capacity of the public building energy supply system. This shows that different actions have different environmental impact effects. According to the different characteristics of the action space, the action space can be divided into two layers: the outer layer is the action of the energy storage system at the moment and the charging power of the electric vehicle; the inner layer is the exchange power between the public building energy supply system and the main grid and the diesel generator power.

[0022] The reinforcement learning algorithm is basically based on the value iteration algorithm. The value iteration algorithm has the advantages of stability and easy training. However, since the action form output by the relevant algorithm is discrete, in more complex scenarios, it will lead to a large action space for the intelligent agent, which will cost more computing resources and affect the training efficiency. According to the different characteristics of the action space, this scheme introduces a quadratic programming algorithm to reduce the dimension of the action space. Since the outer layer action has a direct impact on the subsequent state, the outer layer action is directly output by the reinforcement learning algorithm, and the generalized production and consumption capacity of the public building energy supply system can be directly determined according to the outer layer action. And the production and consumption capacity limits the transaction capacity between the public building energy supply system and the main power grid, so when the outer layer action is determined, the inner layer action can be calculated by the quadratic programming algorithm.

[0023] The action space is represented as follows: A={a out , a in} a out = {P t ESS , P t EV} a in = {P t diesel , P t sell ,P tbuy}; Among them, A represents the action space, a out Indicates outer action, a in Indicates inner action, P t ESS represents the action of the energy storage system at time t, P t EV Represents the charging power of electric vehicles, P t diesel Indicates diesel generator power, P t sell and P t buy Represents the exchange power between the public building energy supply system and the main power grid.

[0024] In the outer layer action, the action of the energy storage system at time t is a discrete action, which is as follows: P r ESS ={-1.0, -0.5, 0.0, 0.5, 1.0}; The charging power of electric vehicles works as follows: P t EV ={0.0, 0.5, 1.0}; When the action value is 1 / -1, it means that the energy storage system and electric vehicles are charged / discharged at maximum power; when the action value is 0.5 / -0.5, it means that the charging power / discharging power is half of the rated power; when the action value is 0, it means that no action is taken.

[0025] The quadratic programming algorithm model is as follows: 0≤P t buy , P t sell ≤|P t GPCC | P t GPCC +P t diesel +P t buy -P t sell =0; Among them, ρ t buy Indicates the electricity purchase price, represents the electricity price, α and β are custom coefficients, P t GPCC Represents the broad production and sales capacity of public building energy supply systems.

[0026] The instant reward of the agent is set as follows: r t =(ρ t sell P t sell -ρ t buy P t buy -α(P t diesel ) 2 -βP t diesel )ΔT; Among them, r t For instant rewards, the cumulative rewards are obtained by accumulating all instant rewards, and the parameters of the reinforcement learning algorithm model are adjusted with the highest cumulative reward as the goal, thereby optimizing the quadratic programming algorithm model.

[0027] The instant reward of the agent also introduces a penalty term to punish the electric vehicle for not meeting the charging requirements, specifically: in, is the penalty coefficient, Is the charging demand. Since the public building energy supply system contains electric vehicle units, the charging demand of electric vehicles needs to be met, so a penalty item needs to be introduced in the reinforcement learning reward to punish the behavior of not meeting the charging demand.

[0028] The effectiveness of the scheme of this embodiment is verified below, wherein real data are selected for wind power generation, photovoltaic power generation, user fixed load data and electricity price data, and the scheduling time interval of the public building energy supply system is set to 30 minutes, that is, the power data and electricity price data are sampled every 30 minutes, so there are 48 scheduling moments in a scheduling cycle of one day.

[0029] The relevant parameters of the controllable units of the public building energy supply system are shown in Table 1: Table 1 Basic parameters of controllable units This scheme uses a neural network as the Q function of reinforcement learning. The parameters of the neural network are explained as follows. In the experiment, both the policy Q value network and the target Q value network have 5 fully connected hidden layers. Each hidden layer has 400 ReLU neurons. When training the Q-network, the variable learning rate training method is used, and the initial learning rate is set to 0.005. The basic parameters of reinforcement learning are shown in Table 2 below.

[0030] Table 2 Reinforcement learning parameter settings parameter value Discount Rate 0.99 Batch size 100 Storage space size 50000 Error function smooth_l1_loss Optimizer Adam The reinforcement learning algorithm mentioned in this scheme has been implemented in Python 3.7.2 using Pytorch 1.6.0. The basic parameters of the computer system are Intel Core i7-8550U (1.80GHz–4.00GHz) equipped with 16GB RAM.

[0031] In the public building energy supply system environment with the same structure, the economic dispatch benefits of this algorithm and other algorithms are shown in Table 3.

[0032] Table 3 Cumulative rewards for different algorithms in one day serial number algorithm Accumulated Rewards 1 IRL-QR 48.83 2 Dueling considering GPCC 46.44 3 DDQN considering GPCC 46.72 4 Dueling 45.78 5 DDQ 43.52 In the above table, Algorithm 1 represents an improved reinforcement learning algorithm that considers the generalized production and consumption capacity of the public building energy supply system. For Algorithm 2 and Algorithm 3, two reinforcement learning algorithms, Dueling and DDQN algorithms, will be used to apply the generalized production and consumption capacity. Dueling and DDQN directly output the actions of energy storage systems, sequentially charged electric vehicles, and diesel generators without the need for quadratic programming, which is also the application idea of ​​traditional reinforcement learning. At the same time, Algorithm 4 and Algorithm 5 mean that when Dueling and DDQN are used to output actions, the generalized production and consumption capacity of the public building energy supply system is not considered. At the same time, in order to avoid the public building energy supply system from obtaining economic benefits through excessive power generation of diesel generators, if the new energy power generation is greater than the fixed load, the action value of diesel generators is set to zero. Unlike the algorithm mentioned in this solution, if the reinforcement learning algorithm is used directly to output actions, the action space of reinforcement learning will become larger. The actions of diesel generators are divided into 4, namely [0kw, 10kw, 20kw, 30kw], and the action dimension of reinforcement learning output is 5*3*4.

[0033] Comparing the data results of different algorithms in Table 3, it can be seen that under the same algorithm, by defining generalized production and consumption capacity, the public building energy supply system can obtain higher economic benefits. And the improved reinforcement learning algorithm can obtain higher cumulative rewards than the traditional idea of ​​directly applying reinforcement learning.

[0034] The cumulative rewards of different algorithms during the training process are as follows: Figure 2 As shown. The improved reinforcement learning algorithm and other reinforcement learning algorithms mentioned above are all based on learning. Figure 2The figure shows the reward changes of the three algorithms during the training process. In the figure, according to the training curves of the five algorithms, it can be seen that the improved reinforcement learning algorithm is close to the optimal strategy at about 1000 steps, and the final cumulative reward convergence is better than other algorithms. Since reinforcement learning takes random actions for exploration during the training process, the early stage of the training curve is mainly the exploration of reinforcement learning. Observing the training curve, it can be seen that before 500 steps, the early reward oscillation of the improved reinforcement learning is relatively small. And after 200 steps, compared with other algorithms, the oscillation amplitude of the reward curve of the improved reinforcement learning begins to decrease, which means that the proposed algorithm can learn the optimal action from exploration faster. This is because Benang stratifies the actions of the agent according to the proposed algorithm, reduces the dimension of the action space, and can reduce the computational resource cost of reinforcement learning to explore the action.

[0035] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0036] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for dispatching a public building energy supply system based on an improved reinforcement learning algorithm, characterized in that: The following steps are involved: Step 1, define the energy management system of the public building energy supply system as an intelligent agent, and the action outputs of the controllable units in the public building energy supply system constitute the action space of the intelligent agent; Step 2: Divide the action space into two layers, the outer layer action is directly output through the reinforcement learning algorithm model, and the inner layer action is calculated through the quadratic programming algorithm model; Step 3: Combine the outer action and the inner action as the action of the agent at time t, and the energy management system dispatches energy for each subsystem in the public building energy supply system according to the action.

2. The method for dispatching a public building energy supply system based on an improved reinforcement learning algorithm according to claim 1 is characterized in that: The action space is represented as follows: A={a out ,a in } a out ={P t ESS ,P t EV } a in ={P t diesel ,P t sell ,P t buy }; Among them, A represents the action space, a out Indicates outer action, a in Indicates inner action, P t ESS represents the action of the energy storage system at time t, P t EV Represents the charging power of electric vehicles, P t diesel Indicates diesel generator power, P t sell and P t buy Represents the exchange power between the public building energy supply system and the main power grid.

3. The method for dispatching a public building energy supply system based on an improved reinforcement learning algorithm according to claim 2 is characterized in that: In the outer layer action, the action of the energy storage system at time t is a discrete action, which is as follows: P t ESS ＝{-1.0,-0.5，0.0，0.5，1.0}； The charging power of electric vehicles works as follows: P t EV ＝{0.0,0.5，1.0}； When the action value is 1 / -1, it means that the energy storage system and electric vehicles are charged / discharged at maximum power; when the action value is 0.5 / -0.5, it means that the charging power / discharging power is half of the rated power; when the action value is 0, it means that no action is taken.

4. The method for dispatching a public building energy supply system based on an improved reinforcement learning algorithm according to claim 2 or 3, characterized in that: The quadratic programming algorithm model is as follows: 0≤P t buy ,P t sell ≤|P t GPCC | P t GPCC +P t diesel +P t buy -P t sell =0; Among them, ρ t buy Indicates the electricity purchase price, represents the electricity price, α and β are custom coefficients, P t GPCC Represents the broad production and sales capacity of public building energy supply systems.

5. The method for dispatching a public building energy supply system based on an improved reinforcement learning algorithm according to claim 4 is characterized in that: The instant reward of the agent is set as follows: Among them, r t For instant rewards, the cumulative rewards are obtained by accumulating all instant rewards, and the parameters of the reinforcement learning algorithm model are adjusted with the highest cumulative reward as the goal, thereby optimizing the quadratic programming algorithm model.

6. The method for dispatching a public building energy supply system based on an improved reinforcement learning algorithm according to claim 5 is characterized in that: The instant reward of the agent also introduces a penalty term to punish the electric vehicle for not meeting the charging requirements, specifically: in, is the penalty coefficient, It is the charging requirement.

Images