Microgrid energy management method based on Rainbow deep Q network

Abstract

The present invention discloses a microgrid energy management method based on Rainbow deep Q network, which takes electric vehicles and energy storage devices in a residential microgrid as controlled carriers, and uses the load demand of the microgrid for 24 hours a day, photovoltaic and wind power output, and charge state prediction data of electric vehicles and energy storage devices as the state space input of the Rainbow deep Q network. Through intelligent learning and execution of the charging and discharging actions of electric vehicles and energy storage devices, the purpose of minimizing the daily operation cost of the microgrid under the premise of ensuring power stability and equipment life safety is achieved. This method can solve the difficulty of modeling the microgrid energy management model caused by the randomness and intermittency of renewable energy, and has fast convergence after training, so as to effectively solve the defects of low energy utilization and high operation cost of the microgrid caused by the uncertainty of renewable energy, energy flow and load diversity.

Description

Technical Field

[0001] The present invention relates to the field of electric power engineering technology, and in particular to the field of microgrid operation control and energy management. Background Art

[0002] In recent years, due to the global coal inventory shortage, the power generation of power plants has continued to show negative growth, causing global power shortages. As a new type of energy Internet coupled with multiple energy systems and energy storage devices, microgrids can not only access clean new energy such as wind power generation and photovoltaic power generation to reduce the power purchase of the power grid, but also use distributed resources and various energy management methods to improve energy utilization and reduce energy and economic waste. When dealing with power shortages, it can play a positive role in increasing revenue and reducing expenditure. However, it is precisely because of the multi-energy coupling characteristics of microgrids, the uncertainty of renewable energy, and the diversity of energy flow and load that the scheduling of energy at different times and spaces has become a major challenge for microgrid energy management. Therefore, it is necessary to find an efficient energy management method to be applied to microgrid energy management. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a microgrid energy management strategy based on the Rainbow deep Q network, model the microgrid structure, and learn an optimal strategy to achieve the purpose of minimizing the daily operating cost of the microgrid and improving energy utilization while ensuring the normal operation of all participating parts of the microgrid.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0005] The microgrid energy management method based on Rainbow deep Q network includes the following steps:

[0006] Step (1): Establish an energy management simulation model corresponding to the residential microgrid. In the day-ahead scheduling stage, obtain the forecast output data of photovoltaic and wind power generation for the next day, the real-time load demand forecast data of the microgrid, and the real-time electricity price forecast data of the main grid, and use them as the training data set of the model.

[0007] Step (2): Define the training environment required for the microgrid energy management simulation model determined by Markov decision in reinforcement learning, including the following sub-steps:

[0008] (2.1) Define the environmental state space: The load demand from 0:00 to 23:00 on the day of scheduling, the output power of new energy sources, and the real-time charge state of electric vehicles and energy storage devices constitute the environmental state space S(t) required for this energy management simulation model.

[0009] (2.2) Define the agent action space: Control the electric vehicle and energy storage device by scheduling the agent to execute charging and discharging actions.

[0010] (2.3) Define a reward function: The reward function guides the agent to achieve a predetermined microgrid energy management optimization.

[0011] (2.4) A microgrid backup safety controller is provided, wherein the microgrid backup safety controller is used to control the intelligent agent to perform overcharging or over-discharging erroneous operations; causing the energy storage device and the electric vehicle to exceed the upper and lower limits of the state of charge, thereby affecting the life and safety of the equipment.

[0012] (2.5) Executing actual control instructions: The agent executes the actions learned from the Rainbow deep Q network to perform charging and discharging operations.

[0013] (2.6) Interaction with the main grid: Connect with the main grid through the public connection point. When there is still surplus power after the output of new energy can meet the load demand of the microgrid, the surplus power will be fed back to the main grid; when the output of new energy, energy storage equipment and electric vehicles cannot meet the load demand of the microgrid at the maximum power output, electricity will be purchased from the main grid at the real-time electricity price. In this way, the real-time power balance of the entire microgrid system can be achieved.

[0014] Step (3): Use the Rainbow deep Q network to train the microgrid energy management model defined in step (2), and divide 0 to 23 hours into 24 scheduling moments, each hour as a step, and one scene as 24 hours; use the real-time prediction data of the day to repeatedly train multiple scenes until the final reward function converges. Step (4): Apply the intelligent agent trained in step (3) to the microgrid energy management of the day. When training each scene, input the model state space data of each scheduling moment into the neural network to obtain the action value function of each executable action in the state space; select the optimal action by horizontally comparing the action value function of each action and convert it into actual instructions, thereby controlling the charging and discharging operations of energy storage equipment and electric vehicles.

[0015] Electric vehicles of residential users participate in the energy management of microgrids. When the load demand is large, they can act as energy storage devices to output power to reduce the amount of electricity purchased from the power grid; when the load demand is small, they can act as interruptible loads to reduce the daily operating costs of the microgrid.

[0016] Wind power and photovoltaic power generation constitute the renewable energy output of the microgrid system; residential users are connected to the microgrid to supply their electricity; the microgrid system is equipped with energy storage equipment; the microgrid is connected to the large power grid through a public connection point to exchange electricity and obtain real-time electricity prices.

[0017] The step (1) comprises the following steps:

[0018] I) Optimize the daily operating cost, which includes three parts: first, the grid interaction cost, which is divided into electricity sales revenue and electricity purchase cost; second, the energy storage equipment operating cost; third, the electric vehicle operating cost; the specific formula is as follows:

[0019]

[0020] Among them, T is the total dispatch period; E b,t is the unit electricity purchase price at time t; E s,t is the unit electricity price at time t; X t is the amount of electricity exchanged with the power grid at time t; C b,t is the operating cost of the energy storage system at time t; C v,t is the operating cost of the electric vehicle at time t.

[0021] II) Balance the power of each participating part with the power of the large power grid. The specific power balance formula is as follows:

[0022] P x,t -P l,t +P n,t -P b,t -P v,t =0 (2)

[0023] Among them, P x,t P is the power exchanged from the large power grid at dispatch time t. Positive numbers represent power purchases and negative numbers represent power sales. l,t is the real-time load demand of the microgrid system at time t; P n,t is the wind power and photovoltaic power output at time t; P b,t is the operating power of the energy storage device at time t, a positive number represents the power in the charging state, and a negative number represents the power in the discharging state; P v,t is the real-time power of the electric vehicle at time t. Positive numbers represent the power in the charging state, and negative numbers represent the power in the discharging state.

[0024] III) While achieving the optimization goal of the lowest daily operating cost, the energy management system needs to reduce the peak-to-valley difference caused by load and production imbalance, and ensure that each device operates in a normal working environment. The optimization goals are as follows:

[0025]

[0026] As shown above, the charging power P of the energy storage device bc,t , discharge power P bd,t The maximum power allowed for normal operation shall not exceed that of the electric vehicle. vc,t , discharge power Pvd,t Nor shall it exceed the maximum power allowed for its normal operation.

[0027] In step (2.1), before each scheduling moment, the agent obtains the wind power and photovoltaic power output P at time t from the environment n,t and the real-time load demand P of the microgrid system at time t l,t , as the external state information of the energy management model. At the same time, according to the action selected by the agent at the last scheduling moment, the charge state B of the electric vehicle at that moment is changed v,t and the charge state B of the energy storage device at that moment b,t , these two are the controllable state information of the energy management model; these four state information form a state space, which is sent to the neural network to train the optimal action and select it.

[0028] In step (2.2), the actions used to optimize the microgrid energy management model include the real-time power P of the electric vehicle at each scheduling time t v,t and the operating power P of the energy storage device at time t b,t , and combined into N according to the charge and discharge gears of each battery v *N b (N v is the number of charging and discharging gears of the electric vehicle, N b is the number of charging and discharging gears of the energy storage device) at each scheduling moment, the actions that can be selected by the intelligent agent constitute the action space.

[0029] In step (2.3), the agent selects actions through a reward function; the reward function is divided into three parts: electricity purchase cost and electricity sales reward, energy storage system operation cost and violation penalty, electric vehicle operation compensation and violation penalty. As shown below:

[0030]

[0031] C e,t is the income or cost of interacting with the power grid at dispatch time t, E s,t is the unit electricity price at time t, E b,t is the unit electricity purchase price at time t, X t is the amount of electricity interacting with the grid at time t; V b,t represents the total electricity cost of the energy storage device at time t, δ1 is the electricity coefficient; D b,t represents the overcharge or over-discharge penalty of the energy storage device at time t, δ2 is D b,t The penalty coefficient, d p1,t is the amount of energy storage system overcharged or over-discharged at time t; similarly, V v,trepresents the operating compensation cost of the electric vehicle at time t, which is used to compensate the user for the loss of battery loss and other problems caused by the participation of electric vehicles in scheduling. δ3 is the compensation coefficient; D v,t represents the overcharge or over-discharge penalty of the electric vehicle at time t, and δ4 is D v,t The penalty coefficient, d p2,t C is the amount of overcharge or overdischarge of the electric vehicle at time t; b,t is the operating cost of the energy storage system at time t, C v,t is the operating cost of the electric vehicle at time t, R t is the reward function.

[0032] Step (3) includes the following sub-steps

[0033] (3.1) Construct the Rainbow deep Q network, including the following parts:

[0034] I) Construct a neural network with two fully connected layers and one hidden layer. Add a Gaussian distributed noise term to the weights of all fully connected layers to interfere with the original ε-greedy random-greedy exploration method to improve the exploration rationality of the algorithm.

[0035] II) Add a competitive network before the output layer Q network to split the output Q function into the average of all Q values ​​under the value function V state and the sum of the number of advantage functions A that are restricted to an average of 0. The advantage is that the value of the action is associated with the real-time state of the agent, making the strategy for selecting actions better; the specific formula is as follows:

[0036]

[0037] Among them, α and β are the parameters of the value function network layer and the advantage function network layer respectively, and 1 / |A| is the average value of the advantage function.

[0038] III) Build two Q networks as the final output layer of the neural network to decouple action selection and value estimation. One network is used to evaluate all actions in the state, and the other is used to select specific actions. This is used to improve the over-estimation error caused by bootstrapping.

[0039] IV) Use the multi-step learning method to update the Q network to accelerate the convergence of the Q network. The formula is as follows:

[0040]

[0041] Among them, n is the update stride length, w is the neural network parameter, and γ is the discount rate.

[0042] (3.2) Prioritized experience replay pool: Set up an experience pool with a fixed capacity and replay each set of data (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool, and according to the size of the error caused by the previous training, the priority is arranged by formula to evaluate the order and probability of each set of data being re-sent to the neural network for training, in order to increase the utilization of data and reduce the correlation of adjacent data. The specific sampling method is as follows:

[0043]

[0044] Among them, X i is the error value caused by this set of experience in the previous training, and ε is to prevent p i The noise is 0, and z is the annealing factor, which is used to adjust the priority.

[0045] The present invention describes the energy management problem of microgrids as a Markov decision process under the framework of reinforcement learning, and learns an optimal control strategy after continuously interacting with the environment. It overcomes the modeling difficulties caused by the uncertainty of new energy output, and after the convergence of the final reward is obtained, the trained intelligent agent can be used for real-time scheduling for a day.

[0046] During the scheduling stage, the real-time status information of each scheduling period is input into the neural network. The neural network evaluates each action and selects the optimal action, and uses this as a basis to execute the charging and discharging operations of electric vehicles and energy storage equipment, while ensuring normal operation and achieving the optimization goal of minimizing daily operating costs.

[0047] This method solves the modeling difficulties caused by the uncertainty of renewable energy output, and has fast convergence after training, and also has excellent generalization capabilities when facing new energy management scenarios.

[0048] The specific technical solutions and beneficial effects of the present invention will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Microgrid structure diagram and energy management strategy block diagram

[0050] Figure 2 One-day wind power generation and photovoltaic power generation forecast map

[0051] Figure 3 Real-time load demand forecast diagram

[0052] Figure 4 Main grid time-of-use electricity price diagram

[0053] Figure 5 Rainbow Deep Q Network Training Results

[0054] Figure 6 Deep Q network training results

[0055] Figure 7 Neural network loss value change chart

[0056] Figure 8 State of charge change diagram of electric vehicles and energy storage devices DETAILED DESCRIPTION

[0057] The technical solutions of the embodiments of the present invention are explained and described below, but the following embodiments are only preferred embodiments of the present invention, not all. Based on the embodiments in the implementation mode, other embodiments obtained by those skilled in the art without creative work are all within the protection scope of the present invention.

[0058] The present invention relates to an energy management method, which describes the energy management problem of a microgrid as a Markov decision process under a reinforcement learning framework, and learns an optimal control strategy after continuously interacting with the environment.

[0059] The purpose of the present invention is to input the real-time status information of each scheduling period into a neural network during the scheduling stage. The neural network evaluates each action and selects the optimal action, and based on this, performs charging and discharging operations of electric vehicles and energy storage devices, thereby achieving the optimization goal of minimizing daily operating costs while ensuring normal operation.

[0060] The microgrid energy management method based on Rainbow deep Q network includes the following steps:

[0061] Step (1): Establish an energy management simulation model corresponding to the residential microgrid. In the day-ahead scheduling stage, obtain the forecast output data of photovoltaic and wind power generation for the next day, the real-time load demand forecast data of the microgrid, and the real-time electricity price forecast data of the main grid, and use them as the training data set of the model.

[0062] I) Optimize the daily operating cost, which includes three parts: first, the grid interaction cost, which is divided into electricity sales revenue and electricity purchase cost; second, the energy storage equipment operating cost; third, the electric vehicle operating cost; the specific formula is as follows:

[0063]

[0064] Among them, T is the total dispatch period; E b,t is the unit electricity purchase price at time t; E s,t is the unit electricity price at time t; X tis the amount of electricity exchanged with the power grid at time t; C b,t is the operating cost of the energy storage system at time t; C v,t is the operating cost of the electric vehicle at time t.

[0065] II) To ensure that the power of each participating part can always maintain balance when managing the energy of the microgrid system, the following power balance relationship is defined:

[0066] P x,t -P l,t +P n,t -P b,t -P v,t =0 (2)

[0067] Among them, P x,t P is the power exchanged from the large power grid at dispatch time t. Positive numbers represent power purchases and negative numbers represent power sales. l,t is the real-time load demand of the microgrid system at time t; P n,t is the wind power and photovoltaic power output at time t; P b,t is the operating power of the energy storage device at time t, a positive number represents the power in the charging state, and a negative number represents the power in the discharging state; P v,t is the real-time power of the electric vehicle at time t. Positive numbers represent the power in the charging state, and negative numbers represent the power in the discharging state.

[0068] Step (2): Define the training environment required for the microgrid energy management simulation model determined by Markov decision making in reinforcement learning.

[0069] (2.1) Define the environmental state space: The load demand from 0:00 to 23:00 on the day of scheduling, the output power of new energy sources, and the real-time charge state of electric vehicles and energy storage devices constitute the environmental state space S(t) required for this energy management simulation model.

[0070] (2.2) Define the agent action space: Control the electric vehicle and energy storage device by scheduling the agent to execute charging and discharging actions.

[0071] (2.3) Define a reward function: used to guide the agent to achieve the predetermined microgrid energy management optimization goal.

[0072] (2.4) A microgrid backup safety controller is provided to prevent the intelligent agent from performing overcharging or over-discharging erroneous operations, causing the energy storage device and the electric vehicle to exceed the upper and lower limits of the state of charge, thereby affecting the life and safety of the equipment.

[0073] (2.5) Executing actual control instructions: The agent executes the actions learned from the Rainbow deep Q network to perform charging and discharging operations.

[0074] (2.6) Interaction with the main grid: Connect with the main grid through the public connection point. When there is still surplus power after the output of new energy can meet the load demand of the microgrid, the surplus power will be fed back to the main grid; when the output of new energy, energy storage equipment and electric vehicles cannot meet the load demand of the microgrid at the maximum power output, electricity will be purchased from the main grid at the real-time electricity price. In this way, the real-time power balance of the entire microgrid system can be achieved.

[0075] Step (3): Use the Rainbow deep Q network to train the microgrid energy management model defined in step (2), divide 0 to 23 o'clock into 24 scheduling moments, each hour as a step, and one scene as 24 hours; use the real-time prediction data of the day to repeatedly train multiple scenes until the final reward function converges.

[0076] Step (4): Apply the intelligent agent trained in step (3) to the microgrid energy management of the day. When training each scene, input the model state space data at each scheduling moment into the neural network to obtain the action value function of each executable action in the state space; select the optimal action by horizontally comparing the action value function of each action and convert it into actual instructions, thereby controlling the charging and discharging operations of the energy storage equipment and electric vehicles.

[0077] In order to illustrate the effect of the present invention, a microgrid system is taken as an implementation object of the present invention to describe the method of the present invention in detail below:

[0078] Step (1): Establish an energy management simulation model corresponding to the residential microgrid. The microgrid structure is as follows: Figure 1 In the day-ahead dispatching stage, the predicted output data of photovoltaic and wind power generation for the next day is obtained, such as Figure 2 As shown; the real-time load demand forecast data of the microgrid, such as Figure 3 as shown; and Figure 4 The real-time electricity price data of the main power grid shown in Figure 2 is used as the training data set for the model.

[0079] I) According to the optimization goal of minimizing the daily operating cost of the microgrid, the composition of the total cost of the microgrid is set, including the grid interaction cost, the energy storage equipment operating cost, and the electric vehicle operating cost, as shown in formula (1):

[0080]

[0081] Since the setting this time takes one-day microgrid energy management as the goal, the total time period T in this invention is 23, that is, one day is divided into 24 time periods for model optimization.

[0082] II) Considering that the power of each participating part should always be balanced in the overall scheduling process, the power balance relationship shown in formula (2) is defined:

[0083] P x,t -P l,t +P n,t -P b,t -P v,t =0 (2)

[0084] Step (2): Define the training environment required for the microgrid energy management simulation model determined by Markov decision in reinforcement learning, including the following sub-steps:

[0085] (2.1) Define the environmental state space: the load demand from 0:00 to 23:00 on the day of dispatch, the output power of new energy, the real-time charge state of electric vehicles and the real-time charge state of energy storage equipment, that is, the state space S(t) = {P n,t, P l,t, B v,t ,B b,t}

[0086] (2.2) Define the agent action space: Control is performed by scheduling the agent to execute the charging and discharging actions of the electric vehicle and the energy storage device, that is, the action space A(t) = {P b,t ,P v,t},Since the reinforcement learning based on value iteration intelligently executes discrete actions, the charging and discharging actions of electric vehicles are set to the following 11 gears:

[0087] {-200,-150,-100,-50,0,50,100,150,200,300,400}

[0088] Similarly, the charging and discharging actions of the energy storage device are also set to 9 gears for control:

[0089] {-200,-150,-100,-50,0,50,100,150,200}

[0090] (2.3) Define the reward function: used to guide the intelligent agent to achieve the predetermined microgrid energy management optimization goal. The reward function is divided into three parts: power purchase cost and power sales reward, energy storage system operation cost and violation penalty, electric vehicle operation compensation and violation penalty. As shown in formula (5):

[0091] R t =C e,t +C v,t +C b,t (5)

[0092] (2.4) A microgrid backup safety controller is set to prevent the intelligent agent from performing overcharging or over-discharging misoperations, causing the energy storage equipment and electric vehicles to exceed the upper and lower limits of the state of charge, affecting the life and safety of the equipment. The setting is shown in formula (6). When it exceeds or equals the set value, the backup controller will set the action given by the model to 0. The reason why the minimum state of charge of electric vehicles is set to 0.35 is that users need to ensure that there is a certain amount of battery remaining for their use when using electric vehicles.

[0093]

[0094] (2.5) Execute actual control instructions: When the state of charge of the electric vehicle and energy storage device does not exceed the set value, the intelligent agent executes the actions learned from the Rainbow deep Q network to perform charging and discharging operations.

[0095] (2.6) Interaction with the main grid: The microgrid is connected to the main grid through a public connection point. When there is excess power after the output of renewable energy can meet the load demand of the microgrid, the remaining power is fed back to the main grid. When the output of renewable energy, energy storage equipment, and electric vehicles cannot meet the load demand of the microgrid at maximum power output, electricity is purchased from the main grid at the real-time electricity price, thereby achieving real-time power balance of the entire microgrid system.

[0096] Step (3): Use the Rainbow deep Q network to train the microgrid energy management model defined in step (2), dividing 0 to 23 o'clock into 24 scheduling moments, with each hour as a step and one scene as 24 hours; use the real-time prediction data of the day to repeatedly train multiple scenes until the final reward function converges. Step 3 includes the following sub-steps

[0097] (3.1) Construct the Rainbow deep Q network, including the following parts:

[0098] I) Construct a neural network consisting of two fully connected layers and one hidden layer. Add a Gaussian distributed noise term to the weights of all fully connected layers to interfere with the original ε-greedy exploration method to improve the exploration rationality of the algorithm. The specific loss function is shown in formula (7).

[0099]

[0100] Among them, the original weight w and error b are transformed from a numerical value to a normal distribution with a mean of μ and a variance of σ. At the same time, there is a certain random noise ε that obeys the Gaussian distribution. The noise ε is a constant generated in each round of training. The calculation method is shown in formula (8):

[0101]

[0102] II) Add a competitive network before the output layer Q network.

[0103] III) Build two Q networks as the final output layer of the neural network.

[0104] IV) Use the multi-step learning method to update the Q network to accelerate the convergence of the Q network. When using multi-step updating, the update step should not be too long. This model uses the double-step updating method to update the Q network.

[0105] (3.2) Prioritized experience replay pool: Before starting training, let the agent perform random actions to accumulate 2,000 sets of experience to fully ensure that there is a certain amount of experience for replay training at the beginning of training. The experience pool size of the model is set to 3,000. When the number of experiences stored in the experience pool reaches 3,000, the new training data will replace the original data in the experience pool in the sampling order. The SumTree method is used to distribute the data to be sampled and its priority, and sampling is performed to achieve prioritized experience replay.

[0106] Step (4): Apply the intelligent agent trained in step (3) to the microgrid energy management of the day, with the initial state of charge of the electric vehicle set to 0.35 and the initial state of charge of the energy storage device set to 0.2. Input the model state space data at each scheduling moment into the neural network to obtain the action value function of each executable action in the state space; select the optimal action by horizontally comparing the action value function of each action and convert it into actual instructions, thereby controlling the charging and discharging operations of the energy storage device and the electric vehicle.

[0107] In order to achieve better training results, this example sets several different random number seeds, compares the model training effects under each random number seed, and obtains the optimal solution. Figure 5 This is the final reward change curve. Figure 7 This is a graph showing the change in the loss value of the neural network. It can be seen that both the convergence speed and the final effect are better than Figure 6 The training results of the deep Q network are shown in Figure 2. Using the trained intelligent agent to manage the energy of the microgrid can obtain better charging and discharging plans for energy storage equipment and electric vehicles, so that they can give full play to their positive role of peak shaving and valley filling on the basis of normal operation, effectively improve the utilization rate of energy, reduce the amount of electricity purchased from the large power grid, and thus improve economic benefits. Figure 8 The charge state change diagram of the electric vehicle and energy storage device when executing optimal control.

[0108] In summary, as a new type of learning-based model-free scheduling strategy generation method, the present invention uses the load demand of the microgrid for 24 hours a day, photovoltaic and wind power output, and the charge state prediction data of electric vehicles and energy storage equipment as the state space output of the Rainbow deep Q network, and learns and executes the charging and discharging actions of electric vehicles and energy storage equipment through intelligent agents, thereby achieving the purpose of minimizing the daily operation cost of the microgrid under the premise of ensuring power stability and equipment life and safety, and improving the energy utilization rate. This method can solve the difficulty of modeling the microgrid energy management model caused by the randomness and intermittency of renewable energy, and has fast convergence after training. It also has excellent generalization ability when dealing with new microgrid energy management models, so that it can effectively solve the defects of low energy utilization and high operating cost of microgrids caused by the uncertainty of renewable energy, energy flow and the diversity of loads.

[0109] The above is only a specific embodiment of the present invention, but the protection scope of the invention is not limited thereto. Those skilled in the art should understand that the present invention includes but is not limited to the contents described in the above specific embodiments. Any modification that does not deviate from the functional and structural principles of the present invention is included in the scope of the claims.

Claims

1. A microgrid energy management method based on Rainbow deep Q network, characterized in that: The steps include: Step (1): Establish an energy management simulation model corresponding to the residential microgrid. In the day-ahead scheduling stage, obtain the photovoltaic and wind power generation forecast data for the next day, the real-time load demand forecast data of the microgrid, and the real-time electricity price forecast data of the large power grid, and use them as the training data set of the model; Step (2): Define the training environment required for the microgrid energy management simulation model determined by Markov decision in reinforcement learning, including the following sub-steps: (2.1) Define the environmental state space, dispatch the load demand from 0:00 to 23:00 on the day, the output power of new energy, the real-time charge state of electric vehicles and the real-time charge state of energy storage equipment, which constitute the environmental state space S(t) required for this energy management simulation model; (2.2) defining the agent action space: controlling the electric vehicle and the energy storage device by scheduling the agent to execute the charging and discharging actions; (2.3) defining a reward function, wherein the reward function guides the agent to achieve a predetermined microgrid energy management optimization; (2.4) Setting a microgrid backup safety controller, wherein the microgrid backup safety controller is used to control the intelligent agent to perform overcharging or over-discharging erroneous operations; (2.5) Executing actual control instructions: The agent executes the actions learned from the Rainbow deep Q network to perform charging and discharging operations; (2.6) Interaction with the main grid: Connect with the main grid through the public connection point. When there is excess power after the output of new energy can meet the load demand of the microgrid, the remaining power will be fed back to the main grid. When the output of new energy, energy storage equipment and electric vehicles cannot meet the load demand of the microgrid at the maximum power output, electricity will be purchased from the main grid at the real-time electricity price, thereby achieving real-time power balance of the entire microgrid system. Step (3): Use the Rainbow deep Q network to train the microgrid energy management model defined in step (2), divide 0 to 23 o'clock into 24 scheduling moments, each hour as a step, and one scene as 24 hours; use the real-time prediction data of the day to repeatedly train multiple scenes until the final reward function converges; Step (3) includes the following sub-steps: (3.1) Construct the Rainbow deep Q network, including the following parts: I) Construct a neural network with two fully connected layers and one hidden layer, and add a Gaussian distributed noise term to the weights of all fully connected layers to interfere with the original ε-greedy random-greedy exploration method to improve the exploration rationality of the algorithm; II) Add a competitive network before the output layer Q network to split the output Q function into the average of all Q values ​​under the value function V state and the sum of the advantage function A that is restricted to an average of 0; the specific formula is as follows: Among them, α and β are the parameters of the value function network layer and the advantage function network layer respectively, and 1 / |A| is the average value of the advantage function; III) Build two Q networks as the final output layer of the neural network to decouple action selection and value estimation. One network is used to evaluate all actions in the state, and the other is used to select specific actions to improve the over-estimation error caused by bootstrapping. IV) Use the multi-step learning method to update the Q network to accelerate the convergence of the Q network. The formula is as follows: Where n is the updated stride length, w is the neural network parameter, and γ is the discount rate; (3.2) Prioritized experience replay pool: Set up an experience pool with a fixed capacity and replay each set of data (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool, and according to the size of the error caused by the previous training, the priority is arranged by formula to evaluate the order and probability of each set of data being re-sent to the neural network for training, in order to increase the utilization of data and reduce the correlation of adjacent data. The specific sampling formula is shown below: Among them, X i is the error value caused by this set of experience in the previous training, and ε is to prevent p i The noise is 0, and z is the annealing factor, which is used to adjust the priority; Step (4): Apply the intelligent agent trained in step (3) to the microgrid energy management of the day. When training each scene, input the model state space data at each scheduling moment into the neural network to obtain the action value function of each executable action in the state space; select the optimal action by horizontally comparing the action value function of each action and convert it into actual instructions, thereby controlling the charging and discharging operations of the energy storage equipment and electric vehicles.

2. The microgrid energy management method based on Rainbow deep Q network according to claim 1 is characterized in that: Electric vehicles of residential users participate in microgrid energy management. They can act as energy storage devices to output power to reduce the amount of electricity purchased from the power grid when the load demand is large, and act as interruptible loads when the load demand is small.

3. The microgrid energy management method based on Rainbow deep Q network according to claim 1 is characterized in that: Wind power and photovoltaic power generation constitute the renewable energy output of the microgrid system; residential users are connected to the microgrid to supply their electricity; the microgrid system is equipped with energy storage equipment; the microgrid is connected to the large power grid through a public connection point to exchange electricity and obtain real-time electricity prices.

4. The microgrid energy management method based on Rainbow deep Q network according to claim 1 is characterized in that: The step (1) comprises the following steps: I) Optimize the daily operating cost, which includes three parts: first, the grid interaction cost, which is divided into electricity sales revenue and electricity purchase cost; second, the energy storage equipment operating cost; third, the electric vehicle operating cost. The specific optimization target is the following formula: Among them, T is the total dispatch period; E b,t is the unit electricity purchase price at time t; E s,t is the unit electricity price at time t; X t is the amount of electricity exchanged with the power grid at time t; C b,t is the operating cost of the energy storage system at time t; C v,t is the operating cost of the electric vehicle at time t; II) Balance the power of each participating part with the power of the large power grid. The specific power balance formula is as follows: P x,t -P l,t +P n,t -P b,t -P v,t =0 (2) Among them, P x,t P is the power exchanged from the large power grid at dispatch time t. Positive numbers represent power purchases and negative numbers represent power sales. l,t is the real-time load demand of the microgrid system at time t; P n,t is the wind power and photovoltaic power output at time t; P b,t is the operating power of the energy storage device at time t, a positive number represents the power in the charging state, and a negative number represents the power in the discharging state; P v,t is the real-time power of the electric vehicle at time t, where positive numbers represent the power in the charging state and negative numbers represent the power in the discharging state; III) The energy management system optimizes the daily minimum operating cost while reducing the peak-to-valley difference caused by load and production imbalance, and ensures that each device operates in a normal working environment. The formula is as follows: Charging power P of energy storage device bc,t , discharge power P bd,t The maximum power allowed for normal operation shall not exceed that of the electric vehicle. vc,t , discharge power P vd,t Nor shall it exceed the maximum power allowed for its normal operation.

5. The microgrid energy management method based on Rainbow deep Q network according to claim 1 is characterized in that: In step (2.1), before each scheduling moment, the agent obtains the wind power and photovoltaic power output P at time t from the environment n,t and the real-time load demand P of the microgrid system at time t l,t , as the external state information of the energy management model; according to the action selected by the agent at the last scheduling moment, change the charge state B of the electric vehicle at that moment v,t and the charge state B of the energy storage device at that moment b,t , these two are the controllable state information of the energy management model; these four state information form a state space, which is sent to the neural network to train the optimal action and select it; In step (2.2), the actions used to optimize the microgrid energy management model include the real-time power P of the electric vehicle at each scheduling time t v,t and the operating power P of the energy storage device at time t b,t , and combined into N according to the charge and discharge gears of each battery v *N b The actions that can be selected by the agent at each scheduling moment constitute the action space; where Nv is the number of charging and discharging gears of the electric vehicle, and Nb is the number of charging and discharging gears of the energy storage device; In step (2.3), the agent selects actions through a reward function; the reward function is divided into three parts: electricity purchase cost and electricity sales reward, energy storage system operation cost and violation penalty, electric vehicle operation compensation and violation penalty; the function is as follows: Among them, C e,t is the revenue or cost of interacting with the power grid at dispatch time t, E s,t is the unit electricity price at time t, E b,t is the unit electricity purchase price at time t, X t is the amount of electricity interacting with the grid at time t; V b,t represents the total electricity cost of the energy storage device at time t, δ1 is the electricity coefficient; D b,t represents the overcharge or over-discharge penalty of the energy storage device at time t, and δ2 is D b,t The penalty coefficient, d p1,t is the amount of energy storage system overcharged or over-discharged at time t; similarly, V v,t represents the operating compensation cost of the electric vehicle at time t, which is used to compensate the user for the loss of battery loss caused by the participation of electric vehicles in scheduling. δ3 is the compensation coefficient; D v,t represents the overcharge or over-discharge penalty of the electric vehicle at time t, and δ4 is D v,t The penalty coefficient, d p2,t C is the amount of overcharge or overdischarge of the electric vehicle at time t; b,t is the operating cost of the energy storage system at time t, C v,t is the operating cost of the electric vehicle at time t, R t is the reward function.

Images