New energy microgrid real-time energy management method based on near-end strategy optimization and Lightgbm

Abstract

The invention relates to a new energy microgrid real-time energy management method based on near-end strategy optimization and Lightgbm. The new energy microgrid real-time energy management method comprises the following steps: S1, forming a training sample library of a Lightgbm model; s2, training a Lightgbm model, and predicting the power shortage of the new energy microgrid; s3, building a reinforcement learning environment required for training the intelligent agent, enabling the intelligent agent to interact with the environment, and forming a training sample library of deep reinforcement learning; s4, improving the reward function designed in S3; s5, training the intelligent agent by using a near-end strategy optimization algorithm, and storing model parameters; and S6, completing real-time energy management of the new energy microgrid. According to the invention, the training efficiency and training effect of the model can be obviously improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of energy management of new energy microgrids, and relates to a real-time energy management method for new energy microgrids, in particular to a real-time energy management method for new energy microgrids based on proximal strategy optimization and Lightgbm. Background Art

[0002] By integrating a variety of renewable energy sources (such as solar energy, wind energy, etc.) and energy storage equipment, new energy microgrids reduce the power system's dependence on traditional fossil energy, reduce carbon emissions, and help achieve my country's "dual carbon" goals as soon as possible.

[0003] In the current deep reinforcement learning framework, using meteorological factors, historical output data of renewable energy (such as wind power and solar energy), and historical loads directly as state variables to train intelligent agents is a common and intuitive strategy in the field of energy management. However, a large number of input features will affect the convergence of the model, increase the difficulty of model training, and thus affect the training effect. In addition, although it is intuitive to directly minimize the operating cost of the microgrid, it is difficult to learn and is not conducive to the convergence of intelligent agent training.

[0004] After searching, no public documents of the prior art that are identical or similar to the present invention were found. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a new energy microgrid real-time energy management method based on proximal strategy optimization and Lightgbm, which significantly improves the training efficiency and training effect of the model.

[0006] The present invention solves the practical problem by adopting the following technical solutions:

[0007] A new energy microgrid real-time energy management method based on proximal strategy optimization and Lightgbm includes the following steps:

[0008] S1. Obtain historical data of new energy microgrids to form a training sample library for the Lightgbm model;

[0009] S2. Based on step 1, a training sample library of the Lightgbm model is formed, the Lightgbm model is trained, and an early stop strategy is introduced to predict the power shortage of the new energy microgrid;

[0010] S3. Describe the energy dispatch problem of the new energy microgrid as a Markov decision process (MDP), build a reinforcement learning environment for training the agent, define rounds, design reward functions, state variables, and action variables; let the agent interact with the environment, and represent the state vector S of the interaction process and the result. t , action vector A t And the reward Rt Store them in the experience replay pool to form a training sample library for deep reinforcement learning;

[0011] Among them, some elements of the state vector are calculated by the historical data in step S1 and the Lightgbm model in step S2;

[0012] S4: Improve the reward function designed in S3, perform densification, and distribute the monthly demand electricity fee to each hour; add penalties for abandoning solar and wind power to guide the agent to actively consume excess renewable energy power generation; add a comparison baseline to eliminate the impact of "background noise" irrelevant to action selection on the agent, and finally complete the construction of the reinforcement learning environment;

[0013] S5. Based on the reinforcement learning environment built in steps S3-S4, use the proximal strategy optimization algorithm to train the agent and save the model parameters;

[0014] S6. Input the real-time meteorological data, historical photovoltaic, wind power, and load data into the trained LightGBM model to predict the power shortage of the new energy microgrid in the next hour. Then input each state variable into the intelligent agent trained in step S5 to control and dispatch the charging and discharging power of the battery and hydrogen storage tank in the next hour, thereby completing the real-time energy management of the new energy microgrid.

[0015] Moreover, the specific method of step S1 is:

[0016] Obtain historical data of new energy microgrids, including historical meteorological data such as temperature, humidity, wind speed, wind direction, solar elevation angle, irradiation intensity, cloud cover, precipitation, historical output data of photovoltaic and wind power generation, and historical load data, to form a training sample library for the Lightgbm model.

[0017] Moreover, the specific method of step S2 is:

[0018] The historical 24-hour meteorological data, photovoltaic and wind power generation data, load data and the hourly changes of the above data of the new energy microgrid are used as input features to predict the power shortage of the microgrid in the next hour.

[0019] The change window size of the hourly change feature is 1 to 24 hours, that is, the difference between each original feature and the past 1 to 24 hours is calculated and added to the training sample as a new feature;

[0020] The model is built and trained based on the Lightgbm library in Python. The objective function is the mean square error (MSE), and its calculation formula is as follows:

[0021]

[0022] Where n is the total number of samples; y true The actual value of photovoltaic output; pred is the predicted value of photovoltaic output.

[0023] Moreover, the granularity of various historical data in steps S1 and S2 is 1 hour, that is, 24 points a day.

[0024] Moreover, the lightGBM model training in step S2 is implemented based on the lightgbm library in Python, and the objective function selects the mean square error MSE; the early stopping strategy in the training process specifically refers to: during training, the test set MSE of each training is first recorded. When the test set MSE does not improve for n consecutive times, the training is stopped, and the number of trainings with the best effect is recorded; then, the model is retrained n times to ensure that the model maintains good performance on the test set.

[0025] Moreover, the state variables of step S3 include: the storage capacity of the battery and the hydrogen storage tank, the power price of the power grid, the historical power shortage peak value and the microgrid power shortage predicted by step S2; the action variables include: the charging and discharging power of the battery and the hydrogen storage tank.

[0026] Moreover, the specific steps of step S3 include:

[0027] (1) First, the new energy microgrid transforms the state variable vector S at time t into t Feedback to the agent, define state variables in, D t+1 , They are the storage capacity of the battery and hydrogen storage tank at time t, the historical power shortage peak value, and the grid electricity price and power shortage forecast value at time t+1; It is predicted by the historical data in S1 and the Lightgbm model trained in S2;

[0028] (2) Then, the agent is represented by the state vector S t Calculate the action vector A t , define the action vector in, are the charging and discharging power of the battery and the hydrogen storage system at time t+1 respectively. Then, the environment is represented by the state vector S t , action vector A t Perform state transfer and calculate the state S at the next moment t+1 status, and calculate the reward R t ;

[0029] (3) Reward function R in reinforcement learning tSimilar to the objective function in the traditional optimization field, in the energy management strategy of the new energy microgrid, the negative value of the monthly operating cost is often used as the reward function, as shown in the following formula:

[0030]

[0031] in:

[0032] P t grid =P t load -P t pv -P t wind -P t battery -P t tank

[0033]

[0034] Where T is the total step length of a single round, i.e. 2160; D t is the grid electricity price at time t; E is the demand electricity price; P t grid is the power flowing from the main grid to the microgrid at time t; is the peak load of the microgrid from time 0 to time t, which is used to calculate the demand electricity fee; P t load , P t pv , P t wind , P t battery , P t tank They are load power, photovoltaic output, wind power output, battery charging power and hydrogen storage tank charging power at time t respectively;

[0035] (4) Then, the state transition process of the environment is defined. The state transition of the environment depends on the mathematical model of each component of the new energy microgrid;

[0036] The constraints of the new energy microgrid are the operation constraints of the energy storage equipment, including the energy storage capacity constraints and the charge and discharge rate constraints, as shown in the following formula:

[0037]

[0038] In the formula, are the minimum and maximum storage capacity of the battery and hydrogen storage system respectively; are the minimum and maximum charge and discharge power of the battery and hydrogen storage system, respectively, both of which are based on discharge as the reference direction, that is, greater than 0 during discharge; η battery , ηtank are the charging efficiencies of the battery and hydrogen storage system, respectively; It is a 0-1 variable, which represents the charging and discharging status of the battery and the hydrogen storage system respectively. It takes 1 when the energy storage device is charging and 0 when it is discharging.

[0039] When the environment is transitioning, The variables are determined by the above mathematical model, and the remaining variables are updated by reading historical data;

[0040] (5) Finally, the state vector S t , action vector A t And the reward R t The reward is stored in the experience replay pool; the above process is repeated until the round ends, thus forming a training sample library for deep reinforcement learning.

[0041] Moreover, the improved reward function in step S4 is specifically:

[0042]

[0043] in:

[0044]

[0045] Where p is the penalty coefficient for abandonment of light and wind; ε is the density coefficient. When ε is 1, the demand cost is completely amortized to each time step, but this amortization method has a certain calculation error. When ε is 0, it is equivalent to not densifying the reward function.

[0046] After improving the reward function, write the relevant programs based on the gym library of OpenAI as described in S3-S4, and finally complete the construction of the reinforcement learning environment.

[0047] Moreover, the specific method of step S5 is:

[0048] The training of the agent using the PPO algorithm is based on the Pytorch implementation in Python, and the environment design is based on the gym library of Openai. In order to enable the agent to consider long-term economic benefits, the length of a single round is one quarter, that is, 90 days, with a total of 2160 steps; the policy network in the PPO algorithm selects the MLP neural network, and the input state is normalized.

[0049] Advantages and beneficial effects of the present invention:

[0050] 1. The present invention proposes a real-time energy management method for a new energy microgrid based on proximal strategy optimization and Lightgbm. By introducing the LightGBM model, the power shortage of the new energy microgrid in the future period is predicted, and the originally huge historical data is refined into a predicted value of the power shortage, which is then introduced into the deep reinforcement learning model as a new state variable, thereby cleverly avoiding the "curse of dimensionality" problem and significantly improving the training efficiency and training effect of the model.

[0051] 2. The present invention improves the reward function, densifies the reward function and adds the penalty for abandoning solar power and wind power and the comparison baseline. The relevant technical measures proposed by the present invention reduce the difficulty of training the intelligent agent, improve the economic benefits of the new energy microgrid, provide new technical ideas for the energy management strategy of the new energy microgrid, and help promote the widespread application and sustainable development of my country's new energy microgrid.

[0052] 3. The present invention builds an energy management strategy for new energy microgrids based on proximal policy optimization in deep reinforcement learning. This strategy can dispatch the charging and discharging power of energy storage units in new energy microgrids in real time to absorb new energy and improve the operating economy of microgrids.

[0053] 4. In order to solve the "curse of dimensionality" problem, the present invention introduces the LightGBM model to predict the power shortage of the new energy microgrid in the future period, refines the originally huge historical data into the predicted value of the power shortage, and introduces it into the deep reinforcement learning model as a new state variable, which reduces the training difficulty of the intelligent agent, thereby improving the training effect of the intelligent agent and the operating economy of the new energy microgrid.

[0054] 5. Compared with the neural network, the lightgbm in the tree model of the present invention has the advantages of short training time, less data required, and not easy to overfit, which helps to improve the prediction accuracy of the power shortage of the new energy microgrid, and thus improve the training effect of the intelligent body.

[0055] 6. The present invention improves the reward function, reduces the difficulty of training the agent, and improves the economic benefits of the new energy microgrid. Specifically, the improvements include the following three aspects: ① Densify the reward function and allocate the monthly demand cost of the microgrid to each reinforcement learning round. ② Add penalties for abandoning solar and wind power to the reward function to artificially guide the agent to use energy storage equipment to store excess new energy power generation. ③ Add a comparison baseline to the reward function to reduce the impact of "background noise" unrelated to action selection on the agent. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a flow chart of the new energy microgrid real-time energy management method proposed by the present invention;

[0057] Figure 2 A schematic diagram of a new energy microgrid structure provided by an embodiment of the present invention;

[0058] Figure 3 A schematic diagram of the interaction between an intelligent agent and an environment provided by an embodiment of the present invention;

[0059] Figure 4 The power load, wind power generation and photovoltaic power generation curve diagram of the test centralized new energy microgrid provided in the embodiment of the present invention;

[0060] Figure 5 A curve chart of the power storage capacity and power shortage prediction of the energy storage system in the new energy microgrid after the intelligent agent participates in the decision-making provided in the embodiment of the present invention. DETAILED DESCRIPTION

[0061] The embodiments of the present invention are further described in detail below with reference to the accompanying drawings:

[0062] A new energy microgrid real-time energy management method based on proximal policy optimization and Lightgbm includes the following steps:

[0063] S1. Obtain historical data of new energy microgrids to form a training sample library for the Lightgbm model.

[0064] The specific method of step S1 is:

[0065] Obtain historical data of new energy microgrids, including historical meteorological data such as temperature, humidity, wind speed, wind direction, solar elevation angle, irradiation intensity, cloud cover, precipitation, historical output data of photovoltaic and wind power generation, and historical load data, to form a training sample library for the Lightgbm model.

[0066] S2. Based on step 1, a training sample library of the Lightgbm model is formed, the Lightgbm model is trained, and an early stop strategy is introduced to predict the power shortage of the new energy microgrid;

[0067] The specific method of step S2 is:

[0068] The historical 24-hour meteorological data, photovoltaic and wind power generation data, load data and the hourly changes of the above data of the new energy microgrid are used as input features to predict the power shortage of the microgrid in the next hour.

[0069] The change window size of the hourly change feature is 1 to 24 hours, that is, the difference between each original feature and the past 1 to 24 hours is calculated and added to the training sample as a new feature;

[0070] The model is built and trained based on the Python Lightgbm library. The objective function is the mean square error (MSE), and its calculation formula is as follows:

[0071]

[0072] Where n is the total number of samples; y true The actual value of photovoltaic output; pred is the predicted value of photovoltaic output.

[0073] In this embodiment, the granularity of various historical data in steps S1 and S2 is 1 hour, that is, 24 points a day.

[0074] In this embodiment, the lightGBM model training in step S2 is implemented based on the lightgbm library in Python, and the objective function is the mean square error (MSE); the early stopping strategy in the training process specifically refers to: first record the MSE of the test set for each training during training, and stop training when the MSE of the test set does not improve for n consecutive times, and record the number of trainings with the best effect. Then, retrain the model n times to keep the model's performance on the test set good and prevent overfitting.

[0075] S3, describe the energy dispatch problem of the new energy microgrid as a Markov decision process MDP, build the reinforcement learning environment required for training the intelligent agent, let the intelligent agent interact with the environment, define the round, design the reward function, state variables and action variables, and form a training sample library for deep reinforcement learning by the historical data of step S1 and the power shortage prediction value of the Lightgbm model in step S2;

[0076] The state variables of step S3 include: the storage capacity of the battery and the hydrogen storage tank, the power price of the power grid, the historical power shortage peak value and the microgrid power shortage predicted by step S2; the action variables include: the charging and discharging power of the battery and the hydrogen storage tank.

[0077] The specific steps of step S3 include:

[0078] (1) First, the new energy microgrid (same as the above environment) transforms the state variable vector S at time t into t Feedback to the agent, define state variables in, D t+1 , They are the storage capacity of the battery and hydrogen storage tank at time t, the historical power shortage peak, and the predicted values ​​of the grid electricity price and power shortage at time t+1. It is predicted by the historical data in S1 and the Lightgbm model trained in S2;

[0079] (2) Then, the agent is represented by the state vector S t Calculate the action vector A t , define the action vector in, are the charging and discharging power of the battery and the hydrogen storage system at time t+1 respectively. Then, the environment is represented by the state vector S t , action vector A t Perform state transfer and calculate the state S at the next moment t+1 status, and calculate the reward R t .

[0080] (3) Reward function R in reinforcement learning t Similar to the objective function in the traditional optimization field, in the energy management strategy of the new energy microgrid, the negative value of the monthly operating cost is often used as the reward function, as shown in the following formula:

[0081]

[0082] in:

[0083] P t grid =P t load -P t pv -P t wind -P t battery -P t tank

[0084]

[0085] Where T is the total step length of a single round, i.e. 2160; D t is the grid electricity price at time t; E is the demand electricity price; P t grid is the power flowing from the main grid to the microgrid at time t; is the peak load of the microgrid from time 0 to time t, which is used to calculate the demand electricity fee; P t load , P t pv , P t wind , P t battery , P t tank They are the load power, photovoltaic output, wind power output, battery charging power and hydrogen storage tank charging power at time t respectively.

[0086] (4) Then, define the state transition process of the environment. The state transition of the environment depends on the mathematical model of each component of the new energy microgrid. For the convenience of description, the present invention still uses the concept of constraint conditions in the traditional optimization model.

[0087] The constraints of the new energy microgrid are mainly the operation constraints of the energy storage equipment, including energy storage capacity constraints and charging and discharging rate constraints, as shown in the following formula.

[0088]

[0089] In the formula, are the minimum and maximum storage capacity of the battery and hydrogen storage system respectively; are the minimum and maximum charge and discharge power of the battery and hydrogen storage system, respectively, both of which are based on discharge as the reference direction, that is, greater than 0 during discharge; η battery , η tank are the charging efficiencies of the battery and hydrogen storage system, respectively; It is a 0-1 variable, which represents the charging and discharging status of the battery and the hydrogen storage system respectively. It takes 1 when the energy storage device is charging and 0 when it is discharging.

[0090] When the environment is transitioning, The variables are determined by the above mathematical model, and the remaining variables are updated by reading historical data.

[0091] (5) Finally, the state vector S t , action vector A t And the reward R t The reward is stored in the experience replay pool; the above process is repeated until the round ends, thus forming a training sample library for deep reinforcement learning.

[0092] In this embodiment, one round lasts for three months (90 days), each step is 1 hour, and there are 2160 steps in total. The decision network of the agent uses an MLP neural network, including an input layer, a hidden layer, and an output layer, and the activation function uses tanh.

[0093] S4: Improve the reward function designed in S3, perform densification, and allocate the monthly demand electricity fee to each hour; add penalties for abandoning solar and wind power to guide the agent to actively consume excess renewable energy power generation; add a comparison baseline to eliminate the impact of "background noise" irrelevant to action selection on the agent, and then complete the construction of the reinforcement learning environment;

[0094] The improved reward function in step S4 is specifically:

[0095]

[0096] in:

[0097]

[0098] Where p is the penalty coefficient for abandonment of light and wind; ε is the density coefficient. When ε is 1, the demand cost is completely amortized to each time step, but this amortization method has a certain calculation error. When ε is 0, it is equivalent to not densifying the reward function.

[0099] After improving the reward function, write the relevant programs based on the gym library of OpenAI as described in S3-S4 to complete the construction of the reinforcement learning environment.

[0100] S5. Based on the reinforcement learning environment built in steps S3-S4, use the Proximal Policy Optimization (PPO) algorithm to train the agent and save the model parameters.

[0101] The specific method of step S5 is:

[0102] The training of the agent using the PPO algorithm is based on the Pytorch implementation in Python, and the environment design is based on the gym library of Openai. In order to enable the agent to consider long-term economic benefits, the length of a single round is one quarter, that is, 90 days, with a total of 2160 steps; the policy network in the PPO algorithm selects the MLP neural network, and the input state is normalized.

[0103] S6. Input the real-time meteorological data, historical photovoltaic, wind power, and load data into the trained LightGBM model to predict the power shortage of the new energy microgrid in the next hour. Then input each state variable into the intelligent agent trained in step S5 to control and dispatch the charging and discharging power of the battery and hydrogen storage tank in the next hour, thereby completing the real-time energy management of the new energy microgrid.

[0104] The present invention is further described below in conjunction with specific examples:

[0105] like Figure 1-5 As shown, this embodiment uses the energy management strategy of the present invention to dispatch and control the energy storage charging and discharging working state of the microgrid in real time on the Rye data set of the pilot project studied by the European Union, including the following steps:

[0106] S1. Obtain historical data of new energy microgrids, including historical meteorological data such as temperature, humidity, wind speed, wind direction, solar elevation angle, irradiation intensity, cloud cover, precipitation, historical output data of photovoltaic and wind power generation, and historical load data, to form a training sample library for the Lightgbm model.

[0107] This example uses the Rye dataset from the EU pilot project. The microgrid structure is as follows: Figure 2As shown. The pilot project includes wind power generation, photovoltaic power generation system, lithium battery energy storage system and hydrogen storage system. The hydrogen storage system includes hydrogen storage tanks, electrolyzers and boilers. The microgrid is connected to the main grid through a distribution transformer. When there is surplus power generation, the microgrid can store energy through electrolysis of water hydrogen storage and battery charging. The two methods have different energy storage efficiency, energy storage capacity and charging and discharging power. When the power generation is insufficient, the microgrid can obtain electricity through energy storage equipment or from the main grid.

[0108] The microgrid equipment parameters are shown in Table 1.

[0109] Table 1 Microgrid equipment parameters

[0110]

[0111] The granularity of various historical data in this dataset is 1 hour, including photovoltaic and wind power output, power load, real-time electricity price and meteorological data from January 1, 2020 to March 8, 2021. Meteorological data include hourly accumulated rainfall, hourly accumulated clear sky flux, hourly accumulated sunshine duration, solar azimuth, solar elevation, and Celsius temperature, relative humidity, wind speed and wind direction at 2, 10, 50 and 100 meters above the ground.

[0112] Based on the original data, time, solar elevation angle, and wind direction angle are further modeled to construct the following features:

[0113]

[0114] Among them, hour is the hour of the day, θ pv ,θ wind are the sun elevation angle and wind direction angle respectively.

[0115] S2. Train the Lightgbm model and introduce the early stop strategy to predict the power shortage of the new energy microgrid. The model uses the weather forecast value at the time to be predicted, the historical 24-hour weather data, photovoltaic and wind power generation data, load data, and the hourly changes of the above data as input features to predict the power shortage of the microgrid in the next hour.

[0116] The change window size of the hourly change feature is 1 to 24 hours, that is, the difference between each original feature and the past 1 to 24 hours is calculated and added to the training sample as a new feature.

[0117] In this embodiment, the model is built and trained based on the Lightgbm library of Python, and the objective function is the mean square error (MSE), and its calculation formula is as follows:

[0118]

[0119] Where n is the total number of samples; y true The actual value of photovoltaic output; pred is the predicted value of photovoltaic output.

[0120] The main hyperparameter values ​​of the Lightgbm model are as follows: learning rate is 0.02, the maximum number of leaf nodes is 128, the feature resampling ratio is 0.8, the data subsampling ratio is 0.9, the bagging frequency is 4, and the number of early stopping rounds is 100.

[0121] S3, describe the energy dispatch problem of the new energy microgrid as a Markov decision process MDP, build the reinforcement learning environment required for training the intelligent agent, define the round, design the reward function, state variables and action variables, and form a training sample library for deep reinforcement learning from the historical data of step S1 and the power shortage prediction value of the Lightgbm model in step S2. The state variables include: the storage capacity of the battery and hydrogen storage tank, the power price of the grid, the historical power shortage peak, and the microgrid power shortage predicted by S2. The action variables include: the charging and discharging power of the battery and hydrogen storage tank.

[0122] In this embodiment, the process of interaction between the agent and the environment is as follows: Figure 3 shown.

[0123] First, the new energy microgrid (same as the above environment) transforms the state variable vector S at time t into t Feedback to the agent, state variables in, D t+1 are the storage capacity of the battery and hydrogen tank at time t, the historical power shortage peak, and the grid electricity price at time t+1. t Calculate the motion vector in, are the charging and discharging power of the battery and the hydrogen storage system at time t+1 respectively. Then, the environment is represented by the state vector S t , action vector A t Perform state transfer and calculate the state S at the next moment t+1 status, and calculate the reward R t Finally, the state vector S t , action vector A t And the reward R t The reward is deposited into the experience replay pool. The above process is repeated until the round ends. In this embodiment, one round lasts for three months (90 days), each step is 1 hour, and there are 2160 steps in total. The decision network of the agent uses an MLP neural network, which includes an input layer, a hidden layer, and an output layer, and the activation function uses tanh.

[0124] In addition, the state transition of the environment depends on the mathematical model of each component of the new energy microgrid. For the convenience of description, the concept of constraint conditions in the traditional optimization model is still used in this embodiment.

[0125] The constraints of the new energy microgrid are mainly the operation constraints of the energy storage equipment, including energy storage capacity constraints and charging and discharging rate constraints, as shown in the following formula.

[0126]

[0127] In the formula, are the minimum and maximum storage capacity of the battery and hydrogen storage system respectively; are the minimum and maximum charge and discharge power of the battery and hydrogen storage system, respectively, both of which are based on discharge as the reference direction, that is, greater than 0 during discharge; η battery , η tank are the charging efficiencies of the battery and hydrogen storage system, respectively; It is a 0-1 variable, which represents the charging and discharging status of the battery and the hydrogen storage system respectively. It takes 1 when the energy storage device is charging and 0 when it is discharging.

[0128] When the environment is transitioning, The variables are determined by the above mathematical model, and the remaining variables are updated by reading historical data.

[0129] S4. Improve the reward function and perform densification to distribute the monthly demand electricity fee to each hour. Add penalties for abandoned solar and wind power to guide the agent to actively consume excess renewable energy generation. Add a comparison baseline to eliminate the impact of "background noise" irrelevant to action selection on the agent.

[0130] After improvement, the reward function adopted by the present invention is as follows:

[0131]

[0132] in:

[0133]

[0134] P t grid =P t load -P t pv -P t wind -P t battery -P t tank

[0135]

[0136] Where T is the total step length of a single round, i.e. 2160; D t is the grid electricity price at time t; p is the penalty coefficient for abandoned solar and wind; E is the demand electricity price; P t grid is the power flowing from the main grid to the microgrid at time t; ε is the density coefficient. When ε is 1, the demand cost is completely amortized to each time step, but this amortization method has a certain calculation error. When ε is 0, it is equivalent to not densifying the reward function. is the peak load of the microgrid from time 0 to time t, which is used to calculate the demand electricity fee; P t load , P t pv , P t wind , P t battery , P t tank They are the load power, photovoltaic output, wind power output, battery charging power and hydrogen storage tank charging power at time t respectively.

[0137] The relevant parameters in the environment and reward function in this embodiment are shown in Table 2.

[0138] Table 2 Parameters related to the embodiment environment and reward function

[0139]

[0140] S5. Based on the established reinforcement learning environment, use the proximal policy optimization algorithm to train the agent and save the model parameters.

[0141] The Proximal Policy Optimization (PPO) algorithm is a deep reinforcement learning algorithm that aims to solve policy optimization problems in continuous action spaces and complex environments. The PPO algorithm has the advantages of high training efficiency, strong training stability, and wide applicability. It is widely used in many fields such as robot learning, game AI, finance, and power systems. PPO combines the efficiency of the policy gradient method with the stability of the trust region method, and avoids instability and performance degradation during training by limiting the difference between the new and old policies (i.e., the clipping mechanism). The clipping mechanism avoids excessive policy updates by limiting the difference between the new and old policies. Specifically, if the ratio of the probability of selecting the same action between the new and old policies exceeds a certain threshold (such as 1.2 or 0.8), it is "clipped" to this threshold, thereby limiting the update amplitude of the policy network and maintaining the stability of training.

[0142] In this embodiment, the training parameters of the PPO algorithm are shown in Table 3.

[0143] Table 3 PPO algorithm training parameters

[0144]

[0145] S6. Input the real-time meteorological data, historical photovoltaic, wind power, and load data into the trained LightGBM model to predict the power shortage of the new energy microgrid in the next hour. Then input each state variable into the trained intelligent agent to control and schedule the charging and discharging power of the battery and hydrogen storage tank in the next hour.

[0146] This example selects December 1, 2020 to March 1, 2021 as the test set to test the economic benefits of the intelligent agent's participation in the energy management of the new energy microgrid. The power load, wind power generation, and photovoltaic power generation curves of the new energy microgrid in the test set are as follows: Figure 4 As shown. Figure 4 It can be seen that the power load, wind power generation and photovoltaic power generation in this embodiment have strong volatility, which will bring great difficulties to the economic operation of the new energy microgrid when taking into account the demand electricity fee.

[0147] After the intelligent agent participates in the decision-making, the predicted values ​​of the battery and hydrogen storage system and the power shortage in the new energy microgrid are as follows: Figure 5 shown.

[0148] After training, the intelligent agent can perceive the operation status of the microgrid in real time and make effect analysis:

[0149] 1. Analysis of Lightgbm's power shortage prediction effect

[0150] The present invention uses the Lightgbm model to predict the power shortage of the microgrid in the future period, reduces the dimension of the original state space, reduces the number of parameters of the decision network, avoids the "curse of dimensionality", and speeds up the training speed. The improvement effect is shown in Table 4.

[0151] Table 4 Comparison of Lightgbm's improvement effect on simplified problems

[0152]

[0153] In addition, the power deficit of the new energy microgrid is directly predicted by the present invention, which is more accurate than predicting the load and new energy power generation separately and then calculating the power deficit. The accuracy comparison between indirect prediction and direct prediction is shown in Table 5. It can be seen that due to the certain correlation between power load, wind power generation and photovoltaic power generation, direct prediction of power deficit is more advantageous than indirect prediction, and the RMSE and accuracy rate are increased by 4.21kW and 3.40% respectively.

[0154] Table 5 Comparison of the accuracy of indirect prediction and direct prediction

[0155]

[0156]

[0157] 2. Analysis of the effect of reward function improvement

[0158] In order to verify the improvement effect of the present invention on the reward function, the original reward function, the reward function with the comparison baseline, the reward function with the abandoned light penalty, the reward function after the reward dense processing, and the improved reward function are used to train the agent. The comparison of the monthly operation cost of the microgrid under different reward functions is shown in Table 6. Compared with the original reward function, the improved reward function of the present invention reduces the monthly operation cost by 518 NOK, which is only 86.80% of the original reward function, and significantly improves the operation economy of the new energy microgrid.

[0159] Table 6 Comparison of monthly operating costs of microgrids under different reward functions

[0160]

[0161] The working principle of the present invention is:

[0162] A real-time energy management strategy for new energy microgrids based on proximal policy optimization and Lightgbm. First, historical data of new energy microgrids are obtained, including historical meteorological data such as temperature, humidity, wind speed, wind direction, solar elevation angle, irradiation intensity, cloud cover, precipitation, historical output data of photovoltaic and wind power generation, and historical load data, to form a training sample library of the Lightgbm model. Then, the Lightgbm model is trained, and the early stopping strategy is introduced to predict the power shortage of the new energy microgrid. The model uses the historical 24-hour meteorological data, photovoltaic, wind power generation data, load data and the hourly change of the above data of the new energy microgrid as input features to predict the power shortage of the microgrid in the next hour. In the present invention, the Lightgbm model plays the role of refining historical features in the energy management strategy, which helps to solve the "curse of dimensionality" problem in deep reinforcement learning. Compared with predicting photovoltaic, wind power output and power load separately and then calculating the power shortage, directly predicting the power shortage helps to improve the prediction accuracy of the power difference. Subsequently, the reinforcement learning environment required for training the intelligent agent is built, the rounds are defined, the reward function, state variables and action variables are designed, and the training sample library for deep reinforcement learning is formed by historical data and the power shortage prediction value of the Lightgbm model. The state variables include: the storage capacity of batteries and hydrogen storage tanks, the power price of the power grid, the historical power shortage peak value, and the predicted value of the microgrid power shortage output by Lightgbm. The action variables include: the charging and discharging power of batteries and hydrogen storage tanks. The present invention aims to minimize the operating cost of the microgrid, densifies the reward function, and adds penalties for abandoning light and wind and comparison baselines, which reduces the difficulty of training the intelligent agent and improves the economic benefits of the new energy microgrid. Based on the built reinforcement learning environment, the proximal policy optimization algorithm (Proximal Policy Optimization) is used to train the intelligent agent and save the model parameters. Finally, the real-time meteorological data, historical photovoltaic, wind power, and load data are input into the trained LightGBM model to predict the power shortage of the new energy microgrid in the next hour. Then, each state variable is input into the trained intelligent agent to control and schedule the charging and discharging power of the battery and hydrogen storage tank in the next hour.

[0163] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0164] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0165] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0166] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

Claims

1. A new energy microgrid real-time energy management method based on proximal strategy optimization and Lightgbm, characterized by: The steps include: S1. Obtain historical data of new energy microgrids to form a training sample library for the Lightgbm model; S2. Based on step 1, a training sample library of the Lightgbm model is formed, the Lightgbm model is trained, and an early stopping strategy is introduced to predict the power shortage of the new energy microgrid; S3. Describe the energy dispatch problem of the new energy microgrid as a Markov decision process (MDP), build a reinforcement learning environment for training the agent, define rounds, design reward functions, state variables, and action variables; let the agent interact with the environment, and represent the state vector S of the interaction process and the result. t , action vector A t And the reward R t Store them in the experience replay pool to form a training sample library for deep reinforcement learning; Among them, some elements of the state vector are calculated by the historical data in step S1 and the Lightgbm model in step S2; S4: Improve the reward function designed in S3, perform densification, and distribute the monthly demand electricity fee to each hour; add penalties for abandoning solar and wind power to guide the agent to actively consume excess renewable energy power generation; add a comparison baseline to eliminate the impact of "background noise" irrelevant to action selection on the agent, and finally complete the construction of the reinforcement learning environment; S5. Based on the reinforcement learning environment built in steps S3-S4, use the proximal strategy optimization algorithm to train the agent and save the model parameters; S6. Input the real-time meteorological data, historical photovoltaic, wind power, and load data into the trained LightGBM model to predict the power shortage of the new energy microgrid in the next hour. Then input each state variable into the intelligent agent trained in step S5 to control and dispatch the charging and discharging power of the battery and hydrogen storage tank in the next hour, thereby completing the real-time energy management of the new energy microgrid.

2. According to claim 1, a new energy microgrid real-time energy management method based on proximal strategy optimization and Lightgbm is characterized by: The specific method of step S1 is: Obtain historical data of new energy microgrids, including historical meteorological data such as temperature, humidity, wind speed, wind direction, solar elevation angle, irradiation intensity, cloud cover, precipitation, historical output data of photovoltaic and wind power generation, and historical load data, to form a training sample library for the Lightgbm model.

3. The real-time energy management method for new energy microgrid based on proximal strategy optimization and Lightgbm according to claim 1 is characterized by: The specific method of step S2 is: The historical 24-hour meteorological data, photovoltaic and wind power generation data, load data and the hourly changes of the above data of the new energy microgrid are used as input features to predict the power shortage of the microgrid in the next hour. The change window size of the hourly change feature is 1 to 24 hours, that is, the difference between each original feature and the past 1 to 24 hours is calculated and added to the training sample as a new feature; The model is built and trained based on the Lightgbm library in Python. The objective function is the mean square error (MSE), and its calculation formula is as follows: Where n is the total number of samples; y true The actual value of photovoltaic output; pred is the predicted value of photovoltaic output.

4. The real-time energy management method for new energy microgrid based on proximal strategy optimization and Lightgbm according to claim 1 is characterized by: The lightGBM model training in step S2 is implemented based on the lightgbm library in Python, and the objective function selects the mean square error MSE; the early stopping strategy in the training process specifically refers to: during training, the test set MSE of each training is first recorded, and when the test set MSE does not increase for n consecutive times, the training is stopped, and the number of trainings with the best effect is recorded; Then, the model is retrained n times to keep the model performing well on the test set.

5. The real-time energy management method for new energy microgrid based on proximal strategy optimization and Lightgbm according to claim 1 is characterized by: The state variables of step S3 include: the storage capacity of the battery and the hydrogen storage tank, the power price of the power grid, the historical power shortage peak value and the microgrid power shortage predicted by step S2; the action variables include: the charging and discharging power of the battery and the hydrogen storage tank.

6. The real-time energy management method for new energy microgrid based on proximal strategy optimization and Lightgbm according to claim 1 is characterized by: The specific steps of step S3 include: (1) First, the new energy microgrid transforms the state variable vector S at time t into t Feedback to the agent, define state variables in, D t+1 , They are the storage capacity of the battery and hydrogen storage tank at time t, the historical power shortage peak value, and the grid electricity price and power shortage forecast value at time t+1; It is predicted by the historical data in S1 and the Lightgbm model trained in S2; (2) Then, the agent is represented by the state vector S t Calculate the action vector A t , define the action vector in, are the charging and discharging power of the battery and the hydrogen storage system at time t+1 respectively; then, the environment is represented by the state vector S t , action vector A t Perform state transfer and calculate the state S at the next moment t+1 status, and calculate the reward R t ; (3) Reward function R in reinforcement learning t Similar to the objective function in the traditional optimization field, in the energy management strategy of the new energy microgrid, the negative value of the monthly operating cost is often used as the reward function, as shown in the following formula: in: Where T is the total step length of a single round, i.e. 2160; D t is the grid electricity price at time t; E is the demand electricity price; is the power flowing from the main grid to the microgrid at time t; is the peak load of the microgrid from time 0 to time t, which is used to calculate the demand electricity fee; They are load power, photovoltaic output, wind power output, battery charging power and hydrogen storage tank charging power at time t respectively; (4) Then, the state transition process of the environment is defined. The state transition of the environment depends on the mathematical model of each component of the new energy microgrid; The constraints of the new energy microgrid are the operation constraints of the energy storage equipment, including the energy storage capacity constraints and the charge and discharge rate constraints, as shown in the following formula: In the formula, are the minimum and maximum storage capacity of the battery and hydrogen storage system respectively; are the minimum and maximum charge and discharge power of the battery and hydrogen storage system, respectively, both of which are based on discharge as the reference direction, that is, greater than 0 during discharge; η battery , η tank are the charging efficiencies of the battery and hydrogen storage system, respectively; is a 0-1 variable, representing the charging and discharging status of the battery and hydrogen storage system, respectively. It takes 1 when the energy storage device is charging and 0 when it is discharging; When the environment is transitioning, The variables are determined by the above mathematical model, and the remaining variables are updated by reading historical data; (5) Finally, the state vector S t , action vector A t And the reward R t The reward is stored in the experience replay pool; the above process is repeated until the round ends, thus forming a training sample library for deep reinforcement learning.

7. The real-time energy management method for new energy microgrid based on proximal strategy optimization and Lightgbm according to claim 1 is characterized by: The improved reward function in step S4 is specifically: in: Where p is the penalty coefficient for abandonment of light and wind; ε is the density coefficient. When ε is 1, the demand cost is completely amortized to each time step, but this amortization method has a certain calculation error. When ε is 0, it is equivalent to not densifying the reward function. After improving the reward function, write the relevant programs based on the gym library of OpenAI as described in S3-S4, and finally complete the construction of the reinforcement learning environment.

8. The real-time energy management method for new energy microgrid based on proximal strategy optimization and Lightgbm according to claim 1 is characterized by: The specific method of step S5 is: The training of the agent using the PPO algorithm is based on the Pytorch implementation in Python, and the environment design is based on the gym library of Openai. In order to enable the agent to consider long-term economic benefits, the length of a single round is one quarter, that is, 90 days, with a total of 2160 steps; the policy network in the PPO algorithm selects the MLP neural network, and the input state is normalized.

Images