Real-time energy management method for new energy microgrid based on proximal policy optimization and lightgbm

Abstract

The present application relates to a kind of new energy microgrid real-time energy management method based on proximal policy optimization and Lightgbm, comprising the following steps: S1, the training sample library of Lightgbm model is formed;S2, train Lightgbm model, the power shortage of new energy microgrid is predicted;S3, the reinforcement learning environment needed for training agent is built, agent and environment are interacted, and the training sample library of deep reinforcement learning is formed;S4, the reward function designed in S3 is improved;S5, using proximal policy optimization algorithm trains agent, and saves model parameter;S6, complete new energy microgrid real-time energy management.The present application can significantly improve the training efficiency and training effect of model.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of new energy micro-grid energy management, and relates to a new energy micro-grid real-time energy management method, in particular to a new energy micro-grid real-time energy management method based on near-end strategy optimization and Lightgbm. BACKGROUND

[0002] The new energy micro-grid reduces the dependence of the power system on traditional fossil energy by integrating various renewable energy (such as solar energy and wind energy) and energy storage devices, reduces carbon emissions, and helps China achieve the "double carbon" goal as soon as possible.

[0003] In the current deep reinforcement learning framework, meteorological factors, historical output data of new energy (such as wind power and solar energy), and historical load are directly used as state variables to train the agent, which is a common and intuitive strategy in the field of energy management. However, the large number of input features can affect the convergence of the model, increase the difficulty of model training, and thus affect the training effect. In addition, directly taking the minimum micro-grid operating cost as the target is intuitive but difficult to learn, which is not conducive to the convergence of agent training.

[0004] After searching, no existing technical disclosure documents similar to the present application have been found. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings of the prior art and provide a new energy micro-grid real-time energy management method based on near-end strategy optimization and Lightgbm, which significantly improves the training efficiency and training effect of the model.

[0006] The present application solves its practical problems by adopting the following technical solutions:

[0007] A new energy micro-grid real-time energy management method based on near-end strategy optimization and Lightgbm, comprising the following steps:

[0008] S1, obtaining historical data of the new energy micro-grid to form a training sample library of the Lightgbm model;

[0009] S2, based on the training sample library of the Lightgbm model formed in step 1, training the Lightgbm model, and introducing an early stopping strategy to predict the power shortage of the new energy micro-grid;

[0010] S3, describing the energy scheduling problem of the new energy micro-grid as a Markov decision process (MDP), building a reinforcement learning environment required for training the agent, defining the round, designing the reward function, state variable and action variable; let the agent interact with the environment, and the state vector S t , action vector A t and reward Rt The experience replay pool is stored, and a training sample library of deep reinforcement learning is formed;

[0011] Part of the elements of the state vector are calculated by the historical data in step S1 and the Lightgbm model in step S2;

[0012] S4, the reward function designed in S3 is improved, and densification processing is performed, the monthly demand electricity charge is allocated to each hour, the light and wind abandonment penalty is added to guide the intelligent agent to actively consume the excess new energy generation, the baseline is added to eliminate the "background noise" irrelevant to the action selection of the intelligent agent, and finally the reinforcement learning environment is built;

[0013] S5, based on the reinforcement learning environment built in steps S3-S4, the intelligent agent is trained using the proximal policy optimization algorithm, and the model parameters are saved;

[0014] S6, real-time weather data, historical photovoltaic, wind power, and load data are input into the trained lightgbm model to predict the new energy microgrid power shortage in the next hour, and each state variable is input into the intelligent agent trained in step S5 to control and schedule the charging and discharging power of the battery and hydrogen storage tank in the next hour, and thus the real-time energy management of the new energy microgrid is completed.

[0015] Moreover, the specific method of step S1 is:

[0016] The historical data of the new energy microgrid are obtained, including historical weather data such as temperature, humidity, wind speed, wind direction, solar elevation angle, irradiance, cloud cover, and precipitation, historical output data of photovoltaic and wind power generation, and historical load data, forming a training sample library of the Lightgbm model.

[0017] Moreover, the specific method of step S2 is:

[0018] The historical 24-hour weather data, photovoltaic, wind power generation data, and load data of the new energy microgrid, as well as the hourly change amount of the above data, are input as input features to predict the microgrid power shortage in the next hour;

[0019] The change window size of the hourly change amount feature is 1 to 24 hours, that is, the difference with the past 1 to 24 hours is calculated for each original feature, and the new feature is added to the training sample;

[0020] The Lightgbm library based on Python is used to complete the model building and training, and the mean square error MSE is selected as the objective function, and the calculation formula is as follows:

[0021]

[0022] where n is the total number of samples; y true actual value of photovoltaic output; y pred is the predicted value of photovoltaic output.

[0023] Moreover, the granularity of each type of historical data in steps S1 and S2 is 1h, i.e., 24 points in a day.

[0024] Moreover, the lightGBM model training in step S2 is implemented based on the lightgbm library in Python, and the objective function is selected as mean square error (MSE); the early stopping strategy in the training process specifically refers to: first record the test set MSE of each training during training, when the test set MSE does not improve for n consecutive times, stop training, and record the best training time; then, retrain the model for n times to keep good performance of the model on the test set.

[0025] Moreover, the state variables in step S3 include: storage capacity of the battery and hydrogen storage tank, power price of the power grid, historical power shortage peak value, and microgrid power shortage predicted in step S2; the action variables include: charging and discharging power of the battery and hydrogen storage tank.

[0026] Moreover, the specific steps of step S3 include:

[0027] (1) First, the new energy microgrid feeds back the state variable vector S t to the agent at time t, and defines the state variable wherein, D t+1 , are the storage capacity of the battery and hydrogen storage tank at time t, the historical power shortage peak value, and the predicted values of the power price and power shortage at time t+1, respectively; obtained by the historical data in step S1 and the Lightgbm model trained in step S2;

[0028] (2) Then, the agent calculates the action vector A t from the state vector S t , and defines the action vector wherein, are the charging and discharging power of the battery and hydrogen storage system at time t+1, respectively. Then, the environment performs state transition from the state vector S t and the action vector A t , calculates the next state S t+1 , and calculates the reward R t .

[0029] (3) The reward function R tSimilar to the objective function in traditional optimization field, the negative value of operation cost is often used as the reward function in energy management strategy of new energy microgrid, as shown in the following formula:

[0030]

[0031] Wherein:

[0032] P t grid P t load P t pv P t wind P t battery P t tank

[0033]

[0034] In the formula, T is the total step length of a round, i.e. 2160; D t is the power price at time t; E is the demand price; P t grid is the power flowing from the main grid to the microgrid at time t; is the peak load of the microgrid from 0 to t, used to calculate the demand charge; P t load , P t pv , P t wind , P t battery , P t tank are the load power, photovoltaic output, wind power output, battery charging power and charging power of the hydrogen storage tank at time t, respectively;

[0035] (4) Then, the state transition process of the environment is defined, and the state transition of the environment depends on the mathematical model of each element of the new energy microgrid;

[0036] The constraint condition of the new energy microgrid is the operation constraint of the energy storage device, including the energy storage capacity constraint and the charging and discharging rate constraint, as shown in the following formula:

[0037]

[0038] In the formula, are the minimum and maximum storage power of the battery and the hydrogen storage system, respectively; are the minimum and maximum charging and discharging power of the battery and the hydrogen storage system, respectively, and are taken as the reference direction of discharging, i.e. greater than 0 when discharging; η battery , ηtank charging efficiency of the battery and the hydrogen storage system, respectively; is a 0-1 variable, representing the charging and discharging state of the battery and the hydrogen storage system, respectively, taking 1 when the energy storage device is charging and 0 when discharging.

[0039] when the environment performs state transition, The remaining variables are updated by reading historical data according to the calculation of the above mathematical model;

[0040] (5) Finally, the state vector S t , the action vector A t and the reward R t are stored in the experience replay pool; the above process is repeated until the round is over, thereby forming a training sample library for deep reinforcement learning.

[0041] Moreover, the improved reward function in step S4 is specifically:

[0042]

[0043] wherein:

[0044]

[0045] In the formula, p is the light abandonment and wind penalty coefficient; epsilon is the density coefficient, when epsilon is 1, the demand cost is completely allocated to each time step, but this allocation method has certain calculation error, when epsilon is 0, it is equivalent to not carrying out the densification treatment on the reward function;

[0046] After improving the reward function, based on the gym library of openai, the related program is written according to S3-S4, and then the reinforcement learning environment is finally completed.

[0047] Moreover, the specific method of step S5 is:

[0048] The training of the intelligent agent using the PPO algorithm is realized based on Pytorch in Python, and the design of the environment is realized based on the gym library of Openai; in order to enable the intelligent agent to consider long-term economic benefits, the length of a single round is taken as a quarter, i.e. 90 days, and there are 2160 steps; the strategy network in the PPO algorithm selects the MLP neural network, and the input state is normalized.

[0049] Advantages and beneficial effects of the present application:

[0050] 1. The application proposes a new energy microgrid real-time energy management method based on proximal policy optimization and Lightgbm, by introducing a LightGBM model, predicting the power shortage of the new energy microgrid in the future period, refining the originally huge historical data into the predicted value of the power shortage, and introducing it as a new state variable into the deep reinforcement learning model, thereby skillfully avoiding the "dimension disaster" problem, significantly improving the training efficiency and training effect of the model.

[0051] 2. The application improves the reward function, densifies the reward function, and adds light and wind abandonment penalties and baseline comparison. The related technical measures proposed by the application reduce the training difficulty of the agent, improve the economic benefit of the new energy microgrid, provide a new technical idea for the energy management strategy of the new energy microgrid, and help promote the widespread application and sustainable development of new energy microgrids in China.

[0052] 3. The application builds an energy management strategy for a new energy microgrid based on proximal policy optimization (Proximal Policy Optimization) in deep reinforcement learning. The strategy can real-time dispatch the charge and discharge power of the energy storage unit in the new energy microgrid to absorb new energy and improve the operation economy of the microgrid.

[0053] 4. To solve the "dimension disaster" problem, the application introduces a LightGBM model to predict the power shortage of the new energy microgrid in the future period, refines the originally huge historical data into the predicted value of the power shortage, and introduces it as a new state variable into the deep reinforcement learning model, reducing the training difficulty of the agent, and thereby improving the training effect of the agent and the operation economy of the new energy microgrid.

[0054] 5. Compared with neural networks, the lightgbm in the tree model has the advantages of short training time, less required data, and less overfitting, which helps to improve the prediction accuracy of the power shortage of the new energy microgrid, and thereby improves the training effect of the agent.

[0055] 6. The application improves the reward function, reduces the training difficulty of the agent, and improves the economic benefit of the new energy microgrid. Specifically, the improvement includes the following three aspects: ① The reward function is densified, and the monthly demand cost of the microgrid is allocated to each reinforcement learning round. ② Add light and wind abandonment penalties to the reward function to artificially guide the agent to use energy storage devices to store excess new energy generation. ③ Add baseline comparison to the reward function to reduce the influence of "background noise" unrelated to action selection on the agent. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 The flowchart of the new energy microgrid real-time energy management method proposed by the application;

[0057] Figure 2 A schematic diagram of a new energy micro-grid structure provided for an embodiment of the present application;

[0058] Figure 3 A schematic diagram of an agent interacting with an environment provided for an embodiment of the present application;

[0059] Figure 4 A curve graph of power consumption, wind power generation and photovoltaic power generation of a new energy micro-grid in a test set provided for an embodiment of the present application;

[0060] Figure 5 A curve graph of the prediction value of the power shortage and the power storage of the energy storage system in the new energy micro-grid after the agent participates in decision-making provided for an embodiment of the present application. DETAILED DESCRIPTION

[0061] The embodiments of the present application are further described in detail below with reference to the accompanying drawings:

[0062] A new energy micro-grid real-time energy management method based on Proximal Policy Optimization (Proximal Policy Optimization) and Lightgbm, comprising the following steps:

[0063] S1, obtaining historical data of the new energy micro-grid to form a training sample library of the Lightgbm model.

[0064] The specific method of the step S1 is:

[0065] Obtaining historical data of the new energy micro-grid, including historical meteorological data such as temperature, humidity, wind speed, wind direction, solar elevation angle, irradiance, cloud cover, precipitation, historical power generation data of photovoltaic and wind power, and historical load data, to form a training sample library of the Lightgbm model.

[0066] S2, based on the training sample library of the Lightgbm model formed in step 1, training the Lightgbm model, and introducing an early stopping strategy to predict the power shortage of the new energy micro-grid;

[0067] The specific method of the step S2 is:

[0068] Taking the historical 24-hour meteorological data, photovoltaic and wind power generation data, load data of the new energy micro-grid and the hourly change amount of the above data as input features, the power shortage of the micro-grid in the future 1 hour is predicted;

[0069] The change window size of the hourly change amount feature is 1 to 24 hours, that is, the difference with the past 1 to 24 hours is calculated for each original feature, and the new feature is added to the training sample;

[0070] The model building and training are completed based on the Lightgbm library of Python, the target function is selected as Mean-Square Error (MSE), and the calculation formula is as follows:

[0071]

[0072] Wherein, n is the total number of samples; y true is the actual value of photovoltaic output; y pred is the predicted value of photovoltaic output.

[0073] In the embodiment, the granularity of each type of historical data in steps S1 and S2 is 1h, i.e. 24 points in a day.

[0074] In the embodiment, the lightGBM model training of step S2 is realized based on the lightgbm library in Python, and the target function is selected as Mean-Square Error (MSE); the early stopping strategy in the training process specifically refers to: first record the test set MSE of each training during training, when the test set MSE does not improve continuously for n times, stop training, and record the best training time. Then, retrain the model for n times to keep the model on the test set good performance and prevent overfitting.

[0075] S3, the energy scheduling problem of the new energy microgrid is described as a Markov decision process MDP, a reinforcement learning environment required for training the agent is built, the agent interacts with the environment, the round is defined, the reward function, state variable and action variable are designed, and the training sample library of deep reinforcement learning is formed by the historical data of step S1 and the power shortage prediction value of the Lightgbm model in step S2;

[0076] The state variable of step S3 includes: the storage capacity of the battery and the hydrogen storage tank, the grid price, the historical power shortage peak value and the microgrid power shortage predicted by step S2; the action variable includes: the charging and discharging power of the battery and the hydrogen storage tank.

[0077] The specific steps of step S3 include:

[0078] (1) First, the new energy microgrid (same as the above environment) feeds back the state variable vector S t to the agent, and the state variable is defined. D t+1 , are the storage capacity of the battery and the hydrogen storage tank at t time, the historical power shortage peak value, and the predicted value of the grid price and power shortage at t+1 time, respectively. Predicted by the Lightgbm model trained in S1 and S2 historical data;

[0079] (2) Then, the agent is in state vector S t Calculate the action vector A t , define the action vector Where, The charging and discharging power of the battery and the hydrogen storage system at t+1, respectively. Then, the environment is in state vector S t , action vector A t State transition, calculate the next state S t+1 And calculate the reward R t .

[0080] (3) The reward function R t In the field of traditional optimization, the objective function is similar to the reward function in reinforcement learning. In the energy management strategy of new energy microgrid, the negative value of monthly operation cost is often used as the reward function, as shown in the following formula:

[0081]

[0082] Where:

[0083] P t grid = P t load -P t pv -P t wind -P t battery -P t tank

[0084]

[0085] In the formula, T is the total step length of a single round, that is, 2160; D t The electricity price at t; E is the demand price; P t grid The power flowing from the main grid to the microgrid at t; The peak load of the microgrid from 0 to t, used to calculate the demand charge; P t load , P t pv , P t wind , P t battery , P t tank The load power, photovoltaic output, wind power, battery charging power and hydrogen tank charging power at t, respectively.

[0086] (4) Then, the state transition process of the environment is defined. The state transition of the environment depends on the mathematical model of each element of the new energy microgrid. For convenience of description, the present application still uses the concept of constraint conditions in the traditional optimization model.

[0087] The constraint conditions of the new energy microgrid are mainly the operation constraints of the energy storage device, including the energy storage capacity constraint and the charge / discharge rate constraint, as shown in the following formula.

[0088]

[0089] In the formula, are the minimum and maximum storage power of the battery and the hydrogen storage system, respectively; are the minimum and maximum charge / discharge power of the battery and the hydrogen storage system, respectively, and are taken as the reference direction of discharge, i.e. greater than 0 when discharging; η battery , η tank are the charging efficiencies of the battery and the hydrogen storage system, respectively; are 0-1 variables, respectively representing the charge / discharge state of the battery and the hydrogen storage system, and taking 1 when charging and 0 when discharging.

[0090] When the environment performs state transition, are determined by the above mathematical model calculation, and the remaining variables are updated by reading historical data.

[0091] (5) Finally, the state vector S t , the action vector A t and the reward R t are stored in the experience replay pool; the above process is repeated until the round is completed, thereby forming a training sample library of deep reinforcement learning.

[0092] In the present embodiment, a round lasts for three months (90 days), each step is 1 hour, and there are a total of 2160 steps. The decision network of the agent uses an MLP neural network, including an input layer, a hidden layer and an output layer, and the activation function uses tanh.

[0093] S4, the reward function designed for S3 is improved, and dense processing is performed, so that the monthly demand electricity fee is allocated to each hour; the light and wind curtailment penalty is added to guide the agent to actively consume excess new energy generation; the baseline is added to eliminate the influence of "background noise" unrelated to action selection on the agent, thereby completing the reinforcement learning environment construction;

[0094] The improved reward function in step S4 is specifically:

[0095]

[0096] Among them:

[0097]

[0098] In the formula, p is the curtailment penalty coefficient for light and wind; ε is the density coefficient. When ε is 1, the demand cost is completely amortized into each time step, but this amortization method has a certain calculation error. When ε is 0, it is equivalent to not performing density processing on the reward function.

[0099] After improving the reward function, based on OpenAI's gym library, the relevant programs were written as described in S3-S4, thereby completing the construction of the reinforcement learning environment.

[0100] S5. Based on the reinforcement learning environment built in steps S3-S4, train the agent using the Proximal Policy Optimization (PPO) algorithm and save the model parameters.

[0101] The specific method for step S5 is as follows:

[0102] The PPO algorithm was used to train the agent based on PyTorch in Python. The environment was designed based on OpenAI's gym library. To enable the agent to consider long-term economic benefits, the length of a single round was set to one quarter, or 90 days, with a total of 2160 steps. The policy network in the PPO algorithm was an MLP neural network, and the input state was normalized.

[0103] S6. Input real-time meteorological data, historical photovoltaic, wind power, and load data into the trained LightGBM model to predict the power deficit of the new energy microgrid in the next hour. Then input the state variables into the agent trained in step S5 to control and schedule the charging and discharging power of batteries and hydrogen storage tanks in the next hour, thereby completing the real-time energy management of the new energy microgrid.

[0104] The invention will be further illustrated below with specific examples:

[0105] like Figures 1-5 As shown, this embodiment utilizes the energy management strategy of the present invention to real-time schedule and control the energy storage charging and discharging operation status of a microgrid on the Rye dataset, a pilot project of the European Union research project. The steps include:

[0106] S1. Acquire historical data of the new energy microgrid, including historical meteorological data such as temperature, humidity, wind speed, wind direction, solar elevation angle, irradiance, cloud cover, and precipitation, historical output data of photovoltaic and wind power generation, and historical load data, to form a training sample library for the Lightgbm model.

[0107] This embodiment uses the Rye dataset from the EU research pilot project, with a microgrid structure as follows: Figure 2The pilot project includes wind power generation, photovoltaic power generation system, lithium battery energy storage system and hydrogen storage system, which includes hydrogen storage tank, electrolytic tank and boiler. The microgrid is connected to the main grid through the distribution transformer. When the power generation is surplus, the microgrid can store energy through water electrolysis and battery charging. The two ways of energy storage have different energy storage efficiency, energy storage capacity and charging and discharging power. When the power generation is insufficient, the microgrid can obtain power from the energy storage equipment or the main grid.

[0108] The microgrid device parameters are shown in Table 1.

[0109] Table 1 Microgrid device parameters

[0110]

[0111] The historical data of the data set has a granularity of 1h, including photovoltaic and wind power generation output, power load, real-time electricity price and weather data from January 1, 2020 to March 8, 2021. The weather data includes hourly cumulative rainfall, hourly cumulative clear sky flux, hourly cumulative sunshine duration, solar azimuth angle, solar elevation angle, and temperature, relative humidity, wind speed and wind direction angle at 2, 10, 50 and 100 meters above ground.

[0112] Based on the original data, the time, solar elevation angle and wind direction angle are further modeled to construct the following features:

[0113]

[0114] Wherein, hour is the hour time in a day, θ pv and θ wind are the solar elevation angle and the wind direction angle respectively.

[0115] S2, train the Lightgbm model and introduce the early stopping strategy to predict the power shortage of the new energy microgrid. The model takes the weather forecast value at the time to be predicted and the historical 24-hour weather data, photovoltaic and wind power generation data, load data and the hourly change of the above data as input features to predict the microgrid power shortage in the next 1 hour.

[0116] The change window size of the hourly change feature is 1 to 24 hours, that is, the difference between each original feature and the past 1 to 24 hours is calculated and added as a new feature to the training sample.

[0117] In this embodiment, the Lightgbm library based on python is used to complete the model building and training, and the target function selects the mean square error (MSE), and its calculation formula is as follows:

[0118]

[0119] Where n is the total number of samples; y true The actual value of photovoltaic output; y pred The predicted value for photovoltaic power output.

[0120] The main hyperparameters of the LightGBM model are set as follows: learning rate 0.02, maximum number of leaf nodes 128, feature resampling ratio 0.8, data subsampling ratio 0.9, bagging frequency 4, and early stopping rounds 100.

[0121] S3. The energy scheduling problem of the new energy microgrid is described as a Markov Decision Process (MDP). A reinforcement learning environment is built to train the agent, defining rounds, designing the reward function, state variables, and action variables. A training sample library for deep reinforcement learning is formed using historical data from step S1 and the power deficit predictions from the LightGBM model in step S2. State variables include: battery and hydrogen storage capacity, grid electricity price, historical peak power deficit, and the microgrid power deficit predicted in S2. Action variables include: charging and discharging power of the battery and hydrogen storage tank.

[0122] In this embodiment, the interaction process between the intelligent agent and the environment is as follows: Figure 3 As shown.

[0123] First, the new energy microgrid (in the same environment as described above) will have its state variable vector S at time t. t Feedback to the agent, state variables in, D t+1 These represent the battery and hydrogen storage capacity at time t, the historical peak power deficit, and the grid electricity price at time t+1, respectively. Subsequently, the agent uses the state vector S... t Calculate action vectors in, These represent the charging and discharging power of the battery and the hydrogen storage system at time t+1, respectively. Then, the environment is defined by the state vector S. t Action vector A t Perform a state transition and calculate the state S at the next time step. t+1 The status is determined, and the reward R is calculated. t Finally, the state vector S t Action vector A t And reward R t The reward is stored in the experience replay pool. The above process is repeated until the end of the round. In this embodiment, one round lasts three months (90 days), with each step lasting one hour, for a total of 2160 steps. The agent's decision network uses an MLP neural network, comprising an input layer, hidden layers, and an output layer, with tanh as the activation function.

[0124] In addition, the state transition of the environment depends on the mathematical model of each element of the new energy microgrid. For convenience of description, the concept of constraint condition in the traditional optimization model is still used in this embodiment.

[0125] The constraint condition of the new energy microgrid is mainly the operation constraint of the energy storage device, including the energy storage capacity constraint and the charge and discharge rate constraint, as shown in the following formula.

[0126]

[0127] In the formula, are the minimum and maximum storage power of the battery and the hydrogen storage system respectively; are the minimum and maximum charge and discharge power of the battery and the hydrogen storage system respectively, and are taken as the reference direction of discharge, that is, greater than 0 when discharging; η battery , η tank are the charging efficiencies of the battery and the hydrogen storage system respectively; are 0-1 variables, representing the charge and discharge state of the battery and the hydrogen storage system respectively, taking 1 when the energy storage device is charging and 0 when discharging.

[0128] When the environment is in state transition, are determined by the above mathematical model calculation, and the remaining variables are updated by reading historical data.

[0129] S4, the reward function is improved, and dense processing is performed, so that the monthly demand electricity fee is allocated to each hour. The light and wind abandonment penalty is added to guide the agent to actively consume the excess new energy generation. The baseline is added to eliminate the influence of "background noise" irrelevant to action selection on the agent.

[0130] After improvement, the reward function used by the present application is as follows:

[0131]

[0132] Among them:

[0133]

[0134] P t grid = P t load -P t pv -P t wind -P t battery -P t tank

[0135]

[0136] In the formula, T is a single round total step length, that is, 2160; D t is the power grid price at time t; p is a light and wind penalty coefficient; E is the demand price; P t grid is the power flowing from the main grid to the microgrid at time t; epsilon is a dense coefficient, when epsilon is 1, the demand charge is completely apportioned to each time step, but this apportionment method has a certain calculation error, when epsilon is 0, it is equivalent to not performing dense processing on the reward function; is the peak load of the microgrid from 0 to t, used to calculate the demand charge; P t load , P t pv , P t wind , P t battery , P t tank are the load power, photovoltaic output, wind power, battery charging power and hydrogen tank charging power at time t respectively.

[0137] The related parameters in the environment and the reward function in the embodiment are shown in Table 2.

[0138] Table 2 Related parameters of the environment and the reward function in the embodiment

[0139]

[0140] S5, based on the built reinforcement learning environment, an agent is trained using a proximal policy optimization algorithm (Proximal Policy Optimization), and model parameters are saved.

[0141] The proximal policy optimization (Proximal Policy Optimization, PPO) algorithm is a deep reinforcement learning algorithm, which aims to solve the problem of policy optimization in continuous action space and complex environment. The PPO algorithm has the advantages of high training efficiency, strong training stability and wide applicability, and is widely used in robot learning, game AI, finance, power system and other fields. PPO combines the efficiency of the policy gradient method and the stability of the trust region method (Trust Region Methods), and avoids the instability and performance degradation in the training process by limiting the difference between the new and old policies (i.e. clipping mechanism). The clipping mechanism avoids the policy update being too drastic by limiting the difference between the new and old policies. Specifically, if the ratio of the probability of selecting the same action between the new and old policies exceeds a certain threshold (such as 1.2 or 0.8), it is "clipped" to this threshold, thereby limiting the update amplitude of the policy network and maintaining the stability of the training.

[0142] In this embodiment, the training parameters of the PPO algorithm are shown in Table 3.

[0143] Table 3 Training parameters of PPO algorithm

[0144]

[0145] S6, input real-time weather data, historical photovoltaic, wind power, load data into the trained lightgbm model, predict the power shortage of the new energy microgrid in the future 1 hour, and then input each state variable into the trained agent to control and schedule the charging and discharging power of the battery and hydrogen storage tank in the future 1 hour.

[0146] In this embodiment, December 1, 2020 to March 1, 2021 are selected as the test set to test the economic benefits of the agent participating in the energy management of the new energy microgrid. The power load, wind power and photovoltaic power curves of the new energy microgrid in the test set are shown in Figure 4 Figure 4 It can be seen from

[0147] After the agent participates in decision-making, the predicted values of the battery and hydrogen storage system power storage and power shortage in the new energy microgrid are shown in Figure 5

[0148] After training, the agent can real-time perceive the running state of the microgrid and make effect analysis:

[0149] 1. Lightgbm prediction power shortage effect analysis

[0150] The present application predicts the power shortage of the microgrid in the future period by means of the Lightgbm model, reduces the dimension of the original state space, reduces the number of parameters of the decision network, avoids the "dimension disaster", speeds up the training speed, and improves the effect as shown in Table 4.

[0151] Table 4 Comparison table of improvement effect of Lightgbm on simplified problem

[0152]

[0153] In addition, by directly predicting the power difference of the new energy microgrid according to the present application, compared with predicting the load and new energy power generation respectively and then calculating the power difference, it is more accurate. The accuracy comparison between indirect prediction and direct prediction is shown in Table 5. It can be seen that, since the power load, wind power and photovoltaic power generation have certain correlation, direct prediction of power shortage has more advantages than indirect prediction, and the RMSE and accuracy rate are improved by 4.21kW and 3.40%, respectively. ​​

[0154] Table 5 Precision comparison of indirect prediction and direct prediction

[0155]

[0156]

[0157] 2. Reward function improvement effect analysis

[0158] In order to verify the improvement effect of the reward function of the present application, the intelligent agent is trained by using the original reward function, the reward function with the comparison baseline, the reward function with the light abandonment penalty, the reward function after the reward density processing and the improved reward function respectively. The microgrid monthly operation cost under different reward functions is shown in Table 6. Compared with the original reward function, the improved reward function of the present application reduces the monthly operation cost by 518 NOK, which is only 86.80% of the original reward function, and significantly improves the operation economy of the new energy microgrid.

[0159] Table 6 Comparison of microgrid monthly operation cost under different reward functions

[0160]

[0161] The working principle of the present application is that:

[0162] The application discloses a new energy micro-grid real-time energy management strategy based on proximal policy optimization (Proximal Policy Optimization) and Lightgbm. First, historical data of the new energy micro-grid is acquired, including historical meteorological data such as temperature, humidity, wind speed, wind direction, solar elevation angle, irradiance, cloud cover and precipitation, historical output data of photovoltaic and wind power generation, and historical load data, to form a training sample library of the Lightgbm model. Then, the Lightgbm model is trained, and an early stopping strategy is introduced to predict the power shortage of the new energy micro-grid. The model takes the historical 24-hour meteorological data, photovoltaic and wind power generation data, load data and hourly change of the above data of the new energy micro-grid as input features to predict the micro-grid power shortage in the next one hour. In the application, the Lightgbm model plays a role in refining historical features in the energy management strategy, which helps to solve the "dimension disaster" problem in deep reinforcement learning. Compared with predicting the photovoltaic, wind power output and power load respectively and then calculating the power shortage, directly predicting the power shortage helps to improve the prediction accuracy of the power shortage. Subsequently, a reinforcement learning environment required for training an intelligent agent is built, a round is defined, a reward function, a state variable and an action variable are designed, and the training sample library of deep reinforcement learning is formed by historical data and the power shortage prediction value of the Lightgbm model. The state variable includes the storage capacity of the battery and the hydrogen storage tank, the grid electricity price, the historical power shortage peak value and the prediction value of the micro-grid power shortage output by the Lightgbm. The action variable includes the charging and discharging power of the battery and the hydrogen storage tank. The application takes the minimum micro-grid operation cost as the target, performs dense processing on the reward function and adds light and wind abandonment penalties and a baseline, thereby reducing the training difficulty of the intelligent agent and improving the economic benefit of the new energy micro-grid. Based on the built reinforcement learning environment, a proximal policy optimization algorithm (Proximal Policy Optimization) is used to train the intelligent agent, and the model parameters are saved. Finally, real-time meteorological data, historical photovoltaic, wind power and load data are input into the trained Lightgbm model to predict the new energy micro-grid power shortage in the next one hour, and the charging and discharging power of the battery and the hydrogen storage tank in the next one hour is controlled and dispatched by inputting the state variables into the trained intelligent agent.

[0163] Those skilled in the art will appreciate that embodiments of the application can be supplied as methods, systems, or computer program products. Accordingly, the application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage media, etc.) having computer-usable program code embodied in the medium.

[0164] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart Figure One one or more flows and / or blocks in the flowchart and / or block diagram. Figure One one or more flows and / or blocks in the flowchart and / or block diagram.

[0165] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart Figure One one or more flows and / or blocks in the flowchart and / or block diagram. Figure One one or more flows and / or blocks in the flowchart and / or block diagram.

[0166] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart Figure One one or more flows and / or blocks in the flowchart and / or block diagram. Figure One one or more flows and / or blocks in the flowchart and / or block diagram.

Claims

1. A new energy microgrid real-time energy management method based on proximal policy optimization and Lightgbm, characterized in that: The method comprises the following steps: S1, obtaining historical data of the new energy microgrid to form a training sample library of the Lightgbm model; S2, based on the training sample library of the Lightgbm model formed in step 1, training the Lightgbm model, and introducing an early stopping strategy to predict the power shortage of the new energy microgrid; S3, the energy scheduling problem of the new energy microgrid is described as a Markov decision process (MDP), a reinforcement learning environment required for training an agent is built, a round is defined, a reward function, a state variable, and an action variable are designed; the agent interacts with the environment, and the interaction process and the state vector S t , the action vector A t , and the reward R t of the result are stored in an experience replay pool, thereby forming a training sample library of deep reinforcement learning; Wherein, part of the elements of the state vector are obtained from the historical data in step S1 and the Lightgbm model in step S2; S4, the reward function designed in S3 is improved and densified, the monthly demand charge is allocated to each hour, the light and wind penalties are added to guide the agent to actively consume excess new energy generation, and the baseline is added to eliminate the "background noise" irrelevant to the action selection, and then the reinforcement learning environment is finally built; S5, based on the reinforcement learning environment built in steps S3-S4, the agent is trained using the proximal policy optimization algorithm, and the model parameters are saved; S6, real-time weather data, historical photovoltaic, wind power and load data are input into the trained lightgbm model to predict the new energy microgrid power shortage in the next one hour, and then the state variables are input into the agent trained in step S5 to control and schedule the charging and discharging power of the battery and hydrogen storage tank in the next one hour, thereby completing the real-time energy management of the new energy microgrid.

2. The new energy microgrid real-time energy management method based on near-end policy optimization and Lightgbm according to claim 1, characterized in that: The specific method of step S1 is: Obtain historical data of the new energy microgrid, including temperature, humidity, wind speed, wind direction, solar elevation angle, irradiance, cloud cover, precipitation and other historical weather data, historical output data of photovoltaic and wind power generation, and historical load data to form a training sample library of the Lightgbm model.

3. The new energy microgrid real-time energy management method based on near-end policy optimization and Lightgbm according to claim 1, characterized in that: The specific method of step S2 is: Using the historical 24-hour weather data, photovoltaic and wind power data, load data of the new energy microgrid and the hourly change of the above data as input features, the microgrid power shortage in the next one hour is predicted; The change window size of the hourly change feature is 1 to 24 hours, that is, the difference with the past 1 to 24 hours is calculated for each original feature and added as a new feature to the training sample; The model is built and trained based on the Lightgbm library of Python, and the objective function selects the mean square error MSE, whose calculation formula is as follows: where n is the total number of samples; y true the actual value of the photovoltaic power output; y pred is the predicted value of the photovoltaic power output.

4. The new energy microgrid real-time energy management method based on near-end policy optimization and Lightgbm according to claim 1, characterized in that: The lightGBM model training in step S2 is realized based on the lightgbm library in Python, and the objective function selects the mean square error MSE; the early stopping strategy in the training process specifically means that during training, the test set MSE of each training is first recorded, and when the test set MSE does not improve for n consecutive times, the training is stopped, and the best training time is recorded; Then, the model is retrained for n times to maintain good performance on the test set.

5. The new energy microgrid real-time energy management method based on near-end policy optimization and Lightgbm according to claim 1, characterized in that: The state variables of step S3 include: the storage capacity of the battery and hydrogen storage tank, the grid electricity price, the historical power shortage peak value and the microgrid power shortage predicted in step S2; the action variables include: the charging and discharging power of the battery and hydrogen storage tank.

6. The new energy microgrid real-time energy management method based on near-end policy optimization and Lightgbm according to claim 1, characterized in that: The specific steps of step S3 include: (1) First, the new energy microgrid feeds back the state variable vector S t to the agent, defining the state variable where, D t+1 , are the storage capacity of the battery and hydrogen tank at time t, the historical peak of power shortage, and the predicted value of the power grid price and power shortage at time t+1, respectively; which is predicted by the Lightgbm model trained by the historical data in S1 and S2; (2) Then, the agent is updated by the state vector S t The action vector A t is calculated Wherein, The charging and discharging power of the battery and the hydrogen storage system at t+1 is respectively; then, the environment is updated by the state vector S t , the action vector A t , the next state S t+1 is calculated, and the reward R t is calculated; (3) Reward function R in reinforcement learning t Similar to the objective function in traditional optimization, the negative value of the monthly operation cost is often used as the reward function in the energy management strategy of the new energy microgrid, as shown in the following equation: Wherein: P t grid = P t load - P t pv - P t wind - P t battery - P t tank In the formula, T is the total step length of a single round, that is, 2160; D t is the power price of the power grid at time t; E is the demand price; P t grid is the power flowing from the main grid to the microgrid at time t; is the peak load of the microgrid from 0 to t, used for calculating the demand charge; P t load , P t pv , P t wind , P t battery , P t tank are the load power, photovoltaic output, wind power output, battery charging power and charging power of the hydrogen storage tank at time t, respectively (4) Then, the state transition process of the environment is defined, and the state transition of the environment depends on the mathematical model of each element of the new energy microgrid; The constraint condition of the new energy microgrid is the operation constraint of the energy storage device, including the energy storage capacity constraint and the charge / discharge rate constraint, as shown in the following formula: wherein, are the minimum and maximum storage power of the battery and hydrogen storage system, respectively; are the minimum and maximum charge-discharge power of the battery and hydrogen storage system, respectively, with the reference direction being discharge, i.e. greater than 0 when discharging; battery are the charge efficiencies of the battery and hydrogen storage system, respectively; tank are the charge efficiencies of the battery and hydrogen storage system, respectively; are 0-1 variables representing the charge-discharge state of the battery and hydrogen storage system, respectively, taking the value 1 when the energy storage device is charging and 0 when discharging. When the environment makes a state transition, The remaining variables are updated by reading historical data, as determined by the mathematical model calculations. (5) Finally, the state vector S t , the action vector A t , and the reward R t are stored into the experience replay pool; the above process is repeated until the episode ends, thereby forming a training sample library for deep reinforcement learning.

7. The new energy microgrid real-time energy management method based on near-end policy optimization and Lightgbm according to claim 6, characterized in that: The improved reward function in the step S4 is specifically: Wherein, p is a light penalty coefficient; and ε is a dense coefficient; when ε is 1, the demand charge is completely apportioned to each time step, but there is a certain calculation error in the apportioning mode; and when ε is 0, it is equivalent to not performing the dense processing on the reward function; After the improvement of the reward function, the related program is written according to S3-S4 based on the gym library of openai, and then the reinforcement learning environment is finally completed. The specific method of the step S5 is:

8. The new energy microgrid real-time energy management method based on near-end policy optimization and Lightgbm according to claim 1, characterized in that: The training of the agent using the PPO algorithm is realized based on Pytorch in Python, and the design of the environment is realized based on the gym library of Openai; in order to enable the agent to consider long-term economic benefits, the length of a single round is taken as a quarter, that is, 90 days, and there are 2160 steps in total; the strategy network in the PPO algorithm selects the MLP neural network, and the input state is normalized. ​

Images