Micro-grid operation strategy optimization method based on deep reinforcement learning, electronic equipment and storage medium

Abstract

The invention discloses a micro-grid operation strategy optimization method based on deep reinforcement learning, electronic equipment and a storage medium, and relates to the field of micro-grids, and the method comprises the steps: 1, building a micro-grid optimization model taking a power generation unit, a storage unit and tie line interaction power as control objects, the method comprises the steps of (1) determining a micro-grid optimization model, (2) determining constraint conditions of the micro-grid optimization model, (3) randomly sampling and generating an initial sampling point set, (4) constructing a PPO-clip algorithm model based on the initial sampling point set, and (5) estimating and solving an optimal solution of the micro-grid optimization model by using the PPO-clip algorithm model and optimizing micro-grid operation. According to the method, the PPO-clip algorithm is adopted to assist the establishment of the micro-grid optimization model, the optimization of the micro-grid optimization model can be realized in a relatively low round, so that the micro-grid optimization model realizes efficient power generation, storage and electric quantity adjustment of the wave crest and the wave trough, and an optimal solution of data is realized.

Description

Technical Field

[0001] The present invention relates to the field of microgrid technology, and in particular to a microgrid operation strategy optimization technology based on deep reinforcement learning, which is specifically used for microgrid operation optimization. Background Art

[0002] With the rapid economic development, fossil fuels and other non-renewable natural resources are becoming exhausted, the global ecological environment is deteriorating, and the problem between energy supply and demand in my country is becoming increasingly prominent. As an important energy consumer and carbon dioxide emitter, it is urgent to carry out energy-saving power generation scheduling and establish a safe, efficient and clean power system that maximizes the use of renewable energy and other clean energy to reduce the consumption of non-renewable energy and the emission of pollutants. Among them, the microgrid system has attracted much attention for its efficient, green and economical energy utilization. On the other hand, it can increase the absorption rate of some renewable energy sources to a certain extent, including light energy and wind energy. China's renewable energy power generation has grown rapidly, especially wind power generation and solar photovoltaic power generation have maintained a relatively fast annual growth, leading the world for many years.

[0003] At present, the research on microgrid operation strategy generally sets the optimization objective function and constructs the model first, and then uses the algorithm to solve it. The solution algorithm includes mathematical programming and heuristic optimization algorithm. Mathematical programming is very efficient when dealing with linear problems, but it is often difficult to jump out of the local optimum when encountering nonlinear problems. The microgrid model contains nonlinear equipment models; while heuristic algorithms can handle such problems well, their calculations are time-consuming and the algorithm's generalization learning ability is insufficient. With the development of artificial intelligence, deep reinforcement learning algorithms with perception and decision-making capabilities have been applied by researchers in the field of power systems and have shown good performance. Summary of the invention

[0004] The purpose of the present invention is to provide a microgrid operation strategy optimization method, electronic device and storage medium based on deep reinforcement learning, so as to solve the problem that the traditional optimization algorithm proposed in the above background technology has difficulty in solving the nonlinear optimization problem of multiple objectives and multiple constraints, and the collection and processing of raw data is slow.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] In the first aspect, the present invention discloses a microgrid operation strategy optimization method based on deep reinforcement learning, specifically:

[0007] Step 1: Establish a microgrid optimization model with the power generation unit, storage unit, and interconnection line interaction power as the control object;

[0008] Step 2: Determine the constraints of the microgrid optimization model;

[0009] Step 3: Random sampling to generate an initial sampling point set;

[0010] Step 4: Construct the PPO-clip algorithm model based on the initial sampling point set;

[0011] Step 5: Use the PPO-clip algorithm model to estimate and solve the optimal solution of the microgrid optimization model to optimize the microgrid operation.

[0012] Furthermore, in step 1, the microgrid operating environment needs to be written: first, a day is divided into 24 time periods, denoted as [1, 2, 3, ..., 24], and the length of each time period is 1 hour. Let the predicted wind power generation power in a certain time period t be denoted as P t wt , the predicted photovoltaic power generation power is P t pv , the predicted load power is P t load , the power of the diesel engine is P t die , the energy storage charging and discharging power is P t ess ;

[0013] The operation strategy followed by deep reinforcement learning to design the microgrid environment at a certain time t is:

[0014] When P t pv +P t wt ≥P t load When P t die is 0, the deep reinforcement learning algorithm will coordinate the excess electricity to charge the energy storage or sell it to the upper-level power grid for profit according to the time-of-use electricity price.

[0015] When P t pv +P t wt <P t load And when SOC is within the normal range, if P t ess Can meet the power difference between load and renewable energy generation, then P t die is 0;

[0016] If P t essIf the power difference is not met, the deep reinforcement learning algorithm will choose the lowest-cost action from the diesel engine power generation cost and the purchase of electricity from the upper-level power grid to make up for the remaining power difference. Among them, the purchase of electricity from the upper-level power grid is related to the time-of-use electricity price.

[0017] When the SOC has reached its limit and cannot be discharged, the deep reinforcement learning algorithm will also coordinate the two methods of diesel engine power generation and purchasing electricity from the superior, so that the microgrid can operate at the lowest cost while meeting the supply and demand balance.

[0018] Furthermore, in step 1, the objective function of the microgrid optimization model is:

[0019] The optimization goal is to minimize the operating cost of the microgrid, that is:

[0020] min(F1+F2)

[0021] Where: F1 is the fuel cost of the diesel generator; F2 is the cost of the microgrid purchasing and selling electricity from the main grid.

[0022]

[0023] Where: T is the scheduling period; is the price at which the main grid buys electricity from the microgrid at time t; is the price of electricity sold by the microgrid to the main grid at time t; Δt is the scheduling time interval; P t grid is the transaction power between the microgrid and the main grid, P t grid <0 means the microgrid sells electricity to the main grid β=0, P t grid >0 means the microgrid purchases electricity from the main grid, β=1. t grid Available through P t load , P t pv , P t wt , P t die and P t ess To calculate, the expression is as follows.

[0024] P t grid =P t load -P t pv -P t wt -P t die +Pt ess .

[0025] Furthermore, the constraint condition in step 2 is

[0026] The optimization goal is to minimize the operating cost of the microgrid, that is:

[0027] min(F1+F2)

[0028] Where: F1 is the fuel cost of the diesel generator; F2 is the cost of the microgrid purchasing and selling electricity from the main grid.

[0029]

[0030] Where: T is the scheduling period; is the price at which the main grid buys electricity from the microgrid at time t; is the price of electricity sold by the microgrid to the main grid at time t; Δt is the scheduling time interval; P t grid is the transaction power between the microgrid and the main grid, P t grid <0 means the microgrid sells electricity to the main grid β=0, P t grid >0 means the microgrid purchases electricity from the main grid, β=1. t grid Available through P t load , P t pv , P t wt , P t die and P t ess To calculate, the expression is as shown in formula (4-3).

[0031] P t grid =P t load -P t pv -P t wt -P t die +P t ess

[0032] (2) Constraints

[0033] In order to achieve safe and reliable operation of the microgrid, the following constraints are established:

[0034] (1) Power balance constraints

[0035] P t wt +P t pv +P t grid +P t die -P t ess =P t load

[0036] (2) Wind power system output power constraints

[0037]

[0038] (3) The output power constraint of the photovoltaic power generation system is:

[0039] 0≤P pv ≤P pv,max

[0040] Where: P pv,max is the maximum output power of photovoltaic power generation.

[0041] (4) Diesel engine power constraints

[0042] 0≤P d (t)≤P d.max

[0043] Where: P d.max The maximum power of the diesel engine

[0044] (5) To ensure the safe operation and service life of the battery, the following constraints must be met:

[0045] SOC min ≤SOC(t)≤SOC max

[0046]

[0047] Where: SOC min , SOC max are the minimum and maximum values ​​of the state of charge respectively; is the maximum discharge power; is the maximum charging power.

[0048] Furthermore, the PPO-clip algorithm model in step 4

[0049] Assume that the initial neural network parameters are: θ = {θ, θ′, θ Q}, θ, θ′ are the Actor neural network parameters, θ Qis the Critic neural network parameter. At the beginning of training, the θ parameter of the Actor deep neural network will be initialized. With the start of training and the iteration of parameters, the θ of the previous time period will be replaced by the θ′ of the new time period. The value of the PPO-clip algorithm policy network can be updated by the following formula:

[0050]

[0051] Critic value function deep neural network parameter θ Q The updated standard formula is:

[0052]

[0053] From the above formula, we can conclude that (V π (s t )-y t ) is the advantage function A(s) mentioned above. t ,a t ), which is also the cleverness of connecting the Actor network and the Critic.

[0054] Then, the network parameters of the PPO-clip algorithm value function can be updated by the following formula:

[0055]

[0056] Where: η Q and η μ The learning rate updated for the value function network and the learning rate for the policy network.

[0057] In a second aspect, the present invention further discloses an electronic device,

[0058] include:

[0059] one or more processors;

[0060] A memory, coupled to the processor, for storing one or more programs;

[0061] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the microgrid operation strategy optimization methods described above.

[0062] In a third aspect, the present invention further discloses a computer-readable storage medium having a computer program stored thereon.

[0063] When the computer program is executed by a processor, the method for optimizing the microgrid operation strategy as described above is implemented.

[0064] The beneficial effects of the present invention are:

[0065] In the present invention, the PPO-clip algorithm is used to assist in the establishment of a microgrid optimization model, which can efficiently and quickly optimize the microgrid optimization model, so that the microgrid optimization model can achieve efficient power generation, storage, and peak and trough power regulation, and achieve optimal data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A schematic diagram of a microgrid operation strategy optimization method based on deep reinforcement learning proposed by the present invention;

[0067] Figure 2 A schematic diagram showing the reward comparison of different deep reinforcement learning algorithms;

[0068] Figure 3 Schematic diagram of microgrid operation cost obtained by each algorithm;

[0069] Figure 4 This is a schematic diagram of the running cost of the 30-day cumulative experiment;

[0070] Figure 5 This is a schematic diagram of the optimization results of the PPO-clip algorithm;

[0071] Figure 6 Schematic diagram of the PPO algorithm optimization results. DETAILED DESCRIPTION

[0072] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0073] Embodiment 1:

[0074] Reference Figure 1-4 , a microgrid operation strategy optimization method based on deep reinforcement learning, specifically:

[0075] Step 1: Establish a microgrid optimization model with the power generation unit, storage unit, and interconnection line interaction power as the control object;

[0076] Step 2: Determine the constraints of the microgrid optimization model;

[0077] Step 3: Random sampling to generate an initial sampling point set;

[0078] Step 4: Construct the PPO-clip algorithm model based on the initial sampling point set;

[0079] Step 5: Use the PPO-clip algorithm model to estimate and solve the optimal solution of the microgrid optimization model to optimize the microgrid operation.

[0080] The power generation unit includes one or more of photovoltaic power generation, wind power generation and diesel power generation.

[0081] Furthermore, in step 1, the microgrid operating environment needs to be written: first, a day is divided into 24 time periods, denoted as [1, 2, 3, ..., 24], and the length of each time period is 1 hour. Let the predicted wind power generation power in a certain time period t be denoted as P t wt , the predicted photovoltaic power generation power is P t pv , the predicted load power is P t load , the power of the diesel engine is P t die , the energy storage charging and discharging power is P t ess ;

[0082] The operation strategy followed by deep reinforcement learning to design the microgrid environment at a certain time t is:

[0083] When P t pv +P t wt ≥P t load When P t die is 0, the deep reinforcement learning algorithm will coordinate the excess electricity to charge the energy storage or sell it to the upper-level power grid for profit according to the time-of-use electricity price.

[0084] When P t pv +P t wt <P t load And when SOC is within the normal range, if P t ess Can meet the power difference between load and renewable energy generation, then P t die is 0; the normal range of SOC is 0.2-0.85.

[0085] If P t ess If the power difference is not met, the deep reinforcement learning algorithm will choose the lowest-cost action from the diesel engine power generation cost and the purchase of electricity from the upper-level power grid to make up for the remaining power difference. Among them, the purchase of electricity from the upper-level power grid is related to the time-of-use electricity price.

[0086] When the SOC has reached its limit and cannot be discharged, the deep reinforcement learning algorithm will also coordinate the two methods of diesel engine power generation and purchasing electricity from the superior, so that the microgrid can operate at the lowest cost while meeting the supply and demand balance.

[0087] Furthermore, in step 1, the objective function of the microgrid optimization model is:

[0088] The optimization goal is to minimize the operating cost of the microgrid, that is:

[0089] min(F1+F2)

[0090] Where: F1 is the fuel cost of the diesel generator; F2 is the cost of the microgrid purchasing and selling electricity from the main grid.

[0091]

[0092] Where: T is the scheduling period; is the price at which the main grid buys electricity from the microgrid at time t; is the price of electricity sold by the microgrid to the main grid at time t; Δt is the scheduling time interval; P t grid is the transaction power between the microgrid and the main grid, P t grid <0 means the microgrid sells electricity to the main grid β=0, P t grid >0 means the microgrid purchases electricity from the main grid, β=1. t grid Available through P t load , P t pv , P t wt , P t die and P t ess To calculate, the expression is as follows.

[0093] P t grid =P t load -P t pv -P t wt -P t die +P t ess .

[0094] Furthermore, the constraint condition in step 2 is

[0095] The optimization goal is to minimize the operating cost of the microgrid, that is:

[0096] min(F1+F2)

[0097] Where: F1 is the fuel cost of the diesel generator; F2 is the cost of the microgrid purchasing and selling electricity from the main grid.

[0098]

[0099] Where: T is the scheduling period; is the price at which the main grid buys electricity from the microgrid at time t; is the price of electricity sold by the microgrid to the main grid at time t; Δt is the scheduling time interval; P t grid is the transaction power between the microgrid and the main grid, P t grid <0 means the microgrid sells electricity to the main grid β=0, P t grid >0 means the microgrid purchases electricity from the main grid, β=1. t grid Available through P t load , P t pv , P t wt , P t die and P t ess To calculate, the expression is as shown in formula (4-3).

[0100] P t grid =P t load -P t pv -P t wt -P t die +P t ess

[0101] (2) Constraints

[0102] In order to achieve safe and reliable operation of the microgrid, the following constraints are established:

[0103] (1) Power balance constraints

[0104] P t wt +P t pv +P t grid +P t die -P t ess =P tload

[0105] (2) Wind power system output power constraints

[0106]

[0107] (3) The output power constraint of the photovoltaic power generation system is:

[0108] 0≤P pv ≤P pv,max

[0109] Where: P pv,max is the maximum output power of photovoltaic power generation.

[0110] (4) Diesel engine power constraints

[0111] 0≤P d (t)≤P d.max

[0112] Where: P d.max The maximum power of the diesel engine

[0113] (5) To ensure the safe operation and service life of the battery, the following constraints must be met:

[0114] SOC min ≤SOC(t)≤SOC max

[0115]

[0116] Where: SOC min , SOC max are the minimum and maximum values ​​of the state of charge respectively; is the maximum discharge power; is the maximum charging power.

[0117] Furthermore, the PPO-clip algorithm model in step 4

[0118] Assume that the initial neural network parameters are: θ = {θ, θ′, θ Q}, θ, θ′ are the Actor neural network parameters, θ Q is the Critic neural network parameter. At the beginning of training, the θ parameter of the Actor deep neural network will be initialized. With the start of training and the iteration of parameters, the θ of the previous time period will be replaced by the θ′ of the new time period. The value of the PPO-clip algorithm policy network can be updated by the following formula:

[0119]

[0120] Critic value function deep neural network parameter θ Q The updated standard formula is:

[0121]

[0122] From the above formula, we can conclude that (V π (s t )-y t ) is the advantage function A(s) mentioned above. t ,a t ), which is also the cleverness of connecting the Actor network and the Critic.

[0123] Then, the network parameters of the PPO-clip algorithm value function can be updated by the following formula:

[0124]

[0125] Where: η Q and η μ The learning rate updated for the value function network and the learning rate for the policy network.

[0126] Embodiment 2:

[0127] When other conditions are the same, the deep reinforcement learning model is replaced by the initial PPO algorithm.

[0128] Embodiment 3:

[0129] Other conditions remain unchanged, and the deep reinforcement learning model is replaced by the classic SAC algorithm.

[0130] from Figure 2 It can be seen that the PPO-clip algorithm reaches stability at around 300 rounds, and the reward value is the highest. The SAC and PPO algorithms reach stability at around 400 and 380 rounds, respectively, and the reward functions are lower than the PPO-clip algorithm. This shows that the PPO-clip algorithm has a strong ability to adapt to the environment and has better performance.

[0131] from Figure 4 It can be concluded that the PPO-clip algorithm still has the best effect within 30 days of operation and is suitable as an algorithm for long-term use.

[0132] from Figures 5 to 6It can be concluded that both the PPO-clip algorithm and the PPO algorithm can enable the energy storage to charge when the electricity price is low and discharge when the electricity price is high, but the PPO-clip algorithm enables the energy storage system to better interact with the upper-level power grid, and has a stronger ability to charge low and sell high, which can reduce operating costs. During the period of 13:00-16:00, when new energy is sufficient, the diesel engine is shut down under both the PPO-clip algorithm and the PPO algorithm, and the remaining power is used to charge the energy storage or sold to the power grid. Under the PPO-clip algorithm, the energy storage can interact better with wind power, photovoltaic power, and time-of-use electricity prices. Compared with the PPO algorithm, the energy storage can be charged more, which can reduce the amount of electricity purchased from the large power grid during subsequent peak electricity prices, thereby reducing the operating costs of the microgrid. In addition, the scheduling of diesel engines will also directly affect the operating costs of the entire microgrid. From Figure 5 It can be concluded that under the optimization of the PPO-clip algorithm, the diesel engine is basically started when the power generation cannot meet the load and the electricity price is at the peak price. However, due to its poor search ability, the PPO algorithm cannot accurately control the diesel engine, thereby increasing the operating cost of the microgrid.

[0133] Embodiment 4: An electronic device,

[0134] comprising: one or more processors;

[0135] A memory, coupled to the processor, for storing one or more programs;

[0136] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the microgrid operation strategy optimization methods described above.

[0137] Embodiment 5: A computer-readable storage medium having a computer program stored thereon,

[0138] When the computer program is executed by a processor, the method for optimizing the microgrid operation strategy as described above is implemented.

[0139] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A microgrid operation strategy optimization method based on deep reinforcement learning, Features: The following steps are involved: (1) Establish a microgrid optimization model with the interactive power of power generation units, storage units, and tie lines as the control objects; (2) Determine the constraints of the microgrid optimization model; (3) Random sampling to generate an initial sampling point set; (4) Construct the PPO-clip algorithm model based on the initial sampling point set; (5) Use the PPO-clip algorithm model to estimate and solve the optimal solution of the microgrid optimization model and optimize the operation of the microgrid.

2. According to the microgrid operation strategy optimization method based on deep reinforcement learning according to claim 1, Features: In step (1), the microgrid operating environment needs to be written: First, a day is divided into 24 time periods, denoted as [1, 2, 3, ..., 24], and the length of each time period is 1 hour; let the predicted wind power generation power in a certain time period t be denoted as P t wt , the predicted photovoltaic power generation power is P t pv , the predicted load power is P t load , the power of the diesel engine is P t die , the energy storage charging and discharging power is P t ess ; The operation strategy followed by deep reinforcement learning to design the microgrid environment at a certain time t is: When P t pv +P t wt ≥P t load When P t die If the value is 0, the deep reinforcement learning algorithm will coordinate the excess electricity to charge the energy storage or sell it to the upper grid for profit according to the time-of-use electricity price; When P t pv +P t wt <P t load And when SOC is within the normal range, if P t ess Can meet the power difference between load and renewable energy generation, then P t die is 0; If P t ess If the power difference is not met, the deep reinforcement learning algorithm will choose the lowest-cost action from the diesel engine power generation cost and the purchase of electricity from the upper-level power grid to make up for the remaining power difference. Among them, the purchase of electricity from the upper-level power grid is related to the time-of-use electricity price. When the SOC has reached its limit and cannot be discharged, the deep reinforcement learning algorithm will also coordinate the two methods of diesel engine power generation and purchasing electricity from the superior, so that the microgrid can operate at the lowest cost while meeting the supply and demand balance.

3. According to the microgrid operation strategy optimization method based on deep reinforcement learning according to claim 1, Features: In step (1), the objective function of the microgrid optimization model is: The optimization goal is to minimize the operating cost of the microgrid, that is: min(F1+F2)(1) In formula (1): F1 is the fuel cost of the diesel generator; F2 is the cost of the microgrid purchasing and selling electricity from the main grid; In formula (2): T is the scheduling period; is the price at which the main grid buys electricity from the microgrid at time t; is the price of electricity sold by the microgrid to the main grid at time t; Δt is the scheduling time interval; P t grid is the transaction power between the microgrid and the main grid, P t grid <0 means the microgrid sells electricity to the main grid β=0, P t grid >0 means the microgrid purchases electricity from the main grid, β=1; P t grid Available through P t load , P t pv , P t wt , P t die and P t ess To calculate, its expression is as follows: P t grid =P t load -P t pv -P t wt -P t die +P t ess (3) 4. According to the microgrid operation strategy optimization method based on deep reinforcement learning according to claim 1, Features: The constraint conditions in step (2) are: The optimization goal is to minimize the operating cost of the microgrid, that is: min(F1+F2)(4) In formula (4): F1 is the fuel cost of the diesel generator; F2 is the cost of the microgrid purchasing and selling electricity from the main grid; In formula (5): T is the scheduling period; is the price at which the main grid buys electricity from the microgrid at time t; is the price of electricity sold by the microgrid to the main grid at time t; Δt is the scheduling time interval; P t grid is the transaction power between the microgrid and the main grid, P t grid <0 means the microgrid sells electricity to the main grid β=0, P t grid >0 means the microgrid purchases electricity from the main grid, β=1; P t grid Available through P t load , P t pv , P t wt , P t die and P t ess To calculate, the expression is as follows: P t grid =P t load -P t pv -P t wt -P t die +P t ess (6) In order to achieve safe and reliable operation of the microgrid, the following constraints are established: Y1: Power balance constraint P t wt +P t pv +P t grid +P t die -P t ess =P t load (7) Y2: Wind power system output power constraint Y3: The output power constraint of the photovoltaic power generation system is: 0≤P pv ≤P pv ,max (9) In formula (9): P pv,max is the maximum output power of photovoltaic power generation; Y4: Diesel engine power constraint 0≤P d (t)≤P d.max (10) In formula (10): P d.max The maximum power of the diesel engine Y5: To ensure the safe operation and service life of the battery, the following constraints must be met: SOC min ≤SOC(t)≤SOC max (11) Where: SOC min , SOC max are the minimum and maximum values ​​of the state of charge respectively; is the maximum discharge power; is the maximum charging power.

5. According to a microgrid operation strategy optimization method based on deep reinforcement learning according to claim 1, Features: The PPO-clip algorithm model in step (4) is: Assume that the initial neural network parameters are: θ = {θ, θ′, θ Q }, θ, θ′ are the Actor neural network parameters, θ Q is the Critic neural network parameter; at the beginning of training, the θ parameter of the Actor deep neural network will be initialized; with the start of training and the iteration of parameters, the θ of the previous time period will be replaced by the θ′ of the new time period; the value of the PPO-clip algorithm policy network can be updated by the following formula: Critic value function deep neural network parameter θ Q The updated standard formula is: From the above formula, we can conclude that (V π (s t )-y t ) is the advantage function A(s) mentioned above. t ,a t ); The network parameters of the PPO-clip algorithm value function are updated by the following formula: Where: η Q and η μ The learning rate updated for the value function network and the learning rate for the policy network.

6. An electronic device, It is characterized in that include: one or more processors; A memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the microgrid operation strategy optimization method as described in any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the microgrid operation strategy optimization method according to any one of claims 1 to 5 is implemented.

Images