A microgrid energy system optimization method based on real-time electricity price incentives

Abstract

The present invention provides a method for optimizing a microgrid energy system based on real-time electricity price incentives, and belongs to the field of electronic information technology. It includes establishing a microgrid energy system and acquiring system data; determining a state set, an action set, a reward function, and a real-time electricity price incentive mechanism; and using the real-time electricity price incentive mechanism and the Dueling Double DQN algorithm to perform offline learning to obtain the optimal strategy, so that the system can achieve maximum economic benefits in any state. The optimization method of the present invention uses real-time buying and selling electricity prices to incentivize exploration of strategies, an online evaluation network to obtain the action corresponding to the optimal action value in the current state, and a target evaluation network to calculate the action value of the action to obtain a target estimated value, which effectively avoids the "overestimation" problem of the algorithm, improves the algorithm estimation accuracy, and thus achieves maximum economic benefits of the microgrid energy system.

Description

Technical Field

[0001] The present invention belongs to the field of electronic information technology, and in particular relates to a microgrid energy system optimization method based on real-time electricity price incentives. Background Art

[0002] Energy systems used for energy interaction with the main power grid are mainly composed of wind turbines, energy storage batteries and various loads. In the application of this type of energy system, the large-scale regional interconnection of the integrated energy system has gradually developed into a large-scale high-dimensional system. Considering that the integrated energy system is a complex dynamic system with high uncertainty, it is very difficult to construct an accurate model. Therefore, the application of related optimization algorithms in solving integrated energy system management problems is limited. The reason is that algorithms such as linear programming and nonlinear programming rely on accurate mathematical models and parameters. The high uncertainty of microgrid energy management systems makes it difficult to construct models; related algorithms based on game theory can effectively solve the optimal strategies of participants in the rational market, thereby achieving optimal management of resources, but they have the disadvantages of complex model structure and slow convergence speed.

[0003] At present, the microgrid energy system management decision-making method based on deep reinforcement learning is the most widely used. There are both academic papers that have made in-depth theoretical analysis on this, and engineering methods for practical application, such as the invention patent application "A multi-time scale optimal decision-making method for energy system management based on deep reinforcement learning" (CN 112488452 A) and the paper "Dynamic demand microgrid energy management based on deep reinforcement learning" (Nakabi TA, Toivanen P. Deeper reinforcement learning for energy management in a microgrid with flexible demand [J]. Sustainable Energy, Grids and Networks, 2021, 25: 100413.). Among them,

[0004] The Chinese invention patent application publication specification CN 112488452 A, published on March 12, 2021, is "An optimal decision-making method for multi-time scales of energy system management based on deep reinforcement learning". It uses a long short-term memory neural network training system model to predict the output power of the photovoltaic energy storage battery group and the power required by the load two steps in advance, and then uses a deep reinforcement learning method to generate the optimal action decision for the charging and discharging actions of the energy storage battery group. However, this method has the following shortcomings:

[0005] 1) First, the output power of the photovoltaic energy storage battery group and the power required by the load are predicted two steps in advance by using a long short-term memory artificial neural network; then the deep Q neural network is used to generate the optimal action decision for the charging and discharging actions of the energy storage battery group. The energy storage battery group model established by this method takes a long time to train, the computational complexity is greatly increased, and the efficiency is low;

[0006] 2) When this method uses deep reinforcement learning to generate the optimal action decision for the charging and discharging actions of the energy storage battery pack, it uses the target evaluation network to obtain the action value of all actions in the state, and then calculates the target value based on the optimal action value. Due to the maximization operation here, the algorithm has an "overestimation" problem;

[0007] 3) The charging and discharging operations of energy storage battery packs in actual energy systems are usually affected by the energy interactions of other energy systems, so the energy system has limitations;

[0008] In the 2021 paper "Energy management of dynamic demand microgrids based on deep reinforcement learning" (Nakabi TA, Toivanen P. Deep reinforcement learning for energy management in a microgrid with flexible demand [J]. Sustainable Energy, Grids and Networks, 2021, 25: 100413.), the authors studied the performance of seven deep reinforcement learning algorithms to enhance the energy management system of microgrids. By adding experience replay and semi-deterministic training mechanisms to the asynchronous advantage actor-critic algorithm (A3C), better model performance and convergence to near-optimal strategies were obtained. However, this method has the following shortcomings:

[0009] 1) In the control mechanism of this method, the use of random exploration of energy trading actions and energy storage charging and discharging actions will result in low economic benefits of the microgrid energy system;

[0010] 2) In the control mechanism of this method, the charging and discharging operations of the energy storage battery pack can also be achieved through energy interaction with the main grid or the adjacent microgrid;

[0011] 3) In the price-responsive load model, the authors used a simple model to describe the responsiveness of residential loads to electricity prices, which can be replaced by an intelligent agent to learn the optimal strategy while considering the trade-off between comfort and electricity costs under the consideration of price uncertainty. Summary of the invention

[0012] The purpose of the present invention is to solve the problem of low economic efficiency of existing complex microgrid energy systems and "over-estimation" of related optimization algorithms, and further provide a microgrid energy system optimization method based on real-time electricity price incentives, to provide the best economic benefit solution for microgrid operation, so as to maximize the economic benefit of the microgrid energy system.

[0013] In order to achieve the purpose of the present invention, the present invention provides a microgrid energy system optimization method based on real-time electricity price incentives, wherein the microgrid energy system includes a main power grid, a wind turbine generator set, an energy storage battery group, N constant temperature control loads, M price response loads and a bus L M , wherein the wind turbine generator set is connected to the busbar L through a transmission line M Bidirectional connection, energy storage battery group through the transmission line and bus L M Bidirectional connection, N constant temperature control loads are connected to the bus L through N transmission lines M , the price response loads are connected to the busbar L through M transmission lines M , busbar L M connected to the main power grid through a transformer; the constant temperature control load refers to the load affected by the ambient temperature, and the price response load refers to the load affected by the electricity price, wherein N and M are both positive integers;

[0014] The optimization method comprises the following steps:

[0015] Step 1: Establish a mathematical model for the energy storage battery pack

[0016] The microgrid energy system is recorded as the system;

[0017] The current time of the system is recorded as t, t = 0, 1, 2...T, T is the time when the system is terminated, then the dynamic storage energy of the energy storage battery group at the current time t is recorded as the current dynamic storage energy B t , the dynamic energy storage of the energy storage battery group at time t-1 is recorded as the previous dynamic energy storage B t-1 , B t ∈(0,B max ), B max is the maximum capacity of the energy storage battery pack, and the current dynamic energy storage B t The expression is as follows:

[0018]

[0019] Where: η c is the charging efficiency coefficient of the energy storage battery pack, η d is the discharge efficiency coefficient of the energy storage battery pack, c t is the current charging power of the energy storage battery pack, c t ∈(0,C max), C max is the maximum charging power of the energy storage battery pack, D t is the current discharge power of the energy storage battery pack, D t ∈(0, D max ), D max is the maximum discharge power of the energy storage battery pack;

[0020] The state of charge of the energy storage battery pack at the current time t is recorded as the current state of charge B sct , establish the mathematical model of the energy storage battery pack and record it as model 1, and its expression is as follows:

[0021]

[0022] Step 2: Establish a mathematical model for the constant temperature control load

[0023] Any one of the N constant temperature control loads is recorded as constant temperature control load Γ i , set the constant temperature control load Γ i The current load consumption is recorded as the current constant temperature load power i=1,2...N;establish constant temperature load power The simplified mathematical model is recorded as Model 2, and its expression is as follows:

[0024]

[0025] Among them, P tcl is the constant temperature control load Γ i The nominal power, is the constant temperature control load Γ i The current working gear is recorded as the current working gear And determined according to the following formula:

[0026]

[0027] In the formula, is the current air temperature, is the lower limit of the given air temperature, is the upper limit of the set air temperature, 0, 1 is the current working position action, 0 means turning off the constant temperature control load Γ i , 1 represents the current constant temperature control load Γ i With nominal power P tcl run, and Indicates the current constant temperature control load Γ i by run, Indicates the current constant temperature control load Γ i by run;

[0028] Set the N current constant temperature control load power The sum is recorded as the total power of the current constant temperature control load Its expression is as follows:

[0029]

[0030] Step 3: Build a mathematical model for price-responsive loads

[0031] Let any one of the M price response loads be denoted as price response load γ γ , γ=1,2...M;the price response load γ γ The current load consumption is recorded as the current price load power The price response load γ γ The current electricity price is recorded as the current price response load electricity price p t , determined according to the following formula:

[0032] p t =p m +δ t

[0033] In the formula, p m is the constant temperature control load electricity price, δ t For price response load γ γ The current price level is denoted as the current response price level δ t , δ t ∈{-3, -1.5, 0, 1.5, 3}, {-3, -1.5, 0, 1.5, 3} is the current price level δ t The five actions of δ t or equal to -3 or equal to -1.5 or equal to 0 or equal to 1.5 or equal to 3;

[0034] Establish current price load power The simplified mathematical model is recorded as Model 3, and its expression is as follows:

[0035]

[0036] Among them, L b，t is the current base load; is the current transfer load, β γ is the load factor; F t γ is the sum of the loads transferred up to the current time t, j is any time, o j γ is the transfer load at time j, wγ，j is the price response load γ from any time j to the current time t γ The transfer state variable of

[0037] Set M current price load power The sum is recorded as the total power of the current price load Its expression is as follows:

[0038]

[0039] Step 4: Obtaining system input data

[0040] Assume that the extraction time range is one year and the extraction interval is one hour, and extract the historical data of the power grid in a certain area within one year. The historical data of the power grid includes wind power generation data G, electricity purchase price data p b , electricity selling price data s , air temperature data T', the extracted 8760 wind power generation data G form the wind power generation data set A1, the extracted 8760 electricity purchase price data p b The electricity purchase price dataset B1 and the 8760 electricity selling price data p extracted are s The electricity price dataset C1 is composed, and the extracted 8760 air temperature T' data constitute the air temperature dataset D1;

[0041] Step 5: Determine the state set S, action set A0 and reward function r t

[0042] According to the model 1, model 2, model 3 obtained in step 1, step 2, step 3 and the input data of the microgrid energy system obtained in step 4, determine the state set S, the action set A0 and the reward function R;

[0043] Step 5.1, determine the state set S

[0044] The state of the system at the current time t is recorded as the current state s t , In the formula, G t is the power generated by the system at the current time t, denoted as the current power generation G t , is the electricity purchase price of the system at the current time t, recorded as the current electricity purchase price is the electricity selling price of the system at the current time t, recorded as the current electricity selling price

[0045] The state set S is T current states s t The set of S = {s1, s2, ...st ,..s T},and

[0046] Step 5.2, determine the action set A0

[0047] The action taken by the system at the current time t is recorded as the current action a t , a t =(a t1 , a t2 , a t3 , a t4 ) t ,

[0048] In the formula, a t1 is the constant temperature control load Γ i The four working gear actions are recorded as the priority action a of the constant temperature control load t1 ;

[0049] a t2 For price response load γ γ The five price level actions of the price response load are recorded as the price level action a t2 ;

[0050] a t3 When there is excess electricity, it is determined whether to sell electricity to the main grid or store electricity in the energy storage battery pack, which is recorded as action a when there is excess energy. t3 ;

[0051] a t4 When there is a power shortage, it is determined to buy power from the main grid or discharge the energy storage battery pack, which is recorded as action a when there is a power shortage. t4 ;

[0052] Introduce action set A0, A0 = {a1, a2, ...a t , ..a T}, and A0∈{(a t1 , a t2 , a t3 , a t4 )};

[0053] Step 5.3, determine the reward function r t

[0054] Reward function r t It represents the net profit obtained by the system at the current time t, and its expression is as follows:

[0055]

[0056] In the formula, is the amount of electricity sold to the main grid at the current time t, pcost The cost of generating electricity for a wind turbine, The current time t is to buy electricity from the main grid, The transmission cost of imported energy from the main grid to the system, the transmission costs for exporting energy from the system to the main grid;

[0057] Step 6: Determine the real-time electricity price incentive mechanism

[0058] The current real-time electricity price incentive mechanism control instruction is recorded as control instruction u p，t , which is expressed as follows:

[0059]

[0060] Where P th is a given electricity price difference threshold;

[0061] The action after applying the real-time electricity price incentive mechanism at the current moment is recorded as the post-regulation action Its expression is as follows:

[0062]

[0063] In the formula, G l is the system power generation G t The power remaining after the system requirements are met is determined by: G l >0 means the system has excess energy, G l ≤0 means the system energy is insufficient, a t3 &u p,t Indicates action a when energy is in excess t3 With control instructions u p,t Perform an "AND" operation, a t4 ||u p,t Indicates action a when energy is in short supply t4 With control instructions u p,t Perform an "or" operation;

[0064] Step 7: Offline learning of Dueling Double DQN algorithm

[0065] Randomly extract D states s from the state set S t The training data set for offline learning is D = 4T / 5; the state set S, action set A0, and reward function r obtained in step 6 t The real-time electricity price incentive mechanism obtained in step 6 is used to perform offline learning using the Dueling Double DQN algorithm of deep reinforcement learning to obtain the optimal strategy π(s y );

[0066] The Dueling Double DQN algorithm includes two neural networks, namely an online evaluation network and a target evaluation network. The neural network parameter of the online evaluation network is denoted by the first neural network parameter θ Q , the neural network parameters of the target evaluation network are recorded as the second neural network parameters θ Q ,;

[0067] Given the training step size step and the maximum step size step max , given the number of training rounds m and the maximum number of training rounds F, step = 1, 2, 3…step max , m = 1, 2, 3...F, that is, each training round contains step max Training times, a total of F training rounds;

[0068] Define the total reward value R in each training round and record it as the daily reward R. During each training round m, the first neural network parameter θ Q , the second neural network parameter θ Q′ All are updated in the direction of maximizing the daily reward R, and finally the optimal strategy π(s y );

[0069] The optimal strategy π(s y ) is as follows:

[0070] π(s y )=a y

[0071] In the formula, s y is the state value of the online evaluation network input corresponding to the optimal strategy, and is the total power of the current constant temperature control load corresponding to the optimal strategy in the state set S Current price response load total power Current price response load electricity price p t 、Current state of charge B sct 、Current power generation G t 、Current electricity purchase price and current electricity prices a y is the action value output by the online evaluation network corresponding to the optimal strategy, denoted as the optimal action a y , and a y =(a t1 , a t2 , a t3 , a t4 ) y , (a t1, a t2 , a t3 , a t4 ) y is the action set A0 with the optimal strategy π(s y ) The priority action a of the corresponding constant temperature control load t1 、Price response load price level action a t2 、Action a when energy is in excess t3 and energy shortage action a t4 ;

[0072] Output the optimal action a y ;

[0073] Step 8, optimal action a y Test

[0074] First, select the state s outside the training data set from the state set S t Recompose a test data set and then randomly extract j from the test data set max States t and redefined as the test state s β , β=1,2,3...j max , test status That is, the test status s β is the total power of the current constant temperature control load Total power of price-responsive load Current price response load electricity price p t 、Current state of charge B sct 、Current power generation G t 、Current electricity purchase price and current electricity prices

[0075] The test state set s β As the input of the online evaluation network trained in step 7, the reward function r obtained in step 5 t , using the Dueling Double DQN algorithm of deep reinforcement learning for offline learning, the system selects the optimal action a y , reward function r t Converge quickly, in any state in the state set S The optimal economic benefits of the microgrid energy system are achieved.

[0076] Preferably, in step 7, the Dueling Double DQN algorithm of deep reinforcement learning is used for offline learning to obtain the optimal strategy π(s y ) are as follows:

[0077] Step 7.1, initialize the first neural network parameter θ Q and the second neural network parameter θ Q′ , and let θ Q′ =θ Q ; Initialize the capacity of the experience replay pool P to V; Initialize the learning rate α and decay coefficient ρ of the online evaluation network, and 0<α<1, 0<ρ≤1; Let the output of the online evaluation network be a, a=μ(s;θ Q ), where a is the action value output by the online evaluation network, a corresponds to the individual in the action set A0, and a=(a t1 , a t2 , a t3 , a t4 );s is the state value input by the online evaluation network, s corresponds to the individual in the state set S, and μ is the first neural network parameter θ that passes the online evaluation network Q , the strategy obtained by inputting the state value s and the real-time electricity price incentive mechanism;

[0078] Step 7.2, the current state s t Input the online evaluation network and get the current output of the online evaluation network And get the final output of the current action a t , the specific expression is as follows:

[0079]

[0080] Step 7.3, the system is based on the current state s t Execute the current action a t , transition to the new state s t+1 , and get the execution of the current action a t The single-step reward value r t , will (s t , a t , r t ,s t+1 ) is called the state transition sequence, and (s t , a t , r t ,s t+1 ) is stored in the experience replay pool P, and the system enters the state s at the next time t+1 t+1 ;

[0081] Loop through steps 7.2 to 7.3, record the number of state transition sequences in the experience replay pool P as E, if E = V, go to step 7.4, otherwise return to step 7.2;

[0082] Step 7.4, randomly extract n state transition sequences from the experience replay pool P, and n<V, use the n state transition sequences as small batch data for training the online evaluation network, and record the kth state transition sequence in the small batch data as (s k , a k , r k ,s k+1 ), k=1, 2, 3, ... n;

[0083] Step 7.5, according to the small batch data (s) obtained in step 8.4 k , a k , r k ,s k+1 ), k = 1, 2, 3...n, calculate the cumulative reward y k and the error function L(θ Q ), the specific expression is as follows:

[0084]

[0085]

[0086] In the formula, is the score value output by the target evaluation network, where Output the action value with the highest score from the online evaluation network, s′ k+1 The state value of the target evaluation network input; Q(s k+1 , a k+1 θ Q ) is the rating value output by the online evaluation network, s k+1 and a k+1 The state value and action value input to the online evaluation network;

[0087] Step 7.6, the online evaluation network minimizes the error function L(θ Q ) uses the stochastic gradient descent algorithm to update θ Q , the specific expression is as follows:

[0088]

[0089]

[0090] In the formula, is the symbol of partial derivative, where Represents the minimization error function L(θ Q ) for θ Q Find the partial derivative, Indicates that the input of the online evaluation network is s k and a k When , the cumulative reward y is calculated kand the rating value Q(s) output by the online evaluation network k , a k θ Q )The square value of the difference is related to the first network parameter θ Q Find the partial derivative, is the first neural network parameter after update;

[0091] remember is the updated second neural network parameter;

[0092] When steps 7.4 to 7.5 are completed once, a step-length training process is completed. max , repeat steps 7.4 to 7.5. When 24 steps of training are completed, a round of training is completed and the reward R for one day is calculated. When step = step max season

[0093] Step 7.7: When steps 7.4 to 7.6 are completed once, the next round of training starts from step 7.2 and ends at step 7.6. When m < F, steps 7.2 to 7.6 are repeated. When m = F, when F rounds of training are completed, the learning process of the Dueling Double DQN algorithm ends.

[0094] Step 7.8, the training algorithm ends and the optimal strategy π(s) is saved y )=a y , let the reward of a training step be r t ;

[0095] In F training rounds, the first neural network parameter θ Q and the second neural network parameter θ Q′ Update in the direction of maximizing the daily reward R, and finally obtain the optimal strategy π(s y ).

[0096] The present invention provides a microgrid energy system optimization method based on real-time electricity price incentives, which improves the economic benefits of the microgrid energy system. Its beneficial effects are specifically reflected in:

[0097] 1. For the first time, we consider using real-time buying and selling electricity prices to incentivize exploration of energy trading actions and energy storage battery charging and discharging actions. Compared with random exploration methods, this method improves the net profit of the microgrid energy system. Through the introduction of the real-time electricity price incentive mechanism, in the case of energy surplus, energy storage battery groups are prioritized for energy storage, and the excess is sold; in the case of energy shortage, the system prioritizes the discharge of energy storage batteries, and then buys electricity when energy is insufficient; this makes the complex microgrid energy system more flexible in the reasonable dispatch of energy, and maximizes the economic benefits of microgrid energy management;

[0098] 2. The online evaluation network and target evaluation network designed by the present invention use independent action value estimates Q (0) (.) and Q (1) (.), so E[Q (1) (φ t+1 , A * )]=Q(φ t+1 , A * ), where A * = argmax a Q (0) (φ t+1 , a), that is, using the online evaluation network to obtain the action corresponding to the optimal action value in the current state, and then using the target evaluation network to calculate the action value of the action, so as to obtain the target value. At the same time, in the process of dual learning, Q (0) (.) and Q (1) (.) The network parameters are gradually updated, which eliminates the estimation bias and effectively avoids the "over-estimation" problem of the algorithm, thereby improving the estimation accuracy of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] Figure 1 is a topological diagram of a microgrid energy system in an embodiment of the present invention;

[0100] Figure 2 It is a flow chart of a microgrid energy system optimization method based on real-time electricity price incentives in the implementation of the present invention;

[0101] Figure 3 Convergence effect diagram of one-day reward in an embodiment of the present invention;

[0102] Figure 4 It is a diagram showing the effect of the charge state change during the action variable training balance in an embodiment of the present invention;

[0103] Figure 5 This is a diagram showing the effect of energy transaction volume and buying and selling electricity price changes when the action variable training is balanced in an embodiment of the present invention. DETAILED DESCRIPTION

[0104] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings.

[0105] Figure 1 : is a topological diagram of a microgrid energy system in an embodiment of the present invention. As can be seen from the diagram, the microgrid energy system of the present invention includes a main power grid, a wind turbine generator set, an energy storage battery set, N constant temperature control loads, M price response loads and a bus L M , wherein the wind turbine generator set is connected to the busbar L through a transmission line M Bidirectional connection, energy storage battery group through the transmission line and bus L M Bidirectional connection, N constant temperature control loads are connected to the bus L through N transmission lines M , the price response loads are connected to the busbar L through M transmission lines M , busbar L M The load is connected to the main power grid through a transformer. The constant temperature control load refers to the load affected by the ambient temperature, and the price response load refers to the load affected by the electricity price, wherein N and M are both positive integers.

[0106] In this embodiment, N=150, M=100. In the figure, the wind turbine generator set, the energy storage battery set, N constant temperature control loads, M price response loads and the bus L M It is also called microgrid system.

[0107] Figure 2 : is a flow chart of a microgrid energy system optimization method based on real-time electricity price incentives in the implementation of the present invention. As can be seen from the figure, the optimization method includes the following steps:

[0108] The microgrid energy system is denoted as system.

[0109] Step 1: Establish a mathematical model for the energy storage battery pack

[0110] The current time of the system is recorded as t, t = 0, 1, 2...T, T is the time when the system is terminated, then the dynamic storage energy of the energy storage battery group at the current time t is recorded as the current dynamic storage energy B t , the dynamic energy storage of the energy storage battery group at time t-1 is recorded as the previous dynamic energy storage B t-1 , B t ∈(0,B max ), B max is the maximum capacity of the energy storage battery pack, and the current dynamic energy storage B t The expression is as follows:

[0111]

[0112] Where: η c is the charging efficiency coefficient of the energy storage battery pack, η dis the discharge efficiency coefficient of the energy storage battery pack, c t is the current charging power of the energy storage battery pack, c t ∈(0,C max ), C max is the maximum charging power of the energy storage battery pack, D t is the current discharge power of the energy storage battery pack, D t ∈(0, D max ), D max is the maximum discharge power of the energy storage battery pack;

[0113] The state of charge of the energy storage battery pack at the current time t is recorded as the current state of charge B sct , establish the mathematical model of the energy storage battery pack and record it as model 1, and its expression is as follows:

[0114]

[0115] In this embodiment, B max =500 kW, η c =0.9,η d =0.9, C max =250 kW, D max =250 kW.

[0116] Step 2: Establish a mathematical model for the constant temperature control load

[0117] Any one of the N constant temperature control loads is recorded as constant temperature control load Γ i , set the constant temperature control load Γ i The current load consumption is recorded as the current constant temperature load power Establish constant temperature load power The simplified mathematical model is recorded as Model 2, and its expression is as follows:

[0118]

[0119] Among them, P tcl is the constant temperature control load Γ i The nominal power, is the constant temperature control load Γ i The current working gear is recorded as the current working gear And determined according to the following formula:

[0120]

[0121] In the formula, is the current air temperature, is the lower limit of the given air temperature, is the upper limit of the set air temperature, 0, 1 is the current working position action, 0 means turning off the constant temperature control load Γ i , 1 represents the current constant temperature control load Γ i With nominal power P tcl run, and Indicates the current constant temperature control load Γ i by run, Indicates the current constant temperature control load Γ i by run;

[0122] Set the N current constant temperature control load power The sum is recorded as the total power of the current constant temperature control load Its expression is as follows:

[0123]

[0124] In this embodiment, p tcl = 1.5 kW, That is, constant temperature control load Γ i The nominal power is 1.5 kilowatts, the lower limit of the air temperature is 19 degrees Celsius, and the upper limit of the air temperature is 25 degrees Celsius. The system selects constant temperature control load Γ i Current working gear like The system selects constant temperature control load Γ i Current working gear

[0125] Step 3: Build a mathematical model for price-responsive loads

[0126] Let any one of the M price response loads be denoted as price response load γ γ , γ=1,2...M;the price response load γ γ The current load consumption is recorded as the current price load power The price response load γ γ The current electricity price is recorded as the current price response load electricity price p t , determined according to the following formula:

[0127] p t =p m +δ t

[0128] In the formula, p m is the constant temperature control load electricity price, δ t For price response load γ γ The current price level is denoted as the current response price level δt , δ t ∈{-3, -1.5, 0, 1.5, 3}, {-3, -1.5, 0, 1.5, 3} is the current price level δ t The five actions of δ t or equal to -3 or equal to -1.5 or equal to 0 or equal to 1.5 or equal to 3;

[0129] Establish current price load power The simplified mathematical model is recorded as Model 3, and its expression is as follows:

[0130]

[0131] Among them, L b，t is the current base load; is the current transfer load, β γ is the load factor; F t γ is the sum of the loads transferred up to the current time t, j is any time, o j γ is the transfer load at time j, w γ，j is the price response load γ from any time j to the current time t γ The transfer state variable of

[0132] Set M current price load power The sum is recorded as the total power of the current price load Its expression is as follows:

[0133]

[0134] In this embodiment, the constant temperature control load electricity price p m is a fixed value, p m = 5.48 EUR / MW; load factor at any time β γ Take a normal distribution random number with a mean of 0.4 and a variance of 0.3, denoted as The price response load γ from any time j to the current time t γ The transition state variable w γ，j Randomly take 0 or 1, that is, w γ，j ∈{0, 1}; Current base load L at each time of the day b，t The values ​​are shown in Table 1:

[0135] Table 1

[0136]

[0137] Step 4: Obtaining system input data

[0138] Assume that the extraction time range is one year and the extraction interval is one hour, and extract the historical data of the power grid in a certain area within one year. The historical data of the power grid includes wind power generation data G, electricity purchase price data p b , electricity selling price data s , air temperature data T', the extracted 8760 wind power generation data G form the wind power generation data set A1, the extracted 8760 electricity purchase price data p b The electricity purchase price dataset B1 and the 8760 electricity selling price data p extracted are s The electricity price dataset C1 and the 8760 air temperature T' data extracted constitute the air temperature dataset D1.

[0139] Step 5: Determine the state set S, action set A0 and reward function r t

[0140] According to the models 1, 2, and 3 obtained in steps 1, 2, and 3 and the input data of the microgrid energy system obtained in step 4, the state set S, the action set A0, and the reward function R are determined.

[0141] Step 5.1, determine the state set S

[0142] The state of the system at the current time t is recorded as the current state s t , In the formula, G t is the power generated by the system at the current time t, denoted as the current power generation G t , is the electricity purchase price of the system at the current time t, recorded as the current electricity purchase price is the electricity selling price of the system at the current time t, recorded as the current electricity selling price

[0143] The state set S is T current states s t The set of S = {s1, s2, ...s t ,..s T},and

[0144] Step 5.2, determine the action set A0

[0145] The action taken by the system at the current time t is recorded as the current action a t , a t =(a t1 , a t2 , a t3, a t4 ) t ,

[0146] In the formula, a t1 is the constant temperature control load Γ i The four working gear actions are recorded as the priority action a of the constant temperature control load t1 ;

[0147] a t2 For price response load γ γ The five price level actions of the price response load are recorded as the price level action a t2 ;

[0148] a t3 When there is excess electricity, it is determined whether to sell electricity to the main grid or store electricity in the energy storage battery pack, which is recorded as action a when there is excess energy. t3 ;

[0149] a t4 When there is a power shortage, it is determined to buy power from the main grid or discharge the energy storage battery pack, which is recorded as action a when there is a power shortage. t4 ;

[0150] Introduce action set A0, A0 = {a1, a2, ...a t , ..a T}, and A0∈{(a t1 , a t2 , a t3 , a t4 )}.

[0151] Step 5.3, determine the reward function r t

[0152] Reward function r t It represents the net profit obtained by the system at the current time t, and its expression is as follows:

[0153]

[0154] In the formula, is the amount of electricity sold to the main grid at the current time t, p cost The cost of generating electricity for a wind turbine, The current time t is to buy electricity from the main grid, The transmission cost of imported energy from the main grid to the system, is the transmission cost of exporting energy from the system to the main grid.

[0155] Step 6: Determine the real-time electricity price incentive mechanism

[0156] The current real-time electricity price incentive mechanism control instruction is recorded as control instruction u p，t, which is expressed as follows:

[0157]

[0158] Where P th is a given electricity price difference threshold;

[0159] The action after applying the real-time electricity price incentive mechanism at the current moment is recorded as the post-regulation action Its expression is as follows:

[0160]

[0161] In the formula, G l is the system power generation G t The power remaining after the system requirements are met is determined by: G l >0 means the system has excess energy, G l ≤0 means the system energy is insufficient, a t3 &u p，t Indicates action a when energy is in excess t3 With control instructions u p，t Perform an "AND" operation, a t4 ||u p，t Indicates action a when energy is in short supply t4 With control instructions u p，t Perform an "OR" operation.

[0162] In this embodiment, the current electricity purchase price The price used is the price of the main grid energy market to increase the bid in order to balance power generation and consumption in real time. The increase means that energy production needs to be increased or consumption needs to be reduced. The current selling price is The main grid energy market uses the price of bids to balance power generation and consumption in real time, and a reduction means increasing consumption or reducing energy production.

[0163] In this embodiment, p cost =32, That is, the cost of wind turbine power generation cost The transmission cost of energy imported from the main grid to the system is 32 euros / MW, and the transmission cost of energy exported from the system to the main grid is 9.7 euros / MW. It is 0.9 euros / MW.

[0164] In this embodiment, P th =0.5, that is, the electricity price difference threshold is set at 0.5 euros / MW.

[0165] Step 7: Offline learning of Dueling Double DQN algorithm

[0166] Randomly extract D states s from the state set S t The training data set for offline learning is D = 4T / 5; the state set S, action set A0, and reward function r obtained in step 6 t The real-time electricity price incentive mechanism obtained in step 6 is used to perform offline learning using the Dueling Double DQN algorithm of deep reinforcement learning to obtain the optimal strategy π(s y );

[0167] The Dueling Double DQN algorithm includes two neural networks, namely an online evaluation network and a target evaluation network. The neural network parameter of the online evaluation network is denoted by the first neural network parameter θ Q , the neural network parameters of the target evaluation network are recorded as the second neural network parameters θ Q′ ;

[0168] Given the training step size step and the maximum step size step max , given the number of training rounds m and the maximum number of training rounds F, step = 1, 2, 3…step max , m = 1, 2, 3...F, that is, each training round contains step max Training times, a total of F training rounds;

[0169] Define the total reward value R in each training round and record it as the daily reward R. During each training round m, the first neural network parameter θ Q , the second neural network parameter θ Q′ All are updated in the direction of maximizing the daily reward R, and finally the optimal strategy π(s y );

[0170] The optimal strategy π(s y ) is as follows:

[0171] π(s y )=a y

[0172] In the formula, s y is the state value of the online evaluation network input corresponding to the optimal strategy, and is the total power of the current constant temperature control load corresponding to the optimal strategy in the state set S Current price response load total power Current price response load electricity price p t 、Current state of charge B sct 、Current power generation G t 、Current electricity purchase price and current electricity prices a y is the action value output by the online evaluation network corresponding to the optimal strategy, denoted as the optimal action a y , and a y =(a t1 , a t2 , a t3 , a t4 ) y , (a t1 , a t2 , a t3 , a t4 )y is the action set A0 with the optimal strategy π(s y ) The priority action a of the corresponding constant temperature control load t1 、Price response load price level action a t2 、Action a when energy is in excess t3 and energy shortage action a t4 ;

[0173] Output the optimal action a y .

[0174] Step 8, optimal action a y Test

[0175] First, select the state s outside the training data set from the state set S t Recompose a test data set and then randomly extract j from the test data set max States t and redefined as the test state s β , β=1,2,3...j max , test status That is, the test status s β is the total power of the current constant temperature control load Total power of price-responsive load Current price response load electricity price p t 、Current state of charge B sct 、Current power generation G t 、Current electricity purchase price and current electricity prices

[0176] The test state set s β As the input of the online evaluation network trained in step 7, the reward function r obtained in step 5 t , using the Dueling Double DQN algorithm of deep reinforcement learning for offline learning, the system selects the optimal action a y , reward function r tConverge quickly, in any state in the state set S The optimal economic benefits of the microgrid energy system are achieved.

[0177] In this embodiment, D = 7000, V = 500, n = 200, α = 0.00025, ρ = 1, step max =100, F=250.

[0178] In the above optimization process, the Dueling Double DQN algorithm of deep reinforcement learning is used for offline learning as described in step 7 to obtain the optimal strategy π(s y ) are as follows:

[0179] Step 7.1, initialize the first neural network parameter θ Q and the second neural network parameter θ Q′ , and let θ Q′ =θ Q ; Initialize the capacity of the experience replay pool P to V; Initialize the learning rate α and decay coefficient ρ of the online evaluation network, and 0<α<1, 0<ρ≤1; Let the output of the online evaluation network be a, a=μ(s;θ Q ), where a is the action value output by the online evaluation network, a corresponds to the individual in the action set A0, and a=(a t1 , a t2 , a t3 , a t4 );s is the state value input by the online evaluation network, s corresponds to the individual in the state set S, and μ is the first neural network parameter θ that passes the online evaluation network Q , the strategy obtained by inputting the state value s and the real-time electricity price incentive mechanism;

[0180] Step 7.2, the current state s t Input the online evaluation network and get the current output of the online evaluation network And get the final output of the current action a t , the specific expression is as follows:

[0181]

[0182] Step 7.3, the system is based on the current state s t Execute the current action a t , transition to the new state s t+1 , and get the execution of the current action a t The single-step reward value r t , will (s t , a t , r t ,st+1 ) is called the state transition sequence, and (s t , a t , r t ,s t+1 ) is stored in the experience replay pool P, and the system enters the state s at the next time t+1 t+1 ;

[0183] Loop through steps 7.2 to 7.3, record the number of state transition sequences in the experience replay pool P as E, if E = V, go to step 7.4, otherwise return to step 7.2;

[0184] Step 7.4, randomly extract n state transition sequences from the experience replay pool P, and n<V, use the n state transition sequences as small batch data for training the online evaluation network, and record the kth state transition sequence in the small batch data as (s k , a k , r k ,s k+1 ), k=1, 2, 3, ... n;

[0185] Step 7.5, according to the small batch data (s k , a k , r k ,s k+1 ), k = 1, 2, 3...n, calculate the cumulative reward y k and the error function L(θ Q ), the specific expression is as follows:

[0186]

[0187]

[0188] In the formula, is the score value output by the target evaluation network, where Output the action value with the highest score from the online evaluation network, s′ k+1 The state value of the target evaluation network input; Q(s k+1 , a k+1 θ Q ) is the rating value output by the online evaluation network, s k+1 and a k+1 The state value and action value input to the online evaluation network;

[0189] Step 7.6, the online evaluation network minimizes the error function L(θ Q ) uses the stochastic gradient descent algorithm to update θ Q , the specific expression is as follows:

[0190]

[0191]

[0192] In the formula, is the symbol of partial derivative, where Represents the minimization error function L(θ Q ) for θ Q Find the partial derivative, Indicates that the input of the online evaluation network is s k and a k When , the cumulative reward y is calculated k and the rating value Q(s) output by the online evaluation network k , a k θ Q )The square value of the difference is related to the first network parameter θ Q Find the partial derivative, is the first neural network parameter after update;

[0193] remember is the updated second neural network parameter;

[0194] When steps 7.4 to 7.5 are completed once, a step-length training process is completed. max , repeat steps 7.4 to 7.5. When 24 steps of training are completed, a round of training is completed and the reward R for one day is calculated. When step = step max season

[0195] Step 7.7: When steps 7.4 to 7.6 are completed once, the next round of training starts from step 7.2 and ends at step 7.6. When m < F, steps 7.2 to 7.6 are repeated. When m = F, when F rounds of training are completed, the learning process of the Dueling Double DQN algorithm ends.

[0196] Step 7.8, the training algorithm ends and the optimal strategy π(s) is saved y )=a y , let the reward of a training step be r t ;

[0197] In F training rounds, the first neural network parameter θ Q and the second neural network parameter θ Q′ Update in the direction of maximizing the daily reward R, and finally obtain the optimal strategy π(s y ).

[0198] In order to prove the beneficial effects of the present invention, the present invention was verified.

[0199] Figure 3 This is a graph showing the convergence effect of one-day rewards in an embodiment of the present invention. Figure 3 The horizontal axis is the number of training rounds m, and the vertical axis is the daily reward R, m = 1, 2, 3...250. Figure 3 It can be seen that with the increase of the number of training rounds m, the daily reward R first oscillates up and down, then gradually increases and finally remains between 70 and 80. When m = 250, R = 74.87751122997476, the training effect has reached the optimal value, and the neural network parameters θ of the online evaluation network and the target evaluation network are Q ,θ Q′ The update has been completed, and the optimal strategy π(s y ).

[0200] Figure 4 and Figure 5 These are two effect diagrams obtained when the action variables are trained to balance in an embodiment of the present invention. Figure 4 In the figure, the horizontal axis is time t and the vertical axis is the state of charge B. sct , t=0, 2, 3…23. Figure 5 In the figure, the horizontal axis is time t, the left vertical axis is energy trading volume, and the right vertical axis is buying and selling electricity price, t = 0, 2, 3...23; Figure 4 , Figure 5 It can be seen that from time 0 to 5, the microgrid energy system gives priority to charging the energy storage battery group when there is excess energy. After the energy storage battery group is fully charged, the remaining energy is sold during the time period with the highest electricity price from time 5 to 10; from time 12 to 23, the microgrid energy system gives priority to supplying power to the energy storage battery group when there is insufficient energy. After the energy storage battery group is powered up, it avoids buying a large amount of energy from the main grid during the time period with the highest electricity price from time 15 to 18. By rationally dispatching the energy in the system, the economic benefits of the microgrid energy system are maximized.

Claims

1. A microgrid energy system optimization method based on real-time electricity price incentives, wherein the microgrid energy system includes a main power grid, a wind turbine generator set, an energy storage battery group, N constant temperature control loads, M price response loads and a bus L M ,in, The wind turbine generator set is connected to the busbar L through a transmission line. M Bidirectional connection, energy storage battery group through the transmission line and bus L M Bidirectional connection, N constant temperature control loads are connected to the bus L through N transmission lines M , the price response loads are connected to the busbar L through M transmission lines M , busbar L M connected to the main power grid through a transformer; the constant temperature control load refers to the load affected by the ambient temperature, and the price response load refers to the load affected by the electricity price, wherein N and M are both positive integers; Characterized in that the optimization method comprises the following steps: The microgrid energy system is recorded as the system; Step 1: Establish a mathematical model for the energy storage battery pack The current time of the system is recorded as t, t = 0, 1, 2...T, T is the time when the system is terminated, then the dynamic storage energy of the energy storage battery group at the current time t is recorded as the current dynamic storage energy B t , the dynamic energy storage of the energy storage battery group at time t-1 is recorded as the previous dynamic energy storage B t-1 , B t ∈(0,B max ), B max is the maximum capacity of the energy storage battery pack, and the current dynamic energy storage B t The expression is as follows: Where: η c is the charging efficiency coefficient of the energy storage battery pack, η d is the discharge efficiency coefficient of the energy storage battery pack, c t is the current charging power of the energy storage battery pack, c t ∈(0,C max ), C max is the maximum charging power of the energy storage battery pack, D t is the current discharge power of the energy storage battery pack, D t ∈(0,D max ), D max is the maximum discharge power of the energy storage battery pack; The state of charge of the energy storage battery pack at the current time t is recorded as the current state of charge B sct , establish the mathematical model of the energy storage battery pack and record it as model 1, and its expression is as follows: Step 2: Establish a mathematical model for the constant temperature control load Any one of the N constant temperature control loads is recorded as constant temperature control load Γ i , set the constant temperature control load Γ i The current load consumption is recorded as the current constant temperature load power i=1,2...N;establish constant temperature load power The simplified mathematical model is recorded as Model 2, and its expression is as follows: Among them, P tcl is the constant temperature control load Γ i The nominal power, is the constant temperature control load Γ i The current working gear is recorded as the current working gear And determined according to the following formula: In the formula, is the current air temperature, is the lower limit of the given air temperature, is the upper limit of the set air temperature, 0, 1 is the current working position action, 0 means turning off the constant temperature control load Γ i , 1 represents the current constant temperature control load Γ i With nominal power P tcl run, and Indicates the current constant temperature control load Γ i by run, Indicates the current constant temperature control load Γ i by run; Set the N current constant temperature control load power The sum is recorded as the total power of the current constant temperature control load Its expression is as follows: Step 3: Build a mathematical model for price-responsive loads Let any one of the M price response loads be denoted as price response load γ γ , γ=1,2...M;the price response load γ γ The current load consumption is recorded as the current price load power The price response load γ γ The current electricity price is recorded as the current price response load electricity price p t , determined according to the following formula: p t =p m +d t In the formula, p m is the constant temperature control load electricity price, δ t For price response load γ γ The current price level is denoted as the current response price level δ t , δ t ∈{-3,-1.5,0,1.5,3}, {-3,-1.5,0,1.5,3} is the current price level δ t The five actions of δ t or equal to -3 or equal to -1.5 or equal to 0 or equal to 1.5 or equal to 3; Establish current price load power The simplified mathematical model is recorded as Model 3, and its expression is as follows: Among them, L b,t is the current base load; is the current transfer load, β γ is the load factor; F t γ is the sum of the loads transferred up to the current time t, j is any time, o j γ is the transfer load at time j, w γ，j is the price response load γ from any time j to the current time t γ The transfer state variable of Set M current price load power The sum is recorded as the total power of the current price load Its expression is as follows: Step 4: Obtaining system input data Assume that the extraction time range is one year and the extraction interval is one hour, and extract the historical data of the power grid in a certain area within one year. The historical data of the power grid includes wind power generation data G, electricity purchase price data p b , electricity selling price data s , air temperature data T', the extracted 8760 wind power generation data G form the wind power generation data set A1, the extracted 8760 electricity purchase price data p b The electricity purchase price dataset B1 and the 8760 electricity selling price data p extracted are s The electricity price dataset C1 is composed, and the extracted 8760 air temperature T' data constitute the air temperature dataset D1; Step 5: Determine the state set S, action set A0 and reward function r t According to the model 1, model 2, model 3 obtained in step 1, step 2, step 3 and the input data of the microgrid energy system obtained in step 4, determine the state set S, action set A0 and reward function r t ; Step 5.1, determine the state set S The state of the system at the current time t is recorded as the current state s t , In the formula, G t is the power generated by the system at the current time t, denoted as the current power generation G t , is the electricity purchase price of the system at the current time t, recorded as the current electricity purchase price is the electricity selling price of the system at the current time t, recorded as the current electricity selling price The state set S is T current states s t The set of S = {s1, s2, ...s t ,..s T },and Step 5.2, determine the action set A0 The action taken by the system at the current time t is recorded as the current action a t , a t =(a t1 , a t2 , a t3 , a t4 ) t , In the formula, a t1 is the constant temperature control load Γ i The four working gear actions are recorded as the priority action a of the constant temperature control load t1 ; a t2 For price response load γ γ The five price level actions of the price response load are recorded as the price level action a t2 ; a t3 When there is excess electricity, it is determined whether to sell electricity to the main grid or store electricity in the energy storage battery pack, which is recorded as action a when there is excess energy. t3 ; a t4 When there is a power shortage, it is determined to buy power from the main grid or discharge the energy storage battery pack, which is recorded as action a when there is a power shortage. t4 ; Introduce action set A0, A0 = {a1, a2, ...a t , ..a T }, and A0∈{(a t1 , a t2 , a t3 , a t4 )}; Step 5.3, determine the reward function r t Reward function r t It represents the net profit obtained by the system at the current time t, and its expression is as follows: In the formula, is the amount of electricity sold to the main grid at the current time t, p cost The cost of generating electricity for a wind turbine, The current time t is to buy electricity from the main grid, The transmission cost of imported energy from the main grid to the system, the transmission costs for exporting energy from the system to the main grid; Step 6: Determine the real-time electricity price incentive mechanism The current real-time electricity price incentive mechanism control instruction is recorded as control instruction u p，t , which is expressed as follows: Where P th is the given electricity price difference threshold; The action after applying the real-time electricity price incentive mechanism at the current moment is recorded as the post-regulation action Its expression is as follows: In the formula, G l is the system's power generation G t The power remaining after the system requirements are met is determined by: G l >0 means the system has excess energy, G l ≤0 means the system energy is insufficient, a t3 &u p,t Indicates action a when energy is in excess t3 With control instructions u p,t Perform an "AND" operation, a t4 ||u p，t Indicates action a when energy is in short supply t4 With control instructions u p，t Perform an "or" operation; Step 7: Offline learning of Dueling Double DQN algorithm Randomly extract D states s from the state set S t The training data set for offline learning is D = 4T / 5; the state set S, action set A0, and reward function r obtained in step 6 t The real-time electricity price incentive mechanism obtained in step 6 is used to perform offline learning using the Dueling Double DQN algorithm of deep reinforcement learning to obtain the optimal strategy π(s y ); The Dueling Double DQN algorithm includes two neural networks, namely an online evaluation network and a target evaluation network. The neural network parameter of the online evaluation network is denoted by the first neural network parameter θ Q , the neural network parameters of the target evaluation network are recorded as the second neural network parameters θ Q′ ; Given the training step size step and the maximum step size step max , given the number of training rounds m and the maximum number of training rounds F, step = 1, 2, 3…step max , m = 1, 2, 3...F, that is, each training round contains step max training times, with a total of F training rounds; define the total reward value R in each training round and record it as the daily reward R, During each training round m, the first neural network parameter θ Q , the second neural network parameter θ Q′ All are updated in the direction of maximizing the daily reward R, and finally the optimal strategy π(s y ); The optimal strategy π(s y ) is as follows: π(s y )=a y In the formula, s y is the state value of the online evaluation network input corresponding to the optimal strategy, and is the total power of the current constant temperature control load corresponding to the optimal strategy in the state set S Current price response load total power Current price response load electricity price p t 、Current state of charge B sct 、Current power generation G t 、Current electricity purchase price and current electricity prices a y is the action value output by the online evaluation network corresponding to the optimal strategy, denoted as the optimal action a y , and a y =(a t1 , a t2 , a t3 , a t4 ) y , (a t1 , a t2 , a t3 , a t4 ) y is the action set A0 with the optimal strategy π(s y ) The priority action a of the corresponding thermostatic control load t1 、Price response load price level action a t2 、Action a when energy is in excess t3 and energy shortage action a t4 ; Output the optimal action a y ; Step 8, optimal action a y Test First, select the state s outside the training data set from the state set S t Recompose a test data set and then randomly extract j from the test data set max States t and redefined as the test state s β , β=1,2,3...j max , test status That is, the test status s β is the total power of the current constant temperature control load Total power of price-responsive load Current price response load electricity price p t 、Current state of charge B sct 、Current power generation G t 、Current electricity purchase price and current electricity prices The test state set s β As the input of the online evaluation network trained in step 7, the reward function r obtained in step 5 t , using the Dueling Double DQN algorithm of deep reinforcement learning for offline learning, the system selects the optimal action a y , reward function r t Converge quickly, in any state in the state set S The optimal economic benefits of the microgrid energy system are achieved.

2. According to claim 1, a microgrid energy system optimization method based on real-time electricity price incentives is characterized in that: Step 7 uses the Dueling Double DQN algorithm of deep reinforcement learning to perform offline learning and obtain the optimal strategy π(s y ) are as follows: Step 7.1, initialize the first neural network parameter θ Q and the second neural network parameter θ Q′ , and let θ Q′ =θ Q ; Initialize the capacity of the experience replay pool P to V; Initialize the learning rate α and decay coefficient ρ of the online evaluation network, and 0<α<1, 0<ρ≤1; Let the output of the online evaluation network be a, a=μ(s;θ Q ), where a is the action value output by the online evaluation network, a corresponds to the individual in the action set A0, and a=(a t1 , a t2 , a t3 , a t4 );s is the state value input by the online evaluation network, s corresponds to the individual in the state set S, and μ is the first neural network parameter θ that passes the online evaluation network Q , the strategy obtained by inputting the state value s and the real-time electricity price incentive mechanism; Step 7.2, the current state s t Input the online evaluation network and get the current output of the online evaluation network And get the final output of the current action a t , the specific expression is as follows: Step 7.3, the system is based on the current state s t Execute the current action a t , transition to the new state s t+1 , and get the execution of the current action a t The single-step reward value r t , will (s t , a t , r t ,s t+1 ) is called the state transition sequence, and (s t , a t , r t ,s t+1 ) is stored in the experience replay pool P, and the system enters the state s at the next time t+1 t+1 ; Loop through steps 7.2 to 7.3, record the number of state transition sequences in the experience replay pool P as E, if E = V, go to step 7.4, otherwise return to step 7.2; Step 7.4, randomly extract n state transition sequences from the experience replay pool P, and n<V, use the n state transition sequences as small batch data for training the online evaluation network, and record the kth state transition sequence in the small batch data as (s k , a k , r k ,s k+1 ), k = 1, 2, 3, ... n; Step 7.5, according to the small batch data (s) obtained in step 8.4 k , a k , r k ,s k+1 ), k = 1, 2, 3...n, calculate the cumulative reward y k and the error function L(θ Q ), the specific expression is as follows: In the formula, is the score value output by the target evaluation network, where Output the action value with the highest score from the online evaluation network, s′ k+1 The state value of the target evaluation network input; Q(s k+1 , a k+1 θ Q ) is the rating value output by the online evaluation network, s k+1 and a k+1 The state value and action value input to the online evaluation network; Step 7.6, the online evaluation network minimizes the error function L(θ Q ) uses the stochastic gradient descent algorithm to update θ Q , the specific expression is as follows: In the formula, is the symbol of partial derivative, where Represents the minimization error function L(θ Q ) for θ Q Find the partial derivative, Indicates that the input of the online evaluation network is s k and a k When , the cumulative reward y is calculated k and the rating value Q(s) output by the online evaluation network k , a k θ Q )The square value of the difference is related to the first network parameter θ Q Find the partial derivative, is the first neural network parameter after update; remember is the updated second neural network parameter; When steps 7.4 to 7.5 are completed once, a step-length training process is completed. max , repeat steps 7.4 to 7.

5. When 24 steps of training are completed, a round of training is completed and the reward R for one day is calculated. When step = step max season Step 7.7: When steps 7.4 to 7.6 are completed once, the next round of training starts from step 7.2 and ends at step 7.

6. When m < F, steps 7.2 to 7.6 are repeated. When m = F, when F rounds of training are completed, the learning process of the Dueling Double DQN algorithm ends. Step 7.8, the training algorithm ends and the optimal strategy π(s) is saved y )=a y , let the reward of a training step be r t ; In F training rounds, the first neural network parameter θ Q and the second neural network parameter θ Q′ Update in the direction of maximizing the daily reward R, and finally obtain the optimal strategy π(s y ).

Images