Deep reinforcement learning based economic dispatch method based on pre-training and knowledge guidance

Abstract

The application belongs to the cross field of deep reinforcement learning and economic dispatch of power system, and particularly relates to a deep reinforcement learning economic dispatch method based on pre-training and knowledge guidance, which comprises the following steps: 1, obtaining high-quality experience data and storing the data in an experience replay pool; 2, constructing a reinforcement learning training environment; 3, training an initial strategy network in advance based on a supervised learning pre-training method; and 4, retraining an Actor-Critic network based on knowledge guidance. The application trains the Actor network in the agent in advance before the agent and the environment interact, so that blind training of the agent in the initial stage of interaction with the environment is avoided; meanwhile, expert knowledge is embedded in the training process of the agent, the search of the agent is limited in the safe operation area of the power system, and the agent is guided to promote new energy consumption.

Description

TECHNICAL FIELD

[0001] The application belongs to the cross field of deep reinforcement learning and economic dispatch of power system, and particularly relates to a deep reinforcement learning economic dispatch method based on pre-training and knowledge guidance. BACKGROUND

[0002] Reinforcement learning obtains feedback and updates the strategy by constantly interacting with the environment, and this environment interaction mode can be described by Markov Decision Process (MDP). Reinforcement learning is mainly divided into two categories: value-based algorithm and policy-based algorithm. Among them, the Deep Deterministic Policy Gradient (DDPG) is a policy-based algorithm that can solve continuous control problems. The DDPG algorithm uses a policy network (Actor) to provide control actions and a value function network (Critic) to evaluate the effect of control actions. DDPG uses a deep neural network to approximate the behavior value function, which can well solve the continuous control problem.

[0003] The training effect of conventional reinforcement learning algorithms such as DDPG is sensitive to the number of actions. When the power grid scale is large, the decision action space is high-dimensional and continuous, and due to the large action exploration space, the agent convergence speed is slow, and even the agent action exploration mechanism fails to find the optimal strategy. Therefore, it is necessary to improve the convergence and convergence speed of the DDPG algorithm in handling large-scale intraday dispatch problems, and avoid model convergence failure due to ineffective gradient training. SUMMARY

[0004] The application provides a deep reinforcement learning economic dispatch method based on pre-training and knowledge guidance, which comprises the following steps:

[0005] Step 1: Obtain high-quality experience data and store it in an experience replay pool;

[0006] Step 2: Construct a reinforcement learning training environment;

[0007] Step 3: Pre-train the initial policy network based on the supervised learning pre-training method;

[0008] Step 4: Retrain the Actor-Critic network based on knowledge guidance.

[0009] Preferably, the step 1 is specifically as follows:

[0010] When the number of dispatch cycle time periods is S NThe unit operation cost and the new energy curtailment rate function in the daily scheduling are respectively:

[0011]

[0012]

[0013] The daily economic dispatch model can be expressed as:

[0014]

[0015] wherein S T represents a set of generator units; S R represents a set of new energy units; represents the output of unit i at time t; a, b, and c represent cost characteristic coefficients; represents the upper limit of the output of a unit; the upper limit relaxation constraint of the output of a unit is:

[0016]

[0017] In the above formula, the first line is the power balance constraint; the second line and the third line are respectively the upper and lower limit constraints of the output and the regulation rate constraint, is the lower limit of the output of a unit, represents the output of unit i at time t-1, and are respectively the maximum downward and upward regulation rates of unit i, and Δt is the regulation period; the fourth line and the fifth line are the line capacity constraints, ΔL k,t is the active power flow increment of line k at time t; L k,t-1 is the active power flow of line k at time t-1, represents the capacity limit of a line, S D is a set of load nodes, is the load increment of node l, S ik represents the sensitivity of the injection power of node i to line k; by adding the relaxation constraint of the output limit of a unit in the daily economic dispatch model, the MIP problem is converted into a linear programming problem, and a commercial solver IPOPT is called to solve, so as to obtain the unit power generation power distribution scheme a t under the historical state s t ; these schemes usually can obtain a higher reward value r t ; these data (s t , a t , r t , s t+1 ) are stored in an experience replay pool for later supervised learning pre-training.

[0018] Preferably, the step 2 specifically comprises:

[0019] Step 2.1: Define the action space, the action space set a of the agent t For:

[0020] a t = [a t,1 , a t,2 , a t,3 ,..., a t,nG ] (5)

[0021] Step 2.2: Define the observation space, the grid state observation s t Including: the current period unit active power P t G , the upper limit of the current period unit active power adjustment value and the lower limit P t and the next period load active power forecast value and the branch current load rate That is:

[0022]

[0023] Step 2.3: Define the reward value, the immediate reward value generated after the agent interacts with the environment in the tth time period in each round is defined as:

[0024] r(t) = c1r1 + c2r2 (7)

[0025]

[0026]

[0027] Preferably, the step 3 specifically comprises the following steps:

[0028] Step 3.1: Establish a neural network As an initial policy network, its network parameters and the Actor network in the agent are consistent, given the state s t , the unit power generation allocation scheme a' t can be obtained through the initial policy network, and its expression is:

[0029]

[0030]

[0031] Step 3.2: To evaluate the effect of the initial policy network learning experience in the replay pool of unit dispatching experience, define the loss value of the tth stage as:

[0032]

[0033] In the above formula: and With the change of a t -a' t1 a t represents the unit commitment scheme in the experience replay pool;a' t represents the unit commitment scheme predicted by the neural network ;

[0034] Step 3.3: Use the state value function to represent the total loss that can be obtained from the initial state S t by executing the policy π i to the final state S T , which is expressed as follows:

[0035]

[0036] In the above formula: the discount factor γ∈[0,1] represents the importance of future rewards to current rewards;

[0037] Step 3.4: Randomly extract N sample data from the experience replay pool, take the state s t as the input of the neural network, and take the unit power generation allocation scheme a t as the target output value, and optimize the initial policy network parameters to maximize The optimization objective is as follows:

[0038]

[0039] Preferably, the step 4 specifically comprises the following steps:

[0040] Step 4.1: Initialize the initial policy network parameters based on supervised learning to the Actor network parameters θ μ in the agent;

[0041] Step 4.2: According to the minimum deviation criterion, update the Critic function network as follows:

[0042]

[0043]

[0044] Step 4.3: Introduce a regularization term based on expert experience knowledge in the loss function of the Actor network, and the regularization term includes power balance and new energy consumption reward, so the loss value of the Actor network is composed of the action value function and two regularization terms, as shown in the following formula:

[0045]

[0046]

[0047]

[0048] In the formula: theta μ is the Actor network parameter; theta Q is the Critic network parameter; w q w1 and w2 represent the regularization term weight; reg1 is the power balance regularization term, representing the square of the difference between the total load and the total output of the unit; reg2 is the new energy consumption regularization term, representing the new energy consumption rate;

[0049] When the Actor-Critic network is retrained, the Actor network can be continuously updated according to the following formula:

[0050]

[0051] Step 4.4: Update of the target network, the update of the target network is carried out according to the following formula, wherein tau is a small update coefficient.

[0052]

[0053] The advantage of the present application is that the proposed method trains the Actor network in the agent in advance before the agent and the environment interact, avoiding the blind training of the agent in the initial stage of interaction with the environment; at the same time, expert knowledge is embedded in the training process of the agent, limiting the search of the agent within the safe operation area of the power system, and guiding the agent to promote new energy consumption. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 is a schematic diagram of the deep reinforcement learning economic dispatch method based on pre-training and knowledge guidance;

[0055] Figure 2 is the initial policy network training error curve;

[0056] Figure 3 is the training effect diagram of the conventional DDPG method;

[0057] Figure 4 is the training effect diagram of the DDPG algorithm based on knowledge guidance;

[0058] Figure 5 is the training effect diagram of the SL-DDPG algorithm based on knowledge guidance. DETAILED DESCRIPTION

[0059] The present application will be further described in detail below with reference to the accompanying drawings.

[0060] like Figure 1 As shown, the deep reinforcement learning-based economic scheduling method based on pre-training and knowledge guidance includes the following steps:

[0061] Step 1: Obtain high-quality experience data and store it in the experience replay pool. The formula for solving the intraday economic scheduling model is as follows:

[0062]

[0063] The unit output limit relaxation constraint is:

[0064]

[0065] By incorporating relaxed constraints on unit output limits into the intraday economic dispatch model, the MIP problem is transformed into a linear programming problem, which is then solved using the commercial solver IPOPT to obtain the historical state s. t The following is a power generation allocation scheme for the generating units: t and reward value r t , will these data (s t ,a t ,r t ,s t+1 ) are stored in the experience replay pool for later supervised learning pre-training;

[0066] Step 2: Construct a reinforcement learning training environment.

[0067] Step 2.1: Action Space. The action space set a of the agent. t for:

[0068] a t =[a t,1 ,a t,2 ,a t,3 ,...,a t,nG ] (twenty four)

[0069] Step 2.2: Observation Space. Power Grid State Observations s t for:

[0070]

[0071] Step 2.3: Reward Value. In each round, the immediate reward value generated after interacting with the environment in the t-th time interval is:

[0072] r(t)=c1r1+c2r2 (26)

[0073]

[0074]

[0075] Step 3: Pre-train the initial policy network in advance based on the supervised learning pre-training method.

[0076] Step 3.1: Establish a neural network As the initial policy network, its network parameters are consistent with the actor network in the agent. Given the state s t , the unit generation power allocation scheme a' t is obtained through the initial policy network, that is:

[0077]

[0078]

[0079] Step 3.2: Calculate the loss value of the t-th stage in the mini-batch:

[0080]

[0081] Step 3.3: Calculate the state value function

[0082]

[0083] Step 3.4: Randomly extract N sample data from the experience replay pool, take the state s t as the input of the neural network, and take the unit generation power allocation scheme a t as the target output value, and optimize the initial policy network parameters to maximize The optimization objective is as follows:

[0084]

[0085] Step 4: Retrain the Actor-Critic network based on knowledge guidance.

[0086] Step 4.1: Initialize the initial policy network parameters based on supervised learning to the actor network parameters θ μ in the agent.

[0087] Step 4.2: Update the Critic network parameters. Initialize the environment to obtain the initial state s t , use the actor network to obtain the action a t = π(s| θ μ ) + ε, and interact with the environment to obtain the next time state s t+1 and the reward r. The data (s t , a t , r t , st+1 ) are stored in the experience replay pool B. J experience samples are randomly extracted from the experience replay pool B. The target value is calculated for each transition experience in the mini-batch.

[0088] The Critic function network is updated according to the minimization bias criterion as follows:

[0089] y i = r i + γQ(s i+1 , π(s i+1 | θ μ ') | θ Q ') (34)

[0090]

[0091] Step 4.3: Knowledge-guided Actor network parameter update. The Actor network can be continuously updated as follows:

[0092]

[0093]

[0094] Step 4.4: Update of the target network. The update of the target network is as follows, where τ is a small update coefficient.

[0095]

[0096] In order for those skilled in the art to better understand the present application and understand the advantages of the present application over the prior art, the applicant further explains in conjunction with specific embodiments.

[0097] An example analysis is carried out taking the modified IEEE 118 node system as an example. The modified IEEE 118 node system has a total of 54 units, of which 36 are conventional units and 18 are new energy units; with a control interval of 5 minutes, the data set constructed contains 100,000 alternating current flow sections for one year; this data set meets the needs of multiple scenarios, with typical scenarios of power grid operation such as tie-line congestion, source and load fluctuation, and new energy power limiting. The test case is implemented in Python language based on the Pytorch framework, the computer hardware conditions are Core i7-1165 CPU, 2.8 GHz, and the number of algorithm iteration events (dispatching period) K is 1.2 x 10 3 times.

[0098] Considering that the state and action space dimensions are 438 and 54, respectively, the Actor network and the Critic network are both set to have 4 layers of neurons, with the number of hidden layer neurons being 1024, 512 and 256, respectively. Except that the activation function of the last layer of the Actor network is a tanh function, the other neural layers of the Actor network and the Critic network all use a ReLU activation function. In addition, the training of the neural network is affected by hyperparameters. Different hyperparameters are suitable for different power grid scales, and a set of parameters with better training effect is selected, as shown in Table 1.

[0099] Table 1 Parameter selection

[0100]

[0101] Improvement of supervised learning pre-training on training effect

[0102] The method uses the loss value L(S) and the round reward value r(t) to evaluate the effect of the initial policy network learning the unit scheduling experience. To avoid the contingency of a single experiment, 10 experimental results are randomly selected, and the error curve with a confidence interval is plotted as shown in Figure 2 The statistical results show that when the initial policy network training reaches stability, the average training time is 0.4 h, the error value is stabilized at about 4%, and the average round reward value is 389.1. The error value is small, and the average score per round is high, which indicates that the initial policy network based on supervised learning pre-training can fully learn the historical scheduling experience. Initializing the parameters of the initial policy network as the parameters of the Actor network in the agent can make the Actor network obtain better initial performance, avoid blind training of the policy network in the early stage, and improve the training speed.

[0103] Influence of knowledge guidance on training effect

[0104] It can be seen from Figure 3 , Figure 4 , Figure 5 that as the training progresses, the conventional DDPG algorithm is difficult to converge, and the scheduling effect is poor; the global reward value of the DDPG algorithm based on knowledge guidance and the SL-DDPG method based on knowledge guidance gradually converges, and the scheduling effect continuously improves. This shows that: ① For learning tasks with high-dimensional and continuous action space, the conventional DDPG method has a large action exploration space, and it is difficult to explore the strategy and has low learning efficiency, which easily leads to convergence failure. ② The introduction of expert experience knowledge can guide the learning of the agent, reduce the action search space, and improve the convergence speed and convergence.

[0105] Comprehensive evaluation of algorithm scheduling results

[0106] In order to verify the effectiveness of the method, five methods shown in Table 2 are used for comparative analysis. The results show that: when the IPOPT optimization solver is used to solve the scheduling plan based on linear programming, the generator output is adjusted based on the principle of minimum adjustment amount, and not enough units are started to provide backup to deal with wind power uncertainty, so the operation cost is the lowest; the SL-DDPG method based on knowledge guidance can dynamically respond to the random changes of source and load, and the new energy consumption rate is the highest among all methods, reaching 97.8%; in addition, in terms of solving efficiency, the SL-DDPG method is significantly faster than the IPOPT method.

[0107] Table 2 Comprehensive evaluation effect comparison table

[0108]

[0109] The above examples have described the technical solutions of the present application in detail. Apparently, the present application is not limited to the described embodiments. Based on the embodiments in the present application, those skilled in the art can make various changes, but any change equivalent or similar to the present application belongs to the protection scope of the present application.

Claims

1. A pre-training and knowledge-guided based deep reinforcement learning economic dispatch method, characterized in that, The method comprises the following steps: Step 1: obtaining high-quality experience data and storing the experience data in an experience replay pool; The step 1 specifically comprises the following steps: In the day-ahead scheduling with the number of dispatching cycle time periods The unit operation cost and the wind curtailment rate function are respectively （ ） （ ） Then, the intraday economic dispatch model can be expressed as: （ ） wherein, denotes a set of generator units; denotes a set of new energy units; denotes a unit at the time t; , , denotes a cost characteristic coefficient; denotes an output upper limit of a unit; the output upper limit of a unit is relaxed as: （ ） In the above formula, the first row is the power balance constraint; the second and third rows are the upper and lower output constraints and the regulation rate constraint, respectively, is the lower output limit of the unit, represents the output of the unit at time , and are the maximum downward and upward regulation rates of the unit , respectively, is the regulation period; the fourth and fifth rows are the line capacity constraints, is the active power flow increment of the line at time ; is the active power flow of the line at time , represents the capacity limit of the line, is the set of load nodes, is the load increment of the node , represents the sensitivity of the injected power of the node to the line ; by adding the relaxation constraint of the unit output limit to the intra-day economic dispatch model, the MIP problem is converted into a linear programming problem, and the unit generation power distribution scheme under the historical state can be obtained by solving ; these schemes usually can obtain higher reward values , and these data are stored in the experience replay pool for later supervised learning pre-training; Step 2: constructing a reinforcement learning training environment; Step 3: pre-training an initial policy network based on a supervised learning method; Step 4: re-training the Actor-Critic network based on knowledge guidance; The step 4 specifically comprises the following steps: Step 4.1 : Initialize the supervised learning based initial policy network parameters Initialize the actor network parameters in the agent ; Step 4.2: updating the Critic function network according to a minimum deviation criterion, as shown in the following formula: （15） （16） Step 4.3: introducing a regularization term based on expert experience knowledge into the loss function of the Actor network, and the regularization term comprises power balance and new energy consumption reward, so that the loss value of the Actor network is composed of an action value function and two regularization terms, as shown in the following formula: （17） （18） （19） In the formula: is the Actor network parameter; is the Critic network parameter; , and denote the regularization term weight; is the power balance regularization term, representing the square of the difference between the total load and the total output of the unit; is the new energy consumption regularization term, representing the new energy consumption rate; When the Actor-Critic network is re-trained, the Actor network can be continuously updated according to the following formula: （20） Step 4.4: updating the target network, and the target network is updated according to the following formula: （21） wherein is a smaller update coefficient.

2. The pre-training and knowledge-guided based deep reinforcement learning economic dispatch method according to claim 1, characterized in that, The step 2 specifically comprises: Step 2.1: Define action space, the set of action spaces for the agent For: （5） Step 2.2: Define the observation space, grid state observations including: current period unit active power , upper limit of current period unit active power adjustment value and lower limit and next period load active power forecast value and branch current load rate i.e.: （6） Step 2.3: Define reward value, in each episode, the instant reward value generated after the first time period of interaction with the environment is defined as: （7） （8） （9）。 3. The pre-training and knowledge-guided based deep reinforcement learning economic dispatch method according to claim 1, characterized in that, The step 3 specifically comprises the following steps: Step 3.1: Establish a neural network As the initial policy network, its network parameters are consistent with the Actor network in the agent, given the state The unit power generation allocation scheme can be obtained through the initial policy network The expression is: （10） （11） Step 3.2: To evaluate the effect of the initial policy network learning experience replay pool of unit commitment experience, define the loss value of the first phase as: （12） In the above formula: and As The value changes; represents the unit commitment scheme in the experience replay pool; represents the unit commitment scheme predicted by the neural network predicted by the neural network Step 3.3: Using the state value function to represent the total loss obtainable from the initial state executing the policy to the final state The expression for the total loss obtainable is as follows: （13） wherein: discount factor represents the importance of future rewards to the current reward. Step 3.4: Randomly extract one sample data from the experience replay pool , state as input to the neural network, unit generation power allocation scheme as target output value, optimize initial policy network parameters by training to maximize , optimization goal as follows: （14）。

Images