An intelligent solving algorithm for unit commitment based on deep reinforcement learning

Abstract

The application discloses an intelligent solving algorithm for unit commitment based on deep reinforcement learning, which comprises the following steps: step 1, performing Markov decision process (MDP) modeling oriented to unit commitment; step 2, performing solving of the MDP model; step 3, performing MDP model solving based on a policy gradient algorithm; and step 4, solving an actual power output scheme of a unit through Lambda iteration; and the intelligent solving algorithm for unit commitment based on deep reinforcement learning is obtained through steps 1 to 4. The application aims to solve the technical problems of low adaptability and low decision precision of a conventional model based on deep learning and constructed for a UC decision field, and provides the intelligent solving algorithm for unit commitment based on deep reinforcement learning.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of electric power system and automation, and particularly relates to an intelligent unit commitment solving algorithm based on deep reinforcement learning. BACKGROUND

[0002] With the rapid development of China's electric power industry, the scale of the power system is increasingly large, and the solution of the unit commitment (UC) problem is increasingly complex and difficult. At the same time, new types of power equipment such as electric vehicles and distributed power sources are connected to the power grid, and the user-side power load in the power system presents a flexible and variable characteristic. The traditional security constrained unit commitment (SCUC) decision-making method based on physical model driving (PMD) faces numerous theoretical and technical challenges. Although it has good physical interpretability, it has problems such as a long research cycle and low applicability. Therefore, under the background of rapid energy technology transformation and deep integration of artificial intelligence technology and the power system, it is of great significance to study an SCUC intelligent decision-making method with strong adaptability and high decision-making accuracy.

[0003] The traditional model based on deep learning (DL) constructed for the UC decision-making field belongs to a kind of supervised learning. Although it has high decision-making accuracy and strong adaptability, it also has some limitations. For example, the traditional data-driven (DD) SCUC decision-making method cannot comprehensively consider multiple influencing factors. The SCUC intelligent expert system based on E-Seq2Seq can fully consider the influence of multiple input factors on the SCUC decision-making result, thereby ensuring the accuracy of the decision-making result. However, this method is essentially a kind of supervised learning, and a large amount of high-quality historical data is needed to train the DL model. However, for the modern power market, although the operation department generally accumulates a large amount of historical decision-making data, it cannot be effectively guaranteed that these historical data are the optimal solution under the circumstances. On the other hand, for some specific scenarios, it is already difficult to obtain historical data.

[0004] Based on this, the application applies deep reinforcement learning (DRL) to the UC decision field and proposes a DRL-based intelligent solving algorithm for UC. Based on data driving, in order to reduce the high dependence of the model on label sample data and further improve the applicability, the DRL is applied to the UC field. The UC problem is calculated in two steps, the first step is to decide the start-stop state of the unit in each period, and the second step is to solve the output of the unit according to the state, which can effectively solve the small-scale UC problem. SUMMARY

[0005] The application aims to solve the technical problems that the conventional model based on deep learning constructed for the UC decision field has poor adaptability and low decision accuracy, and in a large-scale problem similar to unit combination decision, it is difficult to strictly and independently indicate that a certain behavior should be performed in a certain state, and it is difficult to solve the problem in a high-dimensional or continuous action space, and provides an intelligent solving algorithm for unit combination based on deep reinforcement learning.

[0006] In order to solve the above technical problems, the technical scheme adopted by the application is:

[0007] An intelligent solving algorithm for unit combination based on deep reinforcement learning, comprising the following steps:

[0008] Step 1: Markov decision process (MDP) modeling oriented to unit combination; when making a unit combination decision, only the current state is known, all historical information is no longer needed, the current state can determine the future, and the optimal strategy is learned step by step through simulation experiments;

[0009] Step 2: Solve the Markov decision process (MDP) model; the purpose is to maximize the total reward by taking appropriate actions considering the present and future rewards according to the existing state and the reward information of each state;

[0010] Step 3: MDP model solving based on policy gradient algorithm; based on the calculation of the optimal value of the target function to be optimized in decision theory, the strategy maps all time-series states to the optimal start-stop scheme of the unit to obtain the expected unit;

[0011] Step 4: Solve the actual power output scheme of the unit through Lambda iteration;

[0012] The intelligent solving algorithm for unit combination based on deep reinforcement learning is obtained through steps 1 to 4.

[0013] In step 1, when performing Markov decision process (MDP) modeling, the state space, action space, transition function and reward function are set respectively;

[0014] 1-1: Set state space, give the unit start / stop state at corresponding time according to given input unit start / stop time and load data, constitute state space; its mathematical expression is:

[0015] S={U t ,P L} (1)

[0016] In the formula, U t =[u 1,t ,u 2,t ,...u N,t ], wherein u i,t ≠0, indicates the set of unit start / stop time; P L indicates load demand data, since the goal of UC problem is to solve the unit dispatching plan with the lowest total cost according to given load demand under the condition of meeting each constraint, therefore the variable has a very important influence on the problem;

[0017] 1-2: Set action space, at any time, the possible action of each unit is start or stop, and the action space is the combination of all unit start / stop actions;

[0018] At any time, assuming that the system contains N units, the possible action of each unit is start or stop, therefore the action space is the combination of all unit start / stop actions, and the size of the action space is 2 N ; it is expressed as a binary array, namely

[0019] A t =[a 1,t ,a 2,t ,...a N,t ] (2)

[0020] In the formula: A t indicates the set of all unit start / stop action space; a i,t indicates the start / stop action state of the unit; when the action of the unit is start, a i,t =1; when the action of the unit is stop, a i,t =0. But the action needs to comply with the minimum start / stop time constraint of the unit; N indicates the number of all units in the system.

[0021] 1-3: Set transition function, when the MDP model decides the unit start / stop scheme according to the observed state information and obtains the reward value, the transition function will transfer from state s t to state s t according to the unit start / stop action a t+1 under the condition of meeting each constraint, and the state information associated in the present application is the continuous start / stop time u i,t;

[0022] The conversion function of the continuous start / stop time is:

[0023]

[0024] In the formula: u i,t+1 represents the conversion function of the continuous start / stop time of unit i at t+1; u i,t represents the conversion function of the continuous start / stop time of unit i at t; a i,t represents the possible action of each unit as start or stop, when the action of the unit is start, a i,t =1. When the action of the unit is stop, a i,t =0.

[0025] 1-4: Set the reward function, the goal of reinforcement learning RL is to maximize the reward obtained on the unit start / stop action path when the MDP model solves the unit combination UC problem; in the problem studied in the present application, the goal is to minimize the total operation cost of the system;

[0026] The expression is:

[0027] r t =-(F t +λ t ) (4)

[0028] Wherein

[0029]

[0030] In the formula: F t is the operation cost of the system at t; λ t is the penalty value of violating the operation constraint at t; N represents the number of all units in the system; P i is the active power output value of unit i; is the start cost of unit i; a, b, c represent parameters.

[0031] In step 2, the following sub-steps are included:

[0032] Step 2-1: Observe the state information, at each time t, the MDP model observes the state information r in the power system, i.e. the start / stop time of the N units at the current time, the load demand data;

[0033] Step 2-2: Decision scheme, select the optimal action r according to the state information, i.e. the unit start / stop scheme decided at the current time;

[0034] Step 2-3: Iterative solution, according to the start / stop scheme, the Lambda iterative solution is used to solve the economic dispatch problem, and the actual output power of the unit at the current time is obtained.

[0035] Step 2-4: Calculate the operation cost, calculate the system operation cost at the current moment according to the actual output power, which is part of the reward function;

[0036] Step 2-5: Update the start-stop strategy. In order to realize the minimization of the total operation cost of the system, the MDP model needs to continuously update the existing unit start-stop strategy in the continuous interaction with the system environment, and finally obtain the optimal unit start-stop strategy * .

[0037] In step 3, the following sub-steps are included:

[0038] Step 3-1: Parameterize the strategy of the unit start-stop scheme. The core idea of the policy gradient PG algorithm is to parameterize the strategy of the unit start-stop scheme, and by controlling the parameter weights, the unit start-stop scheme with the minimum operation cost is selected;

[0039] Step 3-2: Establish the objective function. The strategy of the unit start-stop scheme is represented by a parameterized neural network θ , and the objective function is established to maximize the expected reward value by adjusting the parameters θ;

[0040] Step 3-2-1: The specific unit start-stop scheme strategy can be described as a function containing parameters θ:

[0041] π θ (s t ,a t )=P(a t |s t ,θ)≈π(s t ,a t ) (6)

[0042] In the formula: π θ represents the unit start-stop scheme strategy; s t represents the state information in the power system at time t; a t represents the unit start-stop scheme decided at time t; π θ (s t ,a t ) represents a function about parameters θ, and its mapping is P(a t |s t ,θ).

[0043] Step 3-2-2: Represent the unit start-stop scheme strategy π θ with a parameterized neural network, and the objective function can be represented as adjusting the parameters θ to maximize the expected reward value, and its mathematical expression is:

[0044]

[0045] J1(θ) represents the expected reward value of the unit start-stop scheme strategy; J1(θ) represents the expected reward value of the unit start-stop scheme strategy; t is the sum of the operating cost and the penalty for violating constraints of the system at time t, where t = 0, 1, 2, L; π θ represents the unit start-stop scheme strategy; γ represents the attenuation coefficient of the policy iteration.

[0046] Step 3-3: Maximize the objective function, that is, search for a set of parameter vectors θ, so that the objective function is maximized. For the maximization problem, the gradient ascent algorithm is used to find the maximum value;

[0047] Step 3-3-1: Maximize the objective function, that is, search for a set of parameter vectors θ, so that the objective function is maximized. Generally, for the maximization problem, the gradient ascent algorithm is used to find the maximum value:

[0048]

[0049] θ represents the parameter vector; J1(θ) represents the expected reward value of the unit start-stop scheme strategy. * represents the final optimal solution of the parameter vector; α represents the step size of the gradient ascent movement; represents the gradient of the parameter θ; J1(θ) represents the expected reward value of the unit start-stop scheme strategy.

[0050] Step 3-3-2: Assume a one-step MDP, and use the gradient ascent algorithm for it. θ (s t ,a t ) represents a function about the parameter θ, and the mapping is P(a t |s t ,θ). It gives the reward value of the unit start-stop scheme a t in the state a t r t =r(s t ,a t ). Then the reward value obtained by selecting the unit start-stop scheme a t is π θ (s t ,a t )r(s t ,a t ), and the weighted reward in the state ∑ a∈A π θ (s t ,a t )r(s t ,a t ) is ∑ a∈A π θ (s ta t )r(s t ,a t ), which is derived as follows:

[0051]

[0052] where J1(θ) denotes the expected reward value of the unit commitment start-stop scheme strategy; the expected value of the unit commitment start-stop scheme reward value; denotes the weighted reward under state ∑ a∈A π θ (s t ,a t )r(s t ,a t ); denotes the state distribution in the strategy.

[0053] Step 3-3-3: the gradient is:

[0054]

[0055] where: denotes the gradient of the expected reward value of the unit commitment start-stop scheme strategy; denotes the weighted reward under state ∑ a∈A π θ (s t ,a t )r(s t ,a t ); denotes the state distribution in the strategy.

[0056] Step 3-3-4: assuming that the gradient is known, the score function is defined as using the likelihood ratio technique, and the relationship between them is as follows:

[0057]

[0058] where: denotes the gradient of the function with respect to the parameter θ; π θ (s t ,a t ) denotes the function with respect to the parameter θ; denotes the score function.

[0059] Therefore, formula (10) can be written as:

[0060]

[0061] where: denotes the gradient of the expected reward value of the unit commitment start-stop scheme strategy;

[0062] This represents the state distribution within the strategy;

[0063] Representing state ∑ a∈A π θ (s t ,a t )r(s t ,a t Weighted rewards under )

[0064] Represents the scoring function; r(s) t ,a t ) indicates that in state a t Unit start-up and shutdown plan a t The reward value.

[0065] Step 3-3-5: Restore the policy gradient to its expected form:

[0066]

[0067] In the formula: This represents the gradient of the expected reward value of the unit start-up and shutdown strategy; Represents the expectation of the policy gradient; Represents the scoring function; r(s) t ,a t ) indicates that in state a t Unit start-up and shutdown plan a t The reward value.

[0068] Steps 3-4: Minimize operating costs by selecting the optimal unit start-up and shutdown scheme to minimize system operating costs;

[0069] By selecting the optimal unit start-up and shutdown scheme To minimize the system's operating cost, the final result is:

[0070]

[0071] In the formula: This represents the gradient of the expected reward value of the unit start-up and shutdown strategy; Represents the expectation of the policy gradient; Represents the scoring function; Indicates that in state a t The optimal unit start-up and shutdown scheme a t The reward value.

[0072] Step 4 includes the following sub-steps:

[0073] Step 4-1: according to the state a solved in step 3-4 t The optimal unit start-stop scheme a t That is, the start-stop action state of the unit 24h is obtained, and the action obtained in step 3-4 is solved by Lambda iteration to obtain the unit scheduling scheme. First, the initial value of the unknown parameter λ is set, which aims to make the cost increment rate of all units equal and equal to the unknown parameter λ;

[0074] Step 4-2: calculate the actual output power P of all units in the on state at this time i ;

[0075] P i = P1+P2+L+P M ,M=1,2,L,n (15)

[0076] In the formula: P i represents the actual output power of all units in the on state; M represents the number of units in the on state.

[0077] Step 4-3: calculate the error value ε between the actual output power value of the unit and the load demand power from formula (16);

[0078]

[0079] In the formula: ε represents the error value between the actual output power value of the unit and the load demand; P Load represents the load demand power; P i represents the actual output power of all units in the on state.

[0080] Step 4-4: if the error value ε between the actual output power value of the unit and the load demand is greater than the allowable error, return to step 4-1 to reiterate and modify the value of the unknown parameter λ;

[0081] Step 4-5: if the error value ε between the actual output power value of the unit and the load demand is less than or equal to the allowable error, stop iteration and output the active power scheduling scheme of all coal-fired units.

[0082] An MDP model solving method based on a policy gradient algorithm, comprising the following steps:

[0083] Step S1: parameterize the policy of the unit start-stop scheme. The core idea of the policy gradient PG algorithm is to parameterize the policy of the unit start-stop scheme, and select the unit start-stop scheme with the minimum operation cost by controlling the parameter weight;

[0084] Step S2: establish a target function, and use a parameterized neural network to represent the unit start-stop scheme policy π θAnd establish the objective function, by adjusting the parameter θ, so that the expected reward value is maximum;

[0085] Step S3: maximize the objective function, the objective function is maximized, that is, a set of parameter vectors θ is searched, so that the objective function is maximum, for the maximization problem, the gradient ascent algorithm is used to find the maximum value;

[0086] Step S4: minimize the operating cost, by selecting the optimal unit start-stop scheme, the operating cost of the system is minimized;

[0087] By selecting the optimal unit start-stop scheme The operating cost of the system is minimized, and finally:

[0088]

[0089] In the formula, Indicates the gradient of the unit start-stop scheme strategy expected reward value; Indicates the expected value of the policy gradient; Indicates the score function; Indicates the reward value of the optimal unit start-stop scheme a t Under the state a t .

[0090] In step S2, the following sub-steps are included:

[0091] Step S2-1: the specific unit start-stop scheme strategy can be described as a function containing parameters θ:

[0092] π θ (s t ,a t )=P(a t |s t ,θ)≈π(s t ,a t ) (6)

[0093] In the formula: π θ Indicates the unit start-stop scheme strategy; s t Indicates the state information in the power system at time t; a t Indicates the unit start-stop scheme decided at time t; π θ (s t ,a t ) indicates the function about parameter θ, and its mapping is P(a t |s t ,θ);

[0094] Step S2-2: use the parameterized neural network to represent the unit start-stop scheme strategy π θThe objective function can then be expressed as adjusting the parameter θ to maximize the expected reward, and its mathematical expression is:

[0095]

[0096] In the formula: J1(θ) represents the expected reward value of the unit start-up and shutdown scheme strategy; This represents the expected reward value during the iterative process of the entire unit start-up and shutdown strategy; r t Let be the sum of the system's operating cost at time t and the penalty for violating constraints, where t = 0, 1, 2, ...; π θ γ represents the unit start-up and shutdown strategy; γ represents the attenuation coefficient of strategy iteration.

[0097] Step S3 includes the following sub-steps:

[0098] Step S3-1: Maximize the objective function, which means searching for a set of parameter vectors θ that maximizes the objective function. Typically, for maximization problems, the gradient ascent algorithm is used to find the maximum value.

[0099]

[0100] In the formula: θ * This represents the final optimal solution of the parameter vector; α represents the step size of the gradient ascent. J1(θ) represents the gradient of parameter θ; J1(θ) represents the expected reward value of the unit start-up and shutdown scheme strategy.

[0101] Step S3-2: Assuming a single-step MDP, apply the gradient ascent algorithm to it; π θ (s t ,a t P(a) represents a function with respect to parameter θ, and the mapping is P(a) t |s t ,θ); it is in state a t The following is a method for starting and stopping the generator unit. t The reward value obtained is r t =r(s t ,a t Then, select unit start-up and shutdown scheme a. t The reward value obtained is π θ (s t ,a t )r(s t ,a t ), in state ∑ a∈A π θ (s t ,a t )r(s t ,a t The weighted reward under ) is ∑a∈A π θ (s t ,a t )r(s t ,a t ), which is derived as follows:

[0102]

[0103] where J1(θ) represents the expected reward value of the unit commitment start-stop scheme strategy; the expected value of the unit commitment start-stop scheme reward value; represents the weighted reward under state ∑ a∈A π θ (s t ,a t )r(s t ,a t ); represents the state distribution in the strategy;

[0104] Step S3-3: the gradient is:

[0105]

[0106] where: represents the gradient of the expected reward value of the unit commitment start-stop scheme strategy; represents the weighted reward under state ∑ a∈A π θ (s t ,a t )r(s t ,a t ); represents the state distribution in the strategy;

[0107] Step S3-4: assuming that the gradient is defined as the score function, and the relationship between the two is as follows:

[0108]

[0109] where: represents the gradient of the function with respect to the parameter θ; π θ (s t ,a t ) represents the function with respect to the parameter θ; represents the score function;

[0110] Therefore, formula (10) can be written as:

[0111]

[0112] where: a gradient of the expected reward value of the unit commitment scheme strategy; a state distribution in the strategy; a state a∈A π θ (s t ,a t ) represents a weighted reward under the state a t ,a t ); a score function; r(s t ,a t ) represents a reward value of the unit commitment scheme a t under the state a t ;

[0113] Step S3-5: restore the strategy gradient to the expected form:

[0114]

[0115] In the formula: a gradient of the expected reward value of the unit commitment scheme strategy; an expectation of the strategy gradient; a score function; r(s t ,a t ) represents a reward value of the unit commitment scheme a t under the state a t .

[0116] The purpose of parameterization is to solve large-scale problems. In a large-scale problem similar to the unit commitment decision, it is not possible to strictly separate each unit state and indicate that a certain action should be performed under a certain state. Therefore, the strategy of the unit commitment scheme needs to be parameterized, and a small number of parameters are used to reasonably approximate the actual objective function. By adjusting these parameters, a better strategy can be obtained by using the parameterized strategy function, and the behavior generated according to the strategy will obtain more rewards. The advantage of this strategy gradient algorithm is that the established Markov decision process MDP model for unit commitment has better convergence, can effectively solve the high-dimensional or continuous action space problem of the unit commitment decision solving problem, and can learn a random strategy.

[0117] Compared with the prior art, the present application has the following technical effects:

[0118] 1) The DRL-based UC intelligent solving algorithm provided by the present application can effectively solve complex small-scale UC problems, and has high applicability;

[0119] 2) Compared with supervised learning, the present application does not need to construct a large number of labeled sample data in advance, avoids the dependence on sample data, and has high generalization performance;

[0120] 3) Compared with the traditional method, the application can directly give action decision through the strategy model of the model, and the solving efficiency is higher.

[0121] 4) The MDP model solving method based on the strategy gradient algorithm effectively solves the solving difficulty of large-scale problems, and the advantages based on the strategy make the model have better convergence, and can effectively solve the high-dimensional or continuous action space problem. BRIEF DESCRIPTION OF DRAWINGS

[0122] The application will be further described below in combination with the drawings and examples:

[0123] Figure 1 is the decision framework diagram of the UC intelligent solving algorithm based on DRL

[0124] Figure 2 is the MDP model solving flowchart

[0125] Figure 3 is the Lambda iteration method solving flowchart

[0126] Figure 4 is the model convergence process comparison diagram under different learning rate parameters

[0127] Figure 5 is the comparison diagram of unit output curve and load demand curve under two conditions

[0128] Figure 6 is the unit output scheme diagram of method 1

[0129] Figure 7 is the unit output scheme diagram of method 2

[0130] Figure 8 is the unit output scheme diagram of method 2 when iterated 300 times

[0131] Figure 9 is the unit output scheme diagram of method 2 when iterated 500 times DETAILED DESCRIPTION

[0132] A unit commitment intelligent solving algorithm based on deep reinforcement learning includes the following steps:

[0133] Step 1: MDP modeling for unit commitment

[0134] Markov decision process (MDP) is composed of state space, reward function, action space and transition function, and the goal of the UC problem studied in the application is to maximize the reward by minimizing the total operating cost of the system.

[0135] Step 1.1: State space

[0136] In this MDP, it is desired that the model gives the unit start-stop state at each time according to the given input data. Therefore, the data input at each time constitutes the state space. Specifically, the state space includes the start / stop time of N generators and the load demand data. The mathematical expression is:

[0137] S = {U t ,P L} (1)

[0138] In the formula, S represents the state space; A t = [a 1,t ,a 2,t ,...a N,t ], wherein A t = [a 1,t ,a 2,t ,...a N,t ] represents the set of unit start / stop times; A t = [a 1,t ,a 2,t ,...a N,t ] represents the load demand data. Since the goal of the UC problem is to solve the unit dispatching plan with the lowest total cost according to the given load demand while meeting various constraints, this variable has a very important influence on this problem.

[0139] Step 1.2: Action space

[0140] In RL, the action space is required to have completeness, efficiency, and legality. 1) Completeness: Completeness refers to ensuring that the action space contains all actions that can complete the target task. In this problem, the goal is to obtain the start-stop state of the unit, so the action space should include all start-stop states of the unit. 2) Efficiency: In the decision variables of this optimization problem, there are discrete variables and continuous variables, which have a high degree of difficulty in solving. Based on this, the present application solves the unit start-stop scheme and the unit output scheme step by step. After obtaining the unit start-stop scheme based on the DRL-based UC solving algorithm, the Lambda iteration is used to solve the unit output scheme. 3) Legality: Legality requires that the actions in the action space meet the constraints.

[0141] At any time, the possible action of each unit is to start or stop, so the action space is the combination of all unit start or stop actions, and the size of the action space is A t = [a 1,t ,a 2,t ,...a N,t ]. It is represented as a binary array, i.e.

[0142] A t =[a 1,t ,a 2,t ,...a N,t (2)

[0143] In the formula: A t This represents the set of all unit start-up or shutdown action spaces; a i,t Indicates the start / stop status of the unit; when the unit is in the start-up state, a i,t =1. When the unit's action is to stop, a i,t =0. However, this action must comply with the unit's minimum start / stop time constraints.

[0144] Step 1.3: Transfer Function

[0145] Once the model determines the unit start-up and shutdown plan based on the observed state information and obtains the reward value, the transfer function will adjust the unit start-up and shutdown actions while satisfying all constraints. From state Switch to status In this invention, the related state information is the continuous start / stop time of the unit. For the unit The conversion function for its continuous start / stop time is:

[0146]

[0147] In the formula: u i,t+1 The conversion function representing the continuous start / stop time of unit i at time t+1; u i,t A conversion function representing the continuous start / stop time of unit i at time t; a i,t This indicates that the possible actions of each unit are starting or stopping. When the unit's action is starting, a i,t =1. When the unit's action is to stop, a i,t =0.

[0148] Step 1.4: Reward Function

[0149] The goal of Reinforcement Learning (RL) is to maximize the rewards the model receives along the path when solving the problem. In the problem studied in this invention, the goal is to minimize the total operating cost of the system. Therefore, the mathematical expression for its cost is:

[0150] r t =-(F t +λ t (4)

[0151] In the formula: r t Let F be the sum of the system's operating cost at time t and the penalty for violating constraints;t is the operation cost of the unit at time t; λ t is the penalty value of violating the operation constraints at time t.

[0152] where

[0153]

[0154] where F t is the operation cost of the unit at time t; N represents the number of units; P i is the active power output value of the unit ; Pris the start-up cost of the unit r; a, b, c represent parameters.

[0155] Step 2: Solution process of the MDP model

[0156] In the MDP of the UC, at each time t, the model observes the state information r in the power system, i.e., the start-up / shut-down time of the N units and the load demand data at the current time, and then selects the optimal action r according to the state information, i.e., the unit start-stop scheme decided at the current time. Finally, according to the start-stop scheme, the economic dispatch problem is solved by Lambda iteration to obtain the actual output power of the unit at the current time, and the system operation cost at the current time is calculated according to the power, i.e., part of the reward function. After receiving the reward value r evaluating the goodness of the current unit start-stop scheme r, the model moves to the next new state s t+1 , which is determined by step 1.3 formula (3). The specific solution process is shown in Figure 2 .

[0157] As shown in Figure 2 , an experience pool mechanism is introduced in the solution process, which mainly includes two processes of sample collection and sampling. The collected unit start-stop state, load data, unit output scheme and reward value are placed in the experience pool in chronological order, and when it is full, the sample data at the earlier time is overwritten. When sampling, a batch of data is uniformly and randomly sampled from the experience pool for learning and updating.

[0158] Definition is the reward value of the entire iteration process of the system, and the reward value at the future time is multiplied by a discount to represent the importance of the future reward value, and its mathematical expression is:

[0159]

[0160] where γ∈[0, 1] represents the reward value of the entire iteration process of the system; γ∈[0, 1] is a discount factor for controlling the relative weight of immediate and future rewards, and the larger the value is, the more important the reward value at the future time is; r t+k+1 ​Let $t$ be the sum of the system's operating cost at time $t+k+1$ and the penalty for violating constraints, as shown in equation (4) in step 1.4.

[0161] To minimize the total system operating cost, the model needs to continuously update the existing unit start-up and shutdown strategy π through ongoing interaction with the system environment, ultimately obtaining the optimal unit start-up and shutdown strategy π. * To evaluate the model and provide the current unit start-up and shutdown plan based on the current time step's state information, a... t The quality of an objective function is usually quantified by using the expectation function.

[0162] Step 3: Solve the MDP model based on the policy gradient algorithm to obtain the optimal start-up and shutdown scheme for the unit;

[0163] Based on the good convergence of the Policy Gradient (PG) algorithm, this invention employs this algorithm to solve the established MDP model. The core idea of ​​the PG algorithm is to parameterize the strategy for solving the unit start-up and shutdown schemes. By controlling the weights of these parameters, the optimal start-up and shutdown scheme with the lowest operating cost is directly selected; that is, the optimal start-up and shutdown scheme is found by learning the gradient information of the policy parameters. The specific unit start-up and shutdown strategy can be described as a function containing the parameter θ:

[0164] π θ (s t ,a t )=P(a t |s t ,θ)≈π(s t ,a t (7)

[0165] Where: π θ The generator start-up and shutdown strategy is represented by π; t represents the state information of the power system at time t; t represents the generator start-up and shutdown strategy decided at time t; π represents the generator start-up and shutdown strategy decided at time t. θ (s t ,a t P(a) represents a function with respect to parameter θ, and its mapping is P(a) t |s t ,θ).

[0166] Representing unit start-up and shutdown strategies using parameterized neural networks π θ The objective function can then be expressed as adjusting the parameter θ to maximize the expected reward, and its mathematical expression is:

[0167]

[0168] In the formula: J1(θ) represents the expected reward value of the unit start-up and shutdown scheme strategy; The expectation of the reward value of the whole unit start-stop scheme strategy iteration process of the system; r t The sum of the operation cost and the punishment of constraint violation of the system at time t, where t = 0, 1, 2, L; π θ The unit start-stop scheme strategy; γ represents the attenuation coefficient of the policy iteration.

[0169] The objective function is maximized, that is, a set of parameter vectors θ is searched to maximize the objective function. Generally, for the maximization problem, the gradient ascent algorithm is used to find the maximum value:

[0170]

[0171] In the formula: θ * The final optimal solution of the parameter vector; α represents the step size of the gradient ascent movement; The gradient of the parameter θ; J1(θ) represents the expected reward value of the unit start-stop scheme strategy.

[0172] Suppose a one-step MDP, which uses the gradient ascent algorithm. π θ (s t ,a t ) represents a function about the parameter θ, and the mapping is P(a t |s t , θ). It gives the reward value of the unit start-stop scheme a t under the state a t r t = r(s t , a t ). Then the reward value of the selected unit start-stop scheme at is π θ (s t , a t ) r(s t , a t ), and the weighted reward under the state ∑ a∈A π θ (s t , a t ) r(s t , a t ) is ∑ a∈A π θ (s t , a t ) r(s t , a t ), which is derived as follows:

[0173]

[0174] In the formula: J1(θ) represents the expected reward value of the unit start-stop scheme strategy; The expectation of the reward value of the unit start-stop scheme; the state ∑ a∈A π θ (s t ,a t )r(s t ,a t ) under the weighting function; the state distribution in the policy.

[0175] The gradient is:

[0176]

[0177] where, the gradient of the expected reward value of the unit commitment scheme policy; the state ∑ a∈A π θ (s t ,a t )r(s t ,a t ) under the weighting function; the state distribution in the policy.

[0178] Assume that the gradient The likelihood ratio technique is applied to define as the score function, and the relationship between them is as follows:

[0179]

[0180] where, the gradient of the function with respect to the parameter θ; π θ (s t ,a t ) represents the function with respect to the parameter θ; the score function.

[0181] Therefore, equation (11) can be written as:

[0182]

[0183] where, the gradient of the expected reward value of the unit commitment scheme policy; the state distribution in the policy; the state ∑ a∈A π θ (s t ,a t )r(s t ,a t ) under the weighting function; the score function; r(s t ,a t) indicates that in state a t Unit start-up and shutdown plan a t The reward value.

[0184] Restore the policy gradient to its expected form:

[0185]

[0186] In the formula, This represents the gradient of the expected reward value of the unit start-up and shutdown strategy; Represents the expectation of the policy gradient; Represents the scoring function; r(s) t ,a t ) indicates that in state a t Unit start-up and shutdown plan a t The reward value.

[0187] By selecting the optimal unit start-up and shutdown scheme To minimize the system's operating cost, the final result is:

[0188]

[0189] In the formula, This represents the gradient of the expected reward value of the unit start-up and shutdown strategy; Represents the expectation of the policy gradient; Represents the scoring function; Indicates that in state a t The optimal unit start-up and shutdown scheme a t The reward value.

[0190] Step 4: Solve the actual power output scheme of the unit through Lambda iteration.

[0191] Before transitioning to the new state, the unit start-up and shutdown scheme a obtained in step 3 will be used. t As the 24-hour start-up and shutdown actions of the unit in the economic dispatch problem, the Lambda iteration is used to solve the problem based on these actions, and the actual output power P of the unit in the start-up state is given.

[0192] The Lambda iterative method, a classic algorithm in economic dispatching, works on the principle of ensuring that the incremental cost rate of all generating units is equal and equal to the unknown parameter λ. λ is continuously adjusted by calculating the difference between the total output of the generating units and the load demand, ultimately yielding the planned active power output of all coal-fired units. The solution process is as follows: Figure 3 As shown.

[0193] According to steps 1-4, the DRL is applied to the unit commitment decision-making field to obtain a DRL-based UC intelligent solving algorithm and construct a DRL-based UC intelligent solving algorithm decision-making framework. The UC problem is calculated in two steps, and the decision-making diagram is as shown in Figure 1 FIG. 1. In the first step, the start-stop scheme of the unit at the current time is determined based on the DRL. In the second step, the economic dispatching problem is solved based on the Lambda iteration according to the start-stop scheme of the unit at the current time.

[0194] Embodiment:

[0195] To verify the correctness and effectiveness of the present application, simulation verification is performed on a system containing 10 thermal power units. The related parameters of the 10 thermal power units are shown in Table 1. The 24h load data for unit commitment decision-making is shown in Table 2.

[0196] Table 1 Characteristic parameters of 10 thermal power units

[0197]

[0198] Table 2 24h load data

[0199]

[0200] 1) Process simulation

[0201] The optimizer used for the PG network is the Adam optimizer, which uses a random optimization method to give an adaptive learning rate for different parameters based on the estimation result of the gradient, so that it can realize efficient calculation and low memory occupation in the optimization process. In order to select a better learning rate parameter, the present application tests lr=0.01 and lr=0.05. In addition, in order to realize fast convergence of the model and decision-making, the number of training cycles Epoch needs to be determined in the training process.

[0202] In order to obtain better training effect, the convergence of the model and the fitting degree of the unit output scheme under the conditions of lr=0.01 and lr=0.05 are compared respectively as follows. The comparison chart of the model convergence process under different learning rate parameters is as shown in Figure 4 FIG. 2.

[0203] From Figure 4It can be seen that under different parameters, the model can quickly converge, indicating that the proposed DRL-based UC intelligent solution algorithm can adapt to the decision of the UC optimal solution in a dynamic environment. When lr=0.01, the reward value obtained by the model is the largest. The reason is that when the learning rate of the neural network is small, the step in the iterative update process at each step is shorter, so it is more accurate to guide the optimal solution. In addition, under different learning rates, the reward value obtained by the model is smaller and fluctuates when Epoch is equal to 1-10. With the increase of the number of iterations, the reward value obtained by each iteration step increases, and finally tends to be stable when the number of training cycles Epoch is equal to about 30. The reason is that in the initial exploration stage of the model, the model explores by trial and error according to the environment state, and there is no experience to follow, so the reward value obtained is low and varies in size. With the continuous deepening of training, the parameters in the strategy model are constantly optimized, and the strategy becomes more and more stable, and finally no longer changes, so the learning rate lr in the present application is set to 0.01.

[0204] To illustrate the advantage of setting lr to 0.01, the unit decision scheme under training cycle 25 times with lr=0.01 and the unit decision scheme under training cycle 200 times with lr=0.05 are given below, as shown in Tables 3 and 4, respectively.

[0205] Table 3 Unit decision scheme under training cycle 25 times with lr=0.01

[0206]

[0207]

[0208] Table 4 Unit decision scheme under training cycle 200 times with lr=0.05

[0209]

[0210] To visually display the difference between the two cases, the sum of the output at each time is compared with the load demand curve, and the results are shown in Figure 5

[0211] Figure 5 ​​As shown, when lr is set to 0.01, the unit output scheme obtained after 25 iterations of training can completely fit the load demand curve at all time points except for a certain difference at the first time point. When lr is set to 0.05, the unit output curve obtained after 200 iterations of training has a large power shortage at the two power consumption peak periods of 9-14 and 19-23, with a maximum of 72.06 MW. The reason is that when lr is set to 0.05, the unit output scheme that meets the current load demand is still not solved after 200 iterations of training, while when lr is set to 0.01, the unit output scheme that meets the constraint condition can be quickly solved after 25 iterations of training.

[0212] After the above large number of simulation tests, it is found that when the related hyperparameters in the DRL algorithm are set according to the data shown in Table 5, the model can quickly converge and has good effect.

[0213] Table 5 DRL algorithm hyperparameters

[0214]

[0215] 2) Comparative analysis

[0216] The advantages of the method of the application over the traditional method are illustrated by comparing the unit output scheme, decision time and cost or reward value of method 1 and method 2.

[0217] Method 1: UC decision-making method based on physical model driving.

[0218] Method 2: UC intelligent decision-making method based on DRL, i.e. the method of the application.

[0219] The unit output schemes obtained by the two methods are shown in Figure 6 , Figure 7 .

[0220] As can be seen from Figure 6 and Figure 7 : in method 1, under the current load demand, most of the output is borne by unit 1, accounting for about 85% of the load demand, and the remaining part is borne by the combination of unit 2, unit 4, unit 5, unit 6, unit 7. Unlike this, in method 2, there is no phenomenon that most of the load is borne by one unit, but is borne by the combination of all units except unit 5, unit 7 and unit 10. To analyze the reason, the decision time and system operation cost or reward value of the two methods are given below.

[0221] Table 6 Decision time and system operation cost / reward value of method 1 and method 2

[0222]

[0223] The difference between the unit output schemes of method 1 and method 2 is also shown in Table 6, in which the reward value obtained by method 2 is 7.6 million yuan higher than the system cost of method 1. The reasons are as follows: on the one hand, compared with the decision result obtained by method 1, the decision result in method 2 has not reached the global optimum, so the system operation cost part contained in the reward value is higher than that of method 1. On the other hand, the reward value in method 2 is composed of the system operation cost and the punishment for violating the constraint condition, and since the unit output scheme in the decision result of the method does not satisfy the load balance constraint at some time, it also contains part of the punishment amount. Based on the above reasons, the reward value obtained in method 2 is higher than the system operation cost in method 1.

[0224] In terms of decision efficiency, the decision time of method 1 is 3938.16s, and although the model in method 2 needs to interact with the environment and constantly explore and try in the limited action space to gradually find the action strategy with the maximum reward value in the training stage, it only takes 2.13s. After the training is completed, according to this strategy, it only takes 0.43s to obtain the UC decision scheme. In summary, although the DRL-based UC intelligent solving algorithm does not solve the final optimal combination state, in terms of training time and decision time, it improves the solving efficiency to a certain extent compared with the UC decision method based on the physical model.

[0225] In order to obtain a more optimal combination state under method 2, the number of iterations is increased to 300 and 500, and the change of the decision result is observed. The unit output schemes under the two conditions are shown in Figure 8 and Figure 9 .

[0226] From Figure 8 and Figure 9 , it can be seen that when the number of iterations is 300, there is a power shortage in the partial power consumption peak period from 9-24, and when the number of iterations is set to 500, the system unit output can meet the load demand at any time in the scheduling period, and there is no power shortage. Therefore, the DRL-based UC intelligent solving algorithm proposed in the application is correct and effective in solving small-scale UC problems.

Claims

1. A method for intelligent solving of unit commitment based on deep reinforcement learning, characterized in that, It comprises the following steps: Step 1: a Markov decision process MDP modeling is performed for unit commitment; Step 2: a Markov decision process MDP model is solved; Step 3: a MDP model solution based on a policy gradient algorithm is performed; Step 4: a Lambda iteration is used to solve the actual power output scheme of the unit; A deep reinforcement learning-based unit commitment intelligent solution method is obtained through steps 1 to 4; In step 1, when performing Markov decision process MDP modeling, the state space, action space, transition function, and reward function are set respectively; 1-1: the state space is set, the unit start / stop state at the corresponding time is given according to the given input unit start / stop time and load data, and the state space is constructed; the mathematical expression is: (1)； In the formula: wherein denotes a set of unit start / stop times; denotes load demand data; 1-2: Set the action space, at any time, assuming that the system contains units, then the possible action for each unit is to start or stop, and the action space is the combination of all unit start / stop actions; At any moment, the possible actions of each unit are start or stop, and the action space is the combination of all unit start or stop actions, whose size is ; which is represented as a binary array, namely (2) ； wherein: denotes the set of all unit start or stop action spaces; denotes the start / stop action state of a unit; when the action of a unit is start, ; when the action of a unit is stop, ; but the action has to respect the minimum start / stop time constraint of the unit; denotes the number of all units in the system; 1-3: Set the transition function, when the MDP model decides the unit start-stop scheme according to the observed state information and gets the reward value, the transition function will be set according to the unit start-stop action From the state To the state , the state information before and after the association is the continuous start / stop time of the unit ; The transition function of the continuous start / stop time is: (3) ； wherein: represents a unit At the conversion function of the continuous start / stop times at the time instant; represents a unit At the conversion function of the continuous start / stop times at the time instant; represents the possible action of each unit as start or stop, when the action of the unit is start, when the action of the unit is stop, ; 1-4: the reward function is set, the goal of reinforcement learning RL is to maximize the reward obtained on the unit start / stop action path when the MDP model solves the unit commitment UC problem; the goal is to minimize the total operating cost of the system; The expression is: (4) ； Wherein (5) ； wherein: is the operating cost of the system at time t; is the penalty value for violating the operating constraints at time t; denotes the number of all units in the system; is the active power value of the unit at time t; is the start-up cost of the unit at time t. 2.The unit commitment intelligent solving method based on deep reinforcement learning according to claim 1, characterized in that: In step 2, the following sub-steps are included: Step 2-1: Observing state information, at each time t, the MDP model observes the state information in the power system i.e. the start-up / shut-down time of N units at the current time, load demand data; Step 2-2: decision scheme, according to which the optimal action is selected based on the state information i.e. the unit start-stop scheme decided at the current time Step 2-3: iterative solution, according to the start / stop scheme, the economic dispatch problem is solved by Lambda iteration, and the actual output power of the unit at the current time is obtained; Step 2-4: calculate the operating cost, calculate the system operating cost at the current time according to the actual output power, which is part of the reward function; Step 2-5: Update the start-stop strategy, in order to achieve the minimization of the total system operation cost, the MDP model needs to update the existing unit start-stop strategy in the continuous interaction with the system environment , and finally obtain the optimal unit start-stop strategy . 3.The unit commitment intelligent solving method based on deep reinforcement learning according to claim 1, characterized in that: In step 3, the following sub-steps are included: Step 3-1: parameterize the policy of the unit start / stop scheme, the core idea of the policy gradient PG algorithm is to parameterize the policy of the unit start / stop scheme, and by controlling the parameter weights, the unit start / stop scheme with the minimum operating cost is selected; Step 3-2: Establish the objective function, and represent the unit start-stop scheme strategy with a parameterized neural network And establish the objective function, adjust the parameters So that the expected reward value is maximum; Step 3-2-1: A specific unit start-stop scheme strategy can be described as a function containing parameters : (6) ； In the formula: represents the unit start-stop scheme strategy; represents state information in the power system at the moment; represents the unit start-stop scheme decided at the moment; represents a function about the parameter , and the mapping is ; Step 3-2-2: Represent the unit commitment start-stop scheme strategy with a parameterized neural network The objective function can be expressed as the adjustment parameters So that the expected reward value is maximum, and the mathematical expression is: (7) ； In the formula: represents the expected reward value of the unit start-stop scheme strategy; represents the expected reward value of the entire unit start-stop scheme strategy iteration process of the system; is the sum of the operation cost and the punishment for violating the constraint of the system at the moment, wherein ; represents the unit start-stop scheme strategy; represents the attenuation coefficient of the strategy iteration; Step 3-3: Maximize the objective function, maximizing the objective function is to search for a set of parameter vectors maximizing the objective function, for the maximization problem, a gradient ascent algorithm is used to find the maximum; Step 3-3-1: Maximize the objective function, that is, search for a set of parameter vectors that maximizes the objective function; typically, for maximization problems, a gradient ascent algorithm is used to find the maximum: (8) ； In the formula: represents the final optimal solution of the parameter vector; represents a step size of a gradient ascent moving amount; represents a gradient of the parameter ; and represents an expected reward value of the unit start-stop scheme strategy. Step 3-3-2: Assume a one-step MDP, and use the gradient ascent algorithm on it; represents a function of the parameters , the mapping is ; it gives the unit commitment scheme at state , and the reward value is ; then the reward value obtained by selecting the unit commitment scheme is , and the weighted reward at state is , which is derived as follows: (9) ； In the formula: represents the expected reward value of the unit start-stop scheme strategy; the expected value of the unit start-stop scheme reward value; represents the state under the weighted reward; represents the state distribution in the strategy; Step 3-3-3: the gradient is: (10) ； wherein: denotes the gradient of the expected reward value of the unit start-stop scheme strategy; denotes the weighted reward in the state ; denotes the state distribution in the strategy; Step 3-3-4: Assume known gradient , define as the score function, with the following relationship: (11) ； wherein: denotes the gradient of the function with respect to the parameter ; denotes the function with respect to the parameter ; denotes the score function; Therefore, equation (10) can be written as: (12) ； wherein: denotes the gradient of the expected reward value of the unit on-off scheme strategy; denotes the state distribution in the strategy; denotes the state under the weighted reward; denotes the score function; denotes the reward value of the unit on-off scheme under the state ; Step 3-3-5: restore the policy gradient to the expected form: (13) ； wherein: denotes the gradient of the expected reward value of the unit commitment start-stop scheme strategy; denotes the expected value of the policy gradient; denotes the score function; denotes the reward value of the unit commitment start-stop scheme in state ; Step 3-4: minimize the operating cost, by selecting the optimal unit start / stop scheme, the operating cost of the system is minimized; By selecting the optimal unit start-stop scheme Make the system running cost minimum, ultimately get: (14) ； In the formula: This represents the gradient of the expected reward value of the unit start-up and shutdown strategy; Represents the expectation of the policy gradient; Represents the scoring function; Indicates the state Optimal unit start-up and shutdown scheme The reward value.

4. The method of claim 3, wherein the method is based on deep reinforcement learning. In step 4, the following sub-steps are included: Step 4-1: the state solved according to step 3-4 The optimal unit start-stop scheme , the start-stop action state of the unit in 24 hours is obtained, and the action obtained in step 3-4 is solved by Lambda iteration to obtain the unit scheduling scheme. First, set the initial value of the unknown parameter , the purpose is to make the cost increment rate of all units equal and equal to the unknown parameter ; Step 4-2: Calculate the actual output power of all units in the on state at this time ; (15) ； wherein: represents the actual output power of all units in the on state; represents the number of units in the on state; Step 4-3: Calculate the error value between the power value actually outputted by the computer group of formula (16) and the required power of the load ; (16) ； In the formula: represents the error value between the actual output power value of the generating unit and the load demand; represents the load demand power; represents the actual output power of all generating units in the on state. Step 4-4: If the error value between the actual output power value of the unit and the load demand is greater than the allowable error, return to step 4-1 to reiterate, modify the value of the unknown parameter Step 4-4: If the error value between the actual output power value of the unit and the load demand is greater than the allowable error, return to step 4-1 to reiterate, modify the value of the unknown parameter Step 4-4: If the error value between the actual output power value of the unit and Steps 4-5: If the error value between the actual power output of the unit and the load demand... If the error is less than or equal to the allowable error, stop the iteration and output the active power output scheduling scheme for all coal-fired units.

Images