A dynamic economic dispatch method with unknown generation cost function

Abstract

The application provides a dynamic economic dispatch method for a power generation cost function unknown, the method contains a new distributed training distributed execution reinforcement learning framework, and under the framework, an economic dispatch algorithm fusing a distributed optimization and a reinforcement learning method is designed. The distributed reinforcement learning optimization algorithm provided by the application solves the dynamic economic dispatch problem of the power generation cost function unknown in a distributed manner based on the distributed optimization theory and the reinforcement learning model, and can improve the power supply of the smart grid. In addition, the distributed reinforcement learning algorithm provided in the application can not only effectively perform power distribution management in the smart grid, but also be applicable to large-scale smart grid scenarios. Through simulation result examples, it is further illustrated that the distributed reinforcement learning optimization algorithm has good performance and effectiveness in the numerical examples of the dynamic economic dispatch problem.

Description

Technical Field

[0001] This invention relates to the field of smart grid distribution management technology, and in particular to a dynamic economic dispatch method for power generation cost functions that are unknown. Background Technology

[0002] With the rise of distributed energy resources and energy management technologies, traditional power grids are undergoing significant transformation. Smart grids fully utilize network communication and sensor technologies to achieve a safe, efficient, stable, and sustainable power supply. Compared to traditional power systems, smart grids have three main characteristics: first, they achieve a diversified energy structure that prioritizes renewable energy and adapts to local conditions; second, they achieve a reliable energy production and supply model that combines centralized and distributed energy sources in a coordinated manner; and third, they achieve comprehensive utilization of various energy sources, as well as an energy-efficient and mutually reinforcing energy consumption pattern that integrates supply and demand.

[0003] Smart distribution networks refer to the distribution network portion of a smart grid. Compared to traditional distribution networks, smart distribution networks have the following advantages: First, they possess strong self-regulation capabilities. Smart distribution networks can promptly detect existing or ongoing grid faults, enabling corresponding adjustments to minimize or eliminate disruptions to users' normal power supply. They solve the "power outage" problem inherent in traditional grids and effectively improve power reliability. In essence, their advantages extend far beyond power reliability. For example, current power reliability management does not account for short-duration outages, but such interruptions often damage or even permanently shut down sensitive high-tech equipment. Second, smart distribution networks offer higher security. They prevent large-scale blackouts, limiting the impact of external disruptions to a certain extent and ensuring normal power supply for critical users. Third, they provide higher power quality. Smart distribution networks monitor and control power quality in real time, ensuring that the effective voltage value and waveform meet user requirements, guaranteeing the normal operation of user equipment and preserving its lifespan. Fourth, they support user interaction. User interaction is also a key feature distinguishing smart distribution networks from traditional distribution networks. This is mainly reflected in two aspects: First, the application of smart meters and the implementation of time-of-use pricing and dynamic real-time pricing allow users to choose their electricity usage periods, saving on electricity costs while contributing to reducing peak grid load. Second, it allows and actively creates conditions for users with distributed energy resources to supply power to the grid during peak hours. Fifth, it provides visualized management of the distribution network and its equipment. Smart distribution networks comprehensively collect real-time operational data of the distribution network and its equipment, as well as data on power quality disturbances and outages, providing operators with an advanced graphical interface that enables them to fully grasp the operational status of the grid and its equipment, overcoming the slow response and low efficiency problems caused by the current "blind management" of distribution networks. Online diagnosis and risk analysis of the grid's operational status provide technical support for operators' scheduling decisions.

[0004] The power distribution problem is particularly important in smart grids. How can we allocate generator power to minimize the total cost of power generation while satisfying certain inequality constraints (e.g., generator output constraints, transmission interface limitations) and equality constraints (e.g., supply-demand balance constraints)? In real-world scenarios, the parameters in the generator cost function are affected by many factors (e.g., generator lifespan, equipment operating mechanisms), making the mathematical expression of the cost function impossible to know precisely. When considering an unknown generator cost function, traditional distributed optimization methods are no longer applicable. The reasons are as follows: 1. An unknown cost function makes it impossible to represent the objective function of the problem with a precise mathematical expression; 2. The cost function may have a complex non-convex form.

[0005] Reinforcement learning is now a hot topic in academia and industry, following deep learning. Both reinforcement learning and supervised learning require large amounts of data for training, but the types of data they need differ. Supervised learning requires diverse labeled data, while reinforcement learning requires interactive rewards. In other words, reinforcement learning only cares about immediate rewards and not the actual environment it faces. The goal of reinforcement learning is to find the best policy to maximize (or minimize) its cumulative reward. Its main learning idea is to exhaust all possibilities, continuously training and improving its policy to find or approximate the optimal policy. Currently, an increasing number of reinforcement learning algorithms are being used in the economic dispatching problem of smart grids.

[0006] This patent draws inspiration from reinforcement learning techniques, particularly state-action value function approximation and nonlinear programming theory, to address the problem of unknown power generation cost function in dynamic economic scheduling. The main contributions are as follows: (1) It combines state-action value function approximation techniques in reinforcement learning with distributed optimization algorithms based on multiplier splitting to propose a distributed reinforcement learning optimization algorithm to handle dynamic economic scheduling problems with unknown power generation cost function; (2) In the distributed reinforcement learning optimization algorithm proposed in this patent, the scheduling strategy of the agent adopts a distributed update method, which is conducive to the expansion of the algorithm to large-scale scenarios; (3) This patent uses time-varying parameters when selecting approximate state-action value functions. Compared with using time-invariant parameters, using time-varying parameters when selecting approximate state-action value functions can reduce errors and maintain the convexity of the approximate function. Summary of the Invention

[0007] This invention provides a dynamic economic dispatch method for problems with unknown generation cost functions. The method includes a novel distributed training and distributed execution reinforcement learning framework, within which an economic dispatch algorithm integrating distributed optimization and reinforcement learning methods is designed. The proposed distributed reinforcement learning optimization algorithm, based on distributed optimization theory and reinforcement learning models, solves the dynamic economic dispatch problem with unknown generation cost functions in a distributed manner, thus improving the power supply of smart grids. Furthermore, the proposed distributed reinforcement learning algorithm is not only effective for distribution management in smart grids but also applicable to large-scale smart grid scenarios. Simulation results further demonstrate the good performance and effectiveness of this distributed reinforcement learning optimization algorithm in numerical examples of dynamic economic dispatch problems.

[0008] This invention addresses the dynamic economic dispatch problem with an unknown generation cost function. First, it combines reinforcement learning and distributed optimization to construct an approximate distributed optimization problem. Then, it designs an economic dispatch algorithm that integrates distributed optimization and reinforcement learning methods.

[0009] A dynamic economic dispatch method with an unknown power generation cost function, characterized by comprising the following steps:

[0010] Step 1: Construct a dynamic economic dispatch model for a smart grid where the generation cost function is unknown;

[0011] Step 2: Use reinforcement learning to transform the proposed dynamic economic scheduling problem into a distributed optimization problem;

[0012] Step 3: Design a distributed reinforcement learning optimization algorithm, obtain a distributed iterative formula, and solve iteratively.

[0013] Specifically, the dynamic economic scheduling problem model in step 1 is referred to as model (1).

[0014]

[0015]

[0016]

[0017]

[0018] Where F i (p i,t ) is the power generation cost function of generator set i, p i,t D is the power output of generator unit i at time t. t It is the total electricity demand at time t. It is the ramp constraint for generator set i. p i and Let them be the minimum and maximum power generation, respectively. definition Let i be the feasible generation range of generator i. In dynamic economic dispatch problems, the most common generation cost function takes the form of: Where a i b i , and c i This is the power generation cost coefficient of generator i, considering the case where the power generation cost function is a sinusoidal cost function:

[0019]

[0020] The additional coefficient e i and f i The valve point effect coefficient of generator i.

[0021] The conversion process in step 2 is as follows:

[0022] The expression for the generation cost function in the dynamic economic dispatch problem model (1) in step 1 is unknown. It is transformed into a constrained approximate optimization problem through distributed reinforcement learning. The specific process is as follows:

[0023] Distributed estimation of total electricity demand at time t: Definition The average consensus protocol is applied to each agent as follows:

[0024]

[0025] Where L is a network communication diagram. The Laplace matrix, According to formula (1), Among them 1 N It is an N-dimensional column vector with all elements equal to 1. According to the dynamic economic dispatch problem model (1), the estimated average power demand of each agent at each time t converges to the actual average power demand, and all agents can obtain the total power demand D at each time t. t .

[0026] Agents in a distributed manner search for feasible power generation combinations at time t: each agent randomly selects... Define the error between electricity demand and supply as m. t [0] = (D 1,t -p 1,t ,…,D N,t -p N,t ) T The agent uses the following iteration:

[0027] m t [k+1]=m t [k]-∈Lm t [k]. (3)

[0028] According to formula (3), Adjust p in the following way i,t :

[0029]

[0030] Where sign(·) is the sign function, min{} represents taking the minimum value, α is the error between demand and supply, and formulas (3) and (4) are repeated until α = 0. When α = 0, P t =(p 1,t ,p 2,t ,…,p N,t ) T This refers to the feasible power generation combinations at time t.

[0031] Calculate the power generation cost at time t: Let Where c i,t =F i (p i,t ),make This is the average power generation cost at time t, using the average consensus protocol:

[0032]

[0033] According to formula (5), That is, the agent's local estimation converges to the average power generation cost at time t, and it is easy to obtain the total power generation cost as follows:

[0034] Update the approximate function parameters at time t: Define J t (D t ,P t ,θ t )=φ(P t ) T θ t Let φ(P) be the approximate state action value function, where φ(P) t ) is the eigenvector, θ t For parameters. Parameter θ t The update is as follows:

[0035]

[0036] make thereby Formula (6) becomes

[0037]

[0038] in These are the parameters of agent i at time t.

[0039] The approximate optimization problem is constructed intelligently. Based on the approximate state-action value function, the following constrained approximate optimization problem is constructed:

[0040]

[0041]

[0042]

[0043]

[0044] in, For the approximate optimal solution of agent i at time t-1, in particular

[0045] Specifically, the distributed algorithm in step 3 is as follows:

[0046] Design a distributed optimization method to solve problem (8), assuming that problem (8) has coefficients. The dual form is:

[0047]

[0048] According to the Laplace matrix L and Establish a constrained optimization problem:

[0049]

[0050] Problem (9) with multipliers The augmented Lagrange duality is:

[0051]

[0052] For agent i, design the following distributed optimization algorithm:

[0053]

[0054] Distributed update scheduling strategy: definition The local policy update for agent i is as follows:

[0055]

[0056] in

[0057] Balancing exploration and application: To balance exploration and application, an ε-greedy strategy is used, that is, selecting (π) with a probability of 1-ε. i (D t ),,π N (D t )) T Other feasible power generation combinations are selected with a probability of ε.

[0058] Compared with existing technologies, the advantages of this invention are as follows: 1) The distributed algorithm structure provides favorable characteristics for the smart grid scheduling process in terms of scalability; 2) The distributed reinforcement learning optimization method proposed in this invention can solve the dilemma of unknown generation cost function in the dynamic economic scheduling problem of power system.

[0059] This invention proposes a novel distributed training and distributed execution reinforcement learning framework, and based on this, designs a distributed reinforcement learning optimization algorithm that combines reinforcement learning and distributed optimization. This algorithm can efficiently and cost-effectively solve the dynamic economic scheduling problem with unknown generation cost functions. Its objective function optimization and fast convergence characteristics bring economic and time benefits to the power system. Using the distributed reinforcement learning scheduling method proposed in this invention, rapid modeling, ease of computation, rapid iteration, and rapid convergence can be achieved, saving computational resources in the power system, reducing the computational load of the smart grid, minimizing the cost of electricity trading in the power system, and bringing considerable economic benefits to the power system. Attached Figure Description

[0060] Figure 1 It is the IEEE 39 bus. Detailed Implementation

[0061] This invention provides a dynamic economic dispatch method for problems with unknown generation cost functions. The method includes a novel distributed training and distributed execution reinforcement learning framework, within which an economic dispatch algorithm integrating distributed optimization and reinforcement learning methods is designed. The proposed distributed reinforcement learning optimization algorithm, based on distributed optimization theory and reinforcement learning models, solves the dynamic economic dispatch problem with unknown generation cost functions in a distributed manner, thus improving the power supply of smart grids. Furthermore, the proposed distributed reinforcement learning algorithm is not only effective for distribution management in smart grids but also applicable to large-scale smart grid scenarios. Simulation results further demonstrate the good performance and effectiveness of this distributed reinforcement learning optimization algorithm in numerical examples of dynamic economic dispatch problems.

[0062] Example 1: This invention addresses the dynamic economic dispatch problem with an unknown generation cost function. First, it combines reinforcement learning and distributed optimization to construct an approximate distributed optimization problem. Then, it designs an economic dispatch algorithm that integrates distributed optimization and reinforcement learning methods.

[0063] Step 1: Construct a dynamic economic dispatch model for a smart grid where the generation cost function is unknown;

[0064] Step 2: Use reinforcement learning to transform the proposed dynamic economic scheduling problem into a distributed optimization problem;

[0065] Step 3: Design a distributed reinforcement learning optimization algorithm, obtain a distributed iterative formula, and solve iteratively.

[0066] Specifically, the dynamic economic scheduling problem model in step 1 is referred to as model (1).

[0067]

[0068]

[0069]

[0070]

[0071] Where F i (p i,t ) is the power generation cost function of generator set i, p i,t D is the power output of generator unit i at time t. t It is the total electricity demand at time t. It is the ramp constraint for generator set i. p i and Let them be the minimum and maximum power generation, respectively. definition Let i be the feasible generation range of generator i. In dynamic economic dispatch problems, the most common generation cost function takes the form of: Where a i b i , and c i This is the power generation cost coefficient of generator i, considering the case where the power generation cost function is a sinusoidal cost function:

[0072]

[0073] The additional coefficient e i and f i The valve point effect coefficient of generator i.

[0074] The conversion process in step 2 is as follows:

[0075] The expression for the generation cost function in the dynamic economic dispatch problem model (1) in step 1 is unknown. It is transformed into a constrained approximate optimization problem through distributed reinforcement learning. The specific process is as follows:

[0076] Distributed estimation of total electricity demand at time t: Definition The average consensus protocol is applied to each agent as follows:

[0077]

[0078] Where L is a network communication diagram. The Laplace matrix, According to formula (1), Among them 1 NIt is an N-dimensional column vector with all elements equal to 1. According to the dynamic economic dispatch problem model (1), the estimated average power demand of each agent at each time t converges to the actual average power demand, and all agents can obtain the total power demand D at each time t. t .

[0079] Agents in a distributed manner search for feasible power generation combinations at time t: each agent randomly selects... Define the error between electricity demand and supply as m. t [0] = (D 1,t -p 1,t ,…,D N,t -p N,t ) T The agent uses the following iteration:

[0080] m t [k+1]=m t [k]-∈Lm t [k]. (3)

[0081] According to formula (3), Adjust p in the following way i,t :

[0082]

[0083] Where sign(·) is the sign function, min{} represents taking the minimum value, and α is the error between demand and supply. Repeat formulas (3) and (4) until α = 0. When α = 0, P t =(p 1,t ,p 2,t ,…,p N,t ) T This refers to the feasible power generation combinations at time t.

[0084] Calculate the power generation cost at time t: Let Where c i,t =F i (p i,t ),make This is the average power generation cost at time t, using the average consensus protocol:

[0085]

[0086] According to formula (5), That is, the agent's local estimation converges to the average power generation cost at time t, and it is easy to obtain the total power generation cost as follows:

[0087] Update the approximate function parameters at time t: Define J t (Dt ,P t ,θ t )=φ(P t ) T θ t Let φ(P) be the approximate state action value function, where φ(P) t ) is the eigenvector, θ t For parameters. Parameter θ t The update is as follows:

[0088]

[0089] make thereby Formula (6) becomes

[0090]

[0091] in These are the parameters of agent i at time t.

[0092] The approximate optimization problem is constructed intelligently. Based on the approximate state-action value function, the following constrained approximate optimization problem is constructed:

[0093]

[0094]

[0095] p i,t ∈P i i = 1, 2, ..., N

[0096]

[0097] in, For the approximate optimal solution of agent i at time t-1, in particular

[0098] Specifically, the distributed algorithm in step 3 is as follows:

[0099] Design a distributed optimization method to solve problem (8), assuming that problem (8) has coefficients. The dual form is:

[0100]

[0101] According to the Laplace matrix L and Establish a constrained optimization problem:

[0102]

[0103] Problem (9) with multipliers The augmented Lagrange duality is:

[0104]

[0105] For agent i, design the following distributed optimization algorithm:

[0106]

[0107] Distributed update scheduling strategy: definition The local policy update for agent i is as follows:

[0108]

[0109] in

[0110] Balancing exploration and application: To balance exploration and application, an ε-greedy strategy is used, that is, selecting (π) with a probability of 1-ε. i (D t ),…,π N (D t )) T Other feasible power generation combinations are selected with a probability of ε.

[0111] This invention proposes a novel distributed training and distributed execution reinforcement learning framework, and based on this, designs a distributed reinforcement learning optimization algorithm that combines reinforcement learning and distributed optimization. This algorithm can efficiently and cost-effectively solve the dynamic economic scheduling problem with unknown generation cost functions. Its objective function optimization and fast convergence characteristics bring economic and time benefits to the power system. Using the distributed reinforcement learning scheduling method proposed in this invention, rapid modeling, ease of computation, rapid iteration, and rapid convergence can be achieved, saving computational resources in the power system, reducing the computational load of the smart grid, minimizing the cost of electricity trading in the power system, and bringing considerable economic benefits to the power system.

[0112] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific numerical simulation implementations.

[0113] The present invention was simulated based on the IEEE 39 bus to verify the performance of the proposed algorithm. Figure 1 This illustrates the network communication diagram between intelligent agents (e.g.) Figure 1 As shown in the figure, Table 1 displays the power generation cost coefficients of each generator in the smart grid.

[0114] Table 1:

[0115]

[0116] Table 2 shows the approximate results of the distributed reinforcement learning optimization algorithm after 1087 training iterations. The agent's scheduling decisions in the distributed reinforcement learning optimization algorithm gradually converge.

[0117] Table 2:

[0118]

[0119] Table 3 shows the optimal solution to the dynamic economic scheduling problem using a distributed optimization algorithm based on multiplier splitting.

[0120] Table 3:

[0121]

[0122] Calculations show that the error between the approximate solution generated by the distributed reinforcement learning optimization algorithm and the exact optimal solution of the problem is 4%. Furthermore, it is known that as the number of training iterations increases, the scheduling strategy of the agent learned by the distributed reinforcement learning optimization algorithm will become better and better.

[0123] It should be noted that the above embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Equivalent substitutions or alternatives made based on the above technical solutions shall all fall within the scope of protection of the present invention.

Claims

1. A dynamic economic dispatch method for power generation cost functions with unknown functions, characterized in that, Includes the following steps: Step 1: Construct a dynamic economic dispatch model for a smart grid where the generation cost function is unknown; Step 2: Use reinforcement learning to transform the proposed dynamic economic scheduling problem into a distributed optimization problem; Step 3: Design a distributed reinforcement learning optimization algorithm, obtain the distributed iterative formula, and solve iteratively. The dynamic economic scheduling problem model in step 1 is specifically referred to as model (1). (1) in It is a generator set The power generation cost function, It is a generator set exist Power generation at any given moment yes Total electricity demand at any given time It is a generator set Climbing constraints, and Let them be the minimum and maximum power generation, respectively. , , ,definition For generator In the dynamic economic dispatch problem, the feasible power generation range is defined by the power generation cost function in the form of: ,in , ,and It is a generator The power generation cost coefficient, considering the case where the power generation cost function is a sinusoidal cost function: The additional coefficient and With generator The valve point effect coefficient; The conversion process in step 2 is as follows: The expression for the generation cost function in the dynamic economic dispatch problem model (1) in step 1 is unknown. It is transformed into a constrained approximate optimization problem through distributed reinforcement learning. The specific process is as follows: Distributed estimation Total electricity demand at any given time: Definition The average consensus protocol is used for each agent as follows: in It is a network communication diagram. The Laplace matrix, According to formula (1), ,in All elements are of dimensional column vector, according to the dynamic economic scheduling problem model (1), each agent for each time step The estimated average electricity demand converges to the actual average electricity demand, and all agents can obtain the value of the average electricity demand at each time step. Total electricity demand , Distributed search by intelligent agents Feasible power generation combinations at any given time: each agent randomly selects The error between electricity demand and supply is defined as... The agent uses the following iteration: According to formula (3), Adjust in the following ways : in It is a symbolic function. This indicates taking the minimum value. It is the error between demand and supply, repeating the formula. and formula until ,when hour, That is Feasible power generation combinations at any given time. calculate Cost of generating electricity at any given time: Assume ,in ,make yes The average cost of generating electricity at any given time, using an average consensus protocol: According to the formula It can be seen that, That is, the agent's local estimation converges to The average cost of generating electricity at any given time, and the total cost of generating electricity is readily obtained. , renew Approximate function parameters at time: Definition Let be the approximate state action value function, where For feature vectors, For parameters, parameters The update is as follows: make , , ,thereby ,formula Become in These are the parameters of agent i at time t. Construct an approximate optimization problem. Based on the approximate state-action value function, construct the following constrained approximate optimization problem: in, For the approximate optimal solution of agent i at time t-1, in particular .

2. The dynamic economic dispatch method for an unknown power generation cost function according to claim 1, characterized in that, The distributed algorithm in step 3 is as follows: Design a distributed optimization method to solve the problem. Assuming the problem band coefficient The dual form is: According to the Laplace matrix and Establish a constrained optimization problem: question multiplier The augmented Lagrange duality is: For intelligent agents Design the following distributed optimization algorithm: Distributed update scheduling strategy: definition , , intelligent agent The local policy update is as follows: in , Balancing exploration and application: To balance exploration and application, use - Greedy strategy, that is, using probability selection ,by The probability of selecting other feasible power generation combinations.

Images