Distributed economic dispatch optimization method based on constraint projection reinforcement learning
By employing a reinforcement learning method based on constraint projection, combined with the Actor-Critic algorithm and the Lagrange multiplier method, the distributed economic dispatch problem is solved, achieving dynamic power optimization and cost minimization under unknown environments, while satisfying grid operation constraints.
Patent Information
- Application Number
- CN202211071171.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-02
- Publication Date
- 2025-12-16
- Estimated Expiration
- 2042-09-02
AI Technical Summary
In existing technologies, the complexity and diversity of distributed economic scheduling problems prevent existing algorithms from effectively solving dynamic distributed economic scheduling problems. In particular, reinforcement learning methods rely on a pre-established finite set of actions and cannot adapt to dynamic changes in unknown environments.
A reinforcement learning method based on constraint projection is adopted, which combines the power generation cost function and the feasible power output of the power generation unit. A distributed economic dispatch mathematical model is constructed through the Actor-Critic algorithm and the Lagrange multiplier method. The average consensus method is used to obtain global information, and the optimal power output is optimized through constraint projection and KKT conditions.
It enables effective coordination and distributed scheduling of generators under unknown load conditions, optimizes power generation costs in real time, meets the constraints of dynamic power demand, and avoids the need for pre-construction of action sets.
Smart Images
Figure CN115473286B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of power systems, and particularly relates to a distributed economic dispatch optimization method based on constraint projection reinforcement learning. BACKGROUND
[0002] Due to the integration of distributed energy resources, the development of intelligent technology and the high demand for energy management, the power grid is undergoing major changes. In this case, smart grid is receiving more and more attention. Smart grid technology makes full use of communication and sensing technology to strive to achieve safe, efficient, stable and sustainable power services. In the smart grid, the economic dispatch problem is of great concern. Economic dispatch refers to minimizing the total generation cost while meeting certain constraints (such as power balance constraints, bus voltage constraints, etc.). Due to time slot and ramp rate constraints, the static economic dispatch problem, as a special case of dynamic economic dispatch, has been more fully studied.
[0003] At present, for the dynamic economic dispatch problem in smart grid, some typical centralized algorithms (such as differential evolution) are widely used. In contrast, due to the complexity and diversity of dynamic economic dispatch problems, there are few reports on distributed economic dispatch problems. For example, a distributed original pair dynamic algorithm is used to solve energy management problems; a distributed original double dynamic algorithm is used for distributed power management in a multi-microgrid environment; the complete decentralized optimization of multi-area distributed economic dispatch is solved by cutting plane consistency algorithm. In fact, most offline algorithms are no longer effective when the exact mathematical form of the optimization target (such as cost and load) is not available. In addition, the calculation results need the complete information support of system data.
[0004] Reinforcement learning does not require any pre-given data, but learns and updates model parameters by receiving rewards (feedback) of actions in an unknown environment (i.e. no prior knowledge of the exact model), where the action set and state set are both limited, and cannot handle large-scale scheduling problems. In order to solve the distributed economic dispatch problem in continuous space, some scholars combine reinforcement learning with constraint projection method to solve the continuous action space and power load problem, or based on diffusion strategy, a distributed reinforcement learning method is proposed to solve dynamic distributed economic dispatch, or using average consensus algorithm to obtain global information, and a distributed Q learning solution is proposed.
[0005] In the prior art, most of the distributed economic dispatch schemes based on reinforcement learning in microgrid rely on the Q learning framework, which requires the system to pre-establish a set of limited actionable sets in some way before scheduling, and results in the cost generated by energy scheduling depending on the availability of the actionable set, which cannot effectively solve the dynamic distributed economic dispatch problem. SUMMARY
[0006] Based on this, it is necessary to provide a distributed economic dispatch optimization method based on constraint projection reinforcement learning for the above technical problems.
[0007] A distributed economic dispatch optimization method based on constraint projection reinforcement learning, comprising the following steps: based on operation constraints, combining a power generation cost function and a feasible power output of a power generation unit to construct a distributed economic dispatch mathematical model, the operation constraints including node power balance restrictions, ramp rate restrictions and power generation capacity restrictions; implementing an average consensus method for each generator in a regional communication network to obtain global information of the microgrid in a distributed manner; using an Actor-Critic algorithm to convert the distributed economic dispatch mathematical model into a distributed economic dispatch optimization problem based on constraint projection; combining a Lagrange multiplier method and a penalty function method to obtain an augmented Lagrange dual form of the distributed economic dispatch optimization problem, and based on KKT conditions to obtain optimal power output of the distributed economic dispatch optimization problem.
[0008] In one of the embodiments, the distributed economic dispatch mathematical model is constructed based on operation constraints, combining a power generation cost function and a feasible power output of a power generation unit, and specifically includes:
[0009]
[0010]
[0011] |po i,t -po i,t-1 |≤pr i (3)
[0012]
[0013] Equations (2)-(4) are respectively node power balance restrictions, ramp rate restrictions and power generation capacity restrictions, wherein i=1,…,N, t=1,…T, T represents total time slots, C i is a power generation cost function, po i,t is a feasible power output of a power generation unit, pd i,t is a power demand of a load unit in a public power grid, pr i represents a ramp rate restriction of the i-th generator, and are minimum and maximum power generation limits of the i-th generator; when line transmission loss is not considered, there is:
[0014]
[0015] wherein PD tis the total power demand of time slot t; the hybrid cost function including multiple energy options is:
[0016]
[0017] wherein, is the cost coefficient of the ith generator, m = 1, …, M, h = 1, …, H; when solving formula (1), assumptions 1 and 2 are set, wherein, assumption 1 is that according to the communication topology structure of the micro-grid, each load bus signal can be transmitted to the corresponding generator bus; assumption 2 is that there is at least one feasible solution in the distributed economic dispatching mathematical model satisfy operation constraints (2) - (4).
[0018] In one embodiment, the average consensus method is implemented for each generator in the regional communication network to obtain the global information of the micro-grid in a distributed manner, specifically including: the dynamic average consensus estimation equation based on the second-order proportional integral is:
[0019]
[0020] wherein, N i represents a set of adjacent nodes, u i is a control input, v i is an estimated state, z i is a decision variable; if the system topology is a directed graph, the estimated state converges to the cumulative average of all control inputs , that is, , when each generator satisfies the foregoing condition, the cumulative average of the control input u i can be calculated by the global variable N.
[0021] In one embodiment, the Actor-Critic algorithm is used to convert the distributed economic dispatching mathematical model into a distributed economic dispatching optimization problem based on constraint projection, specifically including: based on Markov decision, the next process is represented as:
[0022] Pr(s t+1 | s0, a0, …, s t , a t ) = Pr(s t+1 | s t , a t ) (8)
[0023] The reinforcement learning includes a five-tuple (S, A, T, R, Π) of T processes, wherein, S = {s t} includes a state set; A = {a t} includes an action set; a state transition model R = {rt}and policy distribution of action {p t}, representing the probability of moving to the next state s t and action a t under state s t+1 ; p t ~ Pr(a t | s t ) represents the probability of selecting a t according to s t ; the reward r is maximized / minimized by optimizing the policy of action p t at each training; the output of the Actor-Critic network is represented as:
[0024]
[0025]
[0026] wherein, and are the weights of the Actor neural network and the Critic neural network, respectively; S a and S c are activation functions; and are feature vectors; the action probability is the output of the Actor network, and the state value function is the output of the Critic network at time slot t and iteration index k; the time-difference error of each agent is:
[0027]
[0028] wherein, the reward function is λ i ∈ [0, 1] is a discount factor;
[0029] The loss function of the Critic neural network is defined as:
[0030]
[0031] For the Actor network, the time-difference error is used as the evaluation function, and the Actor-Critic is updated based on backpropagation, and the formula is:
[0032]
[0033]
[0034] where la and lc denote the learning rates of Actor and Critic, respectively, at terminal time slot T, satisfies i.e., the state value is set to 0 at time slot T+1; according to constraints (2)-(4), the distributed economic dispatch mathematical model is converted into a distributed economic dispatch optimization problem, which is:
[0035]
[0036] the state value is evaluated by the Critic framework and is the unique projection of the state value to based on the constraint projection property and given .
[0037] In one embodiment, the Lagrange multiplier method and the penalty function method are combined to obtain an augmented Lagrange dual form of the distributed economic dispatch optimization problem, and the optimal power output of the distributed economic dispatch optimization problem is obtained based on the KKT condition, specifically including: converting formula (15) into a dual problem, combining the Lagrange multiplier method and the penalty function method to obtain an augmented Lagrange dual form as:
[0038]
[0039] wherein L is the Laplace matrix of the system network, and the multiplier Y=(y1, y2, …, y N ) T , X=(x1, x2, …, x N ) T ; according to the KKT condition, the optimal power output of the distributed economic dispatch optimization problem is the numerical solution of the equilibrium point of the objective function (15); based on the projection form of the augmented Lagrange function, a distributed projection algorithm is adopted:
[0040]
[0041] wherein, there are:
[0042]
[0043] Assumption 3: Assume that the intersection of the ramp rate constraint interval and the power generation constraint interval is not an empty set, i.e.,
[0044] The input of the Actor network is obtained by constraint projection From formula (15), we have: Then, based on the back-propagation update method, for each generator, we have:
[0045]
[0046] Assume the joint power ramp rate constraint is:
[0047] PR t = PR 1,t × PR 2,t ×... × PR N,t
[0048] Then, the globally feasible power output vector is:
[0049] PO t = (po 1,t , po 2,t ,..., po N,t )
[0050] The global power output vector of the actor policy network is:
[0051] P t = (p 1,t , p 2,t ,..., p N,t )
[0052] The global form is obtained from equation (17) as:
[0053]
[0054] Meanwhile, from and we have:
[0055]
[0056]
[0057] where 1 N = (1,..., 1) T The global form of the local differential system (17) is:
[0058]
[0059] By the LaSalle invariance principle and the projection theorem, it is shown that the iterative trajectory of the constraint projection algorithm is bounded and converges to the optimal power output of the distributed economic dispatch optimization problem under the above assumptions 1-3.
[0060] In one of the embodiments, the iteration trajectory of the constrained projection algorithm is bounded and converges to the optimal power output of the distributed economic dispatch optimization problem by LaSalle invariance principle and projection theorem, specifically including: the global differential system (21) is further integrated, assuming:
[0061] Z = (PO t ,Y,X) T (22)
[0062] The equilibrium point of the differential system (21) is defined as:
[0063]
[0064] The projection interval is extended to (21) and (23) can be obtained:
[0065]
[0066] Equation (21) can be converted to:
[0067]
[0068] Wherein,
[0069]
[0070] The following Lyapunov function is designed:
[0071]
[0072] The continuous differential system (25) is represented as:
[0073]
[0074] Where s is a continuous variable, and equation (26) is:
[0075]
[0076] According to the inner product property:
[0077]
[0078] In addition, (25) gives:
[0079]
[0080] According to the projection property:
[0081]
[0082] Thus the Lyapunov function satisfies:
[0083]
[0084] where, iff Z(s) = Z * then:
[0085]
[0086] The derivative of the function L(s) is given by:
[0087]
[0088] Based on Equation (33) is written as:
[0089]
[0090] where the Jacobian matrix J Q (Z) is given by:
[0091]
[0092] From the projection property of the constraint projection, we have:
[0093]
[0094] We have:
[0095]
[0096] From (34), we have:
[0097]
[0098] We have:
[0099]
[0100] From (30) and (39), we have:
[0101]
[0102] In the vicinity of the equilibrium point, there exists the following compact invariant set:
[0103]
[0104] From (38), we have:
[0105]
[0106] Inserting Q(Z) into (42), we have:
[0107]
[0108] By KKT condition Then we have:
[0109]
[0110] Formula (43) satisfies:
[0111]
[0112] Let Θ be the set of all points such that That is Θ is a maximum invariant set in the invariant set S, Based on the LaSalle invariant set theorem and the convexity of the optimization problem, the feasible power output is obtained by asymptotic convergence, that is
[0113] Compared with the prior art, the advantages and beneficial effects of the present application are that: based on the operation constraints, the distributed economic dispatch mathematical model is constructed by combining the power generation cost function and the feasible power output of the power generation unit, the average consistency method is implemented for each power generator in the regional communication network to obtain the global information of the microgrid in a distributed manner, the Actor-Critic algorithm is used to convert the distributed economic dispatch mathematical model into a distributed economic dispatch optimization problem based on constraint projection, the Lagrange multiplier method and the penalty function are combined to obtain the augmented Lagrange dual form of the distributed economic dispatch optimization problem, and the optimal power output of the distributed economic dispatch optimization problem is obtained based on the KKT condition, the Actor strategy neural network is combined with the distributed optimization based on the multiplier splitting, and the strategy distribution of the action is mapped to the feasible power interval, and finally the real-time scheduling of the continuous action space is realized through the iterative approximation of the Critic evaluation function, thereby realizing the effective coordination of the power generator without the need to focus on the construction of the cost function, and the distributed scheduling task can be completed through the feedback interaction with the unknown load bus. BRIEF DESCRIPTION OF DRAWINGS
[0114] Figure 1 It is a flowchart of a distributed economic dispatch optimization method based on constraint projection reinforcement learning in an embodiment;
[0115] Figure 2 It is a technology roadmap of a distributed economic dispatch optimization method based on constraint projection reinforcement learning in an embodiment;
[0116] Figure 3 It is a reinforcement learning form diagram based on Markov decision in an embodiment;
[0117] Figure 4This is a schematic diagram of the basic reinforcement learning model for dynamic distributed economic scheduling in one embodiment.
[0118] Figure 5 This is a block diagram of the Actor-Critic algorithm based on constraint projection in one embodiment;
[0119] Figure 6 A simplified generator communication diagram of the IEEE 14 bus in one embodiment;
[0120] Figure 7 This is a schematic diagram illustrating the weight evolution of an Actor neural network in one embodiment;
[0121] Figure 8 This is a schematic diagram illustrating the weight evolution of a Critic neural network in one embodiment;
[0122] Figure 9 This is a schematic diagram illustrating the power output scheduling and Q-value evolution of the Actor-Critic algorithm based on constrained projection in one embodiment;
[0123] Figure 10 The total power generation cost is generated by the Actor-Critic algorithm based on constrained projection in one embodiment. Detailed Implementation
[0124] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0125] In one embodiment, considering the real-time energy dispatch requirements of modern power systems and the cost uncertainties arising from multiple fuel resources, such as Figure 1 As shown, a distributed economic scheduling optimization method based on constrained projection reinforcement learning is provided, including the following steps:
[0126] Step S101: Based on operational constraints, a distributed economic dispatch mathematical model is constructed by combining the power generation cost function and the feasible power output of the power generation unit. The operational constraints include node power balance constraints, ramp rate constraints, and power generation capacity constraints.
[0127] Specifically, the distributed economic dispatch problem in a microgrid with unknown non-convex hybrid costs aims to find the optimal power allocation that satisfies the constraint of minimizing the total generation cost of all generators.
[0128] The distributed economic scheduling mathematical model constructed is as follows:
[0129]
[0130]
[0131] |po i,t -po i,t-1 |≤pr i (3)
[0132]
[0133] Equations (2)-(4) are node power balance constraint, ramp rate constraint and generation capacity constraint, respectively, where i = 1,..., N, t = 1,... T, T represents total time slots, C i is generation cost function, po i,t is feasible power output of generation unit, pd i,t is power demand of load unit in public grid, pr i represents ramp rate constraint of the i-th generator, and are minimum and maximum generation limits of the i-th generator; when line transmission loss is not considered, there are:
[0134]
[0135] where PD t is total power demand of time slot t; a hybrid cost function including multiple energy options is:
[0136]
[0137] where is cost coefficient of the i-th generator, m = 1,..., M, h = 1,..., H; when solving equation (1), assumptions 1 and 2 are set, where assumption 1 is that according to the communication topology structure of the microgrid, each load bus signal can be transmitted to the corresponding generator bus; assumption 2 is that there is at least one feasible solution in the distributed economic dispatching mathematical model satisfying operation constraints (2)-(4).
[0138] Specifically, for the operation range of decision variable point, the generation capacity of all generator groups is different, in fact, the coupling between generation cost and feasible power output point is complicated due to the uncertainty related to high popularization rate of intermittent renewable energy, load effect in modern power system and different fuel selection, therefore, a hybrid cost function with multiple energy possible options can be formulated in equation (6), and in order to ensure the existence of optimal solution of equation (1), assumptions 1 and 2 need to be accepted.
[0139] In one embodiment, symbol N Ω(x) is defined as the normal cone at x e Ω, where Ω is a closed convex set; assume that the projection P Ω (y) = argmin x∈Ω ‖y-x‖ exists, then by the projection property we have:
[0140]
[0141] In step S102, an average consensus method is implemented for each generator in the regional communication network to obtain global information of the micro-grid in a distributed manner.
[0142] Specifically, in order to obtain global information of the micro-grid in a distributed manner, such as total power generation cost, total power demand, etc., an average consensus method needs to be introduced and implemented for each generator in the regional communication network, so that the obtained global information has consistency. The dynamic average consensus estimation equation based on the second-order proportional integral is:
[0143]
[0144] In the formula, N i represents a set of adjacent nodes, u i is a control input, v i is an estimated state, and z i is a decision variable; if the system topology is a directed graph, the estimated state converges to the cumulative average of all control inputs u i . When each generator satisfies the foregoing condition, the cumulative of the control input u i can be calculated by the global variable N.
[0145] Specifically, through the average consensus protocol, the state of each generator converges uniformly, so that uniform global information can be obtained. When the system topology is a directed graph, the estimated state converges to the cumulative average of all control inputs; according to the case where the system topology is a directed graph, assuming that the average consensus protocol is implemented for each generator, only the global variable needs to be obtained to calculate the cumulative of the control input.
[0146] In step S103, an Actor-Critic algorithm is used to convert the distributed economic dispatch mathematical model into a distributed economic dispatch optimization problem based on constraint projection.
[0147] Specifically, the basic principle of reinforcement learning is based on a Markov decision process, {s0,a0,r0,s1,a1,...,s T ,a T ,r T}, as shown in Figure 3As shown. For Markov decision, the next process only depends on the current state and is independent of historical information, which reflects the non-causal characteristics of dynamic programming. In order to solve the problem of dimension disaster and overestimation, a preliminary Actor-Critic framework is formed by adding a policy network "Actor" in the Q network, and an Actor-Critic algorithm based on constraint projection is used to calculate the distributed economic dispatch mathematical model. Since the Actor neural network cannot provide feasible power, the action policy distribution is mapped to the actionable interval through the constraint projection algorithm. The strategy distribution of the action output by the Actor neural network is mapped to the feasible power through the constraint projection method, and finally the evaluation function of the Critic neural network is used to evaluate the pros and cons of the strategy, so as to solve the dynamic economic dispatch problem in the microgrid and realize the effective scheduling of the generator.
[0148] where, based on Markov decision, the next process is represented as:
[0149] Pr(s t+1 ∣s0,a0,…,s t ,a t )=Pr(s t+1 ∣s t ,a t ) (8)
[0150] Reinforcement learning includes five tuples (S, A, T, R, Π) of T processes, including state set S = {s t}; action set A = {a t}; state transition model reward set R = {r t} and action policy distribution {π t}, represent the probability of moving to the next state s t+1 under state s t and action a t ; π t ~ Pr(a t ∣s t ) represents the probability of selecting a t according to s t ; as a long-term cumulative reward, the return is maximized / minimized by optimizing the action policy π t at each training; the output of the Actor-Critic network is represented as:
[0151]
[0152]
[0153] wherein, and are the weights of the Actor neural network and the Critic neural network, respectively; S a and S c are activation functions; and are the feature vectors; action probability is the output of the Actor network, state value function is the output of the Critic network at time slot t and iteration index k; the temporal difference error for each agent is:
[0154]
[0155] wherein, the reward function is λ i ∈ [0, 1] is the discount factor; the loss function of the Critic neural network is defined as:
[0156]
[0157] For the Actor network, the temporal difference error is used as the evaluation function, and the Actor-Critic is updated based on backpropagation, and the formula is:
[0158]
[0159]
[0160] wherein, la and lc represent the learning rates of the Actor and the Critic, respectively, at the terminal time slot T, satisfies that is, the state value is set to 0 at time slot T+1; according to the constraint conditions (2)-(4), the distributed economic dispatch mathematical model is converted into a distributed economic dispatch optimization problem, which is:
[0161]
[0162] The state value of the actionable is evaluated through the Critic framework, and based on the constraint projection property and the given state value is the unique projection of to .
[0163] Specifically, since the Actor policy network cannot provide a feasible power, we innovatively map the action policy distribution to the actionable interval through a constraint projection algorithm, and the block diagram of the proposed Actor-Critic algorithm based on constraint projection is as follows:Figure 5 It can be seen that the constrained projection algorithm is treated as a state transition function in reinforcement learning. There are no randomization and minimization operations involved in the whole process of the Actor-Critic algorithm based on the constrained projection. Since the action probability p i,t The constraints (2)-(4) are not satisfied, therefore, the constrained projection method provides an optimal solution for finding the feasible power combination of each generator.
[0164] It is noted that the environment is inevitably unknown due to the uncertainty of the load-side energy demand. Then the state transition probability and the direct reward feedback r t are generally unavailable. In fact, according to the way the agent (generator) interacts with the environment, the corresponding state transition function T t (s t ,a t ) and reward function r t (s t ,a t ) will be proposed.
[0165] According to Figure 4 , the power output po i,t can be seen as the action a t . Under the assumption 1, the load bus (environment side) provides the power demand information pd i,t (state s t ) and the cost function r t (s t ,a t ) calculated by the generation bus (agent end) to obtain the power generation cost C i (t) (reward r i,t ).
[0166] In addition, it is also noted that the iterative formula of Q-learning is derived from the Bellman equation and the greedy policy:
[0167]
[0168] where, denotes the current q-value, lr i denotes the learning rate. For the sake of clarity, we assume that the q-value is optimal, then we have the ideal optimal q-function where, E(·) denotes the expectation, and π denotes the policy of choosing a random or optimal action. However, the optimal q-function should satisfy the Bellman equation, then we have: Therefore, the overestimation problem can be simply attributed to the inequality:
[0169] E(max(q i,1 ,q i,2..., q i,T ))≥max(E(q i,1 ,q i,2 ..., q i,T ))
[0170] It can be concluded that using gradient ascent to fit the q-function will result in a larger estimated expectation.
[0171] In step S104, the augmented Lagrange dual form of the distributed economic dispatch optimization problem is obtained by combining the Lagrange multiplier method and the penalty function method, and the optimal power output of the distributed economic dispatch optimization problem is obtained based on the KKT condition.
[0172] Specifically, in order to meet the feasible power generation under certain constraints, a constraint projection method based on multiplier splitting is introduced, and the LaSalle invariant set theory and projection properties are used to determine that the algorithm can converge to the optimal feasible power. In addition, the constraint projection method is applied only once per gradient update. Since the original objective function (15) is convex, according to the KKT (Kuhn-Tucker conditions) condition, the numerical solution of the equilibrium point is the optimal solution of the distributed economic dispatch optimization problem.
[0173] The formula (15) is converted into a dual problem, and the augmented Lagrange dual form is obtained by combining the Lagrange multiplier method and the penalty function method:
[0174]
[0175] In the formula, L is the Laplace matrix of the system network, and the multiplier Y=(y1, y2, …, y N ) T X=(x1, x2, …, x N ) T According to the KKT condition, the optimal power output of the distributed economic dispatch optimization problem is the numerical solution of the equilibrium point of the objective function (15); based on the projection form of the augmented Lagrange function, the distributed projection algorithm is adopted:
[0176]
[0177] In the formula, there are:
[0178]
[0179] Assumption 3: Assume that the intersection of the slope rate constraint interval and the power generation constraint interval is not an empty set, i.e.
[0180] The input of the Actor network is obtained by constraint projection From formula (15), it can be concluded that: Then, based on the back-propagation update method, for each generator, we have:
[0181]
[0182] Assume the joint power ramp rate constraint is:
[0183] PR t = PR 1,t × PR 2,t ×…× PR N,t
[0184] Then, the globally feasible power output vector is:
[0185] PO t = (po 1,t , po 2,t ,…, po N,t )
[0186] The global power output vector of the Actor policy network is:
[0187] P t = (p 1,t , p 2,t ,…, p N,t )
[0188] The global form is obtained from equation (17) as:
[0189]
[0190] Meanwhile, from and we have:
[0191]
[0192]
[0193] where 1 N = (1,...,1) T , the global form of the local differential system (17) is:
[0194]
[0195] By the LaSalle invariance principle and the projection theorem, it is shown that the iterative trajectory of the constraint projection algorithm (17) is bounded and converges to the optimal power output of the distributed economic dispatch optimization problem under the assumptions 1-3 above.
[0196] Specifically, the optimal solution of the distributed economic dispatch optimization problem is its optimal power output. The input of the Actor network is can be obtained by the constrained projection method (state transition function). From (15) we have . Therefore, based on the backpropagation update method, each generator has .
[0197] where the global differential system (21) is further integrated, assuming:
[0198] Z = (PO t ,Y,X) T (22)
[0199] Then the equilibrium point of the differential system (21) is defined as:
[0200]
[0201] The projection interval is extended to Combining (21) and (23), we have:
[0202]
[0203] Equation (21) can be transformed into:
[0204]
[0205] where,
[0206]
[0207] The following Lyapunov function is designed:
[0208]
[0209] The continuous differential system (25) is represented as:
[0210]
[0211] where s is a continuous variable, and equation (26) is:
[0212]
[0213]
[0214] From the inner product property we have:
[0215]
[0216] In addition, (25) gives:
[0217]
[0218] According to the projection property, we have:
[0219]
[0220] Thus, the Lyapunov function satisfies:
[0221]
[0222] where, iff Z(s) = Z * then:
[0223]
[0224] The derivative of the function L(s) is given by:
[0225]
[0226] Based on Equation (33) is written as:
[0227]
[0228] where the Jacobian matrix J Q (Z) is expressed as:
[0229]
[0230] According to the projection property of the constraint projection, we have:
[0231]
[0232] We have:
[0233]
[0234] From (34), we have:
[0235]
[0236] We get:
[0237]
[0238] From (30) and (39), we have:
[0239]
[0240] In the vicinity of the equilibrium point, there exists the following compact invariant set:
[0241]
[0242] From (38), we have:
[0243]
[0244] Inserting Q(Z) into (42), we have
[0245]
[0246] By KKT condition Then we have
[0247]
[0248] Equation (43) satisfies
[0249]
[0250] Let Θ be the set of all points such that That is Θ is a maximal invariant set in S, Based on the LaSalle invariant set theorem and the convexity of the optimization problem, the feasible power output is obtained by asymptotic convergence, that is
[0251] In this embodiment, based on the operation constraints, the distributed economic dispatch mathematical model is constructed by combining the power generation cost function and the feasible power output of the power generation unit, the average consistency method is implemented for each generator in the regional communication network to obtain the global information of the microgrid in a distributed manner, the Actor-Critic algorithm is used to convert the distributed economic dispatch mathematical model into a distributed economic dispatch optimization problem based on constraint projection, the Lagrange multiplier method and the penalty function are combined to obtain the augmented Lagrange dual form of the distributed economic dispatch optimization problem, and the optimal power output of the distributed economic dispatch optimization problem is obtained based on the KKT condition, the Actor strategy neural network is combined with the distributed optimization based on the multiplier splitting, and the strategy distribution of the action is mapped to the feasible power interval, and finally the continuous action space is realized in real time by iteratively approximating the Critic evaluation function, thereby realizing the effective coordination of the generators without focusing on the construction of the cost function, and completing the distributed scheduling task through the feedback interaction with the unknown load bus.
[0252] In one embodiment, the dynamic distributed economic dispatch problem based on the IEEE 14 bus system is considered, and the effectiveness of the algorithm is verified by the convergence results and multiple sets of comparative experiments. These algorithms are implemented by MATLAB R2014a on a desktop PC with an i5-12400F CPU@2.50GHz, 16GB RAM, and a 64-bit Windows 11 operating system.
[0253] Given the IEEE 14 bus system with 5 generators, the simplified communication topology is as follows Figure 6As shown. Considering the significant differences in characteristics between different types of generators, the total power load in different time slots is PD = [830, 995, 900, 850, 1015] (MW), with a ramp rate of pr. i ={25,20,36,33,75} (MW), capacity constraint parameters are shown in Table 1, and hybrid cost function coefficients are shown in Table 2.
[0254] Table 1 Capacity Limitation Parameters for All Generators
[0255]
[0256] Table 2. Segmented Power Generation Cost Coefficients
[0257]
[0258] For the parameters of the Actor-Critic network, assuming the learning rate la i =lc i =10 -7 The attenuation factor γ = 0.25, training index K = 3500, and time slot T = 5. In this embodiment, a simplified three-layer deep neural network is used to construct the Actor-Critic framework. The sigmoid function is used as the activation function for the output layer of the Actor neural network, and the ReLU function is used as the activation function for the output layer of the Critic neural network.
[0259] Figure 7 and Figure 8 The evolution of weights in the hidden and output layers of the Actor-Critic neural network is shown. Figure 9 illustrates the probability distribution of actions and the Q-values of the Actor-Critic neural network, respectively. In the upper part of the figure, the histogram shows the probability distribution of the final feasible power dispatch generated by the Actor network, while the line graph represents the feasible power dispatch with corresponding probabilities for the five generators. The resulting power outputs are listed in Table 3. It can be seen that the power dispatch between time slots satisfies the ramp rate constraint. In the lower part of the figure, the line graph represents the variation of the Q-values generated by the critic network within the five time slots. Figure 10 The total power generation cost generated by the constrained projection-based Actor-Critic algorithm is shown. It can be seen that both the power generation cost and power output converge. These results verify the feasibility of the designed constrained projection-based Actor-Critic algorithm.
[0260] Table 3 Power output time table of the Actor-Critic algorithm based on constrained projection
[0261]
[0262] Table 4 shows the cost comparison of several representative algorithms for the economic dispatch problem. In the prior art, the lag Q-learning algorithm solves the economic dispatch problem in continuous action space by discretizing the performed actions and mapping the virtual actions to the actual actions (feasible power) through a constraint projection method, in which the virtual action interval is set to 5 (MW). The Q-table-based Q-learning algorithm solves the residential energy management problem using a traditional Q-learning algorithm. Since the distributed power balance constraint (2) cannot be satisfied, local power demand needs to be designed. Considering that the power generation of each generator is different, the power demand allocated to each generator is as follows:
[0263]
[0264] Global term PD t and It can be obtained from the average consensus protocol, and the feasible power interval is discretized at an interval of 1 (MW) to obtain a discrete set of actionable actions. As can be seen from Table 4, the generation cost of the algorithm is lower than that of the lag Q-learning algorithm in the prior art. It is higher than the Q-table-based Q-learning algorithm and cannot satisfy the constraints.
[0265] Table 4 Cost comparison results of distributed reinforcement learning algorithms
[0266]
[0267] It should be noted that assuming that the number of generators in the smart grid grows H times, and each generator needs to be pre-set with M actions, the action set needs to be pre-stored at a rate of HxM, and the Q table is stacked in an amount of HxMxT.
[0268] In this embodiment, the dynamic distributed economic dispatch problem is solved by an Actor-Critic algorithm based on constraint projection. As a state transition function, the constraint projection method provides a unique mapping from the action probability distribution to the combination of actionable actions. In addition, this algorithm does not need to additionally pre-establish a feasible action set. The simulation results show that this method can effectively coordinate the generation cost.
[0269] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM) or a random access memory (RAM), etc.
[0270] Obviously, those skilled in the art should understand that the modules or steps of the present application described above can be realized by general computing devices, which can be centralized on a single computing device or distributed on a network composed of multiple computing devices, and optionally, they can be realized by program codes executable by computing devices, so that they can be stored in computer storage media (ROM / RAM, magnetic disc, optical disc) and executed by computing devices, and in some cases, the steps shown or described can be executed in different order, or they can be made into individual integrated circuit modules, or multiple modules or steps can be made into a single integrated circuit module. Therefore, the present application is not limited to any specific hardware and software combination.
[0271] The above is a further detailed description of the present application in combination with specific embodiments, and the specific implementation of the present application should not be limited to these descriptions. For those skilled in the art, some simple deductions or substitutions can be made without departing from the concept of the present application, which should be considered as falling within the protection scope of the present application.
Claims
1. A distributed economic scheduling optimization method based on constrained projection reinforcement learning, characterized in that, Includes the following steps: Based on operational constraints, a distributed economic dispatch mathematical model is constructed by combining the power generation cost function and the feasible power output of the power generation unit. The operational constraints include node power balance constraints, ramp rate constraints, and power generation capacity constraints. An average consensus method is applied to each generator in the regional communication network to obtain global information in the microgrid in a distributed manner. The Actor-Critic algorithm is used to transform the mathematical model of distributed economic scheduling into a distributed economic scheduling optimization problem based on constraint projection; By combining the Lagrange multiplier method and the penalty function method, the augmented Lagrange dual form of the distributed economic scheduling optimization problem is obtained, and the optimal power output of the distributed economic scheduling optimization problem is obtained based on the KKT conditions.
2. The distributed economic scheduling optimization method based on constrained projection reinforcement learning according to claim 1, characterized in that, The mathematical model for distributed economic dispatch, based on operational constraints and combining the generation cost function and the feasible power output of the generation unit, specifically includes: The constructed distributed economic scheduling mathematical model is as follows: |after i,t -after i,t-1 |≤pr i (3) Equations (2)-(4) represent the node power balance limit, ramp rate limit, and generation capacity limit, respectively, where i = 1, ..., N, t = 1, ..., T, T represents the total time slot, and C i It is the power generation cost function, po i,t It is the feasible power output of the power generation unit, pd i,t It refers to the power demand of load units in the public power grid, pr i This represents the ramp rate limit for the i-th generator. and These are the minimum and maximum power generation limits for the i-th generator; Neglecting line transmission losses, we have: In the formula, PD t This is the total power requirement for time slot t; The mixed cost function that includes multiple energy options is: In the formula, The cost coefficient of the i-th generator, m = 1,...,M, h = 1,...,H. When solving formula (1), assumptions 1 and 2 are set. Assumption 1 is that, based on the communication topology of the microgrid, each load bus signal can be sent to the corresponding generator bus; assumption 2 is that there is at least one feasible solution in the distributed economic dispatch mathematical model. Satisfy the operational constraints (2)-(4).
3. The distributed economic scheduling optimization method based on constrained projection reinforcement learning according to claim 2, characterized in that, The method of implementing average consistency for each generator in the regional communication network to obtain global information of the microgrid in a distributed manner specifically includes: The dynamic average consensus estimation equation based on the second-order proportional integral is as follows: In the formula, N i U represents the set of adjacent nodes. i It is the control input, v i It is an estimated state, z i It is a decision variable; If the system topology is an undirected graph, then the estimated state converges to all control inputs u. i The cumulative average, i.e. When each generator meets the aforementioned conditions, the control input u can be calculated through the global variable N. i The accumulation of.
4. The distributed economic scheduling optimization method based on constrained projection reinforcement learning according to claim 3, characterized in that, The process of transforming the distributed economic scheduling mathematical model into a distributed economic scheduling optimization problem based on constrained projection using the Actor-Critic algorithm specifically includes: Based on Markov decision-making, the next process is represented as: Pr(s t+1 ∣s0,a0,…,s t ,a t )=Pr(s t+1 ∣s t ,a t ) (8) Reinforcement learning involves a quintuple (S, A, T, R, Π) of T processes, where the quintuple includes the state set S = {s...} t }; Action set A = {a t }; State transition model Reward set R = {r t } and the policy distribution of actions {π t }, Indicates that in state s t and action a t Move down to the next state s t+1 The probability of π; t ~Pr(a t |s t ) indicates that according to s t Choose a t The probability; as a long-term cumulative reward, the return By optimizing the action strategy π in each training session t To be maximized / minimized; The output of the Actor-Critic network is represented as follows: In the formula, and These are the weights of the Actor neural network and the Critic neural network, respectively; S a and S c It is an activation function; and It is a feature vector; action probability. It is the output of the Actor network, the state-value function. This is the output of the Critic network at time slot t and iteration index k; The temporal difference error for each agent is: In the formula, the reward function is: λ i ∈[0,1] is the discount factor; The loss function of the Critic neural network is defined as: For the Actor network, the temporal difference error is used as the evaluation function, and the Actor-Critic is updated based on backpropagation, as shown in the following formula: In the formula, la and lc represent the learning rates of the Actor and Critic, respectively, in the terminal time slot T. satisfy That is, state value Set it to 0 at time slot T+1; Based on constraints (2)-(4), the distributed economic scheduling mathematical model is transformed into a distributed economic scheduling optimization problem, which is: Evaluate actionable actions using the Critic framework State value And based on the constraint projection properties and the given State value for arrive The only projection of .
5. The distributed economic scheduling optimization method based on constrained projection reinforcement learning according to claim 4, characterized in that, The method combines the Lagrange multiplier method and the penalty function method to obtain the augmented Lagrange dual form of the distributed economic scheduling optimization problem, and obtains the optimal power output of the distributed economic scheduling optimization problem based on the KKT conditions, specifically including: Transforming formula (15) into a dual problem, and combining the Lagrange multiplier method and the penalty function method, we obtain the augmented Lagrange dual form as follows: In the formula, L is the Laplace matrix of the system network, and the multipliers are Y = (y1, y2, ..., y3). N ) T X = (x1, x2, ..., x N ) T ; According to the KKT conditions, the optimal solution to the distributed economic scheduling optimization problem is the numerical solution of the equilibrium point of the objective function (15). Based on the projection form of the augmented Lagrange function, a distributed projection algorithm is adopted: In the formula, we have: Assumption 3: Assume that the intersection of the slope rate constraint interval and the power generation constraint interval is not an empty set, i.e. The input to the Actor network is obtained through constraint projection. From equation (15), we can derive: Based on the backpropagation update method, for each generator: Assume the combined power ramp rate constraint is: PR t =PR 1,t ×PR 2,t ×…×PR N,t The globally feasible power output vector is then: AFTER t =(after 1,t ,after 2,t ,…,after N,t ) The global power output vector of the Actor policy network is: P t =(p 1,t ,p 2,t ,…,p N,t ) The global form obtained from equation (17) is: At the same time, from have: Among them 1 N =(1,...,1) T The global form of the local differential system (17) is: Based on the LaSalle invariant set theorem and the projection theorem, it is shown that, when the above assumptions 1-3 hold, the iterative trajectory of the constrained projection algorithm is bounded and converges to obtain the optimal solution to the distributed economic scheduling optimization problem.
6. The distributed economic scheduling optimization method based on constrained projection reinforcement learning according to claim 5, characterized in that, Based on LaSalle's invariant set theorem and projection theorem, when assumptions 1-3 hold, the iterative trajectory of the constrained projection algorithm is bounded and converges to obtain the optimal solution to the distributed economic scheduling optimization problem, specifically including: The global differential system (21) is further integrated, assuming: Z=(PO t ,Y,X) T (22) The equilibrium point of the differential system (21) is defined as: The projection range is expanded to Combining (21) and (23), we get: Equation (21) can be transformed into: in, Design the following Lyapunov function: The continuous differential system (25) is represented as: Where s is a continuous variable, equation (26) is: From the properties of inner product, we know that: Furthermore, (25) concludes that: Based on the properties of projection, we have: Therefore, the Lyapunov function satisfies: In the formula, If and only if Z(s) = Z * ,but: The derivative of the function L(s) is given by the following equation: based on Formula (33) can be written as: Among them, the Jacobian matrix J Q (Z) is represented as: According to the projection properties of constrained projection, we have: have: From (34), we can conclude that: get: From (30) and (39): Near the equilibrium point, there exist the following compact invariant sets: From (38), we have: Inserting Q(Z) into (42), we have: Relying on KKT conditions Then we have: Equation (43) satisfies: Let Θ be the set of all points such that Right now Θ is a maximal invariant set in the invariant set S. Based on LaSalle's invariant set theorem and the convexity of the distributed economic scheduling optimization problem, the feasible power obtains the optimal power output through asymptotic convergence, i.e.