Power grid dispatching strategy optimization method and device

Through the design of diversified and hierarchical reward mechanisms, combined with Markov's decision-making process and objective function, the grid scheduling strategy is optimized, and the problem of difficult to balance the consumption capacity of new energy units and the economics of thermal power units in the existing technology is improved. The generalization and training efficiency of power grid scheduling achieve coordinated optimization of safety and economical power grid operation.

CN119010208APending Publication Date: 2024-11-22HOHAI UNIV +2
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411044335.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The existing deep reinforcement learning methods have poor generalization in power grid scheduling, low training efficiency, and cannot effectively take into account the consumption capacity of new energy units and the economics of thermal power units.

Method used

Using diversified and hierarchical reward mechanism design, the Markov decision-making process is constructed, combined with the objective function, reward function and potential function, optimize the grid scheduling strategy, focus on safety and economical early stage of training, and use the weight coefficient to adjust the reward The impact of the function achieves the goal of maximizing the consumption of new energy units and the lowest total cost of thermal power units.

Benefits of technology

The generalization and training efficiency of power grid scheduling strategies are improved, and the safety and economicality of power grid operation can be better taken into account, so as to achieve coordinated optimization of the maximum consumption of new energy units and the lowest total cost of thermal power units.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119010208A_ABST
    Figure CN119010208A_ABST
Patent Text Reader

Abstract

The invention provides a power grid dispatching strategy optimization method and device. The method comprises the following steps: acquiring current state data of a power grid system; inputting the data into a pre-trained power grid dispatching strategy optimization model to obtain an optimized power grid dispatching strategy; the construction of the model comprises the following steps: constructing a target function according to a power grid dispatching strategy optimization target; constructing a Markov decision process including a state space, an action space and a reward mechanism according to the target function; the reward mechanism comprises constructed reward functions and set reward contents, and the reward functions comprise a new energy unit consumption reward, a thermal power unit operation cost reward, a balance unit output reward and an auxiliary reward; the construction of the reward function comprises the design of a reward function weight coefficient, and the reward function weight coefficient is variable in the training process of the power grid dispatching strategy optimization model. According to the method, a power grid dispatching strategy capable of safely realizing consumption maximization of the new energy unit and total cost minimization of the thermal power unit can be obtained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a method and a device for optimizing a power grid dispatching strategy, and belongs to the technical field of power dispatching. Background Art

[0002] With the continuous increase in the penetration rate of renewable energy, the uncertainty at the source and load ends is increasing, the types and number of dispatchable objects are growing exponentially, and the operation of the power grid system is becoming increasingly complex. Traditional dispatching methods are difficult to ensure the balance between the absorption capacity of new energy units and the economic efficiency of thermal power units in the power grid. Therefore, in recent years, deep reinforcement learning (DRL) has received widespread attention as an emerging intelligent dispatching method. However, the existing deep reinforcement learning methods focus on modeling, training result mapping transformation, multi-agent training, etc. The generalization of a single agent is poor, the training efficiency is not high, and it is impossible to balance the absorption capacity of new energy units and the economic efficiency of thermal power units. Summary of the invention

[0003] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a method and device for optimizing a power grid dispatching strategy. Through the diversified and hierarchical design of the reward mechanism, the operational safety of the power grid dispatching strategy and the economic goals of maximizing the consumption of new energy units and minimizing the total cost of thermal power units are comprehensively considered for coordinated optimization, and finally an optimized power grid dispatching strategy is obtained. To achieve the above purpose / solve the above technical problems, the present invention is implemented by adopting the following technical solutions:

[0004] In a first aspect, the present invention provides a method for optimizing a power grid dispatching strategy, comprising:

[0005] Obtain current status data of the power grid system;

[0006] Inputting the current state data of the power grid system into a pre-trained power grid dispatch strategy optimization model to obtain an optimized power grid dispatch strategy;

[0007] The construction of the power grid dispatching strategy optimization model includes:

[0008] Constructing an objective function according to the optimization goal of the power grid dispatching strategy, wherein the optimization goal of the power grid dispatching strategy is to maximize the consumption of new energy units and minimize the total cost of thermal power units;

[0009] Constructing a Markov decision process according to the objective function, wherein the Markov decision process includes a state space, an action space, and a reward mechanism;

[0010] The reward mechanism includes a constructed reward function and set reward content, wherein the reward function includes a reward for the consumption of new energy units, a reward for the operating costs of thermal power units, a reward for the output of balancing units and an auxiliary reward; the construction of the reward function includes the design of a weight coefficient of the reward function, and the weight coefficient of the reward function changes during the training process of the power grid dispatching strategy optimization model; the auxiliary reward is used to reward the balancing unit for safely completing the output of a time period.

[0011] In combination with the first aspect, optionally, the maximization of the consumption of the new energy units adopts different measurement standards according to different types of new energy. When the new energy adopts wind power, the overall wind abandonment rate of wind power is used for measurement. The objective function is designed as follows:

[0012]

[0013] In the formula, and They represent the cost of thermal power units and the overall wind curtailment rate of new energy units, ρ T and ρ W They respectively refer to the weight coefficients of the cost of thermal power units and the overall wind curtailment rate of new energy units. t is the index of the time step in the actual scheduling execution time axis. The value of t is a positive integer within the time period T. τ is the index of the time in the forward-looking prediction time window. The total length of τ is N, and its value is an integer from 0 to N-1.

[0014] The present invention takes the influence of time factors into consideration in both model design and training stages.

[0015] When the absorption level of new energy units is measured by the overall wind abandonment rate of new energy units, minimizing the overall wind abandonment rate of new energy units means maximizing the absorption of new energy-efficient units.

[0016] In combination with the first aspect, optionally, the weight coefficient of the reward function changes during the training process of the power grid dispatch strategy optimization model, which means that the weight coefficient of the reward function is used to improve the influence of the balanced unit output reward and auxiliary reward in the early stage of training, and changes its size in the later stage of training to integrate the influence of various reward functions.

[0017] Training the power grid dispatch strategy optimization model includes focusing on safety rewards in the early stage of model training and focusing on economic rewards in the later stage of training. The safety rewards include balanced unit output rewards and auxiliary rewards. The economic rewards include new energy unit consumption rewards, thermal power unit operating cost rewards, balanced unit output rewards and auxiliary rewards. In order to make the intelligent agent pay more attention to safety learning in the early stage of training, the present invention increases the weight coefficient of safety rewards in the early stage of training; in order to make the intelligent agent pay attention to maximizing the overall reward in the later stage of training, the present invention reduces the weight coefficient of safety rewards in the later stage of training, and pays more attention to the comprehensive impact of various reward functions.

[0018] In combination with the first aspect, optionally, the reward content includes positive rewards and negative rewards, the new energy unit consumption reward is a positive reward, the thermal power unit operating cost reward is a negative reward, the balancing unit output reward includes a balancing unit active output dynamic reward, the reward content of the balancing unit active output dynamic reward is divided according to the relationship between the balancing unit active output value and the upper limit and lower limit of the balancing unit active output, and the auxiliary reward is a positive reward when the balancing unit safely completes the output of a time period.

[0019] The present invention adopts the setting of diversified and multi-level reward mechanisms, so that the power grid dispatch strategy optimization model can take into account the influence of various aspects during training and use, and more synergistically achieve operational safety and achieve the overall optimization goals of maximizing the absorption of new energy units and minimizing the total cost of thermal power units.

[0020] In combination with the above method, optionally, the reward content of the dynamic reward for the active output of the balancing unit is divided according to the relationship between the active output value of the balancing unit and the upper limit and lower limit of the active output of the balancing unit, including:

[0021] Separately divide the range of active output values ​​of balancing units and the content of rewards;

[0022] Establishing a corresponding relationship between the active output value of the balancing unit and the reward content according to the range to which it belongs;

[0023] The corresponding relationship between the active output value of the balancing unit and the reward content is established according to the range to which the active output value of the balancing unit belongs, including judging whether the active output value of the balancing unit is within the range of the upper limit value and the lower limit value of the active output of the balancing unit, the area outside the range of the upper limit value and the lower limit value of the active output of the balancing unit belongs to the restricted area, and the reward content corresponding to the restricted area is a negative reward, and the area within the range of the upper limit value and the lower limit value of the active output of the balancing unit belongs to the non-restricted area, and the reward content corresponding to the non-restricted area is a positive reward;

[0024] The safe completion of the output of a time period means that during the operation of the balancing unit in a time period, there is no situation where the active output falls within the restricted area and the power flow does not converge.

[0025] If the active output value of the balancing unit falls within the restricted area or the balancing unit fails to converge during the training, the training ends directly. In this case, it does not matter whether the specified time period is reached at the current moment. The failure of the flow to converge is information that can be checked in the power grid environment.

[0026] In combination with the above method, optionally, the respectively dividing the range of the active output value of the balancing unit and the reward content includes: the non-restricted area is divided into a safe area and a transition area, the positive reward is divided into a larger reward and a normal reward, and the corresponding relationship between the range of the active output value of the balancing unit and the reward content is established according to which the safe area corresponds to a larger reward, and the transition area corresponds to a normal reward;

[0027] Alternatively, the respective division of the range of active output value of the balancing unit and the reward content includes: the non-restricted area is divided into a safe area, a transition area 1, and a transition area 2, the positive reward is divided into a larger reward, an ordinary reward, and a smaller reward, and the corresponding relationship between the reward content and the range of active output value of the balancing unit is established includes: the safe area corresponds to a larger reward, the transition area 1 corresponds to an ordinary reward, and the transition area 2 corresponds to a smaller reward.

[0028] In combination with the first aspect, optionally, the training of the power grid dispatch strategy optimization model adopts a potential function and a regularization term, wherein the regularization term is a comment network regularization term based on a reward mechanism, and the regularization term F(s t ,s t+1 )The formula is as follows:

[0029] F(s t ,s t+1 )=γΦ(s t+1 )-Φ(s t )

[0030] In the formula, s t represents the system state at time t, Φ(s t ) indicates that the independent variable is s t The potential function, s t+1 represents the system state at time t+1, Φ(s t+1 ) indicates that the independent variable is s t+1 , γ is the reinforcement learning discount factor for agent training on the actual scheduling execution timeline, γ∈[0,1);

[0031] The potential function is a potential function based on the reward mechanism. The potential function Φ(s t ) is as follows:

[0032]

[0033] In the formula, the potential function Φ(s t ) is a piecewise function divided into two layers, and the training period E max represents the maximum round of training, eps represents the current round, and the potential function Φ(s t ) In the first half of the training cycle, the potential function is expressed as Φ 1 (s t), the potential function Φ(s t ) In the second half of the training cycle, the potential function is expressed as Φ 2 (s t ).

[0034] In order to take into account the coordinated optimization of safety and multiple economic objectives, as well as the impact of future moments on the current moment, the model training of the present invention focuses on safety in the early stage and on the coordinated improvement of safety and economy in the later stage. The introduction of potential functions and regularization terms improves the training efficiency. When designing the potential function and regularization terms, the potential function is designed as two layers, and the impact of future moments on the current moment is added to the regularization term.

[0035] Combined with the above method, optionally, the potential function Φ of the first half of the training cycle 1 (s t ) is designed as follows:

[0036]

[0037]

[0038] In the formula, represents the summation operation of τ values ​​from 0 to N-1, P τ BG,t Indicates the active output value of the balancing unit. and They represent the upper and lower limits of the active output of the balancing machine respectively. Indicates the middle value of the active output of the balancing unit;

[0039] Potential function Φ in the second half of the training cycle 2 (s t ) is designed as follows:

[0040]

[0041]

[0042]

[0043]

[0044] In the formula, It represents the accumulated reward from the current round of eps training to the current time t. represents the summation operation of k from 0 to t, r k represents the reward function value at time k, and They represent the maximum and minimum cumulative rewards from the end of training to the current round.

[0045] The potential function of the first half of the training cycle of the present invention focuses on improving the safety of model training, and the potential function of the second half of the training cycle focuses on maximizing the comprehensive reward of the model. In the second half of the training cycle, the model training direction focuses on the coordinated optimization of the safety of the comprehensive power grid dispatching strategy and better realization of multiple economic goals.

[0046] In combination with the aforementioned method, optionally, the training of the power grid dispatch strategy optimization model also adopts a loss function, the design of the loss function is based on the potential function and regularization term of the reward mechanism, and the loss function is used to update the comment network in the SAC algorithm. The loss function is as follows:

[0047]

[0048]

[0049] In the formula, J Q (θ Q ) represents the loss function in updating the Q network, It is represented by the parameter θ Q The state s predicted by the Q network t and action a t The corresponding Q value; Q target Represents the target Q value, which is the target value used to train the Q network; It is part of the Q network loss function and represents half of the square error between the predicted Q value and the target Q value; α is the exploration rate, which represents the weight of the policy entropy and is used to balance exploration and utilization; logπ(a t+1 |s t+1 ) means in state s t+1 Take action a t+1 The logarithm of the strategy probability; θ Q and The network parameters of the main target critic; represents mathematical expectation; F is the weight parameter that controls the regularization term.

[0050] The loss function designed by the present invention introduces a regularization term based on the reward mechanism of the present invention, which can achieve the coordinated realization of comprehensive security of model training and multiple economic objectives.

[0051] In a second aspect, the present invention provides a power dispatching strategy optimization device, comprising:

[0052] Data acquisition module, used to obtain current status data of the power grid system;

[0053] An optimization module, used for inputting the current state data of the power grid system into a pre-trained power grid dispatch strategy optimization model to obtain an optimized power grid dispatch strategy;

[0054] The construction of the power grid dispatching strategy optimization model includes:

[0055] Constructing an objective function according to the optimization goal of the power grid dispatching strategy, wherein the optimization goal of the power grid dispatching strategy is to maximize the consumption of new energy units and minimize the total cost of thermal power units;

[0056] Constructing a Markov decision process according to the objective function, wherein the Markov decision process includes a state space, an action space, and a reward mechanism;

[0057] The reward mechanism includes a constructed reward function and set reward content, wherein the reward function includes a reward for the consumption of new energy units, a reward for the operating costs of thermal power units, a reward for the output of balancing units and an auxiliary reward; the construction of the reward function includes the design of a weight coefficient of the reward function, and the weight coefficient of the reward function changes during the training process of the power grid dispatching strategy optimization model; the auxiliary reward is used to reward the balancing unit for safely completing the output of a time period.

[0058] Beneficial effects:

[0059] The present invention, through the diversified and hierarchical design of the reward mechanism, comprehensively considers the operational safety of the power grid dispatching strategy and the coordinated optimization of the economic goals of maximizing the absorption of new energy units and minimizing the total cost of thermal power units, and finally obtains an optimized power grid dispatching strategy.

[0060] The present invention, through the setting of rich reward functions and reward contents, as well as the consideration of time factors, enables the power grid dispatching strategy optimization model to take into account the influence of various aspects during training and use. On the basis of the reward mechanism of the present invention, the design of potential function, regularization term and loss function is introduced to improve the efficiency of model training, taking into account the coordinated optimization of the safety of power grid system operation and the economy of multiple economic objectives, so that the trained power grid dispatching strategy optimization model can output an optimized power grid dispatching strategy that takes into account both safety and economic objectives. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 is a flow chart of constructing a power grid dispatching strategy optimization model in one embodiment of the present invention;

[0062] Figure 2 It is a schematic diagram of a design of a balanced unit output reward in one embodiment of the present invention;

[0063] Figure 3 It is a schematic diagram of a training process of a power grid dispatching strategy optimization model in one embodiment of the present invention; DETAILED DESCRIPTION

[0064] It should be noted that: the technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations on the technical solution of the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.

[0065] The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0066] Embodiment 1

[0067] Figure 1 This is a flowchart of constructing a power grid dispatching strategy optimization model of an embodiment of the power grid dispatching strategy optimization method of the present invention. This flowchart only shows the logical sequence of the method described in this embodiment. In other possible embodiments of the present invention, different Figure 1 The steps shown or described are accomplished in the order shown.

[0068] refer to Figure 1 As shown, this embodiment introduces a method for optimizing a power grid dispatching strategy, including:

[0069] Obtain current status data of the power grid system;

[0070] Inputting the current state data of the power grid system into a pre-trained power grid dispatch strategy optimization model to obtain an optimized power grid dispatch strategy;

[0071] The construction of the power grid dispatching strategy optimization model includes:

[0072] Constructing an objective function according to the optimization goal of the power grid dispatching strategy, wherein the optimization goal of the power grid dispatching strategy is to maximize the consumption of new energy units and minimize the total cost of thermal power units;

[0073] Constructing a Markov decision process according to the objective function, wherein the Markov decision process includes a state space, an action space, and a reward mechanism;

[0074] The reward mechanism includes a constructed reward function and set reward content, wherein the reward function includes a reward for the consumption of new energy units, a reward for the operating cost of thermal power units, a reward for the output of balanced units, and an auxiliary reward;

[0075] The construction of the reward function includes the design of a reward function weight coefficient, and the reward function weight coefficient changes during the training process of the power grid dispatch strategy optimization model; the auxiliary reward is used to reward the balancing unit for safely completing the output of a time period.

[0076] The present invention can output an optimized grid dispatching strategy according to the current state data of the grid through the hierarchical design of the reward mechanism. The strategy can collaboratively achieve the comprehensive optimization goals of the grid dispatching strategy of maximizing the consumption of new energy units and minimizing the total cost of thermal power units.

[0077] Embodiment 2

[0078] This embodiment introduces the specific implementation process of the power grid dispatching strategy optimization method based on the inventive concept of the first embodiment, which specifically includes the following contents:

[0079] 1. Constructing the objective function based on the optimization goal of the power grid dispatching strategy

[0080] The current optimization goal of the power grid dispatching strategy is to maximize the absorption of new energy units and minimize the total cost of thermal power units. The objective function constructed according to the optimization goal of the power grid dispatching strategy also consists of two parts, the absorption level of new energy units and the operating cost of thermal power units. Among them, the absorption level of new energy units can be divided into wind power, hydropower, photovoltaic power generation absorption level, etc. according to the type of new energy. The absorption level of new energy units in this embodiment is measured by the overall wind power abandonment rate. The objective function represents the minimization of the operating cost of thermal power units and the overall wind abandonment rate of new energy units, where the operating cost of thermal power units is the amount of thermal power used. The specific objective function is expressed as follows:

[0081]

[0082] In the formula, and They represent the cost of thermal power units and the overall wind curtailment rate of new energy units, ρ T and ρ W They refer to the weight coefficients of the cost of thermal power units and the overall wind curtailment rate of new energy units, respectively. Specific values ​​can be set as needed. In one embodiment, ρ T and ρ W Both are set to 1.

[0083] The formula of the cost function of thermal power units is:

[0084]

[0085] The formula for the overall wind curtailment rate function of new energy units is:

[0086]

[0087] Wherein, t is the index of the time step in the actual scheduling execution time axis, and the value of t is a positive integer within the range of the time period T. In one embodiment, the time period T is 288, and the granularity of t is 5 minutes. The granularity is the actual time interval. The time period T is 288, which means that the time period T is 24 hours; τ is the index of the time in the forward prediction time window, and the total length of τ is N, which is an integer from 0 to N-1. In one embodiment, N is 48, and the granularity of τ is 5 minutes. N is 48, which means that the total length of the forward prediction time window is 4 hours; g T is the thermal power unit number, n T is the total number of thermal power units; g W is the new energy unit number, n W is the total number of thermal power generating units; Indicates the number g T The active power output of the thermal power unit at time t is Indicates the number g W The active power output of the new energy unit at time t, It represents the maximum active output of the new energy unit at time t, and They are thermal power units g T The cost coefficient of each unit is a different system value. and All of these are data that can be read in the power grid system environment.

[0088] 2. Constraints of the Grid Dispatching Strategy Optimization Model

[0089] The grid dispatching strategy needs to meet the constraints, which are the prerequisites for dispatching operation. Accordingly, when realizing the objective function, these constraints need to be met first. The constraints include power balance constraints, unit output constraints, and unit ramp constraints:

[0090] 1. Power balance constraints, including active power balance constraints of all units and all loads and reactive power balance constraints of all units and all loads. All units include thermal power units and new energy units. Balanced units belong to thermal power units. Balanced units are units in the power grid system that ensure the balance between the output power of the grid dispatching strategy and the load demand. The constraint formula is as follows:

[0091]

[0092]

[0093] In the formula, all data are exportable data in the power grid system environment, n Bis the total number of nodes in the power grid, i and j represent the i-th and j-th nodes respectively, and both i and j are positive integers within the value range of the total number of nodes in the power grid. i G and P i D They represent the active power of all units and all loads on node i, Q i G and Q i D Represents the reactive power of all units and all loads at node i, V i and V j Represent the voltage amplitudes at nodes i and j, respectively, G ij , B ij and δ ij are the conductance, susceptance and phase angle at nodes i to j respectively.

[0094] 2. Unit output constraints, including line current and voltage constraints, active power constraints of thermal power units and new energy units, reactive power constraints of units, and unit start / shutdown time constraints. The specific formula is as follows:

[0095] Indicates that the current at any time and line should satisfy the constraints,

[0096] In the formula, all data are exportable data in the power grid system environment. It refers to the current on branch l at time t, where l is a positive integer within the range of the total power line. is the maximum current of branch l, ξ thrshld is the preset threshold coefficient, and its value is 1.35.

[0097] Indicates that the voltage amplitude at any time and node should satisfy the constraints,

[0098] In the formula, all data are exportable data in the power grid system environment, V i t is the voltage amplitude at node i at time t, V i min and V i max Refers to the lower and upper limits of the voltage amplitude respectively. Adjusting the active output of the unit will affect the voltage and current of the line, so the current and voltage of the line need to be constrained.

[0099] It means that the active output of any thermal power unit at any time should satisfy the constraint.

[0100] It means that the active output of any new energy unit at any time should meet the constraints,

[0101] In the formula, all data are exportable data in the power grid system environment. and They refer to the minimum and maximum active output of thermal power units respectively. It is the maximum active output of the new energy unit.

[0102] It means that the reactive power of any unit at any time should satisfy the constraint,

[0103] In the formula, all data are exportable data in the power grid system environment. and They refer to the minimum and maximum reactive outputs of all units respectively. Refers to number g G The reactive output value of the unit at time t.

[0104] The constraint of the unit start / shutdown time means that the start / shutdown time of all units should meet the minimum start / shutdown time, which is judged by the number of steps required for the generator to recover. The generator is not allowed to be shut down within the specified time step after starting, and the generator is not allowed to be restarted within the specified time step after shutting down.

[0105] 3. Unit climbing constraints

[0106] It means that the ramp rate of any thermal power unit at any time should meet the constraints,

[0107] In the formula, all data are exportable data in the power grid system environment. Indicates the number g T The ramp rate of the thermal power unit at time t, D gT and U gT They respectively represent the lower and upper limits of the climbing slope of the thermal power unit.

[0108] 3. Markov decision process includes state space, action space and reward mechanism

[0109] 1. State space s t Represents the system state at the current time (time t), which is the set of active output system states of all generator sets in the entire forward-looking prediction time window of the previous time step. The set includes: the active output set of all units, the upper limit set of all unit adjustment values ​​at the current time, the lower limit set of all unit adjustment values, the set of active output changes of all unit loads, and the set of total predicted loads at the current time:

[0110] State space s t The specific contents are as follows:

[0111]

[0112]

[0113]

[0114]

[0115]

[0116]

[0117] In the formula, the total active output of all units is Represents the set of active outputs of all units at the previous moment, and the upper limit set of all unit adjustment values Indicates the upper limit set of the active output adjustment values ​​of all units at the current moment, and the lower limit set of the adjustment values ​​of all units Represents the lower limit set of the active output adjustment value of all units at the current moment, and the set of active output change of all units load It represents the set of load active output changes between the next moment and the current moment, and the total predicted load at the current moment. represents the active output of all unit loads at the current moment, d represents the number of all unit loads, n D Indicates the total number of load numbers of thermal power units and new energy units.

[0118] 2. Action Space a t Represents the dispatch decision at the current moment (time t), which is the set of all unit actions at the current moment, including the set of active output adjustment values ​​of thermal power units and new energy units within the entire forward-looking prediction time window of the current time step:

[0119]

[0120]

[0121]

[0122]

[0123] In the formula, represents the set of active power adjustment values ​​of all units at the current time (time t), and Respectively represent the number g T The thermal power unit has active output at time t+1 and time t, numbered g T The current active output adjustment value of the thermal power unit Indicates the number g TThe difference between the active output of the thermal power unit at time t+1 and time t, and Respectively represent the number g W The new energy unit has active output at time t+1 and time t, numbered g W The current active output adjustment value of the new energy unit Indicates the number g W The difference in active output of the new energy unit at time t+1 and time t.

[0124] 3. The reward mechanism includes the construction of the reward function. The reward mechanism guides the learning direction of the agent through the constructed reward function and the set reward content, so that the agent can learn to perform tasks to maximize the cumulative reward. The formula of the reward function is:

[0125] r t =(ρ W ·r t W +ρ T ·r t T )+λ(r t BG +r t AUX )

[0126] In the formula, the reward function weight coefficient λ is the weight coefficient for measuring safety, and ρ T and ρ W They refer to the weight coefficients of the cost of thermal power units and the overall wind curtailment rate of new energy units, respectively. Specific values ​​can be set as needed. In one embodiment, ρ T and ρ W are all set to 1. The reward content includes positive rewards and negative rewards. The reward function is divided into two parts. The first part includes the cost reward r of the thermal power unit. t T and incentives for new energy units t W , thermal power unit cost reward r t T Negative reward, new energy unit consumption reward r t W is a positive reward, and the second part includes the reward for balancing the unit output r t BG and auxiliary rewards t AUX , auxiliary rewards are used to reward the balanced unit for safely completing the output of a time period.

[0127] The weight coefficient of the reward function changes during the training process of the power grid dispatch strategy optimization model, and changes over time. In the early stage of training, safety rewards are emphasized, and in the later stage of training, economic rewards are emphasized. The safety rewards include balanced unit output rewards and auxiliary rewards. The economic rewards include new energy unit consumption rewards, thermal power unit operating cost rewards, balanced unit output rewards and auxiliary rewards. The weight coefficient of the reward function is used to improve the influence of balanced unit output rewards and auxiliary rewards in the early stage of training, and changes in size in the later stage of training to comprehensively influence the influence of various reward functions. In one embodiment, the first half of the training cycle λ is set to 5, and the second half of the training cycle λ is set to 1.

[0128] 1) Reward function r for new energy unit consumption t W , the formula is as follows:

[0129]

[0130] In the formula, ζ τ is the τth power of the discount coefficient ζ, the discount coefficient represents the importance of the reward in the future period to the current reward, ζ is a preset value, and the value range is (0, 1). In one embodiment, ζ is 0.99, and the new energy unit consumption reward function is a positive reward.

[0131] 2) Reward function of thermal power unit operation cost r t T , the formula is as follows:

[0132]

[0133] In the formula, and It is a thermal power unit g T The cost coefficient of each unit is a different fixed value, which is the data in the power grid system environment. The cost reward of thermal power units is r t T Negative reward.

[0134] 3) Balancer output reward function r t BG , the formula is as follows:

[0135]

[0136] In the formula, ζ τ is the discount factor ζ to the power of τ, To provide dynamic rewards for balancing the active output of the units.

[0137] Combination Figure 2 As shown, Indicates the active output value of the balancing unit, which is a state quantity that can be read in the power grid system environment. The active output dynamic reward of the balancing unit According to the active output value of the balancing unit The upper limit of the active output of the balancing machine and lower limit The relationship between the two is divided into different levels of reward content, including large rewards, normal rewards and negative rewards. Large rewards and normal rewards are positive rewards, balancing the active output value of the unit. The upper limit of the active output of the balancing machine and lower limit The relationship level is divided into safe area, transition area and restricted area.

[0138] Balanced unit active output value Dynamic reward for active output of balancing units τ BG The corresponding relationship is: It is a safe area. Get a bigger reward; Belongs to the transition area, Get normal rewards; It is a restricted area. Receive negative rewards.

[0139] The transition area can be divided into transition area 1 and transition area 2. Correspondingly, the normal reward can be divided into ordinary reward and small reward. Balanced unit active output value Belongs to transition zone 1, Get ordinary rewards; Belongs to transition zone 2, Get a smaller reward. In one embodiment, the dynamic reward for the active output of the balancing unit The specific design is as follows:

[0140]

[0141] In the formula, the dynamic reward for active output of the balancing unit is Larger rewards obtained are set to +1, normal rewards are set to +0.5, smaller rewards are set to +0.1, and negative rewards are set to -1.

[0142] Balanced unit active output value The upper limit of the active output of the balancing machine and lower limit The relationship between the active output of the balanced unit and the and lower limit within the range.

[0143] The active output upper limit of the balance unit and lower limit between, hour, The active output upper limit of the balance unit and lower limit range; otherwise, or hour, The active output upper limit of the balanced unit and lower limit Out of range.

[0144] Balance unit active output upper limit and lower limit Areas outside the range are restricted areas, and dynamic rewards are given for meritorious contributions. Obtain negative rewards to balance the upper limit of the active output of the unit and lower limit The area within the range is a non-restricted area, and there are dynamic rewards for meritorious contributions. Get positive rewards,

[0145] refer to Figure 2 As shown, in one embodiment, and The area in between is a transition area; and The area between them is a safe area; and belongs to the transition area; wherein, in one embodiment, reference Figure 2 As shown, and The area between them belongs to transition area 2; and The areas between them belong to transition area 1; and The areas between them belong to transition area 1; and The area in between belongs to transition area 2.

[0146] 4) Auxiliary rewards t AUX , the design is as follows:

[0147]

[0148] Auxiliary rewards are used to reward balancing units for safely completing the output of a time period. The safe completion of the output of a balancing unit for a time period means that the balancing unit does not have active output in the restricted area and power flow does not converge during the operation of a time period. The upper and lower limits of the active output of the balancing unit and the power flow that does not converge are all existing information that can be checked in the power grid environment.

[0149] 4. Designing Potential Function and Regularization Term

[0150] The training of the power grid dispatching strategy optimization model adopts potential function and regularization term. The potential function is divided into two layers. In the first half of the training cycle, the potential function focuses on safety rewards, and in the second half of the training cycle, the potential function focuses on maximizing the reward value of the overall neural network.

[0151] 1. Regularization term based on reward mechanism

[0152] The regularization term refers to an additional term added to the loss function during model optimization to control the complexity of the model and prevent overfitting.

[0153] F(s t ,s t+1 )=γΦ(s t+1 )-Φ(s t )

[0154] In the formula, s t represents the system state at time t, Φ(s t ) indicates that the independent variable is s t The potential function, s t+1 represents the system state at time t+1, Φ(s t+1 ) indicates that the independent variable is s t+1 , γ is the reinforcement learning discount factor for agent training on the actual scheduling execution timeline, γ∈[0,1). In one embodiment, γ is set to 0.99.

[0155] 2. Potential function based on reward mechanism:

[0156] Potential function based on reward mechanism: The purpose of the potential function is to guide the agent to learn more effectively by introducing additional rewards. The potential function is designed as follows:

[0157]

[0158] In the formula, the potential function Φ(s t ) is a piecewise function divided into two layers, and the training period E max represents the maximum round of training, eps represents the current round, and the potential function Φ(s t ) In the first half of the training cycle, the potential function is expressed as Φ 1 (st ), potential function Φ(s t ) In the second half of the training cycle, the potential function is expressed as Φ 2 (s t ), in one embodiment, E max Set to 3000, max / 2 is 1500.

[0159] Potential function Φ for the first half of the training cycle 1 (s t ) is designed as follows:

[0160]

[0161] In the formula, represents the summation operation of τ values ​​from 0 to N-1, Indicates the active output value of the balancing unit. and They represent the upper and lower limits of the active output of the balancing machine, respectively, and are the setting values ​​of the power grid system environment. Indicates the middle value of the active output of the balancing unit.

[0162] Potential function Φ for the first half of the training cycle 1 (s t ), by making the active power output of the balancing unit as close to the midpoint as possible, additional gradient information is provided for active power output learning, so that the agent can pay more attention to the safety of the learning strategy in the first half of the training process.

[0163] Potential function Φ in the second half of the training cycle 2 (s t ) is designed as follows:

[0164]

[0165]

[0166]

[0167]

[0168] In the formula, It represents the accumulated reward from the current round of eps training to the current time t. represents the summation operation of k from 0 to t, r k represents the reward function value at time k, and They represent the maximum and minimum cumulative rewards from the end of training to the current round.

[0169] Potential function Φ in the second half of the training process 2 (st ), so that the agent can comprehensively improve the safety and economy of the learning strategy in the second half of the training process.

[0170] 5. Design loss function

[0171] The training of the power grid dispatch strategy optimization model also uses a loss function, which is based on the potential function and regularization term mentioned above. The loss function is used to update the comment network in the SAC algorithm. The specific loss function is:

[0172]

[0173]

[0174] In the formula, J Q (θ Q ) represents the loss function in updating the Q network, It is represented by the parameter θ Q The state s predicted by the Q network t and action a t The corresponding Q value. target Represents the target Q value, which is the target value used to train the Q network. It is part of the Q network loss function and represents half of the square error between the predicted Q value and the target Q value. α is the exploration rate, which represents the weight of the policy entropy and is used to balance exploration and exploitation. In one embodiment, α is set to 0.2. logπ(a t+1 |s t+1 ) means in state s t+1 Take action a t+1 The logarithm of the strategy probability. Q and The network parameters of the main target critic, represents the mathematical expectation, γ is the reinforcement learning discount factor for agent training on the actual scheduling execution timeline, γ∈[0,1), in one embodiment, γ is set to 0.99. F To control the weight parameter of the regularization term, in one embodiment, λ F Set to 20.

[0175] 6. Training the Grid Dispatch Strategy Optimization Model

[0176] Use the SAC algorithm to train the dispatching agent and solve the rewards of all power grid dispatching strategies. The strategy with the largest reward is the current optimal power grid dispatching strategy. For the specific training process, refer to

[0177] Figure 3 :

[0178] S1: Initialization: Obtain the topology and initial state data of the power grid.

[0179] S2: The agent makes a scheduling decision: In each time period, the agent selects the optimal action based on the current state and reward function, that is, the current optimal power grid scheduling strategy.

[0180] S3: Execute action: Execute the selected grid dispatch strategy into the grid system and obtain the new state and reward function value.

[0181] S4: Update the agent: Use the SAC algorithm to update the policy network and the comment network through the potential function, regularization term, and loss function based on the reward mechanism.

[0182] S5: Continue scheduling: Completing steps S2-S4 is considered to complete one training, and repeating steps S2-S4 until the training cycle E max End, save step S4 to update the current model parameters output by the intelligent agent, and obtain the trained power grid dispatch strategy optimization model.

[0183] 7. Application of power grid dispatching strategy optimization model in power grid system:

[0184] The present invention uses data provided by the SG-126 power grid reinforcement learning standard example to train and test the power grid dispatch strategy optimization model. The SG-126 power grid reinforcement learning standard example is a set of standard examples provided by the China Electric Power Research Institute, which provides a standardized testing environment for reinforcement learning algorithms in power systems.

[0185] In one embodiment, the current state data of the power grid system includes the following information:

[0186] Load information: active load of load_p node, reactive load of load_q node and voltage amplitude of load_v node;

[0187] Generator information: gen_p generator active output, gen_q generator reactive output and gen_v generator terminal voltage amplitude;

[0188] Network topology information: load factor of rho line;

[0189] Time information: nextstep_load_p is the power load at the next moment;

[0190] Generator status: gen_status generator status information, steps_to_recover_gen the number of steps required for generator recovery;

[0191] Action space information: action_space: the upper and lower limits of power regulation for each generator;

[0192] By inputting the current status data of the above-mentioned power grid system into the trained power grid dispatch optimization model, the optimized power grid dispatch strategy can be quickly obtained.

[0193] Embodiment 3

[0194] Based on the same inventive concept as the first and second embodiments, this embodiment introduces a power grid dispatching strategy optimization device, which includes:

[0195] Data acquisition module, used to obtain current status data of the power grid system;

[0196] An optimization module, used for inputting the current state data of the power grid system into a pre-trained power grid dispatch strategy optimization model to obtain an optimized power grid dispatch strategy;

[0197] The construction of the power grid dispatching strategy optimization model includes:

[0198] Constructing an objective function according to the optimization goal of the power grid dispatching strategy, wherein the optimization goal of the power grid dispatching strategy is to maximize the consumption of new energy units and minimize the total cost of thermal power units;

[0199] Constructing a Markov decision process according to the objective function, wherein the Markov decision process includes a state space, an action space, and a reward mechanism;

[0200] The reward mechanism includes a constructed reward function and set reward content, wherein the reward function includes a reward for the consumption of new energy units, a reward for the operating cost of thermal power units, a reward for the output of balanced units, and an auxiliary reward;

[0201] The construction of the reward function includes the design of a reward function weight coefficient, and the reward function weight coefficient changes during the training process of the power grid dispatch strategy optimization model; the auxiliary reward is used to reward the balancing unit for safely completing the output of a time period.

[0202] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0203] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0204] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0205] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0206] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for optimizing power grid dispatching strategy, Its characteristics are: include: Obtain current status data of the power grid system; Inputting the current state data of the power grid system into a pre-trained power grid dispatch strategy optimization model to obtain an optimized power grid dispatch strategy; The construction of the power grid dispatching strategy optimization model includes: Constructing an objective function according to the optimization goal of the power grid dispatching strategy, wherein the optimization goal of the power grid dispatching strategy is to maximize the consumption of new energy units and minimize the total cost of thermal power units; Constructing a Markov decision process according to the objective function, wherein the Markov decision process includes a state space, an action space, and a reward mechanism; The reward mechanism includes a constructed reward function and set reward content, wherein the reward function includes a reward for the consumption of new energy units, a reward for the operating cost of thermal power units, a reward for the output of balanced units, and an auxiliary reward; The construction of the reward function includes the design of a reward function weight coefficient, and the reward function weight coefficient changes during the training process of the power grid dispatch strategy optimization model; the auxiliary reward is used to reward the balancing unit for safely completing the output of a time period.

2. The power grid dispatching strategy optimization method according to claim 1, Its characteristics are: The maximum consumption of new energy units adopts different measurement standards according to different types of new energy. When the new energy adopts wind power, the overall wind abandonment rate of wind power is used for measurement. The objective function is designed as follows: In the formula, and They represent the cost of thermal power units and the overall wind curtailment rate of new energy units, ρ T and ρ W They respectively refer to the cost of thermal power units and the weight coefficients of the overall wind curtailment rate of new energy units. t is the index of the time step in the actual scheduling execution time axis. The value of t is a positive integer within the time period T. τ is the index of the time in the forward-looking prediction time window. The total length of τ is N, and its value is an integer from 0 to N-1.

3. The power grid dispatching strategy optimization method according to claim 1, Its characteristics are: The reward function weight coefficient changes during the training process of the power grid dispatch strategy optimization model, which means that the reward function weight coefficient is used to improve the influence of the balanced unit output reward and auxiliary reward in the early stage of training, and changes its size in the later stage of training to integrate the influence of various reward functions.

4. The power grid dispatching strategy optimization method according to claim 1, Its characteristics are: The reward content includes positive rewards and negative rewards. The new energy unit consumption reward is a positive reward, the thermal power unit operating cost reward is a negative reward, and the balancing unit output reward includes a balancing unit active output dynamic reward. The reward content of the balancing unit active output dynamic reward is divided according to the relationship between the balancing unit active output value and the upper and lower limits of the balancing unit active output. The auxiliary reward is a positive reward when the balancing unit safely completes the output of a time period.

5. The power grid dispatching strategy optimization method according to claim 4, Its characteristics are: The reward content of the dynamic reward for the active output of the balancing unit is divided into the following according to the relationship between the active output value of the balancing unit and the upper and lower limits of the active output of the balancing unit: Separately divide the range of active output values ​​of balancing units and the content of rewards; Establishing a corresponding relationship between the active output value of the balancing unit and the reward content according to the range to which it belongs; The corresponding relationship between the active output value of the balancing unit and the reward content is established according to the range to which the active output value of the balancing unit belongs, including judging whether the active output value of the balancing unit is within the range of the upper limit value and the lower limit value of the active output of the balancing unit, the area outside the range of the upper limit value and the lower limit value of the active output of the balancing unit belongs to the restricted area, and the reward content corresponding to the restricted area is a negative reward, and the area within the range of the upper limit value and the lower limit value of the active output of the balancing unit belongs to the non-restricted area, and the reward content corresponding to the non-restricted area is a positive reward; The safe completion of the output of a time period means that during the operation of the balancing unit in a time period, there is no situation where the active output falls within the restricted area and the power flow does not converge.

6. The power grid dispatching strategy optimization method according to claim 5, Its characteristics are: The said respectively dividing the ranges of the active output values ​​of the balancing units and the reward contents includes: the said non-restricted area is divided into a safe area and a transition area, the said positive reward is divided into a larger reward and a normal reward, and the said establishing a corresponding relationship between the ranges of the active output values ​​of the balancing units and the said reward contents includes: the safe area corresponds to a larger reward, and the transition area corresponds to a normal reward; Alternatively, the respective division of the range of active output value of the balancing unit and the reward content includes: the non-restricted area is divided into a safe area, a transition area 1, and a transition area 2, the positive reward is divided into a larger reward, an ordinary reward, and a smaller reward, and the corresponding relationship between the reward content and the range of active output value of the balancing unit is established includes: the safe area corresponds to a larger reward, the transition area 1 corresponds to an ordinary reward, and the transition area 2 corresponds to a smaller reward.

7. The power grid dispatching strategy optimization method according to claim 1, Its characteristics are: The training of the power grid dispatch strategy optimization model adopts potential function and regularization term. The regularization term is a comment network regularization term based on the reward mechanism. The regularization term F(s t ,s t+1 )The formula is as follows: F(s t ,with t+1 )=γΦ(s t+1 )-Φ(s t ) In the formula, s t represents the system state at time t, Φ(s t ) indicates that the independent variable is s t The potential function, s t+1 represents the system state at time t+1, Φ(s t+1 ) indicates that the independent variable is s t+1 , γ is the reinforcement learning discount factor for agent training on the actual scheduling execution timeline, γ∈[0,1); The potential function is a potential function based on the reward mechanism. The potential function Φ(s t ) is as follows: In the formula, the potential function Φ(s t ) is a piecewise function divided into two layers, and the training period E max represents the maximum round of training, eps represents the current round, and the potential function Φ(s t ) In the first half of the training cycle, the potential function is expressed as Φ 1 (s t ), the potential function Φ(s t ) In the second half of the training cycle, the potential function is expressed as Φ 2 (s t ).

8. The method for optimizing power grid dispatching strategy according to claim 7, Its characteristics are: Potential function Φ for the first half of the training cycle 1 (s t ) is designed as follows: In the formula, represents the summation operation of τ values ​​from 0 to N-1, P τ BG,t Indicates the active output value of the balancing unit. and They represent the upper and lower limits of the active output of the balancing machine respectively. Indicates the middle value of the active output of the balancing unit; Potential function Φ in the second half of the training cycle 2 (s t ) is designed as follows: In the formula, It represents the accumulated reward from the current round of eps training to the current time t. represents the summation operation of k from 0 to t, r k represents the reward function value at time k, and They represent the maximum and minimum cumulative rewards from the end of training to the current round.

9. The power grid dispatching strategy optimization method according to claim 8, Its characteristics are: The training of the power grid dispatch strategy optimization model also adopts a loss function, the design of which is based on the potential function and regularization term of the reward mechanism. The loss function is used to update the comment network in the SAC algorithm. The loss function is as follows: In the formula, J Q (θ Q ) represents the loss function in updating the Q network, It is represented by the parameter θ Q The state s predicted by the Q network t and action a t The corresponding Q value; Q target Represents the target Q value, which is the target value used to train the Q network; is part of the Q network loss function, which represents half of the square error between the predicted Q value and the target Q value; α is the exploration rate, which represents the weight of the policy entropy and is used to balance exploration and utilization; logπ(at +1 |st +1 ) indicates that in state st +1 Take action at +1 The logarithm of the strategy probability; θ Q and The network parameters of the main target critic; represents mathematical expectation; F is the weight parameter that controls the regularization term.

10. A device for optimizing power dispatching strategy, Its characteristics are: include: Data acquisition module, used to obtain current status data of the power grid system; An optimization module, used for inputting the current state data of the power grid system into a pre-trained power grid dispatch strategy optimization model to obtain an optimized power grid dispatch strategy; The construction of the power grid dispatching strategy optimization model includes: Constructing an objective function according to the optimization goal of the power grid dispatching strategy, wherein the optimization goal of the power grid dispatching strategy is to maximize the consumption of new energy units and minimize the total cost of thermal power units; Constructing a Markov decision process according to the objective function, wherein the Markov decision process includes a state space, an action space, and a reward mechanism; The reward mechanism includes a constructed reward function and set reward content, wherein the reward function includes a reward for the consumption of new energy units, a reward for the operating costs of thermal power units, a reward for the output of balancing units and an auxiliary reward; the construction of the reward function includes the design of a weight coefficient of the reward function, and the weight coefficient of the reward function changes during the training process of the power grid dispatching strategy optimization model; the auxiliary reward is used to reward the balancing unit for safely completing the output of a time period.

Citation Information

Cited By

  • Intelligent substation with automatic control system

    CN119602492A

  • Power system scheduling method and device combined with new energy, and terminal equipment

    CN119944847A