Power grid rolling scheduling deep reinforcement learning decision-making method in extreme weather

Abstract

The invention relates to the technical field of power system automation, and discloses a power grid rolling scheduling deep reinforcement learning decision-making method in extreme weather, which comprises the following steps: constructing a tower vulnerability model and a lead vulnerability model to calculate the fault probability of each power transmission line; constructing a running state time sequence transfer model of each power transmission line, and generating a high-risk cascading fault scene set through Monte Carlo sampling; constructing a power grid rolling scheduling mathematical optimization model considering the successive failure of the power transmission line caused by the extreme weather; constructing a Markov decision process model, and training and constructing a strategy network; and outputting the active output adjustment amount and the load shedding amount of the generator by using the strategy network, executing the control strategy, and circulating the steps until the end moment of the extreme weather is reached. The method has the advantages that the problem of intra-day rolling scheduling decision making of the power grid in an uncertain time sequence scene is solved by using a deep reinforcement learning technology, and the safe operation and reliable power supply capability of the power grid under the condition of power transmission line time sequence cascading failure caused by extreme weather is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of power system automation, and in particular to a deep reinforcement learning decision-making method for rolling dispatch of power grids under extreme weather conditions. Background Art

[0002] In recent years, there have been reports of mass failures such as tower collapse and line breakage caused by extreme weather such as typhoons, ice and snow disasters around the world, and in severe cases, even large-scale power outages. Although large-scale power outages caused by extreme weather are low-probability events, the power industry is paying more and more attention to the safe operation and continuous and reliable power supply of power grids under extreme weather conditions because they are likely to have a greater impact on the economy and society. How to effectively reduce the impact of extreme weather on power equipment, avoid the adverse effects on the safe operation of the power system caused by the successive failures of multiple power equipment due to external severe weather conditions, and achieve safe and orderly power dispatching under extreme weather conditions has become an urgent problem to be solved.

[0003] Existing technologies mainly focus on the formulation of day-ahead power generation plans, and the repair of damaged equipment and power supply restoration strategies after power outages. However, there is little research on the real-time dispatching decision-making of power grids during the development of extreme weather events. The impact of extreme weather events on transmission lines is sequential and uncertain. Therefore, in the real-time dispatching of power grids, it is necessary to roll-optimize its dispatching operation strategy according to the real-time operation status of the power grid, so as to mitigate the adverse effects of sequential failures of transmission lines on the load supply of the power grid. From the perspective of mathematical modeling, the real-time rolling dispatching problem of power grids belongs to a dynamic programming problem with multiple time periods and multiple random scenarios, which is difficult to solve. In practical applications, it is required to complete the optimization problem solution within 15 minutes or even less to ensure that there is sufficient time to implement control measures such as the active output adjustment of the generator. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a deep reinforcement learning decision-making method for rolling dispatch of power grids under extreme weather conditions.

[0005] The purpose of the present invention is achieved through the following technical solutions: A deep reinforcement learning decision-making method for rolling dispatch of power grids under extreme weather conditions, comprising the following steps:

[0006] Step S1, obtaining weather forecast information, and constructing a pole tower vulnerability model and a conductor vulnerability model based on the weather forecast information, and calculating the failure probability of each transmission line based on the pole tower vulnerability model, the conductor vulnerability model and reliability theory;

[0007] Step S2: constructing a time series transition model of the operation state of each transmission line based on the failure probability of each transmission line, and generating a high-risk sequential failure scenario set through Monte Carlo sampling;

[0008] Step S3, constructing a mathematical optimization model for rolling dispatch of power grid considering successive failures of transmission lines caused by extreme weather;

[0009] Step S4: construct a Markov decision process model for the rolling dispatch optimization model of the power grid under extreme weather conditions, and use a deep reinforcement learning algorithm based on a flexible actor-judge to train the Markov decision process model to obtain a strategy network;

[0010] Step S5: acquiring the real-time state of the power grid in each dispatching period, using the strategy network to output the active output adjustment and load shedding of the generator and executing the control strategy.

[0011] Specifically, the specific steps of step S1 include:

[0012] Step S11, obtaining numerical weather forecast results for the next 24 hours from the meteorological forecast department, obtaining geographical location data of each transmission line and tower, and obtaining numerical wind speed forecast results for each transmission line and tower in the next 24 hours based on the geographical location relationship;

[0013] Step S12: constructing a pole tower vulnerability model and a conductor vulnerability model, and calculating the conductor and pole tower failure probabilities based on the pole tower vulnerability model and the conductor vulnerability model;

[0014] Step S13: Based on the series reliability theory and the failure probability of the conductors and towers, the outage probability of the transmission line at time t is further calculated.

[0015] Specifically, the specific steps of step S2 include:

[0016] Step S21: for each transmission line, construct a time sequence state transition model of the operation state of each transmission line:

[0017]

[0018] Among them, prob(s l,t ,s l,t+1 ) represents the transmission line l from time t to s l,t The state is transferred to time t+1 l,t+1 The probability of the state, for the state variable s l,t and l,t+1 If its value is 1, it means that the transmission line l is in normal operation; if its value is 0, it means that the transmission line l is in fault shutdown state. l (t) is the outage probability of the transmission line at time t;

[0019] Step S22: Based on the sequential state transition model of the operation state of each transmission line, a sequential Monte Carlo sampling method is used to generate a large number of scene samples of the sequential operation state of the transmission line, and the same scene samples are removed to obtain a non-redundant scene sample set.

[0020] Step S23: for the set Any scene sample in s:

[0021]

[0022] Among them, N L represents the total number of transmission lines;

[0023] The probability of occurrence of each scene sample is calculated according to the following formula: s :

[0024]

[0025] Among them, Ω line represents a collection of transmission lines;

[0026] Step S24: Setting probability threshold prob th , filter out high-probability scene samples with a probability greater than the probability threshold, and form a high-probability scene set Ω s .

[0027] Specifically, the specific steps of step S3 include:

[0028] Step S31, construct an objective function that minimizes the generator output adjustment cost and the load loss cost:

[0029]

[0030] Among them, t0 represents the current time, T represents the time when the impact of extreme weather on the power grid ends, Ω g and Ω L are the power grid generator set and load set respectively, and are the power amounts of the active output of generator g at time t adjusted upward or downward relative to the planned output on the previous day, and are the unit costs of adjusting the active power upward or downward, and are the load shedding amount and unit load shedding cost at time t respectively;

[0031] Step S32: construct an equation constraint reflecting the power balance of each node:

[0032]

[0033] in, represents the set of generators connected to the grid through node n, represents the active output of generator g at time t, represents the active power demand of the load on node n at time t, represents the load shedding amount on the node at time t, represents the set of neighboring nodes that node n is directly connected to through branches, Indicates branch<n,m> The active power flowing from node n to node m at time t, Ω B represents a set of power grid nodes;

[0034] Step S33: construct an inequality constraint condition considering the upper and lower limits of the generator climbing ability:

[0035]

[0036]

[0037] in, represents the planned active power output of generator g at time t calculated by the day-ahead unit power generation plan, and They represent the minimum and maximum active output of generator g, and They represent the upward climbing ability and downward climbing ability of the generator g respectively;

[0038] Step S34: construct branch power flow safety constraint conditions:

[0039]

[0040] Among them, B n,m Indicates branch<n,m> The equivalent susceptance parameter, Indicates branch<n,m> The active power flowing from node n to node m at time t, u n,m,t Indicates branch<n,m> The running status at time t, u n,m,t =1 indicates branch<n,m> Normal operation, n,m,t =0 means branch<n,m> Failure outage, M represents a sufficiently large number, and Respectively represent branches<n,m> The upper and lower limit safety constraints of active power transmission, Ω Br represents the set of all branches in the power grid, θ n,t represents the node voltage phase angle calculated by the DC power flow model at node n at time t, θ m,t represents the node voltage phase angle calculated by the DC power flow model at node m at time t;

[0041] Step S35, construct the upper and lower limit inequality constraints of the node voltage phase angle in the DC power flow:

[0042]

[0043] Among them, θ n,t represents the node voltage phase angle calculated by the DC power flow model at node n at time t, i n and Indicates the lower and upper limits of the node voltage phase angle in DC power flow calculation;

[0044] Step S36: construct feasibility constraints of control variables:

[0045]

[0046]

[0047] The control variables include: the amount of power adjustment of the active output of generator g at time t relative to the planned output on the previous day The amount of power that the active output of generator g at time t is adjusted downward relative to the planned output on the previous day is the load shedding amount at time t.

[0048] Specifically, the steps of constructing the Markov decision process model in step S4 include:

[0049] Step S411, constructing environment E, where environment E is a power grid timing operation model considering the impact of extreme weather;

[0050] Step S412: construct a state space S, which is a characteristic expression of the real-time operation state of the power grid, including the grid structure, power source planned output, and load demand, and is expressed as:

[0051]

[0052] in,

[0053]

[0054] The grid structure state vector representing the operating status of each transmission line is represents the operating status of the transmission line l at time t. If Indicates that the transmission line l operates normally at time t. It means that the transmission line l fails and stops operating at time t. represents the power supply output state vector at time t, represents the load demand at time t;

[0055] Step S413: construct an action space A, where the action space A is the action strategy that the strategy network can take:

[0056]

[0057] Step S414: construct a reward and punishment function R:

[0058]

[0059] in, Represents the reward associated with the objective function, and the minimization of scheduling cost is converted into maximization of reward through the following formula:

[0060]

[0061] Represents the penalty term associated with the constraint. When any constraint is not met, a negative reward is given as the penalty for the action strategy, as follows:

[0062]

[0063] Among them, F G,cap It represents the penalty term for each generator violating its capacity constraint after executing the action. The calculation formula is as follows:

[0064]

[0065] F G,RU It represents the penalty term for each generator violating its upward climbing capacity constraint after executing the action. The calculation formula is as follows:

[0066]

[0067] F G,RD It represents the penalty term for each generator violating its downward climbing capacity constraint after executing the action. The calculation formula is as follows:

[0068]

[0069] F BR,cap It represents the penalty item of branch power flow exceeding the limit after the action is executed. The calculation formula is as follows:

[0070]

[0071] Specifically, in step S4, a policy network based on a flexible actor-critic is constructed based on the Markov decision process model, and the policy network is trained. The training steps of the policy network include:

[0072] Step S4201: Initialize the experience pool Initialize the value network Vψ ,Q NetworkQ θ and the policy network π φ , and make a copy of the value network as the target value network

[0073] Step S4202: Initialize the loop variable i of the training process to 0, and set the maximum number of iterations of the loop to N. iter ;

[0074] Step S4203: Determine whether i is greater than N iter If yes, go to step S4210, if no, go to step S4204;

[0075] Step S4204, initialize the time step t=0 of the interaction between the policy network and the environment, and set the maximum interaction time step T per round;

[0076] Step S4205, determine whether t is greater than T, if so, proceed to step S4211, if not, proceed to step S4206;

[0077] Step S4206: Calculate the failure rate p of each transmission line at time t l (t), combined with the operating status of each transmission line at time t-1 Determine its state transition probability prob(s at time t l,t ,s l,t+1 );

[0078] Step S4207: Determine the operating state of the transmission line at time t by random sampling to form a state vector The state vector Input to the policy network π φ Get the action vector

[0079] Step S4208: According to the motion vector according to Get the active output of the generator after dispatch, press Get the active power of the node load after load shedding control, calculate the DC power flow of the power grid, and calculate the reward based on the power flow results And according to and Update the state vector

[0080] Step S4209: Record samples And store the samples in the experience pool

[0081] Step S4210, set t=t+1, and return to step S4205;

[0082] Step S4211: From the experience pool The sample is obtained by sampling;

[0083] Step S4212: Calculate the value network V according to the following formula ψ The loss function J V (ψ), and calls the optimizer to update its model parameters ψ:

[0084]

[0085] in, Represents mean calculation, for the experience pool Calculate the mean of all samples in; Indicates that from the experience pool The state s sampled from t Sample; V ψ (s t ) indicates that the environment is in s t Under the state, through the value network V ψ The calculated state value; a t ~π φ represents the action a sampled from the policy distribution t Sample; Q θ (s t ,a t ) indicates that the environment is in s t In the state, take a t After the action, the Q value is calculated by the Q network; log represents logarithmic operation, π φ (a t |s t ) indicates that the environment is in s t In the state, take a t The probability of the action, which is determined by the policy network π φ Calculated;

[0086] Step S4213, calculate Q network Q as follows θ The loss function is used and the optimizer is called to update the parameters θ of the Q network:

[0087]

[0088] in, Indicates that from the experience pool The sample obtained by sampling; Q θ (s t ,a t ) indicates that the environment is in s t In the state, take a t After the action, the Q value calculated by the Q network;

[0089]

[0090] In the formula, r(s t ,a t ) indicates that the environment is in s t In the state, take a t The reward obtained by the action is calculated by the S414 reward and punishment function R; γ is the decay factor, which indicates the impact of the future state on the current action strategy; s t+1 ~p indicates the given state s in the next stage t+1 The probability of occurrence is calculated by the state transition probability formula in step S21; Indicates the given state s in the next stage t+1 The state value of Calculated;

[0091] Step S4214: Calculate the strategy network π according to the following formula φ The loss function is , and the optimizer is called to update the parameters φ of the policy network:

[0092]

[0093] Among them, D KL (h1||h2) represents the KL-divergence between probability distributions h1 and h2; π φ (·|s t ) indicates that the environment is in s t The probability of taking any action in the state is determined by the policy network π φ Calculated; Q θ (s t ,·) indicates that the environment is in s t The Q value obtained when the corresponding action is taken in the state is obtained by the Q network Q θ Calculated; Z θ (s t ) represents the normalization factor:

[0094]

[0095] Step S4215: Update the target value network according to the following formula Model parameters

[0096]

[0097] Among them, τ is the update coefficient, ψ is the value network V ψ Model parameters of

[0098] Step S4216, set i=i+1, and return to step S4203;

[0099] Step S4217: Output the preservation strategy network π φ , ending the training process.

[0100] Specifically, the specific steps of step S5 include:

[0101] Step S51, initializing t=t0;

[0102] Step S52: Obtain the current operating status u of the power transmission line of the power grid n,m,t , generator active output and load demand Form the state vector s t ;

[0103] Step S53: Use the strategy network to output action a t ,according to Get the active output of the generator after scheduling, and get the load shedding amount of each node

[0104] Step S54, executing the generator active power output strategy and load shedding strategy generated in step S53;

[0105] Step S55, set t=t+1;

[0106] Step S56, determine whether t is greater than T, if so, end the current algorithm process, if not, return to step S52.

[0107] The present invention has the following advantages:

[0108] The present invention utilizes deep reinforcement learning technology in the field of artificial intelligence to solve the problem of intraday rolling dispatching decision-making of power grids under uncertain timing scenarios, effectively improving the safe operation and reliable power supply capabilities of the power grid when transmission line timing failures occur one after another due to extreme weather, and effectively promoting the intelligence of power grid dispatching decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0109] Figure 1 It is a flow chart of the deep reinforcement learning decision-making method for rolling dispatch of power grid under extreme weather conditions of the present invention;

[0110] Figure 2 is a flow chart of the strategy network training of the present invention;

[0111] Figure 3 It is a schematic diagram of the rolling scheduling process of the present invention. DETAILED DESCRIPTION

[0112] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention, that is, the embodiments described are only part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0113] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0114] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including a..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0115] The present invention is further described below in conjunction with the accompanying drawings, but the protection scope of the present invention is not limited to the following description. Figures 1 to 3 As shown, a deep reinforcement learning decision-making method for rolling dispatch of power grids under extreme weather conditions includes the following steps: Step S1, obtaining weather forecast information, and constructing a tower vulnerability model and a conductor vulnerability model based on the weather forecast information, and calculating the failure probability of each transmission line based on the tower vulnerability model, the conductor vulnerability model and reliability theory;

[0116] Step S2: constructing a sequential transition model of the operation status of each transmission line based on the failure probability of each transmission line, and generating a high-risk sequential failure scenario set through Monte Carlo sampling; the high-risk sequential failure scenario set is to determine the potential failure scenario of transmission line failure caused by bad weather during the sequential operation of the power grid, which will cause changes in the grid structure during the sequential operation;

[0117] Step S3, constructing a mathematical optimization model for rolling dispatch of power grid considering successive failures of transmission lines caused by extreme weather;

[0118] Step S4: construct a Markov decision process model for the rolling dispatch optimization model of the power grid under extreme weather conditions, and use a deep reinforcement learning algorithm based on a flexible actor-judge to train the Markov decision process model to obtain a strategy network;

[0119] Step S5: acquiring the real-time state of the power grid in each dispatching period, using the strategy network to output the active output adjustment and load shedding of the generator and executing the control strategy.

[0120] Furthermore, the specific steps of step S1 include:

[0121] Step S11, obtaining the numerical weather forecast results for the next 24 hours through the meteorological forecast department, querying the geographical location data of each transmission line and tower through the power grid geographic information system, and obtaining the numerical wind speed forecast results of each transmission line and tower in the next 24 hours according to the geographical location relationship;

[0122] Step S12: constructing a pole tower vulnerability model and a conductor vulnerability model, and calculating the conductor and pole tower failure probabilities based on the pole tower vulnerability model and the conductor vulnerability model;

[0123] Tower vulnerability model:

[0124]

[0125] in,

[0126]

[0127] represents the failure probability of tower i at time t, represents the failure rate of tower i at time t, It represents the forecast wind speed in the area where tower i is located at time t, and represents the design wind speed and limit wind speed of tower i (typical values ​​are 35m / s and 70m / s respectively), K is the model parameter obtained by fitting based on historical data (K can be taken as 0.2 when historical data is missing);

[0128] Wire vulnerability model:

[0129]

[0130] in, represents the failure probability of conductor j at time t, It represents the forecast wind speed in the area where the conductor j is located at time t. and The design wind speed and limit wind speed of the conductors are respectively (according to the current specification GB50545-2010, the design wind speed of 110~330kV transmission lines should not be lower than 23.5m / s, and the design wind speed of 500~750kV transmission lines should not be lower than 27m / s, Determined by destructive test, 3 times can be used in the absence of test data );

[0131] Step S13: Based on the series reliability theory and the failure probability of the conductors and towers, the outage probability p of the transmission line at time t is further calculated. l (t):

[0132]

[0133] in, represents the set of towers that make up the transmission line l, represents the failure probability of the i-th tower at time t, represents the set of wires that make up the transmission line l, represents the fault probability of the j-th conductor at time t. Further, the specific steps of step S2 include:

[0134] Step S21: for each transmission line, construct a time sequence state transition model of the operation state of each transmission line:

[0135]

[0136] Among them, prob(s l,t ,s l,t+1 ) represents the transmission line l from time t to s l,t The state is transferred to time t+1 l,t+1 The probability of the state, for the state variable s l,t and l,t+1 If its value is 1, it means that the transmission line l is in normal operation; if its value is 0, it means that the transmission line l is in fault shutdown state. l (t) is the outage probability of the transmission line at time t;

[0137] Step S22: Based on the sequential state transition model of the operation state of each transmission line, a sequential Monte Carlo sampling method is used to generate a large number of scene samples of the sequential operation state of the transmission line, and the same scene samples are removed to obtain a non-redundant scene sample set.

[0138] Step S23: for the set Any scene sample in s:

[0139]

[0140] Among them, N L represents the total number of transmission lines;

[0141] The probability of occurrence of each scene sample is calculated according to the following formula: s :

[0142]

[0143] Among them, Ω line represents a collection of transmission lines;

[0144] Step S24: Setting probability threshold prob th , filter out high-probability scene samples with a probability greater than the probability threshold, and form a high-probability scene set Ω S .

[0145] Furthermore, the specific steps of step S3 include:

[0146] Step S31, construct an objective function that minimizes the generator output adjustment cost and the load loss cost:

[0147]

[0148] Among them, t0 represents the current time, T represents the time when the impact of extreme weather on the power grid ends, Ω G and Ω L are the power grid generator set and load set respectively, and are the power amounts of the active output of generator g at time t adjusted upward or downward relative to the planned output on the previous day, and are the unit costs of adjusting the active power upward or downward, and are the load shedding amount and unit load shedding cost at time t respectively;

[0149] Step S32: construct an equation constraint reflecting the power balance of each node:

[0150]

[0151] in, represents the set of generators connected to the grid through node n, represents the active output of generator g at time t, represents the active power demand of the load on node n at time t, represents the load shedding amount on the node at time t, represents the set of neighboring nodes that node n is directly connected to through branches (transmission lines or transformers), Indicates branch<n,m> The active power flowing from node n to node m at time t, Ω B represents the set of power grid nodes; the above formula should satisfy the equality constraint for all nodes and all times;

[0152] Step S33: construct an inequality constraint condition considering the upper and lower limits of the generator climbing ability:

[0153]

[0154] in, represents the planned active power output of generator g at time t calculated by the day-ahead unit power generation plan, and They represent the minimum and maximum active output of generator g, and They represent the upward climbing ability and downward climbing ability of the generator g respectively;

[0155] Step S34: construct branch power flow safety constraint conditions:

[0156]

[0157] Among them, B n,m Indicates branch<n,m> The equivalent susceptance parameter, Indicates branch<n,m> The active power flowing from node n to node m at time t, u n,m,t Indicates branch<n,m> The running status at time t, u n,m,t =1 indicates branch<n,m> Normal operation, n,m,t =0 means branch<n,m> Failure outage, M represents a sufficiently large number, and Respectively represent branches<n,m> The upper and lower limit safety constraints of active power transmission, Ω Br represents the set of all branches in the power grid, θ n,t represents the node voltage phase angle calculated by the DC power flow model at node n at time t, θ m,t It represents the node voltage phase angle calculated by the DC power flow model at node m at time t. The grid rolling dispatch optimization model takes into account the grid topology changes caused by high-risk cascading failure scenarios, that is, after the transmission line fails, the line power flow constraints in the optimization model will change, and u will change from 1 to 0.

[0158] Step S35, construct the upper and lower limit inequality constraints of the node voltage phase angle in the DC power flow:

[0159]

[0160] Among them, θ n,trepresents the node voltage phase angle calculated by the DC power flow model at node n at time t, i n and Indicates the lower and upper limits of the node voltage phase angle in DC power flow calculation, which are generally set to -3.1416 radians (rad) and 3.1416 radians (rad);

[0161] Step S36: construct feasibility constraints of control variables:

[0162]

[0163] The control variables include: the amount of power adjustment of the active output of generator g at time t relative to the planned output on the previous day The amount of power that the active output of generator g at time t is adjusted downward relative to the planned output on the previous day is the load shedding amount at time t.

[0164] Furthermore, the steps of constructing the Markov decision process model in step S4 include:

[0165] Step S411, constructing an environment (environment, E), where environment E is a power grid timing operation model that takes into account the impact of extreme weather;

[0166] Step S412: construct a state space (state, S). The state space S is a characteristic expression of the real-time operation state of the power grid, including the grid structure, power source planned output, and load demand. The expression is:

[0167]

[0168] in,

[0169]

[0170] The grid structure state vector representing the operating status of each transmission line is represents the operating status of the transmission line l at time t. If Indicates that the transmission line l operates normally at time t. It means that the transmission line l fails and stops operating at time t. represents the power supply output state vector at time t, represents the load demand at time t;

[0171] Step S413: construct an action space (action, A), where the action space A is the action strategy that the policy network can take:

[0172]

[0173] Step S414: construct a reward and punishment function R:

[0174]

[0175] in, Represents the reward associated with the objective function, and the minimization of scheduling cost is converted into maximization of reward through the following formula:

[0176]

[0177] Represents the penalty term associated with the constraint. When any constraint is not met, a negative reward is given as the penalty for the action strategy, as follows:

[0178]

[0179] Among them, F G,cap It represents the penalty term for each generator violating its capacity constraint after executing the action. The calculation formula is as follows:

[0180]

[0181] F G,RU It represents the penalty term for each generator violating its upward climbing capacity constraint after executing the action. The calculation formula is as follows:

[0182]

[0183] F G,RD It represents the penalty term for each generator violating its downward climbing capacity constraint after executing the action. The calculation formula is as follows:

[0184]

[0185] F BR,cap It represents the penalty item of branch power flow exceeding the limit after the action is executed. The calculation formula is as follows:

[0186]

[0187] Furthermore, the training step of the policy network in step S4 includes:

[0188] Step S4201: Initialize the experience pool Initialize the value network V ψ ,Q NetworkQ θ and the policy network π φ , and make a copy of the value network as the target value network Among them: (1) Value network V ψ and target value network The state vector The deep neural network with value as input and value scalar as output has model parameters represented by ψ and Represents; (2) Q network Qθ is represented by the state vector and motion vector A deep neural network with Q value scalar as input and Q value scalar as output, and its model parameters are represented by θ; (3) policy network π φ The state vector As input, the action vector is the output deep neural network, and its model parameters are represented by φ;

[0189] Step S4202: Initialize the loop variable i of the training process to 0, and set the maximum number of iterations of the loop to N. iter ;

[0190] Step S4203: Determine whether i is greater than N iter If yes, go to step S4210, if no, go to step S4204;

[0191] Step S4204, initialize the time step t=0 of the interaction between the policy network and the environment, and set the maximum interaction time step T per round;

[0192] Step S4205, determine whether t is greater than T, if so, proceed to step S4211, if not, proceed to step S4206;

[0193] Step S4206: Calculate the failure rate p of each transmission line at time t l (t), combined with the operating status of each transmission line at time t-1 Determine its state transition probability prob(s at time t l,t ,s l,t+1 );

[0194] Step S4207: Determine the operating state of the transmission line at time t by random sampling to form a state vector The state vector Input to the policy network π φ Get the action vector

[0195] Step S4208: According to the motion vector according to Get the active output of the generator after dispatch, press Get the active power of the node load after load shedding control, calculate the DC power flow of the power grid, and calculate the reward based on the power flow results And according to and Update the state vector

[0196] Step S4209: Record samples And store the samples in the experience pool

[0197] Step S4210, set t=t+1, and return to step S4205;

[0198] Step S4211: From the experience pool The sample is obtained by sampling;

[0199] Step S4212: Calculate the value network V according to the following formula ψ The loss function JV(ψ) is used and the optimizer is called to update its model parameters ψ:

[0200]

[0201] in, Represents mean calculation, for the experience pool Calculate the mean of all samples in; Indicates that from the experience pool The state s sampled from t Sample; V ψ (s t ) indicates that the environment is in s t Under the state, through the value network V ψ The calculated state value; a t ~π φ represents the action a sampled from the policy distribution t Sample; Qθ(s t ,a t ) indicates that the environment is in s t In the state, take a t After the action, the Q value is calculated by the Q network; log represents logarithmic operation, π φ (a t |s t ) indicates that the environment is in s t In the state, take a t The probability of the action, which is determined by the policy network π φ Calculated;

[0202] Step S4213, calculate the loss function of the Q network Qθ according to the following formula, and call the optimizer to update the parameters θ of the Q network:

[0203]

[0204] in, Indicates that from the experience pool The samples obtained by sampling; Qθ(s t ,a t ) indicates that the environment is in s tIn the state, take a t After the action, the Q value calculated by the Q network;

[0205]

[0206] In the formula, r(s t ,a t ) indicates that the environment is in s t In the state, take a t The reward obtained by the action is calculated by the S414 reward and punishment function R; γ is the decay factor, which indicates the impact of the future state on the current action strategy; s t+1 ~p indicates the given state s in the next stage t+1 The probability of occurrence is calculated by the state transition probability formula in step S21; Indicates the given state s in the next stage t+1 The state value of Calculated;

[0207] Step S4214: Calculate the strategy network π according to the following formula φ The loss function is , and the optimizer is called to update the parameters φ of the policy network:

[0208]

[0209] Among them, D KL (h1||h2) represents the KL-divergence between probability distributions h1 and h2; π φ (·|s t ) indicates that the environment is in s t The probability of taking any action in the state is determined by the policy network π φ Calculated; Qθ(s t ,·) indicates that the environment is in s t The Q value obtained when the corresponding action is taken in the state is calculated by the Q network Qθ; Zθ(s t ) represents the normalization factor:

[0210]

[0211] Step S4215: Update the target value network according to the following formula Model parameters

[0212]

[0213] Among them, τ is the update coefficient, ψ is the value network V ψ Model parameters of

[0214] Step S4216, set i=i+1, and return to step S4203;

[0215] Step S4217: Output the preservation strategy network π φ , ending the training process.

[0216] Furthermore, in the real-time operation scheduling stage, when entering the preset time t0, the rolling scheduling is implemented according to the following steps:

[0217] Step S51, initializing t=t0;

[0218] Step S52: Obtain the current operating status u of the power transmission line of the power grid n,m,t , generator active output and load demand Form the state vector s t ;

[0219] Step S53: Use the strategy network to output action a t ,according to Get the active output of the generator after scheduling, and get the load shedding amount of each node

[0220] Step S54, executing the generator active power output strategy and load shedding strategy generated in step S53;

[0221] Step S55, set t=t+1;

[0222] Step S56, determine whether t is greater than T, if so, end the current algorithm process, if not, return to step S52. Among them, the first four steps are completed in the day-ahead offline stage, and the last step is completed in the day-ahead online stage.

[0223] The above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any technician familiar with the art can make many possible changes and modifications to the technical solution of the present invention by using the above-mentioned technical content without departing from the scope of the technical solution of the present invention, or modify it into an equivalent embodiment of equivalent changes. Therefore, any changes, modifications, equivalent changes and modifications made to the above embodiments based on the technology of the present invention without departing from the content of the technical solution of the present invention belong to the protection scope of the present technical solution.

Claims

1. A deep reinforcement learning decision-making method for rolling dispatch of power grid under extreme weather conditions, characterized by: The following steps are involved: Step S1, obtaining weather forecast information, and constructing a pole tower vulnerability model and a conductor vulnerability model based on the weather forecast information, and calculating the failure probability of each transmission line based on the pole tower vulnerability model, the conductor vulnerability model and reliability theory; Step S2: constructing a time series transition model of the operation state of each transmission line based on the failure probability of each transmission line, and generating a high-risk sequential failure scenario set through Monte Carlo sampling; Step S3, constructing a grid rolling dispatch optimization model considering transmission line failures caused by extreme weather; Step S4: construct a Markov decision process model for the rolling dispatch optimization model of the power grid under extreme weather conditions, and use a deep reinforcement learning algorithm based on a flexible actor-judge to train the Markov decision process model to obtain a strategy network; Step S5: acquiring the real-time state of the power grid in each dispatching period, using the strategy network to output the active output adjustment and load shedding of the generator and executing the control strategy.

2. According to claim 1, a deep reinforcement learning decision-making method for rolling dispatch of power grid under extreme weather conditions is characterized by: The specific steps of step S1 include: Step S11, obtaining numerical weather forecast results for the next 24 hours from the meteorological forecast department, obtaining geographical location data of each transmission line and tower, and obtaining numerical wind speed forecast results for each transmission line and tower in the next 24 hours based on the geographical location relationship; Step S12: constructing a pole tower vulnerability model and a conductor vulnerability model, and calculating the conductor and pole tower failure probabilities based on the pole tower vulnerability model and the conductor vulnerability model; Step S13: Based on the series reliability theory and the failure probabilities of the conductors and towers, the outage probability of the transmission line at time t is further calculated.

3. The deep reinforcement learning decision-making method for rolling dispatch of power grid under extreme weather conditions according to claim 1 is characterized by: The specific steps of step S2 include: Step S21: for each transmission line, construct a time sequence state transition model of the operation state of each transmission line: in, It means that the transmission line l starts at time t l,t The state is transferred to time t+1 l,t+1 The probability of the state, for the state variable s l,t and l,t+1 If its value is 1, it means that the transmission line l is in normal operation; if its value is 0, it means that the transmission line l is in fault shutdown state. l (t) is the outage probability of the transmission line at time t; Step S22: Based on the sequential state transition model of the operation state of each transmission line, a sequential Monte Carlo sampling method is used to generate a large number of scene samples of the sequential operation state of the transmission line, and the same scene samples are removed to obtain a non-redundant scene sample set. Step S23: for the set Any scene sample in s: Among them, N L represents the total number of transmission lines; The probability of occurrence of each scene sample is calculated according to the following formula: s : Among them, Ω line represents a collection of transmission lines; Step S24: Setting probability threshold prob th , filter out high-probability scene samples with a probability greater than the probability threshold, and form a high-probability scene set Ω S .

4. The deep reinforcement learning decision-making method for rolling dispatch of power grid under extreme weather conditions according to claim 1 is characterized by: The specific steps of step S3 include: Step S31, construct an objective function that minimizes the generator output adjustment cost and the load loss cost: Among them, t0 represents the current time, T represents the time when the impact of extreme weather on the power grid ends, Ω G and Ω L are the power grid generator set and load set respectively, and are the power amounts of the active output of generator g at time t adjusted upward or downward relative to the planned output on the previous day, and are the unit costs of adjusting the active power upward or downward, and are the load shedding amount and unit load shedding cost at time t respectively; Step S32: construct an equation constraint reflecting the power balance of each node: in, represents the set of generators connected to the grid through node n, represents the active output of generator g at time t, represents the active power demand of the load on node n at time t, represents the load shedding amount on the node at time t, represents the set of neighboring nodes that node n is directly connected to through branches, Indicates branch<n,m> The active power flowing from node n to node m at time t, Ω B represents a set of power grid nodes; Step S33: construct an inequality constraint condition considering the upper and lower limits of the generator climbing ability: in, represents the planned active power output of generator g at time t calculated by the day-ahead unit power generation plan, and They represent the minimum and maximum active output of generator g, and They represent the upward climbing ability and downward climbing ability of the generator g respectively; Step S34: construct branch power flow safety constraint conditions: Among them, B n,m Indicates branch<n,m> The equivalent susceptance parameter, Indicates branch<n,m> The active power flowing from node n to node m at time t, u n,m,t Indicates branch<n,m> The running status at time t, u n,m,t =1 indicates branch<n,m> Normal operation, n,m,t =0 means branch<n,m> Failure outage, M represents a sufficiently large number, and Respectively represent branches<n,m> The upper and lower limit safety constraints of active power transmission, Ω Br represents the set of all branches in the power grid, θ n,t represents the node voltage phase angle calculated by the DC power flow model at node n at time t; θ m,t represents the node voltage phase angle calculated by the DC power flow model at node m at time t; Step S35, construct the upper and lower limit inequality constraints of the node voltage phase angle in the DC power flow: Among them, θ n,t represents the node voltage phase angle calculated by the DC power flow model at node n at time t; i n and Indicates the lower and upper limits of the node voltage phase angle in DC power flow calculation; Step S36: construct feasibility constraints of control variables: The control variables include: the amount of power adjustment of the active output of generator g at time t relative to the planned output on the previous day The amount of power that the active output of generator g at time t is adjusted downward relative to the planned output on the previous day is the load shedding amount at time t.

5. The deep reinforcement learning decision-making method for rolling dispatch of power grid under extreme weather conditions according to claim 4 is characterized by: The steps of constructing the Markov decision process model in step S4 include: Step S411, constructing environment E, where environment E is a power grid timing operation model considering the impact of extreme weather; Step S412: construct a state space S, which is a characteristic expression of the real-time operation state of the power grid, including the grid structure, power source planned output, and load demand, and is expressed as: in, The grid structure state vector representing the operating status of each transmission line is represents the operating status of the transmission line l at time t. If Indicates that the transmission line l operates normally at time t. It means that the transmission line l fails and stops operating at time t. represents the power supply output state vector at time t, represents the load demand at time t; Step S413: construct an action space A, where the action space A is the action strategy that the strategy network can take: Step S414: construct a reward and punishment function R: in, Represents the reward associated with the objective function, and the minimization of scheduling cost is converted into maximization of reward through the following formula: Represents the penalty term associated with the constraint. When any constraint is not met, a negative reward is given as the penalty for the action strategy, as follows: Among them, F G,cap It represents the penalty term for each generator violating its capacity constraint after executing the action. The calculation formula is as follows: F G,RU It represents the penalty term for each generator violating its upward climbing capacity constraint after executing the action. The calculation formula is as follows: F G,RD It represents the penalty term for each generator violating its downward climbing capacity constraint after executing the action. The calculation formula is as follows: F BR,cap It represents the penalty item of branch power flow exceeding the limit after the action is executed. The calculation formula is as follows:

6. A deep reinforcement learning decision-making method for rolling dispatch of power grids under extreme weather conditions according to claim 5, characterized in that: The training step of the policy network in step S4 includes: Step S4201: Initialize the experience pool Initialize the value network V ψ ,Q NetworkQ θ and the policy network π φ , and make a copy of the value network as the target value network Step S4202: Initialize the loop variable i of the training process to 0, and set the maximum number of iterations of the loop to N. iter ; Step S4203: Determine whether i is greater than N iter If yes, go to step S4210, if no, go to step S4204; Step S4204, initialize the time step t=0 of the interaction between the policy network and the environment, and set the maximum interaction time step T per round; Step S4205, determine whether t is greater than T, if so, proceed to step S4211, if not, proceed to step S4206; Step S4206: Calculate the failure rate p of each transmission line at time t l (t), combined with the operating status of each transmission line at time t-1 Determine its state transition probability prob(s at time t l,t ,s l,t+1 ); Step S4207: Determine the operating state of the transmission line at time t by random sampling to form a state vector The state vector Input to the policy network π φ Get the action vector Step S4208: According to the motion vector according to Get the active output of the generator after dispatch, press Get the active power of the node load after load shedding control, calculate the DC power flow of the power grid, and calculate the reward based on the power flow results And according to and Update the state vector Step S4209: Record samples And store the samples in the experience pool Step S4210, set t=t+1, and return to step S4205; Step S4211: From the experience pool The sample is obtained by sampling; Step S4212: Calculate the value network V according to the following formula ψ The loss function J V (ψ), and calls the optimizer to update its model parameters ψ: in, Represents mean calculation, for the experience pool Calculate the mean of all samples in; Indicates that from the experience pool The state s sampled from t Sample; V ψ (s t ) indicates that the environment is in s t Under the state, through the value network V ψ The calculated state value; a t ~π φ represents the action a sampled from the policy distribution t Sample; Q θ (s t ,a t ) indicates that the environment is in s t In the state, take a t After the action, the Q value is calculated by the Q network; log represents logarithmic operation, π φ (a t |s t ) indicates that the environment is in s t In the state, take a t The probability of the action, which is determined by the policy network π φ Calculated; Step S4213, calculate Q network Q as follows θ The loss function is used and the optimizer is called to update the parameters θ of the Q network: in, Indicates that from the experience pool The sample obtained by sampling; Q θ (s t ,a t ) indicates that the environment is in s t In the state, take a t After the action, the Q value calculated by the Q network; r(s t ,a t ) indicates that the environment is in s t In the state, take a t The reward obtained by the action is calculated by the S414 reward and punishment function R; γ is the decay factor, which indicates the impact of the future state on the current action strategy; s t+1 ~p indicates the given state s in the next stage t+1 The probability of occurrence is calculated by the state transition probability formula in step S21; Indicates the given state s in the next stage t+1 The state value of Calculated; Step S4214: Calculate the strategy network π according to the following formula φ The loss function is , and the optimizer is called to update the parameters φ of the policy network: Among them, D KL (h1||h2) represents the KL-divergence between probability distributions h1 and h2; π φ (·|s t ) indicates that the environment is in s t The probability of taking any action in the state is determined by the policy network π φ Calculated; Q θ (s t ,·) indicates that the environment is in s t The Q value obtained when the corresponding action is taken in the state is obtained by the Q network Q θ Calculated; Z θ (s t ) represents the normalization factor: Step S4215: Update the target value network according to the following formula Model parameters Among them, τ is the update coefficient, ψ is the value network V ψ Model parameters of Step S4216, set i=i+1, and return to step S4203; Step S4217: Output the preservation strategy network π φ , ending the training process.

7. A deep reinforcement learning decision-making method for rolling dispatch of power grids under extreme weather conditions according to claim 5, characterized in that: The specific steps of step S5 include: Step S51, initializing t=t0; Step S52: Obtain the current operating status u of the power transmission line of the power grid n,m,t , generator active output and load demand Form the state vector s t ; Step S53: Use the strategy network to output action a t ,according to Get the active output of the generator after scheduling, and get the load shedding amount of each node Step S54, executing the generator active power output strategy and load shedding strategy generated in step S53; Step S55, set t=t+1; Step S56, determine whether t is greater than T, if so, end the current algorithm process, if not, return to step S52.

Images