Power grid voltage optimization method and device based on hierarchical constraint reinforcement learning

Abstract

The invention discloses a power grid voltage optimization method and device based on hierarchical constraint reinforcement learning, and relates to the technical field of power grid voltage optimization. According to the invention, the hierarchical reinforcement learning model under the long time scale and the short time scale is constructed, and the condition of the power grid voltage is divided into two time scales for adjustment, so that the power grid voltage is optimized in a layer-by-layer coordination and step-by-step optimization mode. The short-time-scale voltage optimization model makes up the deficiency of the long-time-scale reinforcement learning model due to the uncertainty of the load and the distributed power supply, thereby improving the accuracy of power grid voltage regulation. Layered training is carried out on the initial layered voltage optimization model through a layered constraint reinforcement learning solution algorithm, global collaborative optimization of the power grid voltage is effectively realized, and the search space is effectively reduced through a constraint Markov decision model, so that the calculation efficiency and accuracy of the layered voltage optimization model can be improved, and the optimization efficiency of the power grid voltage is improved. And the control efficiency and accuracy of the power grid voltage are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of power grid voltage optimization, and in particular relates to a power grid voltage optimization method and device based on hierarchical constrained reinforcement learning. Background Art

[0002] With the continuous development of the power industry, the proportion of wind and photovoltaic power generation capacity in the power grid has continued to increase, and the output power fluctuations of wind and solar power have an increasingly obvious impact on the voltage of the power grid, which not only affects the consumption of new energy, but also causes the problem of power grid over-limit. In addition, since the new power system of wind and solar power generation presents the characteristics of power electronics, that is, the power of the power system changes quickly and the system inertia is small, it is easy to cause power oscillation. Moreover, the number of objects involved in the control of the power system is large, making it difficult to achieve coordinated control on a time scale.

[0003] In the current grid voltage optimization technology, the control of grid voltage is usually converted into a reactive power optimization problem for processing. Its essence is to control it through mathematical principles. However, when facing the increasingly large-scale power system, this method is prone to the problem of excessive computational complexity, showing a trend of decreasing computational speed and accuracy, resulting in low efficiency and low accuracy in the control of grid voltage. In addition, there is also the control optimization of grid voltage through reinforcement learning, but the traditional reinforcement learning method cannot balance the optimization objectives and constraints well, thereby increasing the difficulty and workload of its regulation, that is, reducing the efficiency and accuracy of grid voltage optimization. Therefore, based on the defects of these prior arts, there is an urgent need for a grid voltage optimization method and device based on hierarchical constraint reinforcement learning to solve the defects of the prior art. Summary of the invention

[0004] The present invention aims to provide a grid voltage optimization method and device based on hierarchical constrained reinforcement learning to solve the above-mentioned technical problems, construct optimization models for long time scales and short time scales, combine hierarchical constrained reinforcement learning to realize the control of grid voltage, and improve the efficiency and accuracy of grid voltage optimization.

[0005] In order to solve the above technical problems, an embodiment of the present invention provides a grid voltage optimization method based on hierarchical constraint reinforcement learning, comprising:

[0006] Acquire historical operating status information of the power grid, and construct a constrained Markov decision model based on the historical operating status information to complete the construction of a long-time scale reinforcement learning model, wherein the long-time scale reinforcement learning model is used for an output voltage control strategy; the output voltage control strategy is a long-time scale strategy;

[0007] Construct a short-time scale voltage optimization model, where the short-time scale is constrained by the long-time scale strategy and the grid is autonomously optimized based on the grid's operating status information;

[0008] An initial hierarchical voltage optimization model is constructed according to the long-time scale reinforcement learning model and the short-time scale optimization model, and hierarchical training is performed on the initial hierarchical voltage optimization model in combination with a preset hierarchical constraint reinforcement learning solution algorithm to obtain an optimal hierarchical voltage optimization model;

[0009] Acquire real-time operation status information of the power grid, input the real-time operation status information into the optimal hierarchical voltage optimization model, and obtain an optimal voltage control strategy;

[0010] The grid voltage is regulated according to the optimal voltage control strategy.

[0011] It can be understood that, compared with the prior art, the present invention constructs a long-time scale reinforcement learning model and a short-time scale optimization model respectively through the historical operation status information of the power grid, thereby forming an initial hierarchical voltage optimization model, and uses a hierarchical constraint reinforcement learning solution algorithm to train the initial hierarchical voltage optimization model, thereby realizing efficient and accurate control of the power grid voltage. Compared with traditional reinforcement learning, the present invention divides the conditions of the power grid voltage into two time scales for adjustment by constructing hierarchical reinforcement learning models under long time scales and short time scales, and compensates for the shortcomings of the long-time scale reinforcement learning model due to the uncertainty of load and distributed power sources through the short-time scale optimization model in a layer-by-layer coordinated and step-by-step optimized manner, thereby improving the accuracy of the power grid voltage regulation. The present invention performs hierarchical training on the initial hierarchical voltage optimization model through a hierarchical constraint reinforcement learning solution algorithm, effectively realizing the global collaborative optimization of the power grid voltage, and effectively reducing the search space through the constrained Markov decision model, thereby improving the calculation efficiency and accuracy of the hierarchical voltage optimization model, and improving the control efficiency and accuracy of the power grid voltage.

[0012] As a preferred solution, the historical operation status information of the power grid is obtained, and a constrained Markov decision model is constructed according to the historical operation status information to complete the construction of a long-time scale reinforcement learning model, wherein the long-time scale reinforcement learning model is used for the output voltage control strategy, specifically including:

[0013] Acquire historical operation status information of the power grid, the historical operation status information including: voltage information, power information, switching gear information, load information, equipment action cost information and equipment action number information;

[0014] Constructing a state space and an action space respectively according to the voltage information, power information, switching gear information and load information;

[0015] Obtain the voltage amplitude, conductance value, susceptance value and phase angle difference of each node in the power grid to calculate the active network loss of the power grid;

[0016] Calculating voltage deviation and operating cost according to the equipment action cost information and voltage information, and taking voltage deviation, active network loss and operating cost minimization as decision targets of the reward function, thereby constructing a reward function;

[0017] Constructing an auxiliary cost function in combination with the state space and the action space according to the device action number information, voltage information and power information;

[0018] According to a preset discount coefficient, a discount benefit function of the voltage control strategy is constructed in combination with the auxiliary cost function;

[0019] A first objective function is constructed according to the discounted revenue function, state space, action space and reward function, and a constrained Markov decision model is constructed in combination with a preset safety threshold and a preset state transition probability function to form a long-time scale reinforcement learning model.

[0020] As a preferred solution, the auxiliary cost function is constructed based on the device action number information, voltage information and power information in combination with the state space and action space, specifically including:

[0021] Calculate the bus voltage deviation value of the power grid, the excess power generation of the power generator set of the power grid, the excess power generation of the distributed power source of the power grid, the energy storage system evaluation value of the power grid and the equipment action number constraint according to the equipment action number information, voltage information and power information;

[0022] Adding the bus voltage deviation value of the power grid, the excess power generation of the power generator set of the power grid, the excess power generation of the distributed power source of the power grid, the energy storage system evaluation value of the power grid and the equipment action number constraint, and constructing an auxiliary cost function in combination with the state space and the action space;

[0023] Wherein, the auxiliary cost function is:

[0024]

[0025] in, is the voltage on the i-th bus during period t, and are the upper and lower constraints respectively; S G,t is the apparent power output by the generator set during period t, is the rated capacity of the generator set; is the apparent power output by the distributed generation in period t, is the rated capacity of the distributed power source; is the state of charge of the energy storage system during period t, and are the upper and lower constraints respectively; N Ci,t is the number of operations of the capacitor in group i during period t, N Ci,max N is the upper limit of its action number; Ti,t is the number of operations of the i-th transformer tap changer in period t, N Ti,max The upper limit of its action number.

[0026] As a preferred solution, the discounted benefit function of the voltage control strategy is constructed according to the preset discount coefficient and in combination with the auxiliary cost function, specifically including:

[0027] According to a preset discount coefficient, a discount benefit function of the voltage control strategy is constructed in combination with the auxiliary cost function;

[0028] Wherein, the discounted revenue function is:

[0029]

[0030] Among them, J C (π) is the discounted benefit function, which is used to characterize the expected discounted benefit corresponding to the auxiliary cost function of the voltage control strategy π; E τ～π It is used to represent the expected value of strategy π; γ is the discount coefficient; c t is the auxiliary cost function; s t is the state space, a t For the action space.

[0031] As a preferred solution, the first objective function is constructed according to the discounted revenue function, state space, action space and reward function, and a constrained Markov decision model is constructed in combination with a preset safety threshold and a state transition probability function to form a long-time scale reinforcement learning model, specifically including:

[0032] Constructing a first objective function according to the discounted revenue function, the state space, the action space and the reward function, and constructing a constrained Markov decision model in combination with a preset safety threshold and a state transition probability function to form a long-time scale reinforcement learning model;

[0033] Wherein, the constrained Markov decision model is:

[0034]

[0035] J C (π)≤d,

[0036] Among them, J(π) is the first objective function; E τ～π It is used to represent the expected value of the voltage control strategy π; γ is the discount coefficient; rt is the reward function; s t is the state space, a t is the action space, d is the safety threshold; J C (π) is the discounted return function.

[0037] As a preferred solution, the short-time scale voltage optimization model is constructed, wherein the short-time scale is constrained by the long-time scale strategy and the power grid is autonomously optimized based on the operation status information of the power grid, specifically including:

[0038] According to the historical operation status information, power flow constraints, voltage constraints, wind turbine output constraints, photovoltaic unit output constraints and energy storage system charging and discharging constraints are established, thereby constructing a constraint set;

[0039] Filter the historical operation status information according to the voltage control strategy data to obtain the historical fluctuation operation status information corresponding to the voltage control strategy data;

[0040] Acquire the voltage regulation coefficient of the power grid, and construct the control variables and the second objective function of the short-time-scale voltage optimization model in combination with the historical fluctuation operation status information to complete the construction of the short-time-scale voltage optimization model;

[0041] Wherein, the second objective function is:

[0042]

[0043] in, minf short is the second objective function, c loss is the network loss coefficient, P loss is the network loss; v is the voltage regulation coefficient, U i is the current voltage at node i, U iN is the rated voltage of node i; n is the total number of nodes; U i and U j represents the voltage amplitude of node i and node j; G ij , B ij and δ ij are the conductance, susceptance and phase angle difference between node i and node j respectively.

[0044] As a preferred solution, the initial hierarchical voltage optimization model is hierarchically trained in combination with a preset hierarchical constraint reinforcement learning solution algorithm to obtain an optimal hierarchical voltage optimization model, specifically including:

[0045] Obtaining Lagrange multipliers, and constructing an unconstrained optimization problem of the initial hierarchical voltage optimization model in combination with the auxiliary cost function and the first objective function;

[0046] constructing a loss function set according to the initial hierarchical voltage optimization model;

[0047] According to a preset hierarchical constraint reinforcement learning solution algorithm, the unconstrained optimization problem of the initial hierarchical voltage optimization model is solved in combination with the loss function set, so as to perform hierarchical training on the initial hierarchical voltage optimization model to obtain an optimal hierarchical voltage optimization model;

[0048] Wherein, the unconstrained optimization problem is:

[0049]

[0050] g(π)=J C (π)-d,

[0051] Among them, λ is the Lagrange multiplier, J L (π,λ) is an unconstrained optimization problem, E τ～π It is used to represent the expected value of the voltage control strategy π; γ is the discount coefficient; r t is the reward function; s t is the state space, a t is the action space, d is the safety threshold; J C (π) is the discounted reward function; f(π) is the cumulative expected reward function, which is used to characterize the total expected reward that the agent can obtain under strategy π; g(π) is used to represent the difference between the total cost brought by strategy π and the safety threshold d.

[0052] As a preferred solution, the unconstrained optimization problem of the initial hierarchical voltage optimization model is solved according to the preset hierarchical constraint reinforcement learning solution algorithm in combination with the loss function set, so as to perform hierarchical training on the initial hierarchical voltage optimization model to obtain the optimal hierarchical voltage optimization model, specifically including:

[0053] Initializing network parameters, temperature coefficients, Lagrange multipliers and experience pools in the initial hierarchical voltage optimization model;

[0054] Inputting historical operating state information into the initial hierarchical voltage optimization model to obtain a voltage control strategy output by a long-time scale reinforcement learning model, and inputting the voltage control strategy into a short-time scale voltage optimization model to obtain a reward feedback value corresponding to a reward function;

[0055] Calculating experience data according to the reward feedback value, and storing the experience data in an experience pool until the sample size in the experience pool is greater than a preset number, completing storage in the experience pool;

[0056] Samples are extracted from the stored experience pool, and the network parameters, temperature coefficients and Lagrange multipliers are updated in combination with the loss function set, and the unconstrained optimization problem of the initial hierarchical voltage optimization model is solved until the voltage control strategy generated by the long-time-scale reinforcement learning model meets the preset requirements, and the hierarchical training of the initial hierarchical voltage optimization model is completed to obtain the optimal hierarchical voltage optimization model.

[0057] As a preferred solution, the method extracts samples from the stored experience pool, updates the network parameters, temperature coefficients and Lagrange multipliers in combination with the loss function set, and solves the unconstrained optimization problem of the initial hierarchical voltage optimization model until the voltage control strategy generated by the long-time scale reinforcement learning model meets the preset requirements, completes the hierarchical training of the initial hierarchical voltage optimization model, and obtains the optimal hierarchical voltage optimization model, which specifically includes:

[0058] The loss function set includes: the loss function of the agent strategy, the critic network loss function, the entropy constraint loss function and the Lagrange multiplier loss function;

[0059] Draw samples from the experience pool of completed storage;

[0060] According to the loss function of the agent strategy and the critic network loss function, the network parameters are updated in combination with the extracted samples;

[0061] According to the entropy-constrained loss function, the temperature coefficient is updated in combination with the extracted samples;

[0062] According to the loss function of the Lagrangian multiplier, the Lagrangian multiplier is updated in combination with the extracted samples;

[0063] Solving the unconstrained optimization problem according to the updated network parameters, temperature coefficients and Lagrange multipliers to update the voltage control strategy;

[0064] When the reward function corresponding to the voltage control strategy converges, the network parameters, temperature coefficients and Lagrange multipliers are updated, and then the hierarchical training of the initial hierarchical voltage optimization model is completed to obtain the optimal hierarchical voltage optimization model.

[0065] Accordingly, an embodiment of the present invention provides a power grid voltage optimization device based on hierarchical constrained reinforcement learning, comprising: a long-time scale reinforcement learning model construction module, a short-time scale voltage optimization model construction module, a hierarchical constrained reinforcement learning module, an optimal voltage control strategy acquisition module and a power grid voltage optimization module;

[0066] The long-time-scale reinforcement learning model construction module is used to obtain the historical operating status information of the power grid, and to construct a constrained Markov decision model based on the historical operating status information to complete the construction of the long-time-scale reinforcement learning model, wherein the long-time-scale reinforcement learning model is used for the output voltage control strategy; the output voltage control strategy is a long-time-scale strategy;

[0067] The short-time scale voltage optimization model construction module is used to construct a short-time scale voltage optimization model, wherein the short-time scale is constrained by the long-time scale strategy and autonomously optimizes the power grid based on the operation status information of the power grid;

[0068] The hierarchical constraint reinforcement learning module is used to construct an initial hierarchical voltage optimization model according to the long-time scale reinforcement learning model and the short-time scale voltage optimization model, and perform hierarchical training on the initial hierarchical voltage optimization model in combination with a preset hierarchical constraint reinforcement learning solution algorithm to obtain an optimal hierarchical voltage optimization model;

[0069] The optimal voltage control strategy acquisition module is used to acquire real-time operation status information of the power grid, input the real-time operation status information into the optimal hierarchical voltage optimization model, and obtain the optimal voltage control strategy;

[0070] The grid voltage optimization module is used to regulate the grid voltage according to the optimal voltage control strategy.

[0071] It can be understood that, compared with the prior art, the present device constructs a long-time scale reinforcement learning model and a short-time scale optimization model respectively through the historical operation status information of the power grid, thereby forming an initial hierarchical voltage optimization model, and uses a hierarchical constraint reinforcement learning solution algorithm to train the initial hierarchical voltage optimization model, thereby realizing efficient and accurate control of the power grid voltage. Compared with traditional reinforcement learning, the present device divides the conditions of the power grid voltage into two time scales for adjustment by constructing hierarchical reinforcement learning models under long time scales and short time scales, and compensates for the shortcomings of the long-time scale reinforcement learning model due to the uncertainty of load and distributed power sources through the short-time scale optimization model in a layered coordinated and step-by-step optimization manner, thereby improving the accuracy of the power grid voltage regulation. The present device hierarchically trains the initial hierarchical voltage optimization model through a hierarchical constraint reinforcement learning solution algorithm, effectively realizing the global collaborative optimization of the power grid voltage, and effectively reducing the search space through the constrained Markov decision model, thereby improving the calculation efficiency and accuracy of the hierarchical voltage optimization model, and improving the control efficiency and accuracy of the power grid voltage. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1: A flowchart of a method for optimizing power grid voltage based on hierarchical constraint reinforcement learning provided by an embodiment of the present invention;

[0073] Figure 2 : A flowchart of another method for optimizing power grid voltage based on hierarchical constraint reinforcement learning provided by an embodiment of the present invention;

[0074] Figure 3 : A schematic diagram of the structure of a power grid voltage optimization device based on hierarchical constraint reinforcement learning provided by an embodiment of the present invention;

[0075] Among them, 201: long-time scale reinforcement learning model construction module; 202: short-time scale voltage optimization model construction module; 203: hierarchical constraint reinforcement learning module; 204: optimal voltage control strategy acquisition module; 205: power grid voltage optimization module. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0077] In recent years, the proportion of wind and photovoltaic power generation capacity has continued to rise, and wind and solar power have been connected to the grid on a large scale. The impact of wind and solar power output power fluctuations on grid voltage has become increasingly obvious, which not only affects the consumption of new energy, but may even cause frequent fluctuations and even over-limit problems due to the large fluctuations in wind and solar power. At the same time, wind turbines, photovoltaic systems and energy storage systems usually use inverters as grid-connected interfaces, which makes the new power system highly electronic, with rapid power changes and small system inertia. If the response is insufficient, it is easy to cause shocks. In addition, in the new power system, there are many objects involved in the control, and the controlled objects are not on the same time scale in the control time scale, making it difficult to achieve coordinated control. The emergence of various emerging problems has made the traditional reactive voltage control method no longer applicable to the new power system, and it is urgent to propose a more practical and effective coordinated control method for the new power system.

[0078] Existing research can summarize the problem as the processing of reactive power optimization problem. Its essence is a multi-constrained nonlinear combination optimization problem. Its optimization objectives are usually to minimize network loss and reduce node voltage deviation. Its constraints include power flow constraints, terminal voltage, transformer tap position, etc. For this problem, the solution methods are mainly divided into: traditional optimization methods and artificial intelligence methods. Traditional optimization methods are difficult to handle large-scale complex systems, especially when involving multiple complex, nonlinear and dynamic constraints. They are prone to high computational complexity and difficult to solve. They are also highly dependent on the problem model, which may not adapt to actual changes in complex and dynamic power systems. Its voltage control strategy is based on the dispatching layer to achieve reactive power optimization. The shortest control instruction cycle is at the minute level, which cannot solve the problem of real-time voltage fluctuation and offset caused by wind and solar fluctuations. The traditional voltage optimization problem based on reinforcement learning can only handle constraints by adding penalty terms to the reward function. However, this method of unifying rewards and constraints into reward functions blurs the boundary between goals and constraints. It is difficult to determine the appropriate penalty coefficient before the penalty term to balance the relationship between goals and constraints. In order to find a suitable penalty coefficient, it is often necessary to continuously adjust the parameters, which increases the difficulty and workload of the algorithm implementation.

[0079] It should be noted that the Constrained Markov Decision Process (CMDP) described in the embodiment of the present invention is an extended form of the Markov Decision Process (MDP).

[0080] Embodiment 1

[0081] Please refer to Figure 1 , is a step flow chart of a grid voltage optimization method based on hierarchical constrained reinforcement learning provided by an embodiment of the present invention, including steps S101 to S105.

[0082] Step S101: Acquire historical operating status information of the power grid, and construct a constrained Markov decision model based on the historical operating status information to complete the construction of a long-time scale reinforcement learning model, wherein the long-time scale reinforcement learning model is used for an output voltage control strategy; the output voltage control strategy is a long-time scale strategy.

[0083] In an optional embodiment, the acquisition of historical operating status information of the power grid and the construction of a constrained Markov decision model based on the historical operating status information to complete the construction of a long-time scale reinforcement learning model, wherein the long-time scale reinforcement learning model is used for the output voltage control strategy, specifically including:

[0084] Acquire historical operation status information of the power grid, the historical operation status information including: voltage information, power information, switching gear information, load information, equipment action cost information and equipment action number information;

[0085] Constructing a state space and an action space respectively according to the voltage information, power information, switching gear information and load information;

[0086] Obtain the voltage amplitude, conductance value, susceptance value and phase angle difference of each node in the power grid to calculate the active network loss of the power grid;

[0087] The voltage deviation and the operating cost are calculated according to the equipment action cost information and the voltage information, and the voltage deviation, the active network loss and the operating cost are minimized as the decision target of the reward function, thereby constructing the reward function;

[0088] Constructing an auxiliary cost function in combination with the state space and the action space according to the device action number information, voltage information and power information;

[0089] According to a preset discount coefficient, a discount benefit function of the voltage control strategy is constructed in combination with the auxiliary cost function;

[0090] A first objective function is constructed according to the discounted revenue function, state space, action space and reward function, and a constrained Markov decision model is constructed in combination with a preset safety threshold and a preset state transition probability function to form a long-time scale reinforcement learning model.

[0091] This optional embodiment fully considers the constraint problem in the reinforcement learning process by constructing a constrained Markov decision model, and can effectively avoid the complex design of the reward function and the repeated adjustment of the penalty coefficient, thereby reducing the complexity of the long-time scale reinforcement learning model and improving the efficiency of solving the long-time scale reinforcement learning model. Through the auxiliary cost function, it can effectively solve the problem that the traditional Markov decision model is difficult to balance the optimization goal of voltage stability and the safety constraint, thereby effectively reducing the search space of the long-time scale reinforcement learning model and improving the efficiency of solving the long-time scale reinforcement learning model.

[0092] It should be noted that the long-time scale reinforcement learning model described in the embodiment of the present invention belongs to the scheduling optimization model under the upper-level long-time scale, which is used for the output voltage control strategy, and its strategy includes the action state and action amount of the slow-motion equipment, the charging and discharging power of the energy storage equipment, and the optimization reference value of the distributed power supply and the energy storage system within a certain period of time; the slow-motion equipment refers to the on-load tap-changing transformer, the parallel capacitor group, and the synchronous generator set.

[0093] In some optional implementations of this embodiment, the state space specifically includes:

[0094]

[0095] in, and is the injected power of the ith node in period t; and is the voltage amplitude and phase angle of the ith node in period t; and is the switching position of the i-th on-load tap-changing transformer and the i-th group of shunt capacitors in time period t; and is the active and reactive loads in period t; and It is the active and reactive output of distributed power sources in the time period; and is the active and reactive power output of the energy storage system during period t; is the terminal voltage of the generator set in period t.

[0096] In some optional implementations of this embodiment, the action space specifically includes:

[0097]

[0098] Where, ΔC i T is the switching position of the parallel capacitor; i It is the tap position of the on-load tap-changing transformer; is the reactive power output of the distributed power source at the i-th location; P E ′ SSi and Q ′ ESSi are respectively the active and reactive charging and discharging power of the energy storage system at the i-th location; is the increase or decrease of the terminal voltage of the i-th generator set.

[0099] In some optional implementations of this embodiment, the active network loss in the time period t specifically includes:

[0100]

[0101] Where n is the total number of nodes; U i and U j represents the voltage amplitude of node i and node j; G ij , B ij and δ ij are the conductance, susceptance and phase angle difference between node i and node j respectively.

[0102] In some optional implementations of this embodiment, the reward function takes voltage deviation, active network loss and operation cost as the minimum decision-making objectives, and specifically includes:

[0103] r t =-(f short +C T,t +C C,t +C G,t ),

[0104]

[0105] Among them, C T,t is the transformer equipment operation cost; C C,t C is the capacitor equipment operation cost; G,t Adjust the cost for the generator set; U i is the current voltage of node i; U iN is the rated voltage of node i; λ v is the voltage regulation coefficient; c loss is the network loss coefficient, P loss It is the active network loss.

[0106] In an optional embodiment, the auxiliary cost function is constructed based on the device action number information, voltage information and power information in combination with the state space and the action space, specifically including:

[0107] Calculate the bus voltage deviation value of the power grid, the excess power generation of the power generator set of the power grid, the excess power generation of the distributed power source of the power grid, the energy storage system evaluation value of the power grid and the equipment action number constraint according to the equipment action number information, voltage information and power information;

[0108] Adding the bus voltage deviation value of the power grid, the excess power generation of the power generator set of the power grid, the excess power generation of the distributed power source of the power grid, the energy storage system evaluation value of the power grid and the equipment action number constraint, and constructing an auxiliary cost function in combination with the state space and the action space;

[0109] Wherein, the auxiliary cost function is:

[0110]

[0111]

[0112] in, is the voltage on the i-th bus during period t, and are the upper and lower constraints respectively; S G,t is the apparent power output by the generator set during period t, is the rated capacity of the generator set; is the apparent power output by the distributed generation in period t, is the rated capacity of the distributed power source; is the state of charge of the energy storage system during period t, and are the upper and lower constraints respectively; N Ci,t is the number of operations of the capacitor in group i during period t, N Ci,max N is the upper limit of its action number; Ti,t is the number of operations of the i-th transformer tap changer in period t, N Ti,max The upper limit of its action number.

[0113] This optional embodiment can effectively solve the problem that the traditional Markov decision model is difficult to balance the optimization goal of voltage stability and safety constraints through an auxiliary cost function, thereby effectively reducing the search space of the long-time scale reinforcement learning model and improving the solution efficiency of the long-time scale reinforcement learning model.

[0114] It should be noted that in the auxiliary cost function, the first summation term To calculate the bus voltage deviation, the second summation term To calculate the amount of power generated by a generator set in excess of its rated capacity, the third summation term To calculate the amount of DG power generation that exceeds its rated capacity, the fourth summation term To evaluate the energy storage system's violation of constraints, the fifth summation term ∑max(0,N Ci,t -N Ci,max ) and the sixth summation term ∑max(0,N Ti,t -N Ti,max ) is the constraint on the number of device actions.

[0115] In an optional embodiment, constructing a discount benefit function of the voltage control strategy according to a preset discount coefficient in combination with the auxiliary cost function specifically includes:

[0116] According to a preset discount coefficient, a discount benefit function of the voltage control strategy is constructed in combination with the auxiliary cost function;

[0117] Wherein, the discounted revenue function is:

[0118]

[0119] Among them, J C (π) is the discounted benefit function, which is used to characterize the expected discounted benefit corresponding to the auxiliary cost function of the voltage control strategy π; E τ～π It is used to represent the expected value of strategy π; γ is the discount coefficient; c t is the auxiliary cost function; s t is the state space, a tFor the action space.

[0120] This optional embodiment can provide a basis for constructing the first objective function of a long-time-scale reinforcement learning model by introducing a discounted benefit function. The discounted benefit function is used to characterize the relationship between the voltage control strategy, the auxiliary cost function and the discounted benefit. This can effectively maximize the cumulative benefit of the long-time-scale reinforcement learning model and improve the accuracy of solving the long-time-scale reinforcement learning model.

[0121] In an optional embodiment, the first objective function is constructed according to the discounted revenue function, the state space, the action space and the reward function, and a constrained Markov decision model is constructed in combination with a preset safety threshold and a state transition probability function to form a long-time scale reinforcement learning model, specifically including:

[0122] Constructing a first objective function according to the discounted revenue function, the state space, the action space and the reward function, and constructing a constrained Markov decision model in combination with a preset safety threshold and a state transition probability function to form a long-time scale reinforcement learning model;

[0123] Wherein, the constrained Markov decision model is:

[0124]

[0125] J C (π)≤d,

[0126] Among them, J(π) is the first objective function; E τ～π It is used to represent the expected value of the voltage control strategy π; γ is the discount coefficient; r t is the reward function; s t is the state space, a t is the action space, d is the safety threshold; J C (π) is the discounted return function.

[0127] This optional embodiment fully considers the constraint problem in the reinforcement learning process by constructing a constrained Markov decision model, and can effectively avoid the complex design of the reward function and the repeated adjustment of the penalty coefficient, thereby reducing the complexity of the long-time scale reinforcement learning model and improving the efficiency of solving the long-time scale reinforcement learning model. Through the auxiliary cost function, it can effectively solve the problem that the traditional Markov decision model is difficult to balance the optimization goal of voltage stability and the safety constraint, thereby effectively reducing the search space of the long-time scale reinforcement learning model and improving the efficiency of solving the long-time scale reinforcement learning model.

[0128] It should be noted that based on the above-mentioned state space, action space, reward function, state transition probability, discount coefficient, auxiliary cost function, safety threshold and first objective function, a Markov model is constructed, and then a long-time scale reinforcement learning model is determined;

[0129] Among them, the long-time scale reinforcement learning model aims to maximize the cumulative benefit. When the auxiliary cost corresponding to the auxiliary cost function meets the safety threshold, the voltage control strategy π is found to maximize the cumulative reward of the entire time series. The discount coefficient and the safety threshold are both hyperparameters. The discount coefficient of the embodiment of the present invention is 0.98; the safety threshold is 0.01 or 0.001.

[0130] Step S102: construct a short-time scale voltage optimization model, wherein the short-time scale is constrained by the long-time scale strategy and autonomously optimizes the power grid based on the operating status information of the power grid.

[0131] In an optional embodiment, the short-time scale voltage optimization model is constructed, wherein the short-time scale is constrained by the long-time scale strategy, and the power grid is autonomously optimized based on the operation status information of the power grid, specifically including:

[0132] According to the historical operation status information, power flow constraints, voltage constraints, wind turbine output constraints, photovoltaic unit output constraints and energy storage system charging and discharging constraints are established, thereby constructing a constraint set;

[0133] Filter the historical operation status information according to the voltage control strategy data to obtain the historical fluctuation operation status information corresponding to the voltage control strategy data;

[0134] Acquire the voltage regulation coefficient of the power grid, and construct the control variables and the second objective function of the short-time-scale voltage optimization model in combination with the historical fluctuation operation status information to complete the construction of the short-time-scale voltage optimization model;

[0135] Wherein, the second objective function is:

[0136]

[0137] in, minf short is the second objective function, c loss is the network loss coefficient, P loss is the network loss; v is the voltage regulation coefficient, U i is the current voltage at node i, U iN is the rated voltage of node i; n is the total number of nodes; U i and U j represents the voltage amplitude of node i and node j; Gij , B ij and δ ij are the conductance, susceptance and phase angle difference between node i and node j respectively.

[0138] This optional embodiment uses the second objective function with the real-time voltage deviation and the minimum network loss as the optimization target, and uses the control variables to characterize the rapid response characteristics of the distributed power supply and the energy storage system, so that the short-time scale voltage optimization model can make up for the deficiencies of the long-time scale reinforcement learning model due to the uncertainty of the load and the distributed power supply, thereby improving the accuracy of the grid voltage regulation.

[0139] It should be noted that the purpose of the short-time scale voltage optimization model is to use the rapid response characteristics of distributed power sources and energy storage systems as control variables to optimize the power grid system in real time when load or distributed power generation fluctuations occur in the control of the long-time scale reinforcement learning model to ensure the safe and stable operation of the power grid system.

[0140] In some optional aspects of this embodiment, the control variables are used to characterize the rapid response characteristics of the distributed power source and the energy storage system, specifically including:

[0141]

[0142] in, is the reactive power output of the distributed power source at the i-th location; P E ′ SSi and Q ′ ESSi are the active and reactive charging and discharging power of the i-th energy storage system respectively.

[0143] In some optional aspects of this embodiment, the power flow constraint specifically includes:

[0144]

[0145] Among them, P G,i and Q G,i are the active power and reactive power of node i respectively; P D,i and Q D,i are the active power and reactive power requirements of node i respectively; G ij , B ij and δ ij are the conductance, susceptance and phase angle difference between node i and node j respectively; U i and U j represents the voltage amplitude of node i and node j; N0 is the node set except the balance node; N PQ It is a set of PQ nodes.

[0146] In some optional aspects of this embodiment, the voltage constraint specifically includes:

[0147]

[0148] in, and are the upper and lower limits of the voltage at node i respectively.

[0149] In some optional aspects of this embodiment, the wind turbine output constraint specifically includes:

[0150]

[0151] Among them, P WT and Q WT are the active output and reactive output of the wind turbine respectively; P WT,max is the upper limit of active output of wind turbine; Q WT,min and Q WT,max They are the lower and upper limits of the reactive power output of the wind turbine respectively.

[0152] In some optional aspects of this embodiment, the photovoltaic group output constraint specifically includes:

[0153]

[0154] Among them, P PV and Q PV are respectively the active output and reactive output of the photovoltaic unit; P PV,max is the upper limit of active output of the photovoltaic unit; Q PV,min and Q PV,max They are the lower and upper limits of the reactive output of the PV unit respectively.

[0155] In some optional aspects of this embodiment, the energy storage system charging and discharging constraints specifically include:

[0156]

[0157] Among them, P +,max and P -,max are the maximum values ​​of charging power and discharging power respectively, δ is a 0-1 variable, when δ=1, the energy storage system is in charging state, and when δ=0, the energy storage system is in discharging state;

[0158] The power of the energy storage system is subject to a maximum capacity limit, specifically:

[0159]

[0160] Among them, P + and P - are the charging power and discharging power of the energy storage system respectively; QESS is the reactive power output of the energy storage system; S ESS,max is the maximum capacity of the energy storage system.

[0161] Step S103: construct an initial hierarchical voltage optimization model according to the long-time scale reinforcement learning model and the short-time scale voltage optimization model, and perform hierarchical training on the initial hierarchical voltage optimization model in combination with a preset hierarchical constraint reinforcement learning solution algorithm to obtain an optimal hierarchical voltage optimization model.

[0162] In an optional embodiment, the initial hierarchical voltage optimization model is hierarchically trained in combination with a preset hierarchical constraint reinforcement learning solution algorithm to obtain an optimal hierarchical voltage optimization model, specifically including:

[0163] Obtaining Lagrange multipliers, and constructing an unconstrained optimization problem of the initial hierarchical voltage optimization model in combination with the auxiliary cost function and the first objective function;

[0164] constructing a loss function set according to the initial hierarchical voltage optimization model;

[0165] According to a preset hierarchical constraint reinforcement learning solution algorithm, the unconstrained optimization problem of the initial hierarchical voltage optimization model is solved in combination with the loss function set, so as to perform hierarchical training on the initial hierarchical voltage optimization model to obtain an optimal hierarchical voltage optimization model;

[0166] The unconstrained optimization problem is:

[0167]

[0168] g(π)=J C (π)-d,

[0169] Among them, λ is the Lagrange multiplier, J L (π,λ) is an unconstrained optimization problem, E π～π It is used to represent the expected value of the voltage control strategy π; γ is the discount coefficient; r t is the reward function; s t is the state space, a t is the action space, d is the safety threshold; J C (π) is the discounted reward function; f(π) is the cumulative expected reward function, which is used to characterize the total expected reward that the agent can obtain under strategy π; g(π) is used to represent the difference between the total cost brought by strategy π and the safety threshold d.

[0170] This optional embodiment fully considers the reward and constraint issues of reinforcement learning in voltage stability optimization, and introduces the Lagrange multiplier method to transform the constrained optimization problem into an unconstrained problem for solution, thereby avoiding adding constraints to the reward in the form of penalty terms, thereby improving the efficiency and accuracy of the solution, and further improving the efficiency and accuracy of the control of the grid voltage. The over-layered constrained reinforcement learning solution algorithm performs layered training on the initial layered voltage optimization model, effectively realizing the global collaborative optimization of the grid voltage and ensuring the accuracy of the control of the grid voltage.

[0171] In an optional embodiment, the unconstrained optimization problem of the initial hierarchical voltage optimization model is solved according to the preset hierarchical constraint reinforcement learning solution algorithm in combination with the loss function set to perform hierarchical training on the initial hierarchical voltage optimization model to obtain the optimal hierarchical voltage optimization model, specifically including:

[0172] Initializing network parameters, temperature coefficients, Lagrange multipliers and experience pools in the initial hierarchical voltage optimization model;

[0173] Inputting historical operating state information into the initial hierarchical voltage optimization model to obtain a voltage control strategy output by a long-time scale reinforcement learning model, and inputting the voltage control strategy into a short-time scale voltage optimization model to obtain a reward feedback value corresponding to a reward function;

[0174] Calculating experience data according to the reward feedback value, and storing the experience data in an experience pool until the sample size in the experience pool is greater than a preset number, completing storage in the experience pool;

[0175] Samples are extracted from the stored experience pool, and the network parameters, temperature coefficients and Lagrange multipliers are updated in combination with the loss function set, and the unconstrained optimization problem of the initial hierarchical voltage optimization model is solved until the voltage control strategy generated by the long-time-scale reinforcement learning model meets the preset requirements, and the hierarchical training of the initial hierarchical voltage optimization model is completed to obtain the optimal hierarchical voltage optimization model.

[0176] This optional embodiment fully considers the reward and constraint issues of reinforcement learning in voltage stability optimization, and introduces the Lagrange multiplier method to transform the constrained optimization problem into an unconstrained problem for solution, thereby avoiding adding constraints to the reward in the form of penalty terms, thereby improving the efficiency and accuracy of the solution, and further improving the efficiency and accuracy of the control of the grid voltage. The over-layered constrained reinforcement learning solution algorithm performs layered training on the initial layered voltage optimization model, effectively realizing the global collaborative optimization of the grid voltage and ensuring the accuracy of the control of the grid voltage.

[0177] In an optional embodiment, extracting samples from the stored experience pool, updating the network parameters, temperature coefficients and Lagrange multipliers in combination with the loss function set, and solving the unconstrained optimization problem of the initial hierarchical voltage optimization model until the voltage control strategy generated by the long-time scale reinforcement learning model meets the preset requirements, completing the hierarchical training of the initial hierarchical voltage optimization model, and obtaining the optimal hierarchical voltage optimization model, specifically includes:

[0178] The loss function set includes: the loss function of the agent strategy, the critic network loss function, the entropy constraint loss function and the Lagrange multiplier loss function;

[0179] Draw samples from the experience pool of completed storage;

[0180] According to the loss function of the agent strategy and the critic network loss function, the network parameters are updated in combination with the extracted samples;

[0181] According to the entropy-constrained loss function, the temperature coefficient is updated in combination with the extracted samples;

[0182] According to the loss function of the Lagrangian multiplier, the Lagrangian multiplier is updated in combination with the extracted samples;

[0183] Solving the unconstrained optimization problem according to the updated network parameters, temperature coefficients and Lagrange multipliers to update the voltage control strategy;

[0184] When the reward function corresponding to the voltage control strategy converges, the network parameters, temperature coefficients and Lagrange multipliers are updated, and then the hierarchical training of the initial hierarchical voltage optimization model is completed to obtain the optimal hierarchical voltage optimization model.

[0185] This optional embodiment fully considers the reward and constraint issues of reinforcement learning in voltage stability optimization, and introduces the Lagrange multiplier method to transform the constrained optimization problem into an unconstrained problem for solution, thereby avoiding adding constraints to the reward in the form of penalty terms, thereby improving the efficiency and accuracy of the solution, and further improving the efficiency and accuracy of the control of the grid voltage. The over-layered constrained reinforcement learning solution algorithm performs layered training on the initial layered voltage optimization model, effectively realizing the global collaborative optimization of the grid voltage and ensuring the accuracy of the control of the grid voltage.

[0186] It should be noted that the preset hierarchical constrained reinforcement learning solution algorithm refers to the LSAC (Langrangian-soft actor-critic) solution algorithm, that is, a constrained reinforcement learning solution method that combines the Lagrangian multiplier method with the flexible actor-critic algorithm.

[0187] In some optional implementations of this embodiment, the loss function of the agent strategy specifically includes:

[0188] Define J as the loss function, and the loss function of the agent strategy is specifically:

[0189]

[0190] Among them, J π (θ) is the loss function of the agent strategy, θ is the parameter of the strategy π, that is, the strategy network; α is the temperature coefficient, which controls the exploration ability of the strategy; D is the experience replay pool, that is, the experience pool, which is used to store sample experience.

[0191] In some optional implementations of this embodiment, the critic network loss function specifically includes:

[0192]

[0193] Among them, the soft Q network (soft Q function) Q w Used to minimize the soft Bellman residual, V θ is the target value function.

[0194] In some optional implementations of this embodiment, the entropy-constrained loss function specifically includes:

[0195]

[0196] Among them, H0 is the minimum expected entropy.

[0197] In some optional implementations of this embodiment, the loss function of the Lagrange multiplier specifically includes:

[0198]

[0199] Where λ is the Lagrange multiplier, Q w is the soft Q function, and d is the safety threshold.

[0200] It should be noted that when the constraints are violated, the Lagrange multiplier λ is increased to increase the influence of the constraint term, thereby enforcing the constraints more strictly. When the constraints are satisfied, the Lagrange multiplier λ is decreased to reduce the influence of the constraint term, so as to focus more on optimizing the objective function.

[0201] In some optional implementations of this embodiment, solving the unconstrained optimization problem of the initial hierarchical voltage optimization model according to the preset hierarchical constraint reinforcement learning solution algorithm in combination with the loss function set specifically includes:

[0202] The first step is to initialize the strategy network θ, soft Q network w, temperature coefficient α, Lagrange multiplier λ and experience pool D of the initial hierarchical voltage optimization model;

[0203] The second step is to control the dispatch center of the initial hierarchical voltage optimization model based on the state space s t Generate action space a t , and input the corresponding voltage control strategy into the short-time scale voltage optimization model to obtain the reward feedback value r of the reward function t , and r t Return to the dispatch center;

[0204] The third step is to calculate the auxiliary cost function c through the dispatch center t and action space a t ;

[0205] The fourth step is to construct empirical data t ,a t ,r t ,c t ,s t+1}, and store them in the experience pool D until the sample size in the experience pool is greater than the preset number, completing the storage of the experience pool;

[0206] The fifth step is to extract samples M(s) from the experience pool D t ,a t ,r t ,c t ,s t+1 ), the state space s is generated by the initial hierarchical voltage optimization model t+1 The corresponding action space a t+1 ;

[0207] The sixth step is to use the loss function J of the agent strategy π (θ) Update the policy network θ through the critic network loss function J Q (w) Update the soft Q network w, update the temperature coefficient α through the entropy constraint loss function, and update the Lagrange multiplier λ through the Lagrange multiplier loss function;

[0208] The seventh step is to update the voltage control strategy according to the updated strategy network θ, soft Q network w, temperature coefficient α and Lagrange multiplier λ. When the voltage control strategy satisfies the convergence of the reward function curve and the number of action constraint violations is 0, the network parameters, temperature coefficients and Lagrange multipliers are updated, and then the hierarchical training of the initial hierarchical voltage optimization model is completed to obtain the optimal hierarchical voltage optimization model.

[0209] Step S104: acquiring real-time operation status information of the power grid, inputting the real-time operation status information into the optimal hierarchical voltage optimization model, and obtaining an optimal voltage control strategy.

[0210] In some optional implementations of this embodiment, the optimal voltage control strategy specifically includes:

[0211]

[0212] Among them, π * is the optimal voltage control strategy, and J(π) is the first objective function of the long-time-scale reinforcement learning model.

[0213] Step S105: regulating the grid voltage according to the optimal voltage control strategy.

[0214] In some optional implementations of this embodiment, regulating the grid voltage according to the optimal voltage control strategy specifically includes:

[0215] According to the real-time operation status information of the power grid, the optimal voltage control strategy π is obtained * ;

[0216] According to the optimal voltage control strategy π * , adjust the action state and action amount of the action equipment in the power grid, the charging and discharging power of the energy storage equipment, and the working parameters of the distributed power supply and energy storage system to complete the regulation of the power grid voltage.

[0217] In some optional implementations of this embodiment, after regulating the grid voltage, the following further includes:

[0218] Monitor the state response data of the power grid in real time, and update the critic network corresponding to the critic network loss function according to the response data;

[0219] If the response data meets the preset control requirements, save the current voltage control strategy;

[0220] If the response data does not meet the preset control requirements, the current grid operation information is added to the experience pool D to perform offline training and strategy improvement on the optimal hierarchical voltage optimization model.

[0221] Please refer to Figure 2, which is a flowchart of another method for optimizing power grid voltage based on hierarchical constrained reinforcement learning provided by an embodiment of the present invention, including steps S301 to S305. First, according to step S301, a dual-time-scale power grid voltage optimization framework is established, and then a constrained Markov decision process for power system voltage stability is constructed based on step S302 to form a long-time-scale deep reinforcement learning agent framework, and then a real-time voltage optimization model under a short-time scale is constructed in step S303; and according to step S304, a constrained Markov decision process (CMDP) is solved by hierarchical constrained reinforcement learning based on LSAC, and finally, according to step S305, the agent generates a control strategy online in the real-time scenario of the power system to optimize the power grid voltage.

[0222] Embodiment 2

[0223] Please refer to Figure 3 , which is a structural schematic diagram of a power grid voltage optimization device based on hierarchical constrained reinforcement learning provided in an embodiment of the present invention, including: a long-time scale reinforcement learning model construction module 201, a short-time scale voltage optimization model construction module 202, a hierarchical constrained reinforcement learning module 203, an optimal voltage control strategy acquisition module 204 and a power grid voltage optimization module 205.

[0224] Among them, the long-time scale reinforcement learning model construction module 201 is used to obtain the historical operating status information of the power grid, and construct a constrained Markov decision model based on the historical operating status information to complete the construction of the long-time scale reinforcement learning model, wherein the long-time scale reinforcement learning model is used for the output voltage control strategy; the output voltage control strategy is a long-time scale strategy.

[0225] In an optional embodiment, the long-time-scale reinforcement learning model construction module 201 includes: a long-time-scale reinforcement learning model construction unit;

[0226] The long-time scale reinforcement learning model building unit is used to obtain historical operation status information of the power grid, and the historical operation status information includes: voltage information, power information, switching gear information, load information, equipment action cost information and equipment action number information;

[0227] Constructing a state space and an action space respectively according to the voltage information, power information, switching gear information and load information;

[0228] Obtain the voltage amplitude, conductance value, susceptance value and phase angle difference of each node in the power grid to calculate the active network loss of the power grid;

[0229] Calculating voltage deviation and operating cost according to the equipment action cost information and voltage information, and taking voltage deviation, active network loss and operating cost minimization as decision targets of the reward function, thereby constructing a reward function;

[0230] Constructing an auxiliary cost function in combination with the state space and the action space according to the device action number information, voltage information and power information;

[0231] According to a preset discount coefficient, a discount benefit function of the voltage control strategy is constructed in combination with the auxiliary cost function;

[0232] A first objective function is constructed according to the discounted revenue function, state space, action space and reward function, and a constrained Markov decision model is constructed in combination with a preset safety threshold and a preset state transition probability function to form a long-time scale reinforcement learning model.

[0233] In an optional embodiment, the long time scale reinforcement learning model construction unit includes: an auxiliary cost function construction subunit;

[0234] The auxiliary cost function construction subunit is used to calculate the bus voltage deviation value of the power grid, the excess power generation of the power grid generator set, the excess power generation of the distributed power source of the power grid, the energy storage system evaluation value of the power grid and the equipment action number constraint according to the equipment action number information, voltage information and power information;

[0235] Adding the bus voltage deviation value of the power grid, the excess power generation of the power generator set of the power grid, the excess power generation of the distributed power source of the power grid, the energy storage system evaluation value of the power grid and the equipment action number constraint, and constructing an auxiliary cost function in combination with the state space and the action space;

[0236] Wherein, the auxiliary cost function is:

[0237]

[0238] in, is the voltage on the i-th bus during period t, and are the upper and lower constraints respectively; S G,t is the apparent power output by the generator set during period t, is the rated capacity of the generator set; is the apparent power output by the distributed generation in period t, is the rated capacity of the distributed power source; is the state of charge of the energy storage system during period t, and are the upper and lower constraints respectively; N Ci,t is the number of operations of the capacitor in group i during period t, NCi,max N is the upper limit of its action number; Ti,t is the number of operations of the i-th transformer tap changer in period t, N Ti,max The upper limit of its action number.

[0239] In an optional embodiment, the long time scale reinforcement learning model construction unit includes: a discounted benefit function construction subunit;

[0240] The discounted benefit function construction subunit is used to construct a discounted benefit function of the voltage control strategy according to a preset discount coefficient in combination with the auxiliary cost function;

[0241] Wherein, the discounted revenue function is:

[0242]

[0243] Among them, J C (π) is the discounted benefit function, which is used to characterize the expected discounted benefit corresponding to the auxiliary cost function of the voltage control strategy π; E τ～π It is used to represent the expected value of strategy π; γ is the discount coefficient; c t is the auxiliary cost function; s t is the state space, a t For the action space.

[0244] In an optional embodiment, the long-time-scale reinforcement learning model construction unit includes: a long-time-scale reinforcement learning model construction subunit;

[0245] The long-time-scale reinforcement learning model construction subunit is used to construct a first objective function according to the discounted benefit function, the state space, the action space and the reward function, and to construct a constrained Markov decision model in combination with a preset safety threshold and a state transition probability function to form a long-time-scale reinforcement learning model;

[0246] Wherein, the constrained Markov decision model is:

[0247]

[0248] J C (π)≤d,

[0249] Among them, J(π) is the first objective function; E τ～π It is used to represent the expected value of the voltage control strategy π; γ is the discount coefficient; r t is the reward function; s t is the state space, a t is the action space, d is the safety threshold; J C (π) is the discounted return function.

[0250] The short-time scale voltage optimization model construction module 202 is used to construct a short-time scale voltage optimization model, wherein the short-time scale is constrained by the long-time scale strategy and autonomously optimizes the power grid based on the operating status information of the power grid.

[0251] In an optional embodiment, the short-time scale voltage optimization model construction module 202 includes: a short-time scale voltage optimization model construction unit;

[0252] The short-time-scale voltage optimization model building unit is used to establish power flow constraints, voltage constraints, wind turbine output constraints, photovoltaic unit output constraints and energy storage system charging and discharging constraints according to the historical operating status information, thereby building a constraint set;

[0253] Filter the historical operation status information according to the voltage control strategy data to obtain the historical fluctuation operation status information corresponding to the voltage control strategy data;

[0254] Acquire the voltage regulation coefficient of the power grid, and construct the control variables and the second objective function of the short-time-scale voltage optimization model in combination with the historical fluctuation operation status information to complete the construction of the short-time-scale voltage optimization model;

[0255] Wherein, the second objective function is:

[0256]

[0257] in, minf short is the second objective function, c loss is the network loss coefficient, P loss is the network loss; v is the voltage regulation coefficient, U i is the current voltage at node i, U iN is the rated voltage of node i; n is the total number of nodes; U i and U j represents the voltage amplitude of node i and node j; G ij , B ij and δ ij are the conductance, susceptance and phase angle difference between node i and node j respectively.

[0258] The hierarchical constrained reinforcement learning module 203 is used to construct an initial hierarchical voltage optimization model based on the long-time scale reinforcement learning model and the short-time scale voltage optimization model, and perform hierarchical training on the initial hierarchical voltage optimization model in combination with a preset hierarchical constrained reinforcement learning solution algorithm to obtain an optimal hierarchical voltage optimization model.

[0259] In an optional embodiment, the hierarchical constraint reinforcement learning module 203 includes: a hierarchical constraint reinforcement learning unit;

[0260] The hierarchical constraint reinforcement learning unit is used to obtain Lagrange multipliers and construct an unconstrained optimization problem of the initial hierarchical voltage optimization model in combination with the auxiliary cost function and the first objective function;

[0261] constructing a loss function set according to the initial hierarchical voltage optimization model;

[0262] According to a preset hierarchical constraint reinforcement learning solution algorithm, the unconstrained optimization problem of the initial hierarchical voltage optimization model is solved in combination with the loss function set, so as to perform hierarchical training on the initial hierarchical voltage optimization model to obtain an optimal hierarchical voltage optimization model;

[0263] Wherein, the unconstrained optimization problem is:

[0264]

[0265] g(π)=J C (π)-d,

[0266] Among them, λ is the Lagrange multiplier, J L (π,λ) is an unconstrained optimization problem, E τ～π It is used to represent the expected value of the voltage control strategy π; γ is the discount coefficient; r t is the reward function; s t is the state space, a t is the action space, d is the safety threshold; J C (π) is the discounted reward function; f(π) is the cumulative expected reward function, which is used to characterize the total expected reward that the agent can obtain under strategy π; g(π) is used to represent the difference between the total cost brought by strategy π and the safety threshold d.

[0267] In an optional embodiment, the hierarchical constrained reinforcement learning unit includes: a hierarchical constrained reinforcement learning subunit;

[0268] The hierarchical constraint reinforcement learning subunit is used to initialize the network parameters, temperature coefficients, Lagrange multipliers and experience pool in the initial hierarchical voltage optimization model;

[0269] Inputting historical operating state information into the initial hierarchical voltage optimization model to obtain a voltage control strategy output by a long-time scale reinforcement learning model, and inputting the voltage control strategy into a short-time scale voltage optimization model to obtain a reward feedback value corresponding to a reward function;

[0270] Calculating experience data according to the reward feedback value, and storing the experience data in an experience pool until the sample size in the experience pool is greater than a preset number, completing storage in the experience pool;

[0271] Samples are extracted from the stored experience pool, and the network parameters, temperature coefficients and Lagrange multipliers are updated in combination with the loss function set, and the unconstrained optimization problem of the initial hierarchical voltage optimization model is solved until the voltage control strategy generated by the long-time-scale reinforcement learning model meets the preset requirements, and the hierarchical training of the initial hierarchical voltage optimization model is completed to obtain the optimal hierarchical voltage optimization model.

[0272] In an optional embodiment, the hierarchical constrained reinforcement learning subunit includes: a hierarchical constrained reinforcement learning component;

[0273] The hierarchical constrained reinforcement learning component is used for the loss function set including: the loss function of the agent strategy, the critic network loss function, the entropy constraint loss function and the Lagrange multiplier loss function;

[0274] Draw samples from the experience pool of completed storage;

[0275] According to the loss function of the agent strategy and the critic network loss function, the network parameters are updated in combination with the extracted samples;

[0276] According to the entropy-constrained loss function, the temperature coefficient is updated in combination with the extracted samples;

[0277] According to the loss function of the Lagrangian multiplier, the Lagrangian multiplier is updated in combination with the extracted samples;

[0278] Solving the unconstrained optimization problem according to the updated network parameters, temperature coefficients and Lagrange multipliers to update the voltage control strategy;

[0279] When the reward function corresponding to the voltage control strategy converges, the network parameters, temperature coefficients and Lagrange multipliers are updated, and then the hierarchical training of the initial hierarchical voltage optimization model is completed to obtain the optimal hierarchical voltage optimization model.

[0280] The optimal voltage control strategy acquisition module 204 is used to acquire real-time operation status information of the power grid, input the real-time operation status information into the optimal hierarchical voltage optimization model, and obtain the optimal voltage control strategy.

[0281] The grid voltage optimization module 205 is used to regulate the grid voltage according to the optimal voltage control strategy.

[0282] In summary, the embodiment of the present invention constructs a long-time scale reinforcement learning model and a short-time scale voltage optimization model respectively through the historical operation status information of the power grid, thereby forming an initial hierarchical voltage optimization model, and uses a hierarchical constraint reinforcement learning solution algorithm to train the initial hierarchical voltage optimization model, thereby realizing efficient and accurate control of the power grid voltage. Compared with traditional reinforcement learning, the embodiment of the present invention divides the conditions of the power grid voltage into two time scales for adjustment by constructing hierarchical reinforcement learning models under long time scales and short time scales, and compensates for the shortcomings of the long-time scale reinforcement learning model due to the uncertainty of load and distributed power sources through the short-time scale voltage optimization model in a layer-by-layer coordinated and step-by-step optimized manner, thereby improving the accuracy of the power grid voltage regulation. The embodiment of the present invention performs hierarchical training on the initial hierarchical voltage optimization model through a hierarchical constraint reinforcement learning solution algorithm, effectively realizing the global collaborative optimization of the power grid voltage, and effectively reducing the search space through the constrained Markov decision model, thereby improving the calculation efficiency and accuracy of the hierarchical voltage optimization model, and improving the control efficiency and accuracy of the power grid voltage.

[0283] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A grid voltage optimization method based on hierarchical constrained reinforcement learning, characterized in that: include: Acquire historical operating status information of the power grid, and construct a constrained Markov decision model based on the historical operating status information to complete the construction of a long-time scale reinforcement learning model, wherein the long-time scale reinforcement learning model is used for an output voltage control strategy; the output voltage control strategy is a long-time scale strategy; Construct a short-time scale voltage optimization model, where the short-time scale is constrained by the long-time scale strategy and the grid is autonomously optimized based on the grid's operating status information; An initial hierarchical voltage optimization model is constructed according to the long-time scale reinforcement learning model and the short-time scale optimization model, and hierarchical training is performed on the initial hierarchical voltage optimization model in combination with a preset hierarchical constraint reinforcement learning solution algorithm to obtain an optimal hierarchical voltage optimization model; Acquire real-time operation status information of the power grid, input the real-time operation status information into the optimal hierarchical voltage optimization model, and obtain an optimal voltage control strategy; The grid voltage is regulated according to the optimal voltage control strategy.

2. A method for optimizing power grid voltage based on hierarchical constraint reinforcement learning according to claim 1, characterized in that: The historical operation status information of the power grid is obtained, and a constrained Markov decision model is constructed according to the historical operation status information to complete the construction of a long-time scale reinforcement learning model, wherein the long-time scale reinforcement learning model is used for the output voltage control strategy, specifically including: Acquire historical operation status information of the power grid, the historical operation status information including: voltage information, power information, switching gear information, load information, equipment action cost information and equipment action number information; Constructing a state space and an action space respectively according to the voltage information, power information, switching gear information and load information; Obtain the voltage amplitude, conductance value, susceptance value and phase angle difference of each node in the power grid to calculate the active network loss of the power grid; Calculating voltage deviation and operating cost according to the equipment action cost information and voltage information, and taking voltage deviation, active network loss and operating cost minimization as decision targets of the reward function, thereby constructing a reward function; Constructing an auxiliary cost function in combination with the state space and the action space according to the device action number information, voltage information and power information; According to a preset discount coefficient, a discount benefit function of the voltage control strategy is constructed in combination with the auxiliary cost function; A first objective function is constructed according to the discounted revenue function, state space, action space and reward function, and a constrained Markov decision model is constructed in combination with a preset safety threshold and a preset state transition probability function to form a long-time scale reinforcement learning model.

3. A method for optimizing power grid voltage based on hierarchical constraint reinforcement learning as claimed in claim 2, characterized in that: The auxiliary cost function is constructed according to the device action number information, voltage information and power information in combination with the state space and the action space, specifically including: Calculate the bus voltage deviation value of the power grid, the excess power generation of the power generator set of the power grid, the excess power generation of the distributed power source of the power grid, the energy storage system evaluation value of the power grid and the equipment action number constraint according to the equipment action number information, voltage information and power information; Adding the bus voltage deviation value of the power grid, the excess power generation of the power generator set of the power grid, the excess power generation of the distributed power source of the power grid, the energy storage system evaluation value of the power grid and the equipment action number constraint, and constructing an auxiliary cost function in combination with the state space and the action space; Wherein, the auxiliary cost function is: in, is the voltage on the i-th bus during period t, and are the upper and lower constraints respectively; S G,t is the apparent power output by the generator set during period t, is the rated capacity of the generator set; is the apparent power output by the distributed generation in period t, is the rated capacity of the distributed power source; is the state of charge of the energy storage system during period t, and are the upper and lower constraints respectively; N Ci,t is the number of operations of the capacitor in group i during period t, N Ci,max N is the upper limit of its action number; Ti,t is the number of operations of the i-th transformer tap changer in period t, N Ti,max The upper limit of its action number.

4. A method for optimizing power grid voltage based on hierarchical constraint reinforcement learning as claimed in claim 2 or 3, characterized in that: The construction of the discount benefit function of the voltage control strategy according to the preset discount coefficient and in combination with the auxiliary cost function specifically includes: According to a preset discount coefficient, a discount benefit function of the voltage control strategy is constructed in combination with the auxiliary cost function; Wherein, the discounted revenue function is: Among them, J C (π) is the discounted benefit function, which is used to characterize the expected discounted benefit corresponding to the auxiliary cost function of the voltage control strategy π; E τ～π It is used to represent the expected value of strategy π; γ is the discount coefficient; c t is the auxiliary cost function; s t is the state space, a t For the action space.

5. A method for optimizing power grid voltage based on hierarchical constraint reinforcement learning as claimed in claim 4, characterized in that: The first objective function is constructed according to the discounted revenue function, the state space, the action space and the reward function, and a constrained Markov decision model is constructed in combination with a preset safety threshold and a state transition probability function to form a long-time scale reinforcement learning model, specifically including: Constructing a first objective function according to the discounted revenue function, the state space, the action space and the reward function, and constructing a constrained Markov decision model in combination with a preset safety threshold and a state transition probability function to form a long-time scale reinforcement learning model; Wherein, the constrained Markov decision model is: stJ C (π)≤d, Among them, J(π) is the first objective function; E τ～π It is used to represent the expected value of the voltage control strategy π; γ is the discount coefficient; r t is the reward function; s t is the state space, a t is the action space, d is the safety threshold; J C (π) is the discounted return function.

6. A method for optimizing power grid voltage based on hierarchical constraint reinforcement learning as claimed in claim 2, characterized in that: The short-time scale voltage optimization model is constructed, wherein the short-time scale is constrained by the long-time scale strategy and the power grid is autonomously optimized based on the operation status information of the power grid, specifically including: According to the historical operation status information, power flow constraints, voltage constraints, wind turbine output constraints, photovoltaic unit output constraints and energy storage system charging and discharging constraints are established, thereby constructing a constraint set; Filter the historical operation status information according to the voltage control strategy data to obtain the historical fluctuation operation status information corresponding to the voltage control strategy data; Acquire the voltage regulation coefficient of the power grid, and construct the control variables and the second objective function of the short-time-scale voltage optimization model in combination with the historical fluctuation operation status information to complete the construction of the short-time-scale voltage optimization model; Wherein, the second objective function is: in, minf short is the second objective function, c loss is the network loss coefficient, P loss is the network loss; v is the voltage regulation coefficient, U i is the current voltage at node i, U iN is the rated voltage of node i; n is the total number of nodes; U i and U j represents the voltage amplitude of node i and node j; G ij , B ij and δ ij are the conductance, susceptance and phase angle difference between node i and node j respectively.

7. A method for optimizing power grid voltage based on hierarchical constraint reinforcement learning according to claim 6, characterized in that: The initial hierarchical voltage optimization model is hierarchically trained in combination with a preset hierarchical constraint reinforcement learning solution algorithm to obtain an optimal hierarchical voltage optimization model, specifically including: Obtaining Lagrange multipliers, and constructing an unconstrained optimization problem of the initial hierarchical voltage optimization model in combination with the auxiliary cost function and the first objective function; constructing a loss function set according to the initial hierarchical voltage optimization model; According to a preset hierarchical constraint reinforcement learning solution algorithm, the unconstrained optimization problem of the initial hierarchical voltage optimization model is solved in combination with the loss function set, so as to perform hierarchical training on the initial hierarchical voltage optimization model to obtain an optimal hierarchical voltage optimization model; Wherein, the unconstrained optimization problem is: g(π)=J C (π)-d, Among them, λ is the Lagrange multiplier, J L (π,λ) is an unconstrained optimization problem, E τ～π It is used to represent the expected value of the voltage control strategy π; γ is the discount coefficient; r t is the reward function; s t is the state space, a t is the action space, d is the safety threshold; J C (π) is the discounted reward function; f(π) is the cumulative expected reward function, which is used to characterize the total expected reward that the agent can obtain under strategy π; g(π) is used to represent the difference between the total cost brought by strategy π and the safety threshold d.

8. A method for optimizing power grid voltage based on hierarchical constraint reinforcement learning according to claim 7, characterized in that: The method of solving the unconstrained optimization problem of the initial hierarchical voltage optimization model according to the preset hierarchical constraint reinforcement learning solution algorithm and combining the loss function set to perform hierarchical training on the initial hierarchical voltage optimization model to obtain the optimal hierarchical voltage optimization model specifically includes: Initializing network parameters, temperature coefficients, Lagrange multipliers and experience pools in the initial hierarchical voltage optimization model; Inputting historical operating state information into the initial hierarchical voltage optimization model to obtain a voltage control strategy output by a long-time scale reinforcement learning model, and inputting the voltage control strategy into a short-time scale voltage optimization model to obtain a reward feedback value corresponding to a reward function; Calculating experience data according to the reward feedback value, and storing the experience data in an experience pool until the sample size in the experience pool is greater than a preset number, completing storage in the experience pool; Samples are extracted from the stored experience pool, and the network parameters, temperature coefficients and Lagrange multipliers are updated in combination with the loss function set, and the unconstrained optimization problem of the initial hierarchical voltage optimization model is solved until the voltage control strategy generated by the long-time-scale reinforcement learning model meets the preset requirements, and the hierarchical training of the initial hierarchical voltage optimization model is completed to obtain the optimal hierarchical voltage optimization model.

9. A method for optimizing power grid voltage based on hierarchical constraint reinforcement learning as claimed in claim 8, characterized in that: The method extracts samples from the stored experience pool, updates the network parameters, temperature coefficients and Lagrange multipliers in combination with the loss function set, and solves the unconstrained optimization problem of the initial hierarchical voltage optimization model until the voltage control strategy generated by the long-time scale reinforcement learning model meets the preset requirements, completes the hierarchical training of the initial hierarchical voltage optimization model, and obtains the optimal hierarchical voltage optimization model, specifically including: The loss function set includes: the loss function of the agent strategy, the critic network loss function, the entropy constraint loss function and the Lagrange multiplier loss function; Draw samples from the experience pool of completed storage; According to the loss function of the agent strategy and the critic network loss function, the network parameters are updated in combination with the extracted samples; According to the entropy-constrained loss function, the temperature coefficient is updated in combination with the extracted samples; According to the loss function of the Lagrangian multiplier, the Lagrangian multiplier is updated in combination with the extracted samples; Solving the unconstrained optimization problem according to the updated network parameters, temperature coefficients and Lagrange multipliers to update the voltage control strategy; When the reward function corresponding to the voltage control strategy converges, the network parameters, temperature coefficients and Lagrange multipliers are updated, and then the hierarchical training of the initial hierarchical voltage optimization model is completed to obtain the optimal hierarchical voltage optimization model.

10. A power grid voltage optimization device based on hierarchical constraint reinforcement learning, characterized in that: include: Long-time scale reinforcement learning model building module, short-time scale voltage optimization model building module, hierarchical constraint reinforcement learning module, optimal voltage control strategy acquisition module and power grid voltage optimization module; The long-time-scale reinforcement learning model construction module is used to obtain the historical operating status information of the power grid, and to construct a constrained Markov decision model based on the historical operating status information to complete the construction of the long-time-scale reinforcement learning model, wherein the long-time-scale reinforcement learning model is used for the output voltage control strategy; the output voltage control strategy is a long-time-scale strategy; The short-time scale voltage optimization model construction module is used to construct a short-time scale voltage optimization model, wherein the short-time scale is constrained by the long-time scale strategy and autonomously optimizes the power grid based on the operation status information of the power grid; The hierarchical constraint reinforcement learning module is used to construct an initial hierarchical voltage optimization model according to the long-time scale reinforcement learning model and the short-time scale voltage optimization model, and perform hierarchical training on the initial hierarchical voltage optimization model in combination with a preset hierarchical constraint reinforcement learning solution algorithm to obtain an optimal hierarchical voltage optimization model; The optimal voltage control strategy acquisition module is used to acquire real-time operation status information of the power grid, input the real-time operation status information into the optimal hierarchical voltage optimization model, and obtain the optimal voltage control strategy; The grid voltage optimization module is used to regulate the grid voltage according to the optimal voltage control strategy.

Images