A real-time optimal power flow method for deep reinforcement learning guided by holomorphic embedding

Abstract

The present application discloses a real-time optimal power flow method for guiding deep reinforcement learning with a holomorphic embedding method, and relates to the technical field of power system optimization operation. The method includes: defining the system state variables and strategy action variables of the power system respectively; constructing a strategy model corresponding to the power system based on the system state variables and the strategy action variables, wherein only the system state is stored in the experience buffer corresponding to the strategy model; converting the nonlinear power flow equation of the power system into a holomorphic function to construct a critic function corresponding to the power system; performing alternating updates and iterations on the critic function and the strategy model, and when the preset convergence conditions are met, outputting the final updated strategy model, and determining the optimal strategy action corresponding to the power system. The present application can solve the problems of adaptability and security of deep reinforcement learning algorithms in the process of power system optimization.

Description

Technical Field

[0001] The present application relates to the technical field of power system optimization operation, and in particular to a real-time optimal power flow method for guiding deep reinforcement learning using a holomorphic embedding method. Background Art

[0002] With the increasing global concern about energy crisis and climate change, modern energy systems are rapidly transforming towards low-carbon, green and sustainable development. This transformation places higher demands on the operation and control of modern power systems. Optimal Power Flow (OPF), as the core issue of power system operation and control, aims to achieve the stability of the power system by optimizing operating conditions. However, with the continuous increase of renewable energy, the complexity of the power market and the dynamic changes of the power network, traditional mathematical programming-based methods face challenges in solving real-time optimization problems. To meet these challenges, advanced machine learning techniques, especially deep reinforcement learning (DRL), have been introduced into the operation and control of power systems in recent years.

[0003] At present, the control strategy of deep reinforcement learning is usually based on deep neural networks. However, directly applying existing deep reinforcement learning algorithms to complex power systems to solve optimal power flow problems will have problems with adaptability and safety, that is, under certain operating conditions, the safety and reliability of predicted control actions cannot be guaranteed. Summary of the invention

[0004] In view of this, the present application provides a real-time optimal power flow method that uses a holomorphic embedding method to guide deep reinforcement learning. The main purpose is to solve the adaptability and safety problems of deep reinforcement learning algorithms in the process of power system optimization, thereby ensuring the safety and reliability of predicted control actions.

[0005] According to a first aspect of the present application, a real-time optimal power flow method for guiding deep reinforcement learning using a holomorphic embedding method is provided, the method comprising:

[0006] Define the system state variables and strategy action variables of the power system respectively;

[0007] Based on the system state variables and the strategy action variables, construct a strategy model corresponding to the power system, wherein the experience buffer corresponding to the strategy model only stores the system state;

[0008] Converting the nonlinear power flow equation of the power system into a holomorphic function to construct a critic function corresponding to the power system;

[0009] The critic function and the strategy model are updated and iterated alternately, and when a preset convergence condition is met, the final updated strategy model is output, and the optimal strategy action corresponding to the power system is determined.

[0010] According to a second aspect of the present application, a real-time optimal power flow device for guiding deep reinforcement learning using a holomorphic embedding method is provided, the device comprising:

[0011] A definition unit, used to define system state variables and strategy action variables of the power system respectively;

[0012] A first construction unit is used to construct a strategy model corresponding to the power system based on the system state variable and the strategy action variable, wherein the experience buffer corresponding to the strategy model only stores the system state;

[0013] A second construction unit is used to convert the nonlinear power flow equation of the power system into a holomorphic function to construct a critic function corresponding to the power system;

[0014] The updating unit is used to alternately update and iterate the critic function and the strategy model, and when a preset convergence condition is met, output the final updated strategy model and determine the optimal strategy action corresponding to the power system.

[0015] According to the third aspect of the present application, a storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the real-time optimal power flow method for guiding deep reinforcement learning using the holomorphic embedding method is implemented.

[0016] According to the fourth aspect of the present application, an electronic device is provided, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein when the processor executes the program, the real-time optimal power flow method for guiding deep reinforcement learning using a holomorphic embedding method is implemented.

[0017] Through the above technical solution, the present application provides a real-time optimal power flow method that uses a holomorphic embedding method to guide deep reinforcement learning. By adopting a physically driven critic function instead of a deep neural network, the adaptability and safety problems of deep reinforcement learning algorithms in the power system optimization process can be solved. That is, this method can ensure that the optimization results are more in line with the actual operation of the power system, thereby providing a more reliable decision-making basis and ensuring the safety and reliability of control actions.

[0018] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 A flow chart of a real-time optimal power flow method for guiding deep reinforcement learning using a holomorphic embedding method provided in an embodiment of the present application is shown;

[0021] Figure 2 A schematic diagram of the algorithm architecture provided by an embodiment of the present application is shown;

[0022] Figure 3 A flow chart of another method for real-time optimal power flow guiding deep reinforcement learning using a holomorphic embedding method provided in an embodiment of the present application is shown;

[0023] Figure 4 A schematic diagram of an update process provided by an embodiment of the present application is shown;

[0024] Figure 5 A structural schematic diagram of a real-time optimal power flow device for guiding deep reinforcement learning using a holomorphic embedding method provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0025] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0026] Directly applying existing deep reinforcement learning algorithms to complex power systems to solve optimal power flow problems will lead to problems of adaptability and safety, that is, the safety and reliability of predicted control actions cannot be guaranteed under certain operating conditions.

[0027] In order to solve the above problems, the embodiment of the present invention provides a real-time optimal power flow method for guiding deep reinforcement learning using a holomorphic embedding method, such as Figure 1 As shown, the method includes:

[0028] Step 101: Define system state variables and strategy action variables of the power system respectively.

[0029] For the embodiment of the present invention, the optimal power flow (OPF) problem is a basic problem of power system operation and control, and its goal is to find the best operating decision that minimizes the operating cost while satisfying all constraints. In order to solve the problems existing in the deep reinforcement learning (DRL) algorithm in the OPF application, the embodiment of the present invention first models the OPF problem as a Markov decision model (MDP), and defines the system state variables, policy action variables and reward functions. Secondly, a new type of critic function is proposed to replace the deep neural network, and the most advanced power flow method is used to find the estimate of the value of the state-action pair, that is, the Q value. Since the critic network must be differentiable, the embodiment of the present invention adopts the holomorphic embedding principle to calculate the Q value. The overall algorithm architecture of the embodiment of the present invention is as follows. Figure 2 shown.

[0030] In conventional SAC algorithms, it is usually necessary to train two critics and two target deep neural networks (DNNs). In order to solve the problems of adaptability and safety of deep reinforcement learning algorithms in the process of power system optimization and ensure that the optimization results are more in line with the actual operation of the power system, the embodiment of the present invention uses a physics-driven critic function to replace the critic's deep neural network. At the same time, the experience buffer is designed to only store the system state, that is, it does not store policy actions and reward values. It can be seen that the algorithm used in the embodiment of the present invention only uses the system state, which is a DRL algorithm guided by a holomorphic embedding algorithm.

[0031] In view of the optimal power flow problem of the power system, the embodiment of the present invention defines three elements in the Markov decision model (MDP), specifically including the system state variable s∈S, the strategy action variable a∈A and the reward value function r. For the specific definition process of the system state variable and the strategy action variable, the method includes: based on the active power variable corresponding to each bus in the power system Reactive power variable and wind power output variables and the current power output variables of each generator Define the system state variables of the power system; based on the active power setting value variables p of each generator other than the balancing machine in the power system g , the voltage setting value variable of the current collection line in each busbar|v i |, and the reactive power setting value variable q of the reactive power compensation device x , defining the strategic action variables of the power system.

[0032] Specifically, the system state variable s is defined as follows:

[0033]

[0034] in, are the active power and reactive power of the ith bus, respectively. is the wind power output of the ith bus, and is the current power output of generator q, Represents the set of all generators. Reflect the working conditions of the generator and constrain the power change rate within the time step. In addition, other parameters can be incorporated into the system state space to reflect other uncertainties, such as real-time electricity prices and line status, etc., which are not specifically limited in the embodiments of the present invention.

[0035] Furthermore, the policy action variables are defined as follows:

[0036]

[0037] Among them, p g Removal of balancing machines on behalf of power systems The active power setting value of all generators except i |Represents the current collection line in each bus (PV bus) and slack busbar The voltage setting value of these buses is connected to the generator or various compensators, q x Represents the reactive power setting value of reactive power compensation equipment such as static compensators.

[0038] In some embodiments, with respect to the specific definition process of the reward value function, the method includes: determining a production cost nonlinear function and a default function corresponding to the power system, wherein the independent variable of the production cost nonlinear function is the active power setting value of each generator other than the balancing machine in the power system; based on the production cost nonlinear function and the default function, defining the reward value function corresponding to the power system.

[0039] Specifically, the reward value function r is used to measure the value of the predicted action under a given power system state, thereby guiding the agent to move in a specific direction. Since OPF is to minimize the operating cost and ensure zero default, the reward value function can be designed as:

[0040]

[0041] Among them, ω0, ω c ,ω z is the weight coefficient, f g (.) is a nonlinear function of the production cost, η is a vector of weighting or normalization factors used to set the relative importance of the operating constraints. Z is the default vector, indicating that the The default value of the constraint is calculated by the following formula for each component of the default vector Z:

[0042]

[0043] Among them, x i It can represent voltage setting value, active power, reactive power, current of different branches, and x i They represent the voltage setting value, active power, reactive power, and upper and lower limits of different branch currents respectively.

[0044] The critic function (Q-value function) of the embodiment of the present invention is designed to focus on the exact reward of the current action. The strategy optimization objective function defined in the embodiment of the present invention is as follows:

[0045]

[0046] Among them, π * Represents the optimal strategy action, that is, the optimal strategy action determined when the cumulative Q value is expected to be maximized.

[0047] Step 102: construct a strategy model corresponding to the power system based on the system state variables and the strategy action variables.

[0048] The experience buffer corresponding to the strategy model only stores the system status.

[0049] For an embodiment of the present invention, based on the system state variables and the strategy action variables, a preset neural network algorithm is used to construct a strategy model corresponding to the power system. The structure of the strategy model is based on a multi-layer deep neural network, with the input being the system state and the output being the strategy action under the current system state.

[0050] Compared with the existing SAC algorithm which stores system status, strategy actions, and reward values ​​in the experience buffer, the embodiment of the present invention only stores the power system status in the experience buffer, thereby reducing the amount of data calculation and storage.

[0051] Step 103: Convert the nonlinear power flow equation of the power system into a holomorphic function to construct a critic function corresponding to the power system.

[0052] For the embodiment of the present invention, according to the definition of the above policy optimization objective function, the critic function needs to use the system state and policy action to calculate the reward. In order to successfully design the critic function, the embodiment of the present invention takes the following measures: 1. Use the holomorphic embedding method to reconstruct the completed decision variables from the predicted control action; 2. Calculate the reward value function through a simple differentiable function; 3. Develop a vectorized version of the critic function to include a small batch sample size for policy learning.

[0053] In some embodiments, with respect to the specific construction process of the commentator function, the method includes: converting the nonlinear power flow equation into an embedded system of complex voltage variables and reactive power variables regarding each bus, wherein the complex voltage variables and the reactive power variables can be expressed as holomorphic functions regarding embedded coefficients; and constructing the commentator function corresponding to the power system based on the embedded system.

[0054] In some embodiments, constructing a critic function corresponding to the power system based on the embedded system includes: expanding the holomorphic functions corresponding to the complex voltage variable and the reactive power variable into Maclaurin series, and bringing them into the embedded system to obtain an expanded embedded system, wherein the expanded embedded system contains target parameters; performing coefficient equationization and imaginary and real part separation on the expanded embedded system in turn to obtain a complex equation corresponding to the power system; and determining the critic function corresponding to the power system based on the complex equation and the reward value function.

[0055] Specifically, the nonlinear power flow equations of the power system are converted into an embedded system with holomorphic functions. These functions can be expanded into polynomial series, and the coefficients can be recursively calculated using linear equations. The specific expression is as follows:

[0056]

[0057] Among them, Y ij represents the admittance matrix element of the power system, v j represent, It is the collection of the slack bus, PV bus (current collection line), and PQ bus. i Represents the complex voltage corresponding to each bus, v i =|v i |∠θ i ,θ i Representative, e i represents the complex power corresponding to each bus, e i =p i +jq i , p i and q i Represent the active power and reactive power corresponding to each bus, and q g represents the reactive power of each generator except the balancing machine, where Q i is a subset of the generators, wind farms, and reactive power compensation equipment connected to bus i. With e iConjugation, With v i Conjugate. The embedded system of formula (6) can be written as:

[0058]

[0059] Among them, v i (α) and q i (α) represent the complex voltage and reactive power, v i (α) and q i (α) is the embedded form and is about the embedding coefficient is a holomorphic function of , where α is essentially a complex variable pre-set according to actual needs. i (α) and q i The image of (α) relative to the complex variable α is the PV curve of the continuous power flow, where α=0 for the no-load state and α=1 for the full-load state. In order to ensure the holomorphism of the embodiment of the present invention, Using alpha * Instead of using α for embedding, is a symmetric transfer matrix.

[0060] Furthermore, v i (α) and q i (α) is expanded into a Maclaurin series as follows:

[0061]

[0062] in, It can be expressed as Among them, W i,n is the target parameter that needs to be updated iteratively. From this conjugate property, the recursive relationship can be obtained as follows:

[0063]

[0064] Among them, for no-load state, V i,0 =1,W i,0 =1,Q i,0 =0.

[0065] Furthermore, v i (α),q i (α), and W i,n The expressions of are substituted into the embedded system of formula (7), and then the equation α is transformed into n The coefficient of and the imaginary part Separation can obtain the complex equation, as shown in formula (10),

[0066]

[0067] in, B i,j =1, Otherwise B i,j is zero. n , U n The definition is as follows:

[0068]

[0069] In formula (10), b n is x n , W n-1 , strategy action a and system state s. By solving the above complex equation, the commentator function can be determined. In the process of solving, for the convenience of calculation, LU decomposition can be used to decompose A into an upper triangular matrix and a lower triangular matrix multiplied, so that A is finally decomposed into all coefficients and data samples at one time. Based on the holomorphic embedding algorithm provided by the embodiment of the present invention, when solving the complex equation, it is necessary to continuously i,n The target parameters are updated iteratively until the following conditions are met:

[0070]

[0071] in, and are the voltage limit value and power limit value respectively. Formula (12a) is the voltage limit condition, formula (12b) is the complex power limit condition, and formula (12c) is the reactive power limit condition.

[0072] Since the traditional power grid power flow equation is a nonlinear equation and sometimes may not find a solution, in order to solve this problem, the embodiment of the present invention embeds a holomorphic function in the power flow equation, that is, both sides of the equation are expressed in the form of a series, and the equation is realized by superposition of the series. In this way, in the process of solving the equation, these series can be solved and superimposed to obtain the final solution. It can be seen that the holomorphic embedding algorithm used in the embodiment of the present invention can ensure that the power flow equation has a solution. In addition, the embodiment of the present invention converts the nonlinear power flow equation into a set of linear equations through the holomorphic embedding algorithm, which makes it possible to apply automatic differential calculation, thereby simplifying the solution process and improving the calculation efficiency. The use of this series makes the algorithm more feasible in a real-time operating environment, significantly improves the operating efficiency of the system, and helps to improve the response speed and flexibility of the power system.

[0073] Furthermore, based on the differentiable holomorphic embedding system (the complex equation corresponding to formula (10)) and the reward function, the critic function corresponding to the power system is constructed. Specifically, the reward function consists of two parts, as shown in formula (3). For the operating cost, except for the slack power output, all active power outputs p g All are predicted by the strategy, and the calculation formula is as follows:

[0074]

[0075] For the violation of constraints, the default vector Z is calculated by formula (4) and normalized by a fixed factor η. The default vector Z includes the PQ bus voltage, the active power of the generator, the reactive power of the PV bus and the slack bus, and the violation of the boundary constraints for the sending and receiving power flows, respectively, and is defined as:

[0076]

[0077] Wherein, formula (14a) is the boundary constraint of PQ bus voltage, formula (14b) is the boundary constraint of generator active power, formula (14c) is the boundary constraint of PV bus and slack bus reactive power, formula (14d) and (14c) are the boundary constraints of sending and receiving power flow current, respectively, Y t is the node admittance matrix of the power grid system, Y f is the branch admittance matrix.

[0078] The embodiments of the present invention can ensure that the power system can maintain stable operation under various circumstances by imposing boundary constraints on control actions and penalties for breach of contract, thereby improving the robustness and reliability of the power system and thus enhancing the adaptability of the power system to external changes.

[0079] Step 104: alternately update and iterate the critic function and the strategy model, and when a preset convergence condition is met, output the final updated strategy model, and determine the optimal strategy action corresponding to the power system.

[0080] Among them, the preset convergence condition can be set according to actual business needs.

[0081] For the embodiment of the present invention, after the critic function and the strategy model are preliminarily defined, the critic function and the strategy model need to be updated and iterated alternately. When a certain number of iterations is reached, the iteration is stopped, and the final updated strategy model is output. The current system state of the power system is input into the final updated strategy model, and the optimal strategy action corresponding to the power system can be obtained. The specific iteration process for the critic function and the strategy model is detailed in Example 2.

[0082] The embodiment of the present invention adopts a physics-driven critic function instead of a deep neural network, which can solve the adaptability and safety problems of the deep reinforcement learning algorithm in the power system optimization process. That is, this method can ensure that the optimization results are more in line with the actual operation of the power system, thereby providing a more reliable decision-making basis and ensuring the safety and reliability of control actions.

[0083] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to fully illustrate the implementation of this embodiment, this embodiment also provides another real-time optimal power flow method for guiding deep reinforcement learning using a holomorphic embedding method, such as Figure 3 As shown, the method includes:

[0084] Step 201: Define system state variables and strategy action variables of the power system respectively.

[0085] For the embodiment of the present invention, the specific process of defining the system state variables, strategy action variables and reward value function of the power system is exactly the same as step 101, and will not be repeated here.

[0086] Step 202: construct a strategy model corresponding to the power system based on the system state variables and the strategy action variables.

[0087] The experience buffer corresponding to the strategy model only stores the system status.

[0088] The policy model of the embodiment of the present invention can be based on the system status s t Predictive control action a t , whose structure is based on a multi-layer deep neural network as shown below:

[0089]

[0090] in, are network parameters, L is the number of layers, is the activation function of the lth layer. In order to update the policy parameters, an experience buffer is first designed to store only the system state. Then, according to the formula (5) of the policy objective, the loss function is defined as:

[0091]

[0092] Among them, M is the size of the mini-batch data, s m is the system state sampled from the experience buffer, a m is the sampled system state s m The corresponding policy action a m .

[0093] Step 203: Convert the nonlinear power flow equation of the power system into a holomorphic function to construct a critic function corresponding to the power system.

[0094] For the embodiment of the present invention, the critic function defined in the above step 103 is vectorized to obtain a vectorized critic function to provide a series of advantages of small batch data, including performance improvement, code simplicity and readability enhancement, loop overhead reduction, memory utilization improvement, GPU or CPU parallel computing potential, and scalability for large power systems. Considering that random initialization of strategy parameters in large-scale power systems may lead to learning divergence, the following are adopted here: 1. Use small data sets for imitation learning; 2. Train multiple strategies based on a narrow uncertainty range; 3. Avoid divergence points at the beginning of training; 4. Use a fast dynamic holomorphic embedding method to reduce the number of iterations.

[0095] At the same time, for the vectorized critic function, the method also includes: alternately updating and iterating the vectorized critic function and the strategy model, and when the preset convergence conditions are met, outputting the final updated strategy model, and determining the optimal strategy action corresponding to the power system.

[0096] Step 204: extract the system state corresponding to the power system from the experience buffer.

[0097] When the critic function and the strategy model are alternately updated in the embodiment of the present invention, the system state corresponding to the power system is firstly extracted from the experience buffer.

[0098] Step 205: input the system state corresponding to the power system into the strategy model to perform action prediction, and obtain the predicted strategy action corresponding to the power system.

[0099] According to the embodiment of the present invention, the extracted system state is input into the strategy model for action prediction, so that the predicted strategy action of the power system under the system state can be obtained.

[0100] Step 206: Based on the system state and the predicted strategic action, the critic function and the strategic model are updated and iterated alternately. When a preset convergence condition is met, the final updated strategic model is output, and the optimal strategic action corresponding to the power system is determined.

[0101] For the embodiment of the present invention, for the alternating update process of the strategy model and the critic function, step 206 specifically includes: based on the system state and the predicted strategy action, updating the critic function to obtain the critic function after this round of update; derivation of the critic function after this round of update to obtain the reference strategy action corresponding to the power system; according to the reference strategy action and the predicted strategy action, updating the strategy model to obtain the strategy model after this round of update; re-determining the system state of the power system under the predicted strategy action, and re-determining the predicted strategy action corresponding to the power system based on the re-determined system state and the strategy model after this round of update; based on the re-determined system state and predicted strategy action, repeating the process of alternatingly updating the critic function and the strategy model, when the preset conditions are met, stopping the update iteration, and outputting the trained predicted strategy model; according to the trained predicted strategy model, determining the optimal strategy action corresponding to the power system.

[0102] Furthermore, the commentator function is updated based on the system state and the prediction strategy action to obtain the commentator function after this round of update, including: solving the complex equation corresponding to the commentator function based on the system state and the prediction strategy action to obtain the complex power corresponding to each bus; based on the complex power and the default function, respectively calculating the default vector of the power system and the active power setting value of each generator; substituting the default vector and the active power setting value into the reward value function to calculate the reward value corresponding to the power system; based on the reward value, determining the commentator function after this round of update.

[0103] Furthermore, based on the system state and the prediction strategy action, the complex equation corresponding to the commentator function is solved to obtain the complex power corresponding to each bus, including: initializing the target parameter in the complex equation to obtain the initial value corresponding to the target parameter; substituting the initial value corresponding to the target parameter into the complex equation for solving, to obtain the initial complex voltage, initial reactive power and initial complex power corresponding to each bus; if the initial complex voltage does not meet the voltage setting value limitation condition of the current collection line, or the initial reactive power does not meet the reactive power of the current collection line If the active power limitation condition is met, or the initial complex power does not meet the complex power limitation condition of the distribution equipment bus, the initial value corresponding to the target parameter is updated and iterated, and the target parameter value after this round of update is re-substituted into the complex equation for solving; the updating and iterative process of the target parameter value and the solving process of the complex equation are repeated until the solved complex voltage meets the voltage setting value limitation condition of the current collection line, the solved reactive power meets the reactive power limitation condition of the current collection line, and the solved complex power meets the complex power limitation condition of the distribution equipment bus, then the complex power is output.

[0104] like Figure 4 As shown, first set the formula (12) according to the actual business needs. ∈, ε, and set n = 1, for W i,n Initialize and get W i,n Then, according to formula (10) and formula (11), solve the complex equation, decompose A when solving, and calculate it according to b n =f(x n-1 ,a,s), calculate x n =A -1 b n According to formula (10), at x n After solving it, we can determine V n and Q n ,because Therefore, the initial complex voltage v corresponding to each bus can be calculated i and initial reactive power q i Then the initial voltage v i Substituting into formula (6), the initial complex power e can be calculated i . In determining the initial complex voltage v i , initial reactive power q i and the initial complex power e i Afterwards, we determine v i Whether the voltage setting value limit condition of PQ bus is met (Formula (12a)), e iWhether the complex power limit condition of the distribution equipment busbar is met (Formula (12b)), q i Whether the reactive power limit condition of PQ bus is met (Formula (12c)). If any of the above conditions is not met, then continue to adjust the target parameter W i,n Update, n = n + 1, and solve the complex equation formula (10) again; if the above conditions are met, stop the iteration and output the final solved complex voltage v i , reactive power q i and complex power e i . Further, in determining the complex voltage v i , reactive power q i and complex power e i Then, the active power setting value p can be calculated according to formula (13): g According to formula (14) and formula (4), the default vector Z can be calculated. Further, the active power setting value p g Substitute the default vector Z into the reward value function (Formula (3)) to calculate the reward value, and finally determine the reviewer function after this round of update based on the reward value and the Bellman equation.

[0105] Furthermore, the critic function after this round of update is derived to obtain the reference strategy action when the critic function Q value is the largest. Then, the reference strategy action is subtracted from the prediction strategy action, and the strategy model is updated to obtain the strategy model after this round of update. Repeat the alternating update process of the strategy model and the critic function until the preset conditions are met, stop the update iteration, and output the trained prediction strategy model. Based on the trained prediction strategy model, determine the optimal strategy action corresponding to the power system.

[0106] The embodiments of the present invention can be applied to power systems of different scales and complexities, and can be adjusted and optimized according to specific circumstances, can meet different system requirements, and have strong scalability and flexibility.

[0107] The embodiment of the present invention adopts a physics-driven critic function instead of a deep neural network, which can solve the adaptability and safety problems of the deep reinforcement learning algorithm in the power system optimization process. That is, this method can ensure that the optimization results are more in line with the actual operation of the power system, thereby providing a more reliable decision-making basis and ensuring the safety and reliability of control actions.

[0108] Further, as Figure 1 and Figure 3 The specific implementation of the method shown in the embodiment provides a real-time optimal power flow device for guiding deep reinforcement learning using a holomorphic embedding method, such as Figure 5As shown, the device includes: a definition unit 31, a first construction unit 32, a second construction unit 33 and an update unit 34.

[0109] The definition unit 31 may be used to define system state variables and strategy action variables of the power system respectively.

[0110] The first construction unit 32 may be configured to construct a strategy model corresponding to the power system based on the system state variables and the strategy action variables, wherein the experience buffer corresponding to the strategy model only stores the system state.

[0111] The second constructing unit 33 may be used to convert the nonlinear power flow equation of the power system into a holomorphic function to construct a critic function corresponding to the power system.

[0112] The updating unit 34 may be used to perform alternate updates and iterations on the critic function and the strategy model, and when a preset convergence condition is met, output a final updated strategy model and determine an optimal strategy action corresponding to the power system.

[0113] In a specific application scenario, the definition unit 31 can be specifically used to define the system state variables of the power system based on the active power variables, reactive power variables and wind power output variables corresponding to each bus in the power system, and the current power output variables of each generator; based on the active power setting value variables of each generator other than the balancing machine in the power system, the voltage setting value variables of the current collection line and the slack bus in each bus, and the reactive power setting value variables of the reactive compensation equipment, define the strategy action variables of the power system.

[0114] In some embodiments, the definition unit 31 can also be used to determine the production cost nonlinear function and default function corresponding to the power system, wherein the independent variable of the production cost nonlinear function is the active power setting value of each generator other than the balancing machine in the power system; based on the production cost nonlinear function and the default function, the reward value function corresponding to the power system is defined.

[0115] In some embodiments, the first construction unit 32 may be specifically configured to construct a strategy model corresponding to the power system based on the system state variables and the strategy action variables using a preset neural network algorithm.

[0116] In some embodiments, the device further comprises: a processing unit.

[0117] The processing unit may be used to perform vectorization processing on the critic function to obtain a vectorized critic function.

[0118] The updating unit 34 may be specifically configured to perform alternate updates and iterations on the vectorized critic function and the strategy model, and when a preset convergence condition is met, output a final updated strategy model, and determine an optimal strategy action corresponding to the power system.

[0119] In some embodiments, the updating unit 34 includes: an extraction module, a prediction module and an updating module.

[0120] The extraction module may be used to extract the system state corresponding to the power system from the experience buffer.

[0121] The prediction module can be used to input the system state corresponding to the power system into the strategy model to perform action prediction, and obtain the predicted strategy action corresponding to the power system.

[0122] The update module can be used to alternately update and iterate the critic function and the strategy model based on the system state and the predicted strategy action. When the preset convergence condition is met, the final updated strategy model is output and the optimal strategy action corresponding to the power system is determined.

[0123] In some embodiments, the update module includes: an update submodule, a derivation submodule and a determination submodule.

[0124] The updating submodule may be used to update the critic function based on the system state and the prediction strategy action to obtain the critic function after this round of update.

[0125] The derivation submodule can be used to derive the critic function after the current round of update to obtain the reference strategy action corresponding to the power system.

[0126] The updating submodule may also be used to update the policy model according to the reference policy action and the predicted policy action to obtain the policy model after this round of updating.

[0127] The determination submodule may be used to redetermine the system state of the power system under the prediction strategy action, and redetermine the prediction strategy action corresponding to the power system based on the redetermined system state and the strategy model after this round of update.

[0128] The updating submodule can also be used to repeat the process of updating the critic function and the strategy model alternately based on the re-determined system state and prediction strategy action, stop the updating iteration when the preset conditions are met, and output the trained prediction strategy model.

[0129] The determination submodule may also be used to determine the optimal strategy action corresponding to the power system according to the trained prediction strategy model.

[0130] In some embodiments, the update submodule can be specifically used to solve the complex equation corresponding to the commentator function based on the system state and the prediction strategy action to obtain the complex power corresponding to each bus; based on the complex power and the default function, respectively calculate the default vector of the power system and the active power setting value of each generator; substitute the default vector and the active power setting value into the reward value function to calculate the reward value corresponding to the power system; based on the reward value, determine the commentator function after this round of update.

[0131] In some embodiments, the update submodule can also be specifically used to initialize the target parameters in the complex equation to obtain the initial values ​​corresponding to the target parameters; substitute the initial values ​​corresponding to the target parameters, the system state and the prediction strategy action into the complex equation for solving, and obtain the initial complex voltage, initial reactive power and initial complex power corresponding to each bus; if the initial complex voltage does not meet the voltage setting value limitation conditions of the current collection line, or the initial reactive power does not meet the reactive power limitation conditions of the current collection line, or the initial complex power does not meet the complex power limitation conditions of the distribution equipment bus, then the initial values ​​corresponding to the target parameters are updated and iterated, and the target parameter values ​​after this round of update are re-substituted into the complex equation for solving; repeat the updating and iterative process of the target parameter values ​​and the solving process of the complex equation until the solved complex voltage meets the voltage setting value limitation conditions of the current collection line, the solved reactive power meets the reactive power limitation conditions of the current collection line, and the solved complex power meets the complex power limitation conditions of the distribution equipment bus, then the complex power is output.

[0132] It should be noted that for other corresponding descriptions of the functional units involved in the real-time optimal power flow device for guiding deep reinforcement learning using a holomorphic embedding method provided in this embodiment, please refer to Figure 1 and Figure 3 The corresponding description in will not be repeated here.

[0133] Based on the above Figure 1 and Figure 3 The method shown in the embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned Figure 1 and Figure 3 The real-time optimal power flow method for deep reinforcement learning guided by the holomorphic embedding method is shown.

[0134] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.), including a number of instructions for enabling an electronic device (which can be a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of the present application.

[0135] Based on the above Figure 1 and Figure 3 The method shown, and Figure 5 In order to achieve the above-mentioned purpose, the embodiment of the present application also provides an electronic device, which can be a personal computer, a tablet computer, a server, or other network equipment, etc. The device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above-mentioned Figure 1 and Figure 3 The real-time optimal power flow method for deep reinforcement learning guided by the holomorphic embedding method is shown.

[0136] Optionally, the above-mentioned physical device may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0137] Those skilled in the art will appreciate that the above-mentioned physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different arrangements of components.

[0138] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the above-mentioned physical device, and supports the operation of the information processing program and other software and / or programs. The network communication module is used to realize the communication between the components inside the storage medium, and the communication with other hardware and software in the information processing physical device.

[0139] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform, or by hardware.

[0140] The embodiment of the present invention adopts a physics-driven critic function instead of a deep neural network, which can solve the adaptability and safety problems of the deep reinforcement learning algorithm in the power system optimization process. That is, this method can ensure that the optimization results are more in line with the actual operation of the power system, thereby providing a more reliable decision-making basis and ensuring the safety and reliability of control actions.

[0141] Those skilled in the art will appreciate that the accompanying drawings are only schematic diagrams of a preferred implementation scenario, and the modules or processes in the accompanying drawings are not necessarily necessary for implementing the present application. Those skilled in the art will appreciate that the modules in the devices in the implementation scenario can be distributed in the devices of the implementation scenario according to the description of the implementation scenario, or can be changed accordingly and located in one or more devices different from the present implementation scenario. The modules of the above-mentioned implementation scenario can be combined into one module, or can be further split into multiple submodules.

[0142] The above serial numbers of this application are only for description and do not represent the advantages and disadvantages of the implementation scenarios. The above disclosure is only a few specific implementation scenarios of this application, but this application is not limited to them, and any changes that can be thought of by technicians in this field should fall within the scope of protection of this application.

Claims

1. A real-time optimal power flow method for deep reinforcement learning guided by a holomorphic embedding method, characterized in that: include: Define the system state variables and strategy action variables of the power system respectively; Based on the system state variables and the strategy action variables, construct a strategy model corresponding to the power system, wherein the experience buffer corresponding to the strategy model only stores the system state; Converting the nonlinear power flow equation of the power system into a holomorphic function to construct a critic function corresponding to the power system; The critic function and the strategy model are updated and iterated alternately, and when a preset convergence condition is met, a final updated strategy model is output, and an optimal strategy action corresponding to the power system is determined; Wherein, the system state variables and strategy action variables of the power system are defined respectively, including: Based on the active power variables, reactive power variables and wind power generation power output variables corresponding to each bus in the power system, and the current power output variables of each generator, define the system state variables of the power system; Based on the active power setting value variables of each generator except the balancing machine in the power system, the voltage setting value variables of the current collecting line and the slack bus in each bus, and the reactive power setting value variables of the reactive compensation device, define the strategy action variables of the power system; The constructing a strategy model corresponding to the power system based on the system state variables and the strategy action variables includes: Based on the system state variables and the strategy action variables, a strategy model corresponding to the power system is constructed using a preset neural network algorithm; The step of converting the nonlinear power flow equation of the power system into a holomorphic function to construct a critic function corresponding to the power system includes: Converting the nonlinear power flow equation into an embedded system of complex voltage variables and reactive power variables about each bus, wherein the complex voltage variables and the reactive power variables can be expressed as a holomorphic function about an embedded coefficient; Based on the embedded system, a commentator function corresponding to the power system is constructed.

2. The method according to claim 1, characterized in that The method further comprises: Determine a production cost nonlinear function and a default function corresponding to the power system, wherein the independent variable of the production cost nonlinear function is the active power setting value of each generator in the power system except the balancing machine; Based on the production cost nonlinear function and the default function, a reward value function corresponding to the power system is defined.

3. The method according to claim 1, characterized in that The method further comprises: Performing vectorization processing on the critic function to obtain a vectorized critic function; The critic function and the strategy model are updated and iterated alternately, and when a preset convergence condition is met, a final updated strategy model is output, and an optimal strategy action corresponding to the power system is determined, including: The vectorized critic function and the strategy model are updated and iterated alternately, and when a preset convergence condition is met, the final updated strategy model is output, and the optimal strategy action corresponding to the power system is determined.

4. The method according to claim 2, characterized in that: The critic function and the strategy model are updated and iterated alternately, and when a preset convergence condition is met, a final updated strategy model is output, and an optimal strategy action corresponding to the power system is determined, including: Extracting a system state corresponding to the power system from the experience buffer; Inputting the system state corresponding to the power system into the strategy model to perform action prediction, and obtaining the predicted strategy action corresponding to the power system; Based on the system state and the predicted strategy action, the critic function and the strategy model are updated and iterated alternately. When a preset convergence condition is met, the final updated strategy model is output, and the optimal strategy action corresponding to the power system is determined.

5. The method according to claim 1, characterized in that The step of constructing a commentator function corresponding to the power system based on the embedded system includes: Expand the holomorphic functions corresponding to the complex voltage variable and the reactive power variable into Maclaurin series, and bring them into the embedded system to obtain an expanded embedded system, wherein the expanded embedded system contains target parameters; The expanded embedded system is sequentially subjected to coefficient equationalization and imaginary and real part separation processing to obtain a complex equation corresponding to the power system; According to the complex equation and the reward value function, a critic function corresponding to the power system is determined.

6. The method according to claim 4, characterized in that The method of performing alternate updating and iteration on the critic function and the strategy model based on the system state and the predicted strategy action, outputting a final updated strategy model when a preset convergence condition is met, and determining an optimal strategy action corresponding to the power system includes: Based on the system state and the prediction strategy action, the critic function is updated to obtain the critic function after this round of update; Deriving the critic function after the current round of update to obtain a reference strategy action corresponding to the power system; According to the reference strategy action and the predicted strategy action, the strategy model is updated to obtain a strategy model after this round of update; Re-determining the system state of the power system under the prediction strategy action, and re-determining the prediction strategy action corresponding to the power system based on the re-determined system state and the strategy model after the current round of update; Based on the re-determined system state and prediction strategy action, the critic function and strategy model are updated repeatedly and alternately. When the preset conditions are met, the update iteration is stopped and the trained prediction strategy model is output; According to the trained prediction strategy model, the optimal strategy action corresponding to the power system is determined.

7. The method according to claim 6, characterized in that The updating of the critic function based on the system state and the prediction strategy action to obtain the critic function after this round of update includes: Based on the system state and the prediction strategy action, solving the complex equation corresponding to the commentator function to obtain the complex power corresponding to each bus; Based on the complex power and the default function, respectively calculating the default vector of the power system and the active power setting value of each generator; Substituting the default vector and the active power setting value into the reward value function to calculate the reward value corresponding to the power system; Based on the reward value, the updated commentator function of this round is determined.

8. The method according to claim 7, characterized in that The method of solving the complex equation corresponding to the commentator function based on the system state and the prediction strategy action to obtain the complex power corresponding to each bus includes: Initializing a target parameter in the complex equation to obtain an initial value corresponding to the target parameter; Substituting the initial value corresponding to the target parameter, the system state and the prediction strategy action into the complex equation for solving, to obtain the initial complex voltage, initial reactive power and initial complex power corresponding to each bus; If the initial complex voltage does not satisfy the voltage setting value limitation condition of the current collection line, or the initial reactive power does not satisfy the reactive power limitation condition of the current collection line, or the initial complex power does not satisfy the complex power limitation condition of the distribution equipment bus, then the initial value corresponding to the target parameter is updated and iterated, and the target parameter value after this round of update is re-substituted into the complex equation for solving; Repeat the iterative process of updating the target parameter value and the process of solving the complex equation until the solved complex voltage meets the voltage setting value limitation condition of the current collection line, the solved reactive power meets the reactive power limitation condition of the current collection line, and the solved complex power meets the complex power limitation condition of the distribution equipment bus, then output the complex power.

9. A real-time optimal power flow device for guiding deep reinforcement learning using a holomorphic embedding method, characterized in that: include: A definition unit, used to define system state variables and strategy action variables of the power system respectively; A first construction unit is used to construct a strategy model corresponding to the power system based on the system state variable and the strategy action variable, wherein the experience buffer corresponding to the strategy model only stores the system state; A second construction unit is used to convert the nonlinear power flow equation of the power system into a holomorphic function to construct a critic function corresponding to the power system; An updating unit, used for alternately updating and iterating the critic function and the strategy model, outputting a final updated strategy model when a preset convergence condition is met, and determining an optimal strategy action corresponding to the power system; The definition unit is specifically used to define the system state variables of the power system based on the active power variables, reactive power variables and wind power output variables corresponding to each bus in the power system, and the current power output variables of each generator; based on the active power setting value variables of each generator except the balancing machine in the power system, the voltage setting value variables of the current collection line and the slack bus in each bus, and the reactive power setting value variables of the reactive compensation equipment, define the strategy action variables of the power system; The first construction unit is specifically configured to construct a strategy model corresponding to the power system using a preset neural network algorithm based on the system state variables and the strategy action variables; The second construction unit is specifically used to convert the nonlinear power flow equation into an embedded system of complex voltage variables and reactive power variables about each bus, wherein the complex voltage variables and the reactive power variables can be expressed as holomorphic functions about embedded coefficients; based on the embedded system, a commentator function corresponding to the power system is constructed.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

11. An electronic device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

Images