A method and apparatus for obtaining dispatch decisions in a renewable energy power system

Abstract

The application discloses a kind of renewable energy power system scheduling decision acquisition method and device, belong to power system control technical field, the method includes: by the constraint Markov process is nested into Lagrange-depth deterministic policy gradient algorithm framework;Further, initial decision network is reinforced learning training under the Lagrange-depth deterministic policy gradient algorithm framework;The implicit connection between the system state containing uncertainty information and actual decision is learned using the powerful fitting ability of neural network, and finally the current system state in the constraint Markov process corresponding to the renewable energy power system is input into the target decision network to obtain actual scheduling decision.This application considers the constraint in the process of power system scheduling state transition, realizes the purpose that agent guarantees action economy and safety at the same time when deciding and the efficient solution of scheduling plan.

Description

Technical Field

[0001] This invention belongs to the field of power system control technology, and more specifically, relates to a method and apparatus for obtaining dispatch decisions for a renewable energy power system. Background Technology

[0002] In recent years, renewable energy has been introduced into the power grid to address climate change and reduce dependence on fossil fuels. The volatility and randomness of renewable energy pose new challenges to the reliability and economical operation of power systems, and place higher demands on the solution of day-ahead dispatch schemes.

[0003] Traditional mathematical methods rely on constructing scene sets to describe uncertainty; however, the computational difficulty increases rapidly with the scale of the scene. Furthermore, these methods ignore the historical similarity of scheduling problems, consuming significant computational resources with each solution attempt. Deep reinforcement learning combines the fitting ability of deep learning with the decision-making ability of reinforcement learning, offering a significant advantage in solving optimization problems with uncertainty. This type of method avoids precisely modeling uncertainties, shifting the computational burden from online to offline solutions.

[0004] A well-trained agent can make decisions quickly and does not need to resolve the optimization problem when boundary conditions change. However, deep reinforcement learning algorithms typically update policies during training with an emphasis on economic optimization, but the actions taken by the agent are difficult to adapt to various scenarios, which may lead to power system safety issues. Therefore, existing methods struggle to improve solution efficiency while ensuring decision security. Summary of the Invention

[0005] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a method and apparatus for obtaining dispatch decisions of renewable energy power systems. Its purpose is to solve the technical problem that existing methods are unable to improve the solution efficiency while ensuring the security of the decision.

[0006] To achieve the above objectives, according to one aspect of the present invention, a method for obtaining dispatch decisions for a renewable energy power system is provided, comprising:

[0007] S1: The scheduling model corresponding to the minimum system operating cost of the renewable energy power system is reconstructed as a constrained Markov process;

[0008] S2: The constrained Markov process is nested within the framework of the Lagrange-deep deterministic policy gradient algorithm;

[0009] S3: The initial decision network is trained by reinforcement learning under the framework of the Lagrange-deep deterministic policy gradient algorithm to obtain the target decision network; the initial decision network includes the following sequentially connected: actor network, evaluation network and cost evaluation network;

[0010] S4: Input the current system state in the constrained Markov process corresponding to the renewable energy power system into the target decision network to obtain the actual scheduling decision.

[0011] In one embodiment, the actor network includes an encoder with a first type of domain knowledge embedding; the encoder includes a graph attention network and a graph convolutional network guided by the first type of domain knowledge; the first type of domain knowledge includes: a linearized AC power flow equation.

[0012] In one embodiment, the graph convolutional network is represented as:

[0013] ;;

[0014] in, Let the node degree matrix be... Let be the self-conductance matrix of the node. Let be the self-susceptance matrix of the node. Let be the mutual conductance matrix of the nodes. Let be the mutual susceptance matrix of the nodes. The active power matrix of the nodes. The active power matrix of the nodes. The active power characteristics of the surrounding area of ​​the node are aggregated. , Aggregate reactive power characteristics around the node. , and The graph convolutional network guided by the power flow equation The output of the layer, , , and The graph convolutional network guided by the power flow equation The trainable parameters of the layer, This represents the Schul product.

[0015] In one embodiment, the step S3 is preceded by:

[0016] The original decision network is pre-trained using a second type of domain knowledge; the second type of domain knowledge includes optimal scheduling decisions based on the actual wind power curve as the boundary condition; the original decision network has the same network structure as the initial decision network, but the network parameters are different.

[0017] The loss function for the actor network pre-training is: ;

[0018] in, The parameter is The policy network, For the loss function of the actor network pre-training, and For the sample exist The target action and state corresponding to each moment. The number of samples.

[0019] In one embodiment, the state in the constrained Markov process Represented as Action space The constraints are: ;

[0020] in, and for The status and dispatching actions of the power system during different time periods. and For the system in time period Network layer characteristics and device layer characteristics, and For generator During the period Active and reactive power, and For energy storage During the period The charging and discharging power, To update the agent's policy, For cumulative rewards, For the first The constraints that need to be satisfied, A set of constraints.

[0021] In one embodiment, S2 includes:

[0022] The constrained Markov process is reconstructed into an augmented Lagrangian form:

[0023] ;

[0024] Auxiliary cost function Calculation method:

[0025] ;

[0026] ;

[0027] in, Calculate the cost of the current strategy. To augment the action-value function, This indicates the degree to which the constraints of the current strategy are violated. For Lagrange multipliers, and For the updated policy network and Lagrange multipliers, , and It is a collection of generators, energy storage, and nodes. and For generator Uphill / downhill power, For energy storage During the period State of charge ( ), and For energy storage of Upper and lower limits, and These are the slack variables used to ensure the power balance constraints at system nodes. and For generator During the period The upper and lower limits of permissible output, and For time period The system backup requirements For nodes During the period The load demand, For nodes During the period voltage, and For nodes The upper and lower limits of voltage, For the node To the node During the time period on the line The meritorious trend, and For the node To the node The upper and lower limits of the current flow along the line.

[0028] In one embodiment, the loss function for reinforcement training includes:

[0029] Loss function for actor network reinforcement learning training: ;

[0030] Evaluate the training loss function of a network reinforcement learning system: ;

[0031] Cost-effective network reinforcement learning training loss function: ;

[0032] in, To evaluate the network's loss function, and The parameter is The evaluation network and its corresponding parameters are: The target network, The parameter is Target actor network The loss function of the cost evaluation network, and The parameter is The cost evaluation network and its corresponding parameters are The target network, , To evaluate the target value of the network and cost evaluation network, , , Let be the loss function of the actor network.

[0033] According to another aspect of the present invention, a dispatch decision acquisition device for a renewable energy power system is provided, comprising:

[0034] The reconfiguration module is used to reconfigure the scheduling model corresponding to the minimum system operating cost of the renewable energy power system into a constrained Markov process.

[0035] Nested modules are used to nest the constrained Markov process within the framework of the Lagrange-deep deterministic policy gradient algorithm.

[0036] The learning module is used to train the initial decision network through reinforcement learning within the framework of the Lagrange-deep deterministic policy gradient algorithm to obtain the target decision network; the initial decision network includes, in sequence, an actor network, an evaluation network, and a cost evaluation network.

[0037] The input module is used to input the current system state of the constrained Markov process corresponding to the renewable energy power system into the target decision network to obtain the actual scheduling decision.

[0038] According to another aspect of the invention, a renewable energy power system is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method of the claim.

[0039] According to another aspect of the invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method.

[0040] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0041] (1) This invention provides a method for obtaining dispatch decisions for a renewable energy power system. The method involves nesting the constrained Markov process within a Lagrange-deep deterministic policy gradient algorithm framework. Further, the initial decision network is trained using reinforcement learning within this framework. The powerful fitting ability of neural networks is utilized to learn the implicit relationship between the system state containing uncertain information and the actual decision. Finally, the current system state in the constrained Markov process corresponding to the renewable energy power system is input into the target decision network to obtain the actual dispatch decision. This application considers the constraints in the power system dispatch state transition process, achieving the goal of ensuring both the economy and safety of actions during decision-making, as well as the efficient solution of the dispatch plan.

[0042] (2) The actor network in this scheme includes an encoder with the first type of domain knowledge embedding; the encoder includes a graph attention network and a graph convolutional network guided by the first type of domain knowledge; this design takes into account the different physical information reflected by the features at different levels of the power system, realizes the separate representation and encoding of different types of features, and enables each feature to focus more on the information it wants to express through the processing of different types of networks.

[0043] (3) The graph convolutional network described in this scheme is represented as: This design takes into account the actual physical rules in power system dispatching, provides guidance for the decision-making process of intelligent agents, and helps intelligent agents obtain better strategies.

[0044] (4) The S3 section of this scheme also includes: pre-training the original decision network using the second type of domain knowledge; the loss function for pre-training the actor network: This design takes into account historical cases and expert knowledge, enabling the agent to obtain a reasonable initial strategy and achieve faster convergence during the training process.

[0045] (5) The state in the constrained Markov process described in this scheme Represented as This design takes into account the constraints that the agent needs to satisfy when making decisions, and achieves the goal of maximizing cumulative rewards while ensuring that actions meet the constraints.

[0046] (6) In this scheme, the constrained Markov process is reconstructed into an augmented Lagrange form: This design takes into account the auxiliary cost function to evaluate the violation of constraints during training, thus quantifying the degree of constraint violation.

[0047] (7) The evaluation loss function for network reinforcement learning training in this scheme: Cost evaluation network reinforcement learning training loss function: This design takes into account both the economy and security of decision-making, enabling higher quality training of the agent. Attached Figure Description

[0048] Figure 1 A flowchart of the method for obtaining dispatch decisions for a renewable energy power system provided in Embodiment 1 of the present invention;

[0049] Figure 2 The topology and node information of the IEEE-39 node system provided in Embodiment 1 of the present invention;

[0050] Figure 3 This is a schematic diagram of the pre-training results of all networks in the IEEE-39 node system provided in Embodiment 1 of the present invention;

[0051] Figure 4 This is a graph showing the reward changes during the training of different types of reinforcement learning algorithms in the IEEE-39 node system provided in Embodiment 1 of the present invention.

[0052] Figure 5 This is a schematic diagram illustrating the scenario and uncertainty information of stochastic optimization of the IEEE-39 node system in a typical day, as provided in Embodiment 1 of the present invention.

[0053] Figure 6 This is a comparison chart of the proposed method for IEEE-39 nodes provided in Embodiment 1 of the present invention and the results of random optimization scheduling;

[0054] Figure 7 This is a comparison diagram of the proposed method for the IEEE-39 node provided in Embodiment 1 of the present invention and the results of stochastic optimized power flow. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0056] Example 1

[0057] This embodiment provides a method for obtaining dispatch decisions for a renewable energy power system, such as... Figure 1As shown, the process includes: S1: Reconstructing the scheduling model corresponding to the minimum system operating cost of the renewable energy power system into a constrained Markov process; S2: Nesting the constrained Markov process into the framework of the Lagrange-deep deterministic policy gradient algorithm; S3: Training the initial decision network through reinforcement learning within the framework of the Lagrange-deep deterministic policy gradient algorithm to obtain the target decision network; the initial decision network includes, in sequence, an actor network, an evaluation network, and a cost evaluation network; S4: Inputting the current system state from the constrained Markov process corresponding to the renewable energy power system into the target decision network to obtain the actual scheduling decision.

[0058] This embodiment uses the IEEE-39-node system model to demonstrate the solution capability of the proposed domain knowledge-embedded renewable energy power system scheduling solution method. The system parameters are shown in Table 1.

[0059] The topology and node information of the IEEE-39 node system are as follows: Figure 2 As shown, the system maintains a renewable energy penetration rate of around 20%, with wind curtailment costs and load shedding costs of $1,000 / MW and $100,000 / MW, respectively.

[0060] The renewable energy power system dispatch problem is modeled as follows:

[0061] The objective function is to achieve the most economical system operation:

[0062] ;

[0063] in, For time period Total cost and For generator During the period The active power and start-stop costs, and For energy storage During the period The charging and discharging power, and Cost per unit of charge / discharge power and For renewable energy During the period The amount discarded and the corresponding cost per unit power, and For nodes During the period Reduced active and reactive power, and It is used to calculate the cost of generator fuel and load reduction. , , , and It is a collection of scheduling intervals, generators, energy storage, renewable energy sources, and nodes.

[0064] Generator power upper and lower limit constraints: ;in, , and For generator During the period The status of the system's operation, startup status, and shutdown status. For generator During the period The reactive power.

[0065] Generator ramping constraints: ;in, / and / For generator The upper and lower limits of active power and reactive power, and For generator Uphill / downhill power.

[0066] Minimum start-stop constraints for generators: ;in, and For generator The minimum start-stop time.

[0067] Renewable energy constraints: ;in, and For renewable energy During the period Actual and predicted power values.

[0068] Energy storage charge and discharge constraints: ;in, and For energy storage During the period The charging and discharging states, and For energy storage The upper limit of charging and discharging power.

[0069] Energy storage constraint: ;in, For energy storage During the period State of charge ( ), and For energy storage The charge and discharge efficiency, and For energy storage of Upper and lower limits, For the scheduling period, This represents the energy storage capacity. The first formula represents the energy storage capacity. State update constraints, the second formula represents energy storage Upper and lower limit constraints.

[0070] System node power balance constraints: ;in , and For the node To the node During the time period on the line The trend of meritorious and ineffective actions and For nodes During the period The active and reactive load demand, , , , and For nodes During the period The active and reactive power of the generator, the power of renewable energy, and the charging and discharging power of energy storage.

[0071] System backup constraints: ;in, and For generator During the period The upper and lower limits of permissible output, and For time period The system backup requirements and A function to select the minimum and maximum values.

[0072] Linearized AC power flow constraints: ;in, , and For nodes and nodes The conductivity and susceptance between them and For nodes During the period The voltage and power angle.

[0073] Voltage and power flow upper and lower limit constraints: ;in, and For nodes The upper and lower limits of voltage, and For the node To the node The upper and lower limits of the current flow along the line.

[0074] In one embodiment, the actor network includes an encoder with a first type of domain knowledge embedding; the encoder includes a graph attention network and a graph convolutional network guided by the first type of domain knowledge; the first type of domain knowledge includes: a linearized AC power flow equation.

[0075] To make the agent's decision-making process more consistent with the actual physical system, a physical knowledge embedding encoder was derived as part of the actor network to complete the embedding of physical knowledge. The derivation process is as follows:

[0076] 1) Graph Convolutional Network Guided by Power Flow Equations

[0077] The original AC power flow equations are shown below.

[0078]

[0079] The original AC power flow is linearized according to the following approximate equation.

[0080]

[0081] The node power injection equation can be obtained as follows.

[0082]

[0083]

[0084] Combining the above two equations, we can... and Treating it as a variable, we obtain the following expression.

[0085]

[0086] in,

[0087]

[0088] From the perspective of feature aggregation in graph convolution, the above formula only aggregates the features of connected nodes. The characteristics of the nodes themselves should also be considered. Based on the idea of ​​degree matrix normalization, the operator of graph convolution network guided by the power flow equation can be expressed as follows.

[0089]

[0090] in,

[0091]

[0092] The corresponding matrix form is as follows.

[0093]

[0094] in,

[0095]

[0096] By reorganizing it into the form of neural network forward propagation, we obtain a graph convolutional network guided by the power flow equation.

[0097]

[0098] in, For nodes The number of connected nodes, and The graph convolutional network guided by the power flow equation The output of the layer, , , and The graph convolutional network guided by the power flow equation The trainable parameters of the layer, This represents the Schul product.

[0099] 2) Graph Attention Network

[0100] For nodes Calculate the similarity coefficient between adjacent nodes. To reflect the nodes and The correlation between them.

[0101]

[0102] in, This represents the operation of concatenating vectors. It is a forward computing network.

[0103] Used to calculate attention coefficient .

[0104]

[0105] in, The LeakyReLU activation function is used. It is an exponential function. For nodes The set of connected nodes.

[0106] based on The aggregated features are obtained by weighted summation of the features. .

[0107]

[0108] in, It is a non-linear activation function.

[0109] 3) Encoders embedding physical knowledge

[0110] We use graph convolutional networks and graph attention networks guided by power flow equations to process them respectively. and The processed power system features are obtained by concatenating the vectors. .

[0111]

[0112] in, and This represents the state convolution process of a graph convolutional network and a graph attention network guided by power flow equations. The high-dimensional features of the power system obtained through processing with different types of networks and vector concatenation can better reflect the internal physical information, making each feature more focused on the information it wants to express; this process can be viewed as encoding. Therefore, the graph convolutional network and graph attention network guided by power flow equations combine to form an encoder for embedding physical knowledge, which will act as part of the actor network to complete the embedding of physical knowledge.

[0113] In one embodiment, before S3, the method further includes: pre-training the original decision network using a second type of domain knowledge; the second type of domain knowledge includes optimal scheduling decisions based on the actual wind power curve as the boundary condition; the original decision network has the same network structure as the initial decision network, but different network parameters; the loss function for pre-training the actor network is: ;in, The parameter is The policy network, For the loss function of the actor network pre-training, and For the sample exist The target action and state corresponding to each moment. The number of samples.

[0114] To enable the agent to converge to a better final policy more quickly during reinforcement learning training, an expert knowledge base was constructed based on historical cases and expert knowledge to pre-train all networks, completing the embedding of expert knowledge. The process is as follows:

[0115] Expert knowledge refers to optimal dispatch decisions based on actual wind power curves as boundary conditions. It can be used to solve optimization problems by querying historical cases and leveraging expert knowledge. Based on this, the original problem can be simplified to reduce computation time. According to expert knowledge, the minimum number of generators that need to be started each day can be determined, and in most cases, the DC power flow can meet safety requirements. The original model can be simplified by adding the following constraints.

[0116]

[0117] in, This determines the minimum number of units that need to be activated during the scheduling cycle. By replacing the original linearized AC power flow model with DC power flow constraints, the original problem is transformed into a simplified deterministic optimization problem, which significantly reduces the time required to acquire the expert knowledge base.

[0118] First, the actor network is pre-trained. Based on historical data and the scheduling results obtained from the simplified model, the power system state is constructed. and actions Combine the above data into tuples. And store the tuple into the initial expert knowledge dataset. The input for actor network pre-training is... , tag as We select mean squared error (MSE) as the metric to construct a loss function. As shown below.

[0119]

[0120] Secondly, the evaluation network and cost evaluation network need to be pre-trained. Without pre-training, their output values ​​will be highly inaccurate. To pre-train the evaluation network and cost evaluation network, In Will be the initial strategy Update and verify the linearized AC power flow constraints. Then calculate the corresponding... , Based on this, Expandable to Correspondingly The content in the previous section has also been expanded. The mean squared error (MSE) is still selected as the metric for constructing the loss function.

[0121] The input for evaluating network pre-training is and , tag as loss function As shown below.

[0122]

[0123]

[0124] in, and The parameter is The evaluation network and its corresponding parameters are: The target network, The parameter is The target actor network.

[0125] The input for pre-training the cost evaluation network is and , tag as loss function As shown below.

[0126]

[0127]

[0128] in, and The parameter is The cost evaluation network and its corresponding parameters are The target network, The target value for the cost evaluation network.

[0129] Pre-training of all networks within the Lagrange-deep deterministic policy gradient algorithm framework using an expert knowledge base and loss function completes the embedding of expert knowledge.

[0130] In one embodiment, the state of the constrained Markov process is... Represented as Action space The constraints are: ;in, and for The status and dispatching actions of the power system during different time periods. and For the system in time period Network layer characteristics and device layer characteristics, and For generator During the period Active and reactive power, and For energy storage During the period The charging and discharging power, To update the agent's policy, For cumulative rewards, For the first The constraints that need to be satisfied, A set of constraints.

[0131] Specifically, the original scheduling problem can be constructed as a constrained Markov process, with the following form:

[0132] 1) State Space

[0133] A power system can be viewed as consisting of a power system network and various electrical devices. To reflect the physical connections between these components and to align with the network structure of a power system, graph data is chosen to construct state characteristics, and the characteristics of power system dispatching are divided into network-level and device-level features. Each bus in the power system possesses data such as active load, reactive load, voltage value, and power angle; these features are categorized as network-level features. These are highly relevant to the power flow and security of the power grid. Nodes contain various electrical devices, such as generators, energy storage, and renewable energy stations. Depending on the different load demands of the power system, they will have different operating states, including unit availability, generator active and reactive power output, reactive power output, ES charging and discharging power, SOC, predicted renewable energy output power, and corresponding uncertainties. These characteristics can be categorized as equipment-level characteristics. They can determine the state transitions of the power system:

[0134] .

[0135] Power system during time period The state is described by the following two feature matrices:

[0136] ;

[0137] in, The number of system nodes, and For the system in time period Network-level and device-level characteristics. Observed state. It can be represented as .

[0138] 2) Action Space

[0139] After observing and extracting environmental features, the agent will generate an action based on the current policy. It consists of the outputs of various dispatching devices: the active and reactive power of the generator, and the charging and discharging power of the energy storage. .

[0140] 3) Reward function

[0141] The goal of scheduling is to minimize the system cost, and the reward function is... It guides the agent to make more economical behaviors, and the evaluation of actions during training is based on real wind force curves.

[0142] 4) Constrained strategies

[0143] Agent's strategy Not only to maximize future cumulative rewards Furthermore, it is also necessary to satisfy the constraints of the constrained Markov process.

[0144] ;

[0145] in, For the first The constraints that need to be satisfied, To update the agent's policy, A set of constraints.

[0146] In one embodiment, S2 includes: the constructed constrained Markov process can be reformulated as an augmented Lagrange form to obtain the final solution by alternately updating the parameters of the policy network and the Lagrange multipliers.

[0147] ;

[0148] The parameter is The policy network, Calculate the cost of the current strategy. To augment the action-value function, This indicates the degree to which the constraints of the current strategy are violated. These are Lagrange multipliers. , The learning rate of the policy network. For the update step size of the Lagrange multipliers, .

[0149] To quantify violations of different types of constraints, an auxiliary cost function is introduced. This function is used for the constraints mentioned earlier, including generator power upper and lower limits, generator minimum start / stop constraints, renewable energy constraints, energy storage charge / discharge constraints, and energy storage... State update constraints can be determined or satisfied through the design of the actor network and state transitions.

[0150] For generator ramping constraints and energy storage Upper and lower limit constraints, system node power balance constraints, and system reserve constraints need to be calculated using corresponding auxiliary cost functions.

[0151] ;

[0152] After assessing the above constraints, power flow calculations are performed based on the current scheduling plan, and the following auxiliary costs are calculated: ;in, Used to describe the degree of constraint violation. and These are the slack variables used to ensure the power balance constraints of the system nodes.

[0153] The Lagrange-Deep Deterministic Policy Gradient Algorithm framework includes an actor network (i.e., a policy network), an evaluation network, and a cost evaluation network. The actor network makes decisions by observing the current state of the system, while the evaluation and cost evaluation networks assess the actions taken by the actor network and the current state of the environment, evaluating whether the actions and constraints are violated.

[0154] In one embodiment, relying solely on expert knowledge datasets may not enable the agent to learn comprehensive scheduling policies. The purpose of pre-training is to enable the agent to learn general scheduling rules, and training still needs to be performed based on the framework of the Lagrange-deep deterministic policy gradient algorithm to solve the constrained Markov process that has been reconstructed into an augmented Lagrange form.

[0155] The evaluation network is used to approximate the action-value function, and its parameter update process is shown below.

[0156] ;

[0157] ;

[0158] , ;

[0159] in, As a discount factor, To evaluate the learning rate of the network, This refers to the soft update rate.

[0160] The cost evaluation network is used to approximate the auxiliary cost function, and its parameter update process is shown below:

[0161] ;

[0162] ;

[0163] , ;

[0164] in, The learning rate of the network is used to evaluate its cost.

[0165] The goal of the actor network is to make actions that minimize the objective function, and its parameters should be updated in the direction of higher rewards and fewer constraint violations. Therefore, the update process of the actor network needs to consider both the evaluation network and the cost evaluation network simultaneously.

[0166] ;

[0167] ;

[0168] .

[0169] By continuously interacting with the environment, network parameters are updated and a reasonable scheduling strategy is obtained.

[0170] The simulation results of the method provided by this invention are described below:

[0171] Figure 3 Analysis of the pre-training results shows that the network parameters in the proposed method converge. The actor network's pre-training is label-based and converges quickly with fewer epochs. The other two networks, however, use nested functions to guide updates, requiring more training time. This step embeds expert knowledge, giving the agent a reasonable initial strategy and reducing unnecessary exploration during reinforcement learning training, thus improving training efficiency. The trained agent is obtained, and the changes in reward during training are recorded. To demonstrate the superiority of the proposed method, ablation experiments are conducted for comparison. The settings and differences between the other three algorithms and the algorithm of this invention are shown below.

[0172] The methods proposed in this invention are: a Lagrange-Deep Deterministic Policy Gradient Algorithm embedding domain knowledge, denoted as DK-CRL. Method 1: The basic Lagrange-Deep Deterministic Policy Gradient Algorithm, denoted as CRL. Method 2: A Deep Deterministic Policy Gradient Algorithm embedding domain knowledge, denoted as DK-DDPG. Method 3: A basic Deep Deterministic Policy Gradient Algorithm, denoted as DDPG.

[0173] The other three algorithms were also used to solve the renewable energy power system dispatch problem. The reward changes of each algorithm during training were recorded, and the results are as follows: Figure 4As shown in the diagram. Analysis reveals that the proposed method, DK-CRL, exhibits the best and most stable training performance, accumulating the most rewards. Due to the embedding of domain knowledge, DK-DDPG and DK-CRL have higher initial rewards than DDPG and CRL, enabling them to converge to better policies more quickly. Although a better initial policy results in a higher initial reward for DK-DDPG than for CRL, the introduction of a cost evaluation network better handles constraints, allowing CRL to obtain better policies and more rewards. Therefore, reinforcement learning algorithms that consider constraints yield higher rewards than ordinary reinforcement learning algorithms. Simply embedding prior expert knowledge is insufficient to fully equip an agent with the ability to handle uncertain scheduling; there is still significant room for improvement in reinforcement learning training after pre-training. Since different scheduling days require different optimal solutions, the stable reward curve exhibits fluctuations. In conclusion, compared to the other three algorithms, DK-CRL achieves the best final policy by embedding domain knowledge while considering scheduling constraints.

[0174] The solution capability of the method proposed in this invention will be demonstrated in terms of both solution speed and solution quality. The scheduling schemes obtained by different algorithms will be evaluated in terms of both economic scheduling and real-time scheduling.

[0175] First, the economic scheduling of the scheduling scheme is evaluated. This is done based on uncertainty information. For each scenario, an economic scheduling problem with the following objective will be solved:

[0176] ;

[0177] The average cost across all scenarios is then summed with the start-stop costs corresponding to the scheduling scheme to obtain the final expected cost as the final result. ;Pick With a value of 500, the economic scheduling evaluation results of different algorithms under typical daytime conditions are shown in Table 2.

[0178] In addition to the four algorithms mentioned above, stochastic optimization and robust optimization algorithms, denoted as SO and RO respectively, are introduced for comparison. The computation time in the table refers to the time spent obtaining the scheduling plan, and the renewable energy absorption rate is the average of the economic scheduling results across all scenarios. It can be seen that the proposed method, DK-CRL, can quickly obtain an economical scheduling scheme, exhibiting low operating costs and high wind power absorption in the test scenario. The results of the other three reinforcement learning algorithms correspond to the previous training results, and the quality of the solutions gradually deteriorates. However, their solution times are very short, approximately 5 seconds. For traditional mathematical algorithms, SO and RO require iterative solutions considering various scenarios, resulting in a significant time consumption. Since SO's scenario set is difficult to cover all possible wind power curves, the resulting scheduling scheme has poor economic efficiency and renewable energy absorption rate. RO, on the other hand, requires ensuring that all possible scenarios in the uncertain set are absorbed, thus achieving a 100% absorption rate. This leads to a relatively conservative scheduling scheme and consequently, higher expected costs.

[0179] Secondly, the real-time scheduling of the scheduling scheme is evaluated. Based on the obtained scheduling scheme, the generator start-up and shutdown plan is determined first, and uncertainties are revealed time-by-time. Under typical conditions, real-time scheduling is performed using the DK-CRL method and the SO scheduling scheme proposed in this invention, respectively. The specific 20 SO scenarios and wind power uncertainty information in the SO scenario set are as follows: Figure 5 As shown, it can be seen that due to the difference between the prediction and the actual situation, the generated scene is difficult to accurately cover the real wind curve.

[0180] Figure 6 The outputs of each component in the scheduling results of the two methods are presented. The results show that DK-CRL can activate fewer generators and utilize sufficient energy storage to absorb wind energy during periods of low load demand and high wind speeds. When load increases and wind power decreases, it can utilize energy storage to fill the power gap caused by insufficient generator ramping capability. In contrast, SO's scheduling scheme will activate more generators, leading to wind curtailment during periods 1–3 and 20–24.

[0181] Figure 7 The power flow of each branch at each time period is shown in the scheduling results of the two methods. The results indicate that DK-CRL can limit the power flow within a safe range and is safer than SO (Self-Solving). The power flow of DK-CRL does not reach the upper or lower limits of the branches, and some SOs appear at the boundaries of the power flow constraints. Figure 2 The topology of the IEEE-39 node system shows that when there is a generator on a branch, the power flow is relatively large and it is easier to reach the boundary.

[0182] To further demonstrate the real-time scheduling capabilities of the proposed method, 100 days were randomly selected for real-time scheduling. This represents the difference between the calculated cost and the optimal cost. The optimal cost is the global optimal solution based on the actual wind power output curve. The average numerical results are shown in Table 3.

[0183] Clearly, the proposed method DK-CRL can find a solution closer to the optimal solution in a shorter time and absorb more wind power. For reinforcement learning algorithms, the trained agent can quickly obtain scheduling schemes, so the computation time is very short. DK-CRL has better solution speed and solution quality than other methods. DDPG, which does not consider constraints and pre-training, converges to a local optimum policy and has the worst solution performance. SO requires the solution to be feasible for all scenarios, thus requiring more computation time. RO can achieve complete wind power absorption in economic scheduling evaluation with uncertain sets, but it will produce wind curtailment in real-time scheduling evaluation. Both of these methods also ignore the relationship between prediction information and actual data, making it difficult to guarantee high-quality real-time scheduling schemes. In contrast, DK-CRL overcomes these problems and can quickly generate solutions with smaller... The scheduling scheme.

[0184] Example 2

[0185] This embodiment provides a device for obtaining dispatch decisions for a renewable energy power system, including: a reconstruction module, a nesting module, a learning module, and an input module. The reconstruction module is used to reconstruct the dispatch model corresponding to the minimum system operating cost of the renewable energy power system into a constrained Markov process. The nesting module is used to nest the constrained Markov process within the framework of a Lagrange-deep deterministic policy gradient algorithm. The learning module is used to train the initial decision network through reinforcement learning within the Lagrange-deep deterministic policy gradient algorithm framework to obtain a target decision network. The initial decision network includes, in sequence, an actor network, an evaluation network, and a cost evaluation network. The input module is used to input the current system state of the constrained Markov process corresponding to the renewable energy power system into the target decision network to obtain the actual dispatch decision.

[0186] Example 3

[0187] This embodiment provides a renewable energy power system, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method.

[0188] Example 4

[0189] This embodiment provides a computer-readable storage medium storing a computer program thereon, the steps of a method implemented when the computer program is executed by a processor.

[0190] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for obtaining scheduling decisions for a renewable energy power system, characterized in that, The method comprises the following steps: S1: reconstructing a scheduling model corresponding to minimum system operation cost of a renewable energy power system into a constrained Markov process; S2: embedding the constrained Markov process into a Lagrange-deep deterministic policy gradient algorithm framework; S3: performing reinforcement learning training on an initial decision network under the Lagrange-deep deterministic policy gradient algorithm framework to obtain a target decision network; the initial decision network comprises an actor network, a critic network and a cost evaluation network connected in sequence; S4: inputting a current system state in the constrained Markov process corresponding to the renewable energy power system into the target decision network to obtain an actual scheduling decision; the actor network comprises a first domain knowledge embedded encoder; the encoder comprises a graph attention network and a graph convolution network guided based on the first domain knowledge; the first domain knowledge comprises linearized alternating current flow equations; and the graph convolution network is represented as: ； in, Let the node degree matrix be... Let be the self-conductance matrix of the node. Let be the self-susceptance matrix of the node. Let be the mutual conductance matrix of the nodes. Let be the mutual susceptance matrix of the nodes. The active power matrix of the nodes. The active power matrix of the nodes. The active power characteristics are aggregated around the node. , Aggregate reactive power characteristics around the node. , and The graph convolutional network guided by the power flow equation The output of the layer, , , and The graph convolutional network guided by the power flow equation The trainable parameters of the layer, Represents the Schul product; the state in the constrained markov process is represented as , the action space , the constraint is: ; and is the state and dispatch action of the power system in the time period, and is the network-level features and the device-level features of the system in the time period , and is the active and reactive power of the generator in the time period , and is the charge and discharge power of the energy storage in the time period , is the policy of the updated agent, is the accumulated reward, is the constraint in the th constraint needs to be satisfied, is the set of constraints; the constraints include: generator power upper and lower limit constraints, generator ramping constraints, generator minimum start-stop constraints, renewable energy constraints, energy storage charge and discharge constraints, energy storage constraints, system node power balance constraints, system reserve constraints, linearized alternating current flow constraints, and voltage and flow upper and lower limit constraints.

2. The method of claim 1, wherein, The method further comprises the following steps before S3: pre-training an original decision network by using second domain knowledge; the second domain knowledge comprises an optimal scheduling decision with an actual wind power curve as a boundary condition; the original decision network has the same network structure as the initial decision network but different network parameters; The loss function of the actor network pre-training: ; wherein, represents a parameter for a policy network, a loss function for pre-training of the actor network, and is a sample corresponding to a target action and state at a time instant, is a number of samples.

3. The method of claim 1, wherein, S2 comprises: reconstructing the constrained Markov process into an augmented Lagrange form: ； Auxiliary cost function is calculated in the following way: ； ； wherein, calculating a cost corresponding to the current policy, for augmenting the action-value function, representing a constraint violation degree of the current policy, for Lagrange multipliers, and for the updated policy network and Lagrange multipliers, , and are sets of generators, storages and nodes, and are ramp-up / down power of generators , are state-of-charge (SoC) of storages in time period , , and are upper / lower limits of storages , , and are slack variables used to ensure system node power balance constraints, and are allowed output upper / lower limits of generators in time period , and are system reserve demand in time period , are load demand of node in time period , are voltage of node in time period , and are voltage upper / lower limits of node , are active power flow on line from node to node in time period , and are flow upper / lower limits on line from node to node .

4. The method of claim 1-3, wherein, the loss function of reinforcement training comprises: Actor network reinforcement learning training loss function: ; Evaluating network reinforcement learning training loss function: ; Cost evaluation network reinforcement learning training loss function: ; wherein, is a loss function of the evaluation network, and denotes the evaluation network with parameters and the target network with corresponding parameters , denotes the target actor network with parameters , is a loss function of the cost evaluation network, and denotes the cost evaluation network with parameters and the target network with corresponding parameters , , is the target value of the evaluation network and the cost evaluation network, , , is a loss function of the actor network.

5. A device for obtaining dispatch decisions for a renewable energy power system, characterized in that, A computer program product for performing the scheduling decision acquisition method of any one of claims 1-4, comprising: a reconstruction module configured to reconstruct a scheduling model corresponding to minimum system operation cost of a renewable energy power system into a constrained Markov process; a nesting module configured to embed the constrained Markov process into a Lagrange-deep deterministic policy gradient algorithm framework; a learning module configured to perform reinforcement learning training on an initial decision network under the Lagrange-deep deterministic policy gradient algorithm framework to obtain a target decision network; the initial decision network comprises an actor network, a critic network and a cost evaluation network connected in sequence; an input module configured to input a current system state in the constrained Markov process corresponding to the renewable energy power system into the target decision network to obtain an actual scheduling decision.

6. A renewable energy power system comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1-4.

7. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1-4.

Images