Power grid optimal scheduling method and device based on reinforcement learning, computer equipment, readable storage medium and program product

Abstract

The invention relates to a power grid optimal scheduling method and device based on reinforcement learning, computer equipment, a readable storage medium and a program product, and relates to the technical field of power scheduling. According to the invention, the solving speed and convergence stability of power grid dispatching can be improved. The method comprises the steps of determining a target, a target function and a constraint condition of optimal scheduling of a power grid; performing power grid optimization scheduling modeling based on an Actor-Critic structure according to the objective function and the constraint condition to obtain an initial scheduling model, performing Actor network and physical layer embedding design, Critic network design and physical constraint loss design on the initial scheduling model to obtain a to-be-trained scheduling model, performing preliminary training on the to-be-trained scheduling model to obtain a to-be-trained scheduling model, and performing training on the to-be-trained scheduling model to obtain a to-be-trained scheduling model; obtaining a preliminarily trained scheduling model; and performing iterative training and parameter updating on the preliminarily trained scheduling model by using a pre-constructed experience pool until the updated network parameters converge to obtain a trained target scheduling model, and performing optimal scheduling on the new energy power grid system according to the target scheduling model.

Description

Technical Field

[0001] This application relates to the field of power dispatching technology, and in particular to a power grid optimal dispatching method, apparatus, computer equipment, computer-readable storage medium, and computer program product based on reinforcement learning. Background Technology

[0002] Optimal power flow dispatch is a core issue in power grid system operation. Its goal is to minimize generation costs or optimize energy efficiency while satisfying voltage constraints and line power flow limitations. In traditional power grid systems, optimal power flow (OPF) is generally solved using analytical optimization methods, such as Newton's method, Lagrange multiplier method, and interior-point method.

[0003] However, the above methods mainly rely on static modeling and linearization approximation, which are difficult to adapt to the randomness, rapid change and strong coupling characteristics brought about by the current high proportion of renewable energy access. They have problems such as slow solution speed and low convergence stability, and cannot meet the requirements of real-time accurate solution. Summary of the Invention

[0004] Therefore, it is necessary to provide a power grid optimal scheduling method, device, computer equipment, computer-readable storage medium, and computer program product based on reinforcement learning to address the above-mentioned technical problems.

[0005] Firstly, this application provides a power grid optimal scheduling method based on reinforcement learning, including:

[0006] In response to dispatch instructions for the new energy power grid system, the objective of optimal power grid dispatch, the objective function corresponding to the objective, and the constraints that need to be satisfied are determined.

[0007] Based on the objective function and the constraints, and using the Actor-Critic structure, power grid optimization scheduling modeling is performed to obtain an initial scheduling model. The initial scheduling model is then subjected to Actor network and physical layer embedding design, Critic network design, and physical constraint loss design to obtain a scheduling model to be trained. The scheduling model to be trained is then subjected to preliminary training to obtain a preliminary trained scheduling model.

[0008] An experience pool is formed based on the grid status information, reward function, grid dispatching scheme, and new grid status information of the new energy grid system.

[0009] The initial training scheduling model is iteratively trained and its parameters are updated using the experience pool until the updated network parameters converge, resulting in a trained target scheduling model. The new energy power grid system is then optimally scheduled based on the target scheduling model.

[0010] In one embodiment, the design of the Actor network and physical layer embedding, the Critic network, and the physical constraint loss for the initial scheduling model includes:

[0011] Based on the power grid state information, the Actor network and physical layer are embedded and designed using a combination of neural networks and physical models; the objective value function of the Critic network is determined, and the Critic network is designed based on the objective value function; a physical consistency loss term, including generator power amplitude loss, voltage amplitude loss, generator power ramping loss, and line power flow constraint loss, is introduced into the optimization objective of the Critic network, and physical constraint loss is designed based on the physical consistency loss term.

[0012] In one embodiment, the preliminary training of the scheduling model to be trained to obtain a pre-trained scheduling model includes:

[0013] The loss function of the Critic network is determined, and the Critic network is trained by minimizing the Q-value error based on the loss function. The Actor network is trained in stages according to the economic-driven stage, the constraint-enhancing stage, and the constraint-driven stage using a staged training strategy. Based on the trained Critic network and the staged trained Actor network, the preliminary training scheduling model is obtained.

[0014] In one embodiment, the step of employing a phased training strategy to train the Actor network in three phases—an economy-driven phase, a constraint-reinforcement phase, and a constraint-driven phase—includes:

[0015] In the economic-dominated phase, a first optimization objective loss function of the Actor network is determined based on the initial constraint weights, and the Actor network is trained using the first optimization objective loss function. In the constraint-strengthening phase, a second optimization objective loss function of the Actor network is determined based on the strengthened constraint weights, and the Actor network is trained using the second optimization objective loss function. In the constraint-dominated phase, a third optimization objective loss function of the Actor network is determined based on the second strengthened constraint weights, and the Actor network is trained using the third optimization objective loss function.

[0016] In one embodiment, the iterative training and parameter update of the initially trained scheduling model using the experience pool includes:

[0017] If the total number of training steps of the current model meets the first threshold condition, the network parameters of the staged Actor network are updated using the first optimization objective loss function; if the total number of training steps meets the second threshold condition, the network parameters are updated using the second optimization objective loss function; otherwise, the network parameters are updated using the third optimization objective loss function; using mini-batch data in the experience pool, the network parameters of the trained Critic network are updated by minimizing the loss function of the trained Critic network.

[0018] In one embodiment, before forming the experience pool based on the grid state information, reward function, grid dispatching scheme, and new grid state information of the new energy grid system, the method further includes:

[0019] Obtain the power grid state information and the reward function, determine the power grid scheduling scheme based on the Actor network; execute the power grid scheduling scheme according to the power grid state information and the reward function, and obtain the new power grid state information after the scheme is executed.

[0020] Secondly, this application also provides a power grid optimal scheduling device based on reinforcement learning, comprising:

[0021] The instruction response module is used to respond to the dispatch instructions for the new energy power grid system, and to determine the objective of optimal power grid dispatch, the objective function corresponding to the objective, and the constraints that need to be satisfied.

[0022] The preliminary training module is used to perform power grid optimization scheduling modeling based on the objective function and the constraints and the Actor-Critic structure to obtain an initial scheduling model. The initial scheduling model is then used for Actor network and physical layer embedding design, Critic network design and physical constraint loss design to obtain a scheduling model to be trained. The scheduling model to be trained is then used for preliminary training to obtain a preliminary trained scheduling model.

[0023] The data processing module is used to form an experience pool based on the grid status information, reward function, grid dispatching scheme and new grid status information of the new energy grid system;

[0024] The optimal scheduling module is used to iteratively train and update the parameters of the initially trained scheduling model using the experience pool until the updated network parameters converge, thereby obtaining the trained target scheduling model, and performing optimal scheduling of the new energy power grid system according to the target scheduling model.

[0025] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0026] In response to dispatch instructions for a new energy power grid system, the objective of optimal grid dispatch, the objective function corresponding to the objective, and the constraints to be satisfied are determined. Based on the objective function and the constraints, and using an Actor-Critic structure, a grid optimization dispatch model is constructed to obtain an initial dispatch model. The initial dispatch model is then subjected to Actor network and physical layer embedding design, Critic network design, and physical constraint loss design to obtain a dispatch model to be trained. This model is then preliminarily trained to obtain a preliminarily trained dispatch model. An experience pool is formed based on the grid state information, reward function, grid dispatch scheme, and new grid state information of the new energy power grid system. The preliminarily trained dispatch model is iteratively trained and its parameters are updated using the experience pool until the updated network parameters converge, resulting in a trained target dispatch model. The new energy power grid system is then optimally dispatched based on this target dispatch model.

[0027] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0028] In response to dispatch instructions for a new energy power grid system, the objective of optimal grid dispatch, the objective function corresponding to the objective, and the constraints to be satisfied are determined. Based on the objective function and the constraints, and using an Actor-Critic structure, a grid optimization dispatch model is constructed to obtain an initial dispatch model. The initial dispatch model is then subjected to Actor network and physical layer embedding design, Critic network design, and physical constraint loss design to obtain a dispatch model to be trained. This model is then preliminarily trained to obtain a preliminarily trained dispatch model. An experience pool is formed based on the grid state information, reward function, grid dispatch scheme, and new grid state information of the new energy power grid system. The preliminarily trained dispatch model is iteratively trained and its parameters are updated using the experience pool until the updated network parameters converge, resulting in a trained target dispatch model. The new energy power grid system is then optimally dispatched based on this target dispatch model.

[0029] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0030] In response to dispatch instructions for a new energy power grid system, the objective of optimal grid dispatch, the objective function corresponding to the objective, and the constraints to be satisfied are determined. Based on the objective function and the constraints, and using an Actor-Critic structure, a grid optimization dispatch model is constructed to obtain an initial dispatch model. The initial dispatch model is then subjected to Actor network and physical layer embedding design, Critic network design, and physical constraint loss design to obtain a dispatch model to be trained. This model is then preliminarily trained to obtain a preliminarily trained dispatch model. An experience pool is formed based on the grid state information, reward function, grid dispatch scheme, and new grid state information of the new energy power grid system. The preliminarily trained dispatch model is iteratively trained and its parameters are updated using the experience pool until the updated network parameters converge, resulting in a trained target dispatch model. The new energy power grid system is then optimally dispatched based on this target dispatch model.

[0031] The aforementioned reinforcement learning-based optimal power grid scheduling method, apparatus, computer equipment, computer-readable storage medium, and computer program product determine the objective of optimal power grid scheduling, the corresponding objective function, and the constraints to be satisfied. Then, based on the objective function and constraints, and using an Actor-Critic structure, an initial scheduling model is obtained. This initial model undergoes Actor network and physical layer embedding design, Critic network design, and physical constraint loss design to obtain a scheduling model to be trained. This model is then preliminarily trained to obtain a preliminarily trained scheduling model. An experience pool is formed based on the power grid state information, reward function, power grid scheduling scheme, and new power grid state information of the renewable energy power grid system. Finally, the preliminarily trained scheduling model is iteratively trained and its parameters are updated using the experience pool until the updated network parameters converge, resulting in a trained target scheduling model. Optimal scheduling of the renewable energy power grid system is then performed based on this target scheduling model. This application effectively improves the solution speed, intelligence level, and convergence stability of model parameters in power grid scheduling, while reducing scheduling costs. It ensures the safe and stable operation of the renewable energy power grid system even with renewable energy integration, and meets the real-time and accurate solution requirements of the renewable energy power grid system. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is an application environment diagram of a power grid optimal scheduling method based on reinforcement learning in one embodiment;

[0034] Figure 2 This is a flowchart illustrating a reinforcement learning-based optimal power grid scheduling method in one embodiment.

[0035] Figure 3 This is a schematic diagram of the topology of a new energy power grid system in one embodiment;

[0036] Figure 4 This is a schematic diagram of the load curves for Bus 3, Bus 4, and Bus 15 nodes in one embodiment;

[0037] Figure 5 This is a schematic diagram of the renewable energy aggregation power generation curve of Bus 16 nodes in one embodiment;

[0038] Figure 6 This is a schematic diagram of the generator power curve at Bus 0 node in one embodiment;

[0039] Figure 7 This is a schematic diagram of the generator power curve at Bus 17 node in one embodiment;

[0040] Figure 8 This is a schematic diagram of the generator power curve at Bus 18 node in one embodiment;

[0041] Figure 9 This is a schematic diagram of the voltage amplitude curve of Bus 3 node in one embodiment;

[0042] Figure 10 This is a schematic diagram of the voltage amplitude curve of Bus 4 node in one embodiment;

[0043] Figure 11 This is a schematic diagram of the voltage amplitude curve at Bus 15 node in one embodiment;

[0044] Figure 12 This is a schematic diagram illustrating the daily power generation cost statistics over 20 days in one embodiment;

[0045] Figure 13 This is a flowchart illustrating the model design steps in one embodiment;

[0046] Figure 14 This is a flowchart illustrating a reinforcement learning-based optimal power grid scheduling method in a specific embodiment.

[0047] Figure 15 This is a block diagram of a power grid optimal scheduling device based on reinforcement learning in one embodiment;

[0048] Figure 16 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0050] The reinforcement learning-based optimal power grid scheduling method provided in this application can be applied to, for example, Figure 1 The application environment shown illustrates this. In this environment, the terminal can communicate with the server via a network. The data storage system can store the data that the server needs to process. The data storage system can be integrated onto the server or located on the cloud or other network servers. In situations such as... Figure 1 In the application environment shown, the terminal can be, but is not limited to, various personal computers, laptops, smartphones, and tablets. The server can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0051] In one embodiment, such as Figure 2 As shown, a reinforcement learning-based optimal power grid scheduling method is presented, which can be applied to... Figure 1 In the terminal, the method may include the following steps:

[0052] Step S201: In response to the dispatching instructions for the new energy power grid system, determine the objective of optimal power grid dispatch, the objective function corresponding to the objective, and the constraints that need to be satisfied.

[0053] The new energy power grid system consists of generators, new energy generators, load nodes, and power lines, such as... Figure 3 As shown, there are 19 nodes in this embodiment. Among them, Bus 0, Bus 17 and Bus 18 are generator sets that need to be scheduled, and Bus 1, Bus 2, Bus 3, Bus 4, Bus 5, Bus 6, Bus 7, Bus 8, Bus 9, Bus 10, Bus 11, Bus 12, Bus 13, Bus 14, Bus 15 and Bus 16 are load nodes. Bus 16 will be connected to a new energy generator set to generate electricity through photovoltaic and wind power.

[0054] Specifically, the objective of the optimal power grid scheduling problem can be to minimize the generator operating cost while satisfying constraints, and the objective function can be expressed as:

[0055]

[0056] In the above formula, and The first The active power output and reactive power output of each generator. This is the cost coefficient for the generator. For generator sets, This represents the total operating cost of the generator set.

[0057] Optimal power grid dispatch needs to satisfy the following constraints:

[0058] ① Active power balance constraint equations:

[0059]

[0060] ②Equations without power balance constraints:

[0061]

[0062] ③ Voltage amplitude constraint:

[0063] ④ Generator power limit: ,

[0064] ⑤ Generator ramping constraints:

[0065] ⑥ Line power flow restrictions:

[0066] In the above formula, For nodes voltage amplitude, For nodes voltage phase angle, It is a set of power grid nodes. It is the power flow of the lines in the power grid. A collection of power grid lines. and For nodes Minimum and maximum voltage values and For nodes Minimum and maximum active power output of the generator. and For nodes Minimum and maximum reactive power output of the generator. The maximum power flow of lines in a power grid. and These are the values ​​of the real and imaginary parts of the admittance matrix, respectively. and They are nodes The active and reactive power demands of the load are assessed, and renewable energy generation is integrated into the load during this process. For generator power ramping constraints, for nodes not connected to the generator... ,set up To meet the power balance calculation requirements.

[0067] For example, the entire new energy power grid system can be dispatched every 5 minutes. During this process, the power output of each generator at the previous moment will be obtained first. Voltage amplitude and phase angle at each node of the power grid Current load demand Finally, based on the given power values ​​of each generator... And complete the power grid dispatching tasks.

[0068] Step S202: Based on the objective function and constraints, and using the Actor-Critic structure, perform power grid optimization scheduling modeling to obtain an initial scheduling model. Then, design the Actor network and physical layer embedding, the Critic network, and the physical constraint loss to obtain a scheduling model to be trained. Perform preliminary training on the scheduling model to be trained to obtain a preliminarily trained scheduling model.

[0069] Specifically, the terminal models the power grid optimization scheduling model framework as a deep reinforcement learning model. In this embodiment, the deep reinforcement learning model is built based on an Actor-Critic structure and consists of five network models: the Actor policy network representing the generation of scheduling policy actions, the Critic1 value network estimating the state value function, the Critic2 value network estimating the state value function, the Critic1 network's target network Critic1-target, and the Critic2 network's target network Critic2-target. The parameters of the above five networks are as follows: , , , and .

[0070] In this embodiment, the optimal power grid scheduling problem is modeled as a deep reinforcement learning interactive environment. The power grid state information includes: the power output of each generator at the previous time step. Voltage amplitude and phase angle at each node of the power grid and current load demand Current power grid status information The vector consists of the variable values, and the action of the reinforcement learning environment is the target voltage and phase angle of each node at the current time. and the target power output of each generator. The environmental reward function is designed to be the negative of the generator cost function, i.e. ,in Execute the scheduling strategy at the current moment. Post-generator operating costs.

[0071] Step S203: Based on the grid status information, reward function, grid dispatching scheme, and new grid status information of the new energy grid system, an experience pool is formed.

[0072] Specifically, the terminal obtains the current power grid status information. And calculate the reward function. Then, a power grid dispatching scheme is obtained based on the Actor network. According to power grid status information and reward function Execute power grid dispatching scheme The new power grid status information after the implementation of the plan is obtained. , and then , , and Store in experience pool middle.

[0073] Step S204: Iteratively train and update the parameters of the initially trained scheduling model using the experience pool until the updated network parameters converge to obtain the trained target scheduling model. Optimal scheduling of the new energy power grid system is then performed based on the target scheduling model.

[0074] Specifically, the terminal uses the experience pool to iteratively train and update the parameters of the initially trained scheduling model, and determines whether the parameters of the Actor network, Critic1 network, and Critic2 network have converged. If they have converged, the training ends; if they have not converged, the above training steps are repeated until the updated network parameters converge, thus obtaining the trained target scheduling model. Finally, the new energy power grid system is optimally scheduled according to the target scheduling model.

[0075] For example, once the entire model converges, it can be loaded and tested on a randomly selected day. The test includes three models: the first is an optimal power flow model based on mathematical optimization; the second is the method of this invention; and the third is a traditional security reinforcement learning method. During the test... Figure 4 The load curves for Bus 3, Bus 4, and Bus 15 nodes are randomly selected for a particular day. Figure 5 This is the power generation curve of renewable energy aggregated at Bus 16 nodes on a randomly selected day. (Experimental results are in...) Figure 6 The generator power curve at Bus 0 node. Figure 7 The generator power curve at Bus 17 node. Figure 8 The output power curves for the generators at the 18-node Bus are shown. Comparing the three results, it can be seen that the overall output power of each generator in this invention is closer to the result of the optimal power flow model, with smaller errors and closer to the theoretical optimal value. Figure 9 The voltage amplitude curve at Bus node 3 is shown. Figure 10 The voltage amplitude curve at Bus 4 node is shown. Figure 11 The voltage amplitude curves for Bus node 15 are shown. Comparing the voltage amplitudes of the three nodes, it can be seen that neither the method of this invention nor the optimal power flow model exceeds the limit range [0.95 pu, 1.05 pu], meaning that no limit violations occur, thus better satisfying the scheduling constraints. In contrast, traditional safety reinforcement learning may produce a small number of limit violations at certain times, failing to fully satisfy the constraints.

[0076] Based on this, another 20 days will be randomly selected for testing. Figure 12 The daily power generation cost was statistically analyzed over 20 days. Experimental results show that the method proposed in this application increases the average daily power generation cost by 0.2413% compared to the optimal power flow model, but reduces it by 1.5286% compared to the traditional secure reinforcement learning method, effectively demonstrating the advantage of this application in scheduling cost. Furthermore, by comparing the average time consumed per scheduling task for the three methods, the optimal scheduling model averaged 190.3 ms, this application averaged 1.12 ms, and the traditional secure reinforcement learning method averaged 1.11 ms. This demonstrates that the method proposed in this application has a significant efficiency advantage compared to the optimal power flow model.

[0077] In this embodiment, by embedding the AC power flow equations into an Actor network and employing the Soft Actor-Critic (SAC) algorithm for optimization, the generation cost can be minimized without violating physical constraints. To overcome the difficulties of traditional methods in constraint handling, this application introduces a staged constraint learning strategy, gradually increasing the weight of constraint conditions to ensure constraint satisfaction and improve the model's convergence speed. This application can effectively improve the intelligence level of power grid dispatching, reduce dispatching costs, and ensure the safe and stable operation of the power grid even with the integration of renewable energy, providing a new solution for the intelligent real-time dispatching of new energy power grid systems.

[0078] In one embodiment, such as Figure 13 As shown, step S202 above, which involves designing the Actor network and physical layer embedding, the Critic network, and the physical constraint loss for the initial scheduling model, may include the following steps:

[0079] Step S1301: Based on the power grid state information, the Actor network and physical layer are embedded using a combination of neural network and physical model design.

[0080] Step S1302: Determine the objective value function of the Critic network, and design the Critic network based on the objective value function.

[0081] Step S1303: Introduce a physical consistency loss term, including generator power amplitude loss, voltage amplitude loss, generator power ramping loss, and line power flow constraint loss, into the optimization objective of the Critic network, and design the physical constraint loss based on the physical consistency loss term.

[0082] Specifically, the design of the initial scheduling model in this embodiment can include three parts:

[0083] ①Actor Network and Physical Layer Embedding Design: This application uses a combination of neural networks and physical models to output scheduling actions. The combination method is that the output sample of the Actor network is used as the input of the physical layer. First, in the Actor network part, its input is the state vector. The output is the target voltage of each node at the current moment. and phase angle The variance and mean of the Gaussian distribution to which it belongs And satisfy and This approach not only satisfies the smooth exploration mechanism but also allows the agent to make effective decisions in the face of uncertainty. Secondly, based on the output of the Actor network, the target voltage of each node is determined through random sampling. and phase angle Corresponding network output The data is input into the physical layer, which then processes it using the power balance equation. Calculate and obtain the target power generation of each node. Since the target power generation of each node is calculated based on the power flow equation, it ensures that the power generation meets the physical constraints of the power balance of the power grid.

[0084] ② Critic network design: In this application, Critic1 and Critic2 networks are used to evaluate the value function of the current policy. The guidance strategy has been updated, including The purpose of constructing two Critic value networks (Critic1 and Critic2) is to alleviate the overestimation problem of the Q-value. Furthermore, when calculating the target value function for each network, the minimum value from the Critic1-target network and the Critic2-target network is used in the calculation.

[0085]

[0086] In the above formula, It is entropy weight. It is the network target value. It is the immediate reward at time t. The parameter is The policy network, It's a strategy. The logarithmic probability of a state.

[0087] ③ Physical Constraint Loss Design: To explicitly embed the physical constraints of power system operation during the training process of reinforcement learning agents, enabling the generated scheduling strategy to achieve economic optimization while satisfying safe operation conditions, this application proposes a multi-constraint physical loss modeling method based on a physical information neural network, and guides the Actor network for optimization through physical losses. This method introduces a physical consistency loss term, composed of power balance equations, voltage constraints, ramping constraints, and line flow constraints, into the network's optimization objective. This automatically satisfies the basic physical laws of the power system during the forward propagation phase of the Actor network, thus avoiding the shortcomings of traditional methods based on ex-post correction of penalty terms. Physical constraint losses include generator power amplitude loss, voltage amplitude loss, generator power ramping loss, and line flow constraint loss. The total physical loss of the power system is defined as:

[0088]

[0089] In the above formula, the generator active power amplitude loss is: Reactive power loss of generator: Voltage amplitude loss function: Generator power ramping loss: Line power flow constraint loss: ; These are the weighting coefficients for each constraint loss.

[0090] In one embodiment, step S202 above, which involves performing preliminary training on the scheduling model to be trained to obtain a preliminarily trained scheduling model, may include the following steps:

[0091] The loss function of the Critic network is determined, and the Critic network is trained by minimizing the Q-value error based on the loss function. A phased training strategy is adopted to train the Actor network in three stages: economic-driven stage, constraint-enhancing stage, and constraint-driven stage. Based on the trained Critic network and the phased trained Actor network, a preliminary training scheduling model is obtained.

[0092] Specifically, the updates of the Critic1 and Critic2 networks are performed by minimizing the Q-value error, where the loss functions of the Critic1 and Critic2 networks are respectively... and And satisfy:

[0093]

[0094]

[0095] Furthermore, the loss function of the Actor network is designed as a phased strategy optimization model, with the first phase being the economic-driven phase, the second phase being the constraint-enhancing phase, and the third phase being the constraint-driven phase.

[0096] In one embodiment, as described above, the Actor network is trained in stages according to an economic-dominated stage, a constraint-reinforcement stage, and a constraint-dominated stage using a phased training strategy. This may include the following steps:

[0097] In the economic-dominated phase, the first optimization objective loss function of the Actor network is determined based on the initial constraint weights, and the Actor network is trained using the first optimization objective loss function. In the constraint reinforcement phase, the second optimization objective loss function of the Actor network is determined based on the reinforced constraint weights, and the Actor network is trained using the second optimization objective loss function. In the constraint-dominated phase, the third optimization objective loss function of the Actor network is determined based on the second reinforcement constraint weights, and the Actor network is trained using the third optimization objective loss function.

[0098] Specifically, the phased training strategy in this embodiment includes the following steps:

[0099] ① The economic-driven stage belongs to the initial stage of model training. The main objective of the Actor network is to minimize the power generation cost. The constraint weights are relatively small, while maintaining a slight physical pull to prevent the policy from deviating too far from the feasible region. The corresponding first optimization objective loss function is:

[0100]

[0101] In the above formula, , Let be the regularization coefficient for the first-stage physical constraint loss, and .

[0102] because In the middle Q value Depend on and Combination, and and From the power balance equation To better optimize the Actor network parameters through gradient feedback, we use constraints... Introduction right The derivative of Q is used to correct the Actor network parameters, that is, the gradient of Q with respect to the Actor network parameters is calculated as follows:

[0103]

[0104] In the above formula, and .

[0105] ② The constraint reinforcement phase is the mid-training phase. Constraints are gradually increased, and the weight of physical constraint losses is increased to push the strategy into the feasible region, ensuring significant convergence of the main constraint residuals. Simultaneously, efforts are made to prevent cost rebound. The corresponding second optimization objective loss function is:

[0106]

[0107] In the above formula, Let be the regularization coefficient for the physical constraint loss in the second stage, and , This represents the regularization coefficient for the physical constraint loss in the third stage. This represents the number of steps the model has taken at the end of the first phase. This represents the total number of steps in the current model training. Adjustment coefficient.

[0108] ③ The constraint-driven phase is the later stage of training. Under the condition that all constraints are fully satisfied, the goal is to minimize the power generation cost. The corresponding loss function for the third optimization objective is:

[0109]

[0110] When all physical constraint equations are satisfied in the later stage of this phase The loss will be zero, and the entire optimization objective will fully achieve the optimization cost under physical constraints.

[0111] In one embodiment, step S204 above, which involves iteratively training and updating the parameters of the initially trained scheduling model using the experience pool, may include the following steps:

[0112] If the total number of steps in the current model training meets the first threshold condition, the network parameters of the Actor network after phased training are updated using the first optimization objective loss function; if the total number of steps meets the second threshold condition, the network parameters are updated using the second optimization objective loss function; otherwise, the network parameters are updated using the third optimization objective loss function; using mini-batch data in the experience pool, the network parameters of the Critic network after training are updated by minimizing the loss function of the Critic network after training.

[0113] Specifically, the terminal randomly selects a small batch of data from the experience pool, if the total number of steps in the current model training is... By minimizing loss Update Actor network parameters ,like and Then by minimizing the loss Update Actor network parameters Otherwise, minimize the loss Update Actor network parameters Then, the terminal randomly selects a small batch of data from the experience pool, minimizing the loss functions of the Critic1 and Critic2 networks. and The parameters of the two networks are optimized and updated, and the network parameters of the target networks Critic1-target and Critic2-target are updated using a soft update method. Finally, the terminal determines whether the network parameters of the Actor network, Critic1 network, and Critic2 network have converged. If they have converged, the training ends; if they have not converged, the above steps are repeated for the next interaction cycle to continue training.

[0114] In one embodiment, before forming an experience pool based on the grid state information, reward function, grid dispatching scheme, and new grid state information of the new energy grid system, the method of this application further includes the following steps:

[0115] Obtain power grid state information and reward function, determine power grid dispatching scheme based on Actor network; execute power grid dispatching scheme according to power grid state information and reward function, and obtain new power grid state information after scheme execution.

[0116] Specifically, the terminal obtains the current power grid status information. And calculate the reward function. Then, a power grid dispatching scheme is obtained based on the Actor network. According to power grid status information and reward function Execute power grid dispatching scheme The new power grid status information after the implementation of the plan is obtained. , and then , , and Store in experience pool middle.

[0117] In one embodiment, such as Figure 14 As shown, a reinforcement learning-based optimal power grid scheduling method is provided in a specific embodiment, which includes the following steps:

[0118] Step S1401: In response to the dispatching instructions for the new energy power grid system, determine the objective of optimal power grid dispatching, the objective function corresponding to the objective, and the constraints that need to be satisfied; based on the objective function and constraints and the Actor-Critic structure, perform power grid optimal dispatching modeling to obtain the initial dispatching model.

[0119] Step S1402: Based on the power grid state information, the Actor network and physical layer are embedded using a combination of neural network and physical model; the objective value function of the Critic network is determined, and the Critic network is designed based on the objective value function; a physical consistency loss term, including generator power amplitude loss, voltage amplitude loss, generator power ramping loss, and line power flow constraint loss, is introduced into the optimization objective of the Critic network, and physical constraint loss is designed based on the physical consistency loss term to obtain the scheduling model to be trained.

[0120] Step S1403: Determine the loss function of the Critic network, and train the Critic network by minimizing the Q-value error according to the loss function; adopt a phased training strategy to train the Actor network in three phases: economic-driven phase, constraint-enhancing phase, and constraint-driven phase; based on the trained Critic network and the phased trained Actor network, obtain the preliminary trained scheduling model.

[0121] Step S1404: Obtain the grid state information and reward function of the new energy power grid system, and determine the grid dispatching scheme based on the Actor network; execute the grid dispatching scheme according to the grid state information and reward function to obtain the new grid state information after the scheme is executed; form an experience pool based on the grid state information, reward function, grid dispatching scheme and new grid state information of the new energy power grid system.

[0122] Step S1405: If the total number of training steps of the current model meets the first threshold condition, then update the network parameters of the Actor network after phased training through the first optimization objective loss function; if the total number of training steps meets the second threshold condition, then update the network parameters through the second optimization objective loss function; otherwise, update the network parameters through the third optimization objective loss function; using the mini-batch data in the experience pool, update the network parameters of the trained Critic network by minimizing the loss function of the trained Critic network until the updated network parameters converge, thereby obtaining the trained target scheduling model, and performing optimal scheduling of the new energy power grid system according to the target scheduling model.

[0123] The beneficial effects of the above embodiments are as follows:

[0124] (1) This application embeds AC power flow equations and operating constraints in the Actor network, realizing deep coupling between strategy generation and the physical laws of the power system. Compared with the traditional deep reinforcement learning method that corrects the penalty term later, this application can ensure that the constraints such as power balance, voltage amplitude and power flow limit are met during the network training process, thus improving the feasibility and stability of the scheduling strategy.

[0125] (2) This application proposes a phased constraint progressive training strategy. By dynamically adjusting the physical constraint loss weight, a continuous transition from economic goal-driven to constraint-driven is achieved. This effectively solves the gradient conflict and convergence instability problems in traditional reinforcement learning in multi-constraint optimization. Finally, optimized scheduling is achieved under the condition of satisfying all physical constraints. This is a key technological breakthrough for realizing the safe and efficient operation of smart grids. Compared with traditional safety reinforcement learning algorithms, the scheduling cost is reduced by an average of 1.5268%.

[0126] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0127] Based on the same inventive concept, this application also provides a reinforcement learning-based power grid optimal scheduling device for implementing the reinforcement learning-based power grid optimal scheduling method described above. The solution provided by this device is similar to the implementation scheme described in the above method. Therefore, the specific limitations in one or more reinforcement learning-based power grid optimal scheduling device embodiments provided below can be found in the limitations of the reinforcement learning-based power grid optimal scheduling method described above, and will not be repeated here.

[0128] In one exemplary embodiment, such as Figure 15 As shown, a power grid optimal scheduling device based on reinforcement learning is provided, which may include:

[0129] The instruction response module 1501 is used to respond to the dispatch instructions for the new energy power grid system, and to determine the objective of the optimal dispatch of the power grid, the objective function corresponding to the objective, and the constraints that need to be satisfied.

[0130] The preliminary training module 1502 is used to perform power grid optimization scheduling modeling based on the objective function and constraints and the Actor-Critic structure to obtain an initial scheduling model. The initial scheduling model is then used for Actor network and physical layer embedding design, Critic network design, and physical constraint loss design to obtain a scheduling model to be trained. The scheduling model to be trained is then subjected to preliminary training to obtain a preliminary trained scheduling model.

[0131] Data processing module 1503 is used to form an experience pool based on the grid status information, reward function, grid dispatching scheme and new grid status information of the new energy grid system;

[0132] The optimal scheduling module 1504 is used to iteratively train and update the parameters of the initially trained scheduling model using the experience pool until the updated network parameters converge, thereby obtaining the trained target scheduling model. The target scheduling model is then used to perform optimal scheduling of the new energy power grid system.

[0133] In one embodiment, the preliminary training module 1502 is further configured to embed the Actor network and physical layer into the power grid based on the power grid state information using a combination of neural network and physical model; determine the objective value function of the Critic network; design the Critic network based on the objective value function; introduce a physical consistency loss term, including generator power amplitude loss, voltage amplitude loss, generator power ramping loss, and line power flow constraint loss, into the optimization objective of the Critic network; and design the physical constraint loss based on the physical consistency loss term.

[0134] In one embodiment, the preliminary training module 1502 is further used to determine the loss function of the Critic network, and train the Critic network by minimizing the Q-value error according to the loss function; adopt a phased training strategy to train the Actor network in stages according to the economic-dominated stage, the constraint-strengthening stage, and the constraint-dominated stage; and obtain the preliminary training scheduling model based on the trained Critic network and the phased trained Actor network.

[0135] In one embodiment, the initial training module 1502 is further configured to: determine a first optimization objective loss function of the Actor network based on the initial constraint weights during the economic-dominated phase, and train the Actor network using the first optimization objective loss function; determine a second optimization objective loss function of the Actor network based on the strengthened constraint weights during the constraint-enhanced phase, and train the Actor network using the second optimization objective loss function; and determine a third optimization objective loss function of the Actor network based on the strengthened constraint weights during the constraint-dominated phase, and train the Actor network using the third optimization objective loss function.

[0136] In one embodiment, the optimal scheduling module 1504 is used to update the network parameters of the staged Actor network through a first optimization objective loss function if the total number of training steps of the current model meets a first threshold condition; update the network parameters through a second optimization objective loss function if the total number of training steps meets a second threshold condition; otherwise, update the network parameters through a third optimization objective loss function; and update the network parameters of the trained Critic network by minimizing the loss function of the trained Critic network using mini-batch data in the experience pool.

[0137] In one embodiment, the device may further include: an information acquisition module, configured to acquire power grid state information and a reward function, determine a power grid dispatching scheme based on the Actor network, execute the power grid dispatching scheme according to the power grid state information and the reward function, and obtain new power grid state information after the scheme is executed.

[0138] The modules in the aforementioned reinforcement learning-based optimal power grid scheduling device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can invoke and execute the corresponding operations of each module.

[0139] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 16As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a reinforcement learning-based optimal power grid scheduling method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0140] Those skilled in the art will understand that Figure 16 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0141] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0142] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0143] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0144] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0145] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0146] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0147] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A power grid optimal scheduling method based on reinforcement learning, characterized in that, The method includes: In response to dispatch instructions for the new energy power grid system, the objective of optimal power grid dispatch, the objective function corresponding to the objective, and the constraints that need to be satisfied are determined. Based on the objective function and the constraints, and using the Actor-Critic structure, power grid optimization scheduling modeling is performed to obtain an initial scheduling model. The initial scheduling model is then subjected to Actor network and physical layer embedding design, Critic network design, and physical constraint loss design to obtain a scheduling model to be trained. The scheduling model to be trained is then subjected to preliminary training to obtain a preliminary trained scheduling model. An experience pool is formed based on the grid status information, reward function, grid dispatching scheme, and new grid status information of the new energy grid system. The initial training scheduling model is iteratively trained and its parameters are updated using the experience pool until the updated network parameters converge, resulting in a trained target scheduling model. The new energy power grid system is then optimally scheduled based on the target scheduling model.

2. The method according to claim 1, characterized in that, The design of the Actor network and physical layer embedding, the Critic network, and the physical constraint loss for the initial scheduling model includes: Based on the power grid state information, the Actor network and physical layer are embedded in a design that combines neural networks and physical models. Determine the objective value function of the Critic network, and design the Critic network based on the objective value function; The optimization objective of the Critic network incorporates a physical consistency loss term, which includes generator power amplitude loss, voltage amplitude loss, generator power ramping loss, and line power flow constraint loss. The physical constraint loss is designed based on the physical consistency loss term.

3. The method according to claim 2, characterized in that, The preliminary training of the scheduling model to be trained to obtain a preliminarily trained scheduling model includes: Determine the loss function of the Critic network, and train the Critic network by minimizing the Q-value error based on the loss function; The Actor network is trained in stages according to the economic-dominated stage, the constraint-reinforcement stage, and the constraint-dominated stage, using a phased training strategy. The scheduling model is obtained based on the trained Critic network and the staged trained Actor network.

4. The method according to claim 3, characterized in that, The phased training strategy involves training the Actor network in three phases: an economy-driven phase, a constraint-reinforcement phase, and a constraint-driven phase. In the economic-dominated phase, a first optimization objective loss function of the Actor network is determined based on the initial constraint weights, and the Actor network is trained using the first optimization objective loss function. In the constraint reinforcement stage, the second optimization objective loss function of the Actor network is determined according to the reinforced constraint weights, and the Actor network is trained using the second optimization objective loss function; In the constraint-dominated phase, the third optimization objective loss function of the Actor network is determined based on the constraint weights after the second reinforcement, and the Actor network is trained based on the third optimization objective loss function.

5. The method according to claim 4, characterized in that, The iterative training and parameter update of the initially trained scheduling model using the experience pool includes: If the total number of training steps of the current model meets the first threshold condition, the network parameters of the staged Actor network are updated through the first optimization objective loss function; if the total number of training steps meets the second threshold condition, the network parameters are updated through the second optimization objective loss function; otherwise, the network parameters are updated through the third optimization objective loss function. Using mini-batch data from the experience pool, the network parameters of the trained Critic network are updated by minimizing the loss function of the trained Critic network.

6. The method according to any one of claims 1 to 5, characterized in that, Before forming the experience pool based on the grid state information, reward function, grid dispatching scheme, and new grid state information of the new energy grid system, the process also includes: Obtain the power grid status information and the reward function, and determine the power grid scheduling scheme based on the Actor network; The power grid scheduling scheme is executed based on the power grid status information and the reward function to obtain the new power grid status information after the scheme is executed.

7. A power grid optimal scheduling device based on reinforcement learning, characterized in that, The device includes: The instruction response module is used to respond to the dispatch instructions for the new energy power grid system, and to determine the objective of optimal power grid dispatch, the objective function corresponding to the objective, and the constraints that need to be satisfied. The preliminary training module is used to perform power grid optimization scheduling modeling based on the objective function and the constraints and the Actor-Critic structure to obtain an initial scheduling model. The initial scheduling model is then used for Actor network and physical layer embedding design, Critic network design and physical constraint loss design to obtain a scheduling model to be trained. The scheduling model to be trained is then used for preliminary training to obtain a preliminary trained scheduling model. The data processing module is used to form an experience pool based on the grid status information, reward function, grid dispatching scheme and new grid status information of the new energy grid system; The optimal scheduling module is used to iteratively train and update the parameters of the initially trained scheduling model using the experience pool until the updated network parameters converge, thereby obtaining the trained target scheduling model, and performing optimal scheduling of the new energy power grid system according to the target scheduling model.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images