Power grid simulation environment scheduling optimization method and system based on reinforcement learning

Abstract

The invention discloses a power grid simulation environment scheduling optimization method and system based on reinforcement learning. A multi-target reward function system and a double insurance mechanism combined with reward constraint limitation are designed. In the training process of the A3C algorithm, modification is carried out in combination with the actual condition of high-proportion new energy grid connection, safe and stable operation of a power system can be effectively maintained by introducing a power grid safety index, passive action screening is carried out to reduce the single-step action space dimension, the convergence speed of reinforcement learning is increased, and the method is suitable for large-scale popularization and application. And the real-time performance of the A3C algorithm in power grid dispatching optimization is enhanced. Then, by combining economic cost, output limitation of a generator set, capacity limitation of a power transmission line and the like, it is ensured that the algorithm can comprehensively consider and effectively process the constraint conditions in the training process; through the standardized constraint processing mechanism, the generated scheduling strategy not only meets the actual operation requirements of the power system, but also can be optimized under various complex constraint conditions.

Description

Technical Field

[0001] The present invention belongs to the field of power system optimization and dispatching, and relates to a power grid simulation environment dispatching optimization method and system based on reinforcement learning. Background Art

[0002] With the continuous expansion of the scale and complexity of power systems, traditional grid dispatching methods are facing huge challenges. Grid dispatch optimization involves multiple aspects such as load forecasting, power generation dispatching, and transmission dispatching. The purpose is to achieve economical and efficient allocation of power resources while ensuring the safe and stable operation of the power system. However, the high dimensionality, nonlinearity, dynamics and uncertainty of the power system make traditional optimization methods (such as linear programming, dynamic programming and heuristic algorithms) show certain limitations in dealing with complex grid dispatching problems.

[0003] Reinforcement Learning (RL) is a machine learning method that continuously learns and optimizes strategies through the interaction between the agent and the environment to maximize the cumulative reward. RL has unique advantages in dealing with high-dimensional state space and decision sequence problems, and therefore has shown broad application prospects in the field of power grid dispatch optimization. In recent years, with the improvement of computing power and the continuous advancement of algorithms, the application research of RL in power grid dispatch optimization has made significant progress. In power grid dispatch optimization, RL algorithms solve the dispatch of the operating environment of the simulated power system, so that the agent can learn the optimal dispatch strategy through continuous trial and error. The key to the RL algorithm is to guide the behavior of the agent through reward signals so that it gradually approaches the optimal dispatch plan. For example, the agent can adjust the output of the generator set according to factors such as the operating status of the power grid, load demand, and power generation cost to achieve economic dispatch and stable operation of the power grid. In terms of specific applications, RL algorithms are used to solve a variety of power grid dispatch problems, such as load forecasting, power generation dispatch, energy storage dispatch, demand response, etc. For example, in power generation scheduling optimization, the intelligent agent can optimize the output of each generator set by learning the operating rules of the power grid and the power generation cost curve, so as to minimize the overall power generation cost; in energy storage scheduling, the intelligent agent can achieve optimal utilization of energy storage resources by learning the charging and discharging rules of energy storage equipment.

[0004] A3C (Asynchronous Advantage Actor-Critic) algorithm is an advanced algorithm in RL. It combines the Actor-Critic method with the asynchronous update strategy and has the characteristics of high efficiency and fast convergence. The A3C algorithm can effectively improve the learning efficiency and the diversity of strategy exploration through the asynchronous interaction of multiple agents in different environments. In power grid dispatch optimization, the A3C algorithm can quickly learn and converge to the optimal dispatch strategy through multiple parallel agents trained in different power grid dispatch environments. Specifically, the A3C algorithm can be applied to power generation dispatch optimization. By learning the dispatch strategy of generator sets under different load demands, the output of the generator sets can dynamically adapt to load changes and maximize the economy and stability of the power system; in energy storage dispatch optimization, by learning the optimal charging and discharging strategy of energy storage equipment, the A3C algorithm can optimize the utilization of energy storage resources and improve the regulation capacity and economic benefits of the power system; in demand response management, the A3C algorithm can optimize the demand response strategy by learning the user's electricity consumption behavior pattern, balance the supply and demand of electricity, reduce the peak-valley difference of the power system, and improve the operation efficiency of the power grid.

[0005] Although reinforcement learning (RL) algorithms have shown great potential in the field of power grid dispatch optimization, they still have some significant defects in practical applications. These defects are mainly reflected in the lack of attention to safety priority and the need for additional design of complex constraint processing.

[0006] Current reinforcement learning algorithms often lack the emphasis on safety priority in the field of power grid dispatch optimization. The operational safety of the power system is the primary consideration, and any dispatch strategy must ensure the stability and safety of the system. However, many RL algorithms focus mainly on maximizing economic benefits and ignore the safety of power system operation. For example, when the exploration factor is high, reinforcement learning will have a certain probability of selecting a power grid dispatch solution that is not suitable for the current state, in order to jump out of the local optimal solution and explore the feasibility of other solutions. Although the final result of this may be more in line with the goal than following the rules, the power system safety failure phenomena such as over-limit that occur in this process cannot be accepted in reality. RL algorithms often face the problem of lack of constraints in power grid dispatch optimization. The power grid dispatch problem is essentially a complex multi-constraint optimization problem involving multiple constraints such as the output limit of the generator set, the capacity limit of the transmission line, and environmental regulations. However, current RL algorithms usually find it difficult to fully consider and effectively handle these constraints during the training process. Although the A3C algorithm improves learning efficiency through asynchronous interaction of multiple agents, it does not explicitly incorporate safety constraints into the reward function during the learning process, which may lead to the neglect of safety in the pursuit of optimal economic benefits. For example, under high load conditions, the A3C algorithm will dispatch some generators in critical states, thereby increasing the instability and failure risk of the power system. Although the A3C algorithm has advantages in policy learning, it often needs to design additional complex constraint processing mechanisms when faced with multiple complex constraints. Otherwise, it is easy to cause the learned dispatch strategy to fail to meet the operating constraints of the power system in actual applications. For example, when optimizing power generation dispatch, the A3C algorithm may generate dispatch instructions that exceed the maximum output capacity of some generators, making it impossible to execute in actual operations.

[0007] Therefore, the reinforcement learning algorithm represented by the A3C algorithm has shown good application prospects in the field of power grid dispatch optimization, but it still has significant defects in terms of insufficient emphasis on safety priority and the need for additional design of complex constraint processing. The solution to these problems requires further research and improvement to improve the actual application effect of RL algorithms in power grid dispatch optimization and ensure the safe, stable and efficient operation of the power system. Summary of the invention

[0008] The purpose of the present invention is to solve the problem that the existing power grid dispatching optimization methods in the prior art do not pay enough attention to safety priority and require additional design of complex constraint processing, and to provide a power grid simulation environment dispatching optimization method and system based on reinforcement learning.

[0009] In order to achieve the above object, the present invention adopts the following technical solutions:

[0010] The present invention proposes a method for optimizing power grid simulation environment scheduling based on reinforcement learning, comprising:

[0011] Construct the state space and action space of the power grid simulation environment;

[0012] Constructing a target-based reward function and a constraint-based reward function according to the action space and the state space of the power grid simulation environment;

[0013] The weighted reward value is obtained according to the goal-based reward function and the constraint-based reward function, and the Actor-Critic network structure combined with the weighted reward value is used to optimize the scheduling of the power grid simulation environment.

[0014] The present invention proposes a power grid simulation environment dispatch optimization system based on reinforcement learning, comprising:

[0015] A power grid parameter acquisition module, wherein the power grid parameter acquisition module is used to construct a state space of a power grid simulation environment and an action space of a power grid simulation environment;

[0016] A reward function construction module, wherein the reward function construction module is used to construct a target-based reward function and a constraint-based reward function according to an action space of a power grid simulation environment and a state space of a power grid simulation environment;

[0017] The scheduling optimization module is used to obtain weighted reward values ​​according to a target-based reward function and a constraint-based reward function, and to perform scheduling optimization in a power grid simulation environment using an Actor-Critic network structure combined with the weighted reward values.

[0018] A terminal device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of a method for optimizing scheduling in a power grid simulation environment based on reinforcement learning are implemented.

[0019] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a method for optimizing scheduling in a power grid simulation environment based on reinforcement learning.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] The present invention proposes a method for optimizing power grid simulation environment scheduling based on reinforcement learning, which optimizes the operation and control of the power system through the A3C method that combines weighted rewards, action space screening and other methods. Power system optimization scheduling refers to the technical field of maximizing energy utilization efficiency, ensuring safe and stable operation of the power grid and reducing environmental impact by dynamically adjusting the operating state of the power system. The present invention aims to solve several key deficiencies of existing reinforcement learning algorithms in the field of power grid scheduling optimization, including insufficient attention to safety, difficulty in achieving real-time scheduling and lack of constraints. In order to meet these challenges, first of all, the present invention designs a multi-objective reward function system and a double insurance mechanism combining constraints and restrictions with rewards to address the problem that existing reinforcement learning algorithms do not pay enough attention to the safety of power grid operation. A reward function system that comprehensively considers indicators such as economy, low-carbon benefits and operational safety is designed. In the training process of the A3C algorithm, the actual conditions of high-proportion new energy grid connection are combined for modification. By introducing grid security indicators such as line load rate and line over-limit rate, it is ensured that the dispatching strategy can effectively maintain the safe and stable operation of the power system while achieving economic optimization, and passive action screening is performed to reduce the action space dimension of a single step, accelerate the convergence speed of reinforcement learning, and enhance the real-time performance of the A3C algorithm in grid dispatch optimization. Then, the present invention proposes a standardized multi-objective constraint system for the complex constraints of the existing reinforcement learning algorithm. In the design of the A3C algorithm, by combining economic costs, output limits of generator sets, and capacity limits of transmission lines, it is ensured that the algorithm can fully consider and effectively handle these constraints during the training process. Through this standardized constraint processing mechanism, the generated dispatching strategy not only meets the actual operation requirements of the power system, but also can be optimized under a variety of complex constraints. In summary, the present invention aims to overcome the main defects of the existing reinforcement learning algorithm in the field of grid dispatch optimization by introducing a double insurance mechanism and a standardized reward function design, and provide a solid guarantee for the efficient and safe operation of the power system.

[0022] The invention proposes a power grid simulation environment dispatch optimization system based on reinforcement learning. By dividing the system into a power grid parameter acquisition module, a reward function construction module and a dispatch optimization module, the Actor-Critic network structure is combined with a weighted reward value to perform power grid simulation environment dispatch optimization. The modular concept is adopted to make each module independent of each other, which is convenient for unified management of each module. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 This is a flow chart of the power grid simulation environment scheduling optimization method based on reinforcement learning of the present invention.

[0025] Figure 2 Construct a graph for the state space of the present invention.

[0026] Figure 3 It is the position diagram of the passive discriminator of the present invention.

[0027] Figure 4 It is a flow chart of the action screening of the present invention.

[0028] Figure 5 Schematic diagram of the A3C algorithm structure of the present invention.

[0029] Figure 6 This is a diagram of the power grid simulation environment scheduling optimization system based on reinforcement learning of the present invention.

[0030] Figure 7 The figure is a schematic structural diagram of an electronic device of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0032] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0034] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0035] The present invention proposes a method for optimizing the dispatching of a power grid simulation environment based on reinforcement learning. By constructing a state space and an action space, establishing a reward function and training a neural network, a reinforcement learning algorithm framework is built to optimize the dispatching of the power grid simulation environment. Figure 1 As shown, the following steps are included:

[0036] S1. Construct the state space and action space of the power grid simulation environment;

[0037] The state space of the power grid simulation environment and the action space of the power grid simulation environment are constructed as follows:

[0038] Acquire numerical features and non-numerical features, and construct the state space of the power grid simulation environment based on the numerical features and non-numerical features; wherein the numerical features include generator active power, generator reactive power, load active power, load reactive power, predicted load, line power and constrained power; and the non-numerical features include line disconnection status;

[0039] In the construction of the action space of the power grid simulation environment, a passive discriminator is set to run in the environment. The specific process of the passive discriminator's judgment is as follows: after the agent takes an operation control action, all the system power flow constraints are calculated while calculating the reward; if the action taken by the agent causes the constraint to exceed the bounds, the probability of each action is arranged in descending order, and the operation control action with the second highest action probability is selected until the agent finds an action that meets the constraint conditions; if all actions cannot meet the constraint conditions, the agent executes the action that maximizes the current reward function.

[0040] S2. constructing a target-based reward function and a constraint-based reward function according to the action space of the power grid simulation environment and the state space of the power grid simulation environment;

[0041] According to the action space and state space of the power grid simulation environment, a goal-based reward function is constructed, which is as follows:

[0042] The target-based reward function includes the first dispatching reward, second dispatching reward and third dispatching reward of the new energy system. The first dispatching reward of the grid includes the power divergence reward, power control reward and voltage balance control reward; the second dispatching reward of the grid includes the abandoned wind and solar cost reward and the unit operation cost reward; the third dispatching reward of the grid includes the renewable energy output reward and the carbon emission reward;

[0043] Among them, the constraints of the new energy system are:

[0044]

[0045] Among them, C1 to C5 represent constraints; represents the lower limit of active power output of each generator, represents the upper limit of active power output of each generator, represents the lower limit of reactive power output of each generator, Represents the upper limit of reactive power output of each generator; represents the lower voltage limit of each generator node, represents the voltage upper limit of each generator node, represents the lower voltage limit of each generator load node, Indicates the voltage upper limit of each generator load node; ΔP i is the imbalance between active powers at node i, ΔQ i is the imbalance between reactive powers at node i, P i is the active power injected at node i, Q i is the reactive power injected at node i, V i is the voltage amplitude at node i, V j is the voltage amplitude at node j, G ij is the conductance of the branch, B ij is the branch susceptance; θ ij is the voltage phase angle difference; represents the active power of the generator, represents the active power of the load node, is the reactive power of the generator, Represents the reactive power of the load node, Δ>0 indicates a normal value close to 0; n represents the generator number; N represents the number of generators; m represents the load number; M represents the number of loads.

[0046] The trend divergence reward As shown below:

[0047]

[0048]

[0049] The power control rewards As shown below:

[0050]

[0051] The voltage balancing control rewards As shown below:

[0052]

[0053] Among them, R1 represents the power flow divergence reward function, R2 represents the power control reward function, and R3 represents the voltage balance reward function; () represents exponential normalization; R min Represents the minimum reward during the system operation; R g represents the penalty imposed on the agent when power flow divergence occurs; ρ w represents the current transmission power of the transmission line w; and represents the rated maximum transmission power of the transmission line w, χ w is the penalty coefficient for exceeding the power flow limit of transmission line w; R0 is the default reward function reflecting the consistency of grid operation; w represents the number of power lines; W represents the number of power lines, P G Indicates the active power of the generator, Q G Indicates the reactive power of the generator.

[0054] The abandoned wind and solar cost reward As shown below:

[0055]

[0056]

[0057] The unit operating cost reward As shown below:

[0058]

[0059] Among them, R4 represents the power generation cost reward function, and R5 represents the wind and solar abandonment cost reward function; a n represents the cost coefficient of the quadratic term, b n represents the cost coefficient of the first-order term, c n represents the cost coefficient of the constant term; Indicates the active power of the generator; c q It represents the unit cost coefficient of abandoned wind and solar power; Indicates the maximum output value of the new energy generator; P new Indicates the actual output value of the new energy generator; n new Indicates the number of the new energy generator; N new Indicates the number of renewable energy generators.

[0060] Renewable Energy Output Incentive As shown below:

[0061]

[0062] Carbon Emissions Rewards As shown below:

[0063]

[0064]

[0065] Among them, R6 represents the renewable energy output reward function, R7 represents the carbon emission reward function; c p P represents the unit cost coefficient of conventional units; con Indicates the actual output value of a conventional generator; n con Indicates the number of conventional generators; N con Indicates the number of conventional generators.

[0066] According to the action space and state space of the power grid simulation environment, a constraint-based reward function is constructed, which is as follows:

[0067]

[0068] Among them, λ is a fixed reward value, For constraint-based rewards, C1 to C5 represent the constraints of the new energy system.

[0069] S3. Obtain weighted reward values ​​based on the goal-based reward function and the constraint-based reward function, and use the Actor-Critic network structure combined with the weighted reward value to optimize the scheduling of the power grid simulation environment.

[0070] The weighted reward value is obtained according to the target-based reward function and the constraint-based reward function, as follows:

[0071]

[0072] Among them, r is the weighted reward value, α i is the calculation result of the normalized reward value, is the weight coefficient of each reward item, and the weight coefficient is usually in the range of 0 to 1.

[0073] The method is described in detail below with reference to the accompanying drawings:

[0074] 1) State space construction

[0075] The input received by the agent determines the performance to a certain extent, such as Figure 2As shown, the state space required by the present invention includes both numerical features and non-numerical features as inputs of the intelligent agent, so that the intelligent agent can access the entire state of the power network at each time step. The state features used in the present invention can be divided into numerical features and non-numerical features. Numerical features include generator active power and reactive power, load active power and reactive power, predicted load, line power and constrained power. Non-numerical features include line disconnection status, which is directly incorporated into the topological features as an additional vector.

[0076]

[0077] In the formula, S is the vector space, P L , Q L Indicates the load active power and reactive power, P G , Q G Represents the active power and reactive power of the generator, p Lk t+1 represents the predicted active demand value of the kth load node at the next moment, P ρ Indicates line power, Q C represents the constrained power, PR is an additional vector such as the line disconnection state, and represents the topological structure of the power system.

[0078] 2) Action space construction

[0079] In the construction of action space, such as Figure 3 As shown, a passive discriminator is set to run in the environment, and only when the power flow exceeds the limit, it starts to screen the feasible actions and continues to simulate until the iteration ends. Here, it plays the first insurance role in terms of power grid security.

[0080] like Figure 4 As shown in the figure, the specific process of the passive discriminator is as follows: after the agent takes the running control action, all the power flow constraints of the system are calculated while calculating the reward. If the action taken by the agent will cause the constraint to exceed the bounds, the probability of each action is sorted in descending order, and the running control action with the second highest action probability is selected. And so on, until the agent finds an action that can meet the constraints. If all actions cannot meet the constraints, the agent will perform the action that maximizes the current reward function.

[0081] In the action screening process, the topological actions that can restore the system from the power flow over-limit state to the safe operation state and the power generation output adjustment actions together constitute the action space used for agent training:

[0082] Ω t =Ω t-1 ∪(Ω A,t-1 ∩Ω TP,t-1 )∪(Ω A,t-1 ∩ΩG,t )

[0083] Where: Ω t and Ω t+1 are the feasible action sets for the simulation of step t and step t+1 respectively; Ω A,t is the set of actions that can satisfy all constraints and whose current reward is better than not performing the action in the simulation of step t; Ω TP,t is the set of all feasible topological actions in the t-th step simulation; Ω G,t It is the set of all feasible power output adjustment actions in the t-th simulation step.

[0084] 3) Construction of reward function system

[0085] 3.1 Goal-based Reward Function

[0086] (1) Security Scheduling Rewards

[0087] In the dispatch reward of safety, the three main aspects involved are the new power system flow divergence, active and reactive power control, and voltage balance control. According to the constraints involved in each aspect, the corresponding evaluation indicators are normalized and converted into the corresponding reward function. Constitute a new type of power system with rewards for safe dispatch.

[0088] The constraints of the new energy system are:

[0089]

[0090] Among them, C1 to C5 represent constraints; represents the lower limit of active power output of each generator, represents the upper limit of active power output of each generator, represents the lower limit of reactive power output of each generator, Represents the upper limit of reactive power output of each generator; represents the lower voltage limit of each generator node, represents the voltage upper limit of each generator node, represents the lower voltage limit of each generator load node, Indicates the voltage upper limit of each generator load node; ΔP i is the imbalance between active powers at node i, ΔQ i is the imbalance between reactive powers at node i, P i is the active power injected at node i, Q i is the reactive power injected at node i, V i is the voltage amplitude at node i, V j is the voltage amplitude at node j, G ijis the conductance of the branch, B ij is the branch susceptance; θ ij is the voltage phase angle difference; represents the active power of the generator, represents the active power of the load node, is the reactive power of the generator, Represents the reactive power of the load node, Δ>0 indicates a normal value close to 0; n represents the generator number; N represents the number of generators; m represents the load number; M represents the number of loads.

[0091] New Energy System Trend Divergence Rewards As shown below:

[0092]

[0093]

[0094] About Power Control Rewards As shown below:

[0095]

[0096] About Voltage Balance Control Rewards As shown below:

[0097]

[0098] in, The values ​​are all normalized negative values. The closer they are to 0, the better the system is running and the closer it is to the optimization target. R1 represents the power flow divergence reward function, R2 represents the power control reward function, and R3 represents the voltage balance reward function; e () represents exponential normalization; R min Represents the minimum reward during the system operation; R g represents the penalty imposed on the agent when power flow divergence occurs; ρ w represents the current transmission power of the transmission line w; and represents the rated maximum transmission power of the transmission line w, χ w is the penalty coefficient for exceeding the power flow limit of transmission line w; R0 is the default reward function reflecting the consistency of grid operation; w represents the number of power lines; W represents the number of power lines, P G Indicates the active power of the generator, Q G Indicates the reactive power of the generator.

[0099] (2) Economic dispatch reward

[0100] In the economic dispatch reward, the two main aspects involved are the unit operation cost and the abandoned wind and solar power cost. The corresponding evaluation indicators are normalized and converted into the corresponding reward function.

[0101] Rewards for abandoned wind and solar power costs As shown below:

[0102]

[0103] Unit operating expense reward As shown below:

[0104]

[0105] in, The values ​​are all normalized negative values. The closer they are to 0, the better the current system operation status is and the closer it is to the optimization target. R4 represents the power generation cost reward function, and R5 represents the wind and solar abandonment cost reward function; a n represents the cost coefficient of the quadratic term, b n represents the cost coefficient of the first-order term, c n represents the cost coefficient of the constant term; Indicates the active power of the generator; c q Indicates the unit cost coefficient of abandoned wind and solar power; Indicates the maximum output value of the new energy generator; P new Indicates the actual output value of the new energy generator; n new Indicates the number of the new energy generator; N new Indicates the number of renewable energy generators.

[0106] (3) Low-carbon scheduling rewards

[0107] In low-carbon scheduling rewards, the two main aspects involved are renewable energy output and carbon emissions. The corresponding evaluation indicators are normalized and converted into corresponding reward functions.

[0108] Renewable Energy Output Incentive As shown below:

[0109]

[0110]

[0111] Carbon Emissions Rewards As shown below:

[0112]

[0113] in, The value is between 0 and 1. The closer it is to 1, the higher the utilization rate of renewable energy in the current system. The value is between -1 and 0. The closer it is to 0, the lower the carbon emissions of the current system. R6 represents the renewable energy output reward function, and R7 represents the carbon emissions reward function; c p P represents the unit cost coefficient of conventional units; con Indicates the actual output value of a conventional generator; n con Indicates the number of conventional generators; N con Indicates the number of conventional generators.

[0114] 3.2 Constraint-based Rewards

[0115] Since the above-mentioned target-based scheduling rewards are mostly negative, in order to enable the agent to fully explore the simulated environment, when all target constraints are met, positive rewards are used to strengthen the encouragement of the agent's exploration in terms of meeting the constraints. This plays a second role in terms of grid security, and prevents over-limits at the action selection point, and appropriately encourages exploration at the reward point, ensuring a balance between exploration and utilization:

[0116]

[0117] Among them, λ is a fixed reward value, For constraint-based rewards, C1 to C5 represent the constraints of the new energy system.

[0118] Based on the above 8 reward functions, the weighted reward value finally constructed for the reinforcement learning agent is as follows:

[0119]

[0120] Among them, r is the weighted reward value, α i is the calculation result of the normalized reward value, is the weight coefficient of each reward item, and the weight coefficient is usually in the range of 0 to 1.

[0121] 4) A3C-based neural network training and algorithm flow

[0122] The A3C algorithm used in the present invention adopts an Actor-Critic network structure, which consists of two networks, namely, the actor network and the critic network. The goal of the actor network is to learn the policy function, that is, the probability distribution of selecting an action under a given state. The goal of the critic network is to learn the state value function or the state-action value function, which is used to evaluate the value of different states or state-action pairs. The training framework of the A3C algorithm used in the present invention is as follows: Figure 5 shown.

[0123] The A3C algorithm adopted by the present invention adopts an asynchronous training method, each thread interacts with the environment independently, and realizes gradient update through parameter sharing. In addition to the initialized neural network parameters, multiple parallel training threads are created, each thread independently runs an intelligent agent to interact with the environment, and uses the Actor and Critic network to realize the approximation of the strategy and value function. Each thread independently interacts with the environment of the power grid, selects the scheduling action after action screening according to the current strategy network, and observes the new state and reward according to the feedback of the power grid environment, and stores this information in the local experience replay buffer. When a thread reaches a predetermined number of time steps or the end of the trajectory, the thread samples data from the experience replay buffer and performs gradient update by calculating the advantage function. After each thread performs several gradient updates, the updated parameters are passed to the main thread for global parameter update. Repeat the above process until the predetermined training round is reached or the termination condition is met. This training method can improve the efficiency and stability of training, and can learn better strategies and value functions.

[0124] The specific algorithm flow in the present invention is as follows:

[0125] Algorithm input part: In the global A3C neural network, the corresponding parameter based on the policy π is θ, and the corresponding parameter based on the value network is ω; in the A3C neural network of the branch thread, the corresponding parameters are θ′, ω′, the global shared iteration round is T, and the global maximum number of iterations is T max , the action space is Ω t , attenuation factor γ;

[0126] Algorithm output part: power grid dispatch strategy π and value function V.

[0127] 1: Update time series t=1.

[0128] 2: Reset the gradient updates dθ←0, dω←0 of the global Actor network and the global Critic network. The purpose of this step is to clear the previously accumulated gradient information at the beginning of each new iteration to ensure that the new calculation is not interfered by the old gradient.

[0129] 3: Synchronize specific parameters from the global Actor network and the global Critic network to the neural network of the branch thread θ′=θ, ω′=ω. This step ensures that the neural network of the branch thread has the same initial parameters as the global network.

[0130] 4: Initialize t start = t, and get the state s of time series t t .

[0131] 5: Based on strategy π(Ωt |s t ;θ) Execute actionΩ t .

[0132] 6: Perform action space screening to detect whether the action exceeds the limit.

[0133] 7: Execute action Ω t Get multiple target rewards r t and the next state s given by the simulation environment t+1 .

[0134] 8: Update the time step, let t←t+1,T←T+1.

[0135] 9: If s t If it is the terminal state, or the time series t reaches the maximum value, then go to step 10, otherwise go back to step 510: calculate the position s at the last time series t t R(s,t), where:

[0136]

[0137] 11: From back to front, for each time series i∈(t-1, t-2, ..., t start ), do the following:

[0138] 12: Calculate R(s,i)=r at each moment i +γR(s,i+1).

[0139] 13: Accumulate the gradient update of Actor:

[0140]

[0141] 14: Accumulated Critic gradient update:

[0142]

[0143] 15: If i = t start , then go to step 16, otherwise go to step 11.

[0144] 16: Update the model parameters of the global neural network: Use dθ to perform asynchronous updates of θ and use dω to perform asynchronous updates of ω.

[0145] 17: If T>T max , end the loop and output the global A3C neural network parameters θ and ω, otherwise go to step 3.

[0146] The present invention compares the proposed A3C algorithm with the Random algorithm, i.e., a random power grid dispatching algorithm. The Random algorithm randomly selects a dispatching action when the power flow exceeds the limit without considering any strategy.

[0147] (1) Comparison of average cumulative returns

[0148] This paper uses the A3C algorithm to conduct simulation experiments on the IEEE-5 and IEEE-36 datasets, and calculates the average cumulative return of each episode. The results are shown in the figure:

[0149] The reward function value of the multi-objective is a large negative number at the beginning. During the optimization process, it gradually converges to a positive value after multiple oscillations. This proves the effectiveness of the algorithm.

[0150] (2) Power outage time comparison

[0151] Since IEEE-5 lacks renewable energy generators, this paper uses the IEEE-36 dataset to compare the sustainability of new energy systems and randomly selects 6 sets of time series data as test sets.

[0152] It can be seen that the Random algorithm experienced power outages in all five data segments, and the power outage time accounted for more than 50%. However, the A3C algorithm after weight adjustment had no power outages, which effectively achieved safe operation of the system, demonstrating the good adaptability of the algorithm combined with weight adjustment, which can guarantee the safe operation of the new energy system to the greatest extent.

[0153] The IEEE-5 node dataset includes data related to 5 substations, 8 transmission lines, 2 units, and 3 loads.

[0154] The IEEE-36 node data includes relevant data of 36 substations, 59 transmission lines, 22 generators (including 12 wind, light and other renewable energy generators), and 37 loads.

[0155] Actual case analysis and model effect verification, taking a provincial power grid SG-126 standard example and its operation as the experimental object, the figure shows the power grid topology of the SG-126 standard example, which includes 54 generators (18 new energy units), 91 loads (including ordinary loads and adjustable loads) and 5 energy storage devices, 126 buses and 185 branches, a total of 145 nodes. The proportion of new energy installed capacity exceeds 30%. In addition to various types of generators, the adjustable units of this example include energy storage devices and adjustable loads, which have a larger decision space. In addition, the SG-126 example is based on the volatility of new energy output in the real power grid, load curve characteristics, energy storage and flexible load control strategies, and simulates the open characteristics of line congestion, line random failures, weather changes, etc. in the real power grid environment, covering typical power grid operation scenarios such as tie line congestion, tie line N-1 failure, source load drastic fluctuations, and new energy power restrictions. It contains 100,000 consecutive converged AC flow sections and prediction data for one year, which further increases the difficulty of decision-making.

[0156] In order to verify the effectiveness of this technical solution, an actual case analysis and model effect verification were carried out on the example. The experiment was based on the actual operation data of the power grid, and the improved A3C algorithm was used for training and testing, and the evaluation index was the average cumulative return. The experimental process includes data preprocessing, model training and model testing. In the training stage, the double insurance mechanism of action screening and constraint restrictions combined with rewards is used to prevent constraint violations; in the testing stage, the performance of the model in different scenarios is evaluated. The results show that as the training progresses, the average cumulative return of the model gradually increases, and finally reaches a high level in the testing stage, proving the effectiveness and robustness of the model. The experimental results show that the A3C algorithm using this technical solution performs well on the SG-126 power grid, can effectively optimize action space screening, and prevent constraint violations, further verifying the feasibility and effectiveness of this technical solution in practical applications, especially the innovation and uniqueness in data analysis, action space screening and constraint processing.

[0157] Example 2

[0158] The present invention proposes a power grid simulation environment dispatch optimization system based on reinforcement learning, such as Figure 6 As shown, including:

[0159] A power grid parameter acquisition module, wherein the power grid parameter acquisition module is used to construct a state space of a power grid simulation environment and an action space of a power grid simulation environment;

[0160] The state space of the power grid simulation environment and the action space of the power grid simulation environment are constructed as follows:

[0161] Acquire numerical features and non-numerical features, and construct the state space of the power grid simulation environment based on the numerical features and non-numerical features; wherein the numerical features include generator active power, generator reactive power, load active power, load reactive power, predicted load, line power and constrained power; and the non-numerical features include line disconnection status;

[0162] In the construction of the action space of the power grid simulation environment, a passive discriminator is set to run in the environment. The specific process of the passive discriminator's judgment is as follows: after the agent takes an operation control action, all the system power flow constraints are calculated while calculating the reward; if the action taken by the agent causes the constraint to exceed the bounds, the probability of each action is arranged in descending order, and the operation control action with the second highest action probability is selected until the agent finds an action that meets the constraint conditions; if all actions cannot meet the constraint conditions, the agent executes the action that maximizes the current reward function.

[0163] A reward function construction module, wherein the reward function construction module is used to construct a target-based reward function and a constraint-based reward function according to an action space of a power grid simulation environment and a state space of a power grid simulation environment;

[0164] According to the action space and state space of the power grid simulation environment, a goal-based reward function is constructed, which is as follows:

[0165] The target-based reward function includes the first dispatching reward, second dispatching reward and third dispatching reward of the new energy system. The first dispatching reward of the grid includes the power divergence reward, power control reward and voltage balance control reward; the second dispatching reward of the grid includes the abandoned wind and solar cost reward and the unit operation cost reward; the third dispatching reward of the grid includes the renewable energy output reward and the carbon emission reward;

[0166] Among them, the constraints of the new energy system are:

[0167]

[0168] Among them, C1 to C5 represent constraints; represents the lower limit of active power output of each generator, represents the upper limit of active power output of each generator, represents the lower limit of reactive power output of each generator, Represents the upper limit of reactive power output of each generator; represents the lower voltage limit of each generator node, represents the voltage upper limit of each generator node, represents the lower voltage limit of each generator load node, Indicates the voltage upper limit of each generator load node; ΔPi is the imbalance between active powers at node i, ΔQ i is the imbalance between reactive powers at node i, P i is the active power injected at node i, Q i is the reactive power injected at node i, V i is the voltage amplitude at node i, V j is the voltage amplitude at node j, G ij is the conductance of the branch, B ij is the branch susceptance; θ ij is the voltage phase angle difference; represents the active power of the generator, represents the active power of the load node, Represents the reactive power of the generator, Represents the reactive power of the load node, Δ>0 indicates a normal value close to 0; n represents the generator number; N represents the number of generators; m represents the load number; M represents the number of loads.

[0169] The trend divergence reward As shown below:

[0170]

[0171] The power control rewards As shown below:

[0172]

[0173] The voltage balancing control rewards As shown below:

[0174]

[0175]

[0176] Among them, R1 represents the power flow divergence reward function, R2 represents the power control reward function, and R3 represents the voltage balance reward function; () represents exponential normalization; R min Represents the minimum reward during the system operation; R g represents the penalty imposed on the agent when power flow divergence occurs; ρ w represents the current transmission power of the transmission line w; and represents the rated maximum transmission power of the transmission line w, χ w is the penalty coefficient for exceeding the power flow limit of transmission line w; R0 is the default reward function reflecting the consistency of grid operation; w represents the number of power lines; W represents the number of power lines, P GIndicates the active power of the generator, Q G Indicates the reactive power of the generator.

[0177] The abandoned wind and solar cost reward As shown below:

[0178]

[0179] The unit operating cost reward As shown below:

[0180]

[0181] Among them, R4 represents the power generation cost reward function, and R5 represents the wind and solar abandonment cost reward function; a n represents the cost coefficient of the quadratic term, b n represents the cost coefficient of the first-order term, c n represents the cost coefficient of the constant term; Indicates the active power of the generator; c q Indicates the unit cost coefficient of abandoned wind and solar power; Indicates the maximum output value of the new energy generator; P new Indicates the actual output value of the new energy generator; n new Indicates the number of the new energy generator; N new Indicates the number of renewable energy generators.

[0182] Renewable Energy Output Incentive As shown below:

[0183]

[0184] Carbon Emissions Rewards As shown below:

[0185]

[0186] Among them, R6 represents the renewable energy output reward function, R7 represents the carbon emission reward function; c p P represents the unit cost coefficient of conventional units; con Indicates the actual output value of a conventional generator; n con Indicates the number of conventional generators; N con Indicates the number of conventional generators.

[0187] According to the action space and state space of the power grid simulation environment, a constraint-based reward function is constructed, which is as follows:

[0188]

[0189] Among them, λ is a fixed reward value, For constraint-based rewards, C1 to C5 represent the constraints of the new energy system.

[0190] The scheduling optimization module is used to obtain weighted reward values ​​according to a target-based reward function and a constraint-based reward function, and to perform scheduling optimization in a power grid simulation environment using an Actor-Critic network structure combined with the weighted reward values.

[0191] The weighted reward value is obtained according to the target-based reward function and the constraint-based reward function, as follows:

[0192]

[0193] Among them, r is the weighted reward value, α i is the calculation result of the normalized reward value, is the weight coefficient of each reward item.

[0194] Example 3

[0195] See also Figure 7 As shown, the present invention also provides an electronic device 100 for a power grid simulation environment scheduling optimization method based on reinforcement learning; the electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on the at least one processor 102, and at least one communication bus 104.

[0196] The memory 101 can be used to store the computer program 103, and the processor 102 implements the steps of the power grid simulation environment dispatch optimization method based on reinforcement learning described in Example 1 by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101. The memory 101 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data (such as audio data) created according to the use of the electronic device 100, etc. In addition, the memory 101 can include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices.

[0197] The at least one processor 102 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 102 may be a microprocessor or any conventional processor, etc. The processor 102 is the control center of the electronic device 100, and uses various interfaces and lines to connect various parts of the entire electronic device 100.

[0198] The memory 101 in the electronic device 100 stores a plurality of instructions to implement a method for optimizing power grid simulation environment scheduling based on reinforcement learning, and the processor 102 can execute the plurality of instructions to implement:

[0199] Construct the state space and action space of the power grid simulation environment;

[0200] Constructing a target-based reward function and a constraint-based reward function according to the action space and the state space of the power grid simulation environment;

[0201] The weighted reward value is obtained according to the goal-based reward function and the constraint-based reward function, and the Actor-Critic network structure combined with the weighted reward value is used to optimize the scheduling of the power grid simulation environment.

[0202] Example 4

[0203] If the module / unit integrated in the electronic device 100 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory and read-only memory (ROM, Read-Only Memory).

[0204] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0205] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0206] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0207] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0208] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for optimizing power grid simulation environment scheduling based on reinforcement learning, characterized in that: include: Construct the state space and action space of the power grid simulation environment; Constructing a target-based reward function and a constraint-based reward function according to the action space and the state space of the power grid simulation environment; The weighted reward value is obtained according to the goal-based reward function and the constraint-based reward function, and the Actor-Critic network structure combined with the weighted reward value is used to optimize the scheduling of the power grid simulation environment.

2. The power grid simulation environment scheduling optimization method based on reinforcement learning according to claim 1 is characterized in that: The state space of the power grid simulation environment and the action space of the power grid simulation environment are constructed as follows: Acquire numerical features and non-numerical features, and construct the state space of the power grid simulation environment based on the numerical features and non-numerical features; wherein the numerical features include generator active power, generator reactive power, load active power, load reactive power, predicted load, line power and constrained power; and the non-numerical features include line disconnection status; In the construction of the action space of the power grid simulation environment, a passive discriminator is set to run in the environment. The specific process of the passive discriminator's judgment is as follows: after the agent takes an operation control action, all the system power flow constraints are calculated while calculating the reward; if the action taken by the agent causes the constraint to exceed the bounds, the probability of each action is arranged in descending order, and the operation control action with the second highest action probability is selected until the agent finds an action that meets the constraint conditions; if all actions cannot meet the constraint conditions, the agent executes the action that maximizes the current reward function.

3. The power grid simulation environment scheduling optimization method based on reinforcement learning according to claim 1 is characterized in that: According to the action space and state space of the power grid simulation environment, a target-based reward function is constructed, which is as follows: The target-based reward function includes the first dispatching reward, second dispatching reward and third dispatching reward of the new energy system. The first dispatching reward of the grid includes the power divergence reward, power control reward and voltage balance control reward; the second dispatching reward of the grid includes the abandoned wind and solar cost reward and the unit operation cost reward; the third dispatching reward of the grid includes the renewable energy output reward and the carbon emission reward; Among them, the constraints of the new energy system are: in, to Indicates constraints; represents the lower limit of active power output of each generator, represents the upper limit of active power output of each generator, represents the lower limit of reactive power output of each generator, Represents the upper limit of reactive power output of each generator; represents the lower voltage limit of each generator node, represents the voltage upper limit of each generator node, represents the lower voltage limit of each generator load node, Indicates the voltage upper limit of each generator load node; Is a node i The imbalance between active power, Is a node i The imbalance between reactive power, Is a node i The active power injected at Is a node i The reactive power injected at Is a node i The voltage amplitude at Is a node j The voltage amplitude at is the conductance of the branch, is the branch circuit susceptance; is the voltage phase angle difference; represents the active power of the generator, represents the active power of the load node, is the reactive power of the generator, represents the reactive power of the load node, Indicates normal values ​​close to 0; Indicates the generator number; Indicates the number of generators; Indicates the number of the load; Indicates the amount of load.

4. The method for optimizing power grid simulation environment scheduling based on reinforcement learning according to claim 3 is characterized in that: The trend divergence reward r t 1 As shown below: The power control rewards r t 2 As shown below: The voltage balancing control rewards r t 3 As shown below: in, represents the power flow divergence reward function, represents the power control reward function, represents the voltage balance reward function; represents exponential normalization; Indicates the minimum reward during the operation of the system; represents the penalty imposed on the agent when power flow divergence occurs; Indicates transmission line w The current transmission power of Indicates transmission line w The rated maximum transmission power, Over the transmission line w Penalty factor for power flow limitation; As a default reward function that reflects the consistency of grid operation; w Indicates the number of the power line; represents the number of power lines, represents the active power of the generator, Indicates the reactive power of the generator.

5. The method for optimizing power grid simulation environment scheduling based on reinforcement learning according to claim 3, characterized in that: The abandoned wind and solar cost reward r t 4 As shown below: The unit operating cost reward r t 5 As shown below: in, represents the power generation cost reward function, represents the cost reward function for abandoned wind and solar power; represents the cost coefficient of the quadratic term, represents the cost coefficient of the linear term, represents the cost coefficient of the constant term; Indicates the active power of the generator; Indicates the unit cost coefficient of abandoned wind and solar power; Indicates the maximum output value of the new energy generator; Indicates the actual output value of the new energy generator; Indicates the number of the new energy generator; Indicates the number of renewable energy generators.

6. The method for optimizing power grid simulation environment scheduling based on reinforcement learning according to claim 3, characterized in that: Renewable Energy Output Incentive r t 6 As shown below: Carbon Emissions Rewards r t 7 As shown below: in, represents the renewable energy output reward function, represents the carbon emission reward function; It represents the unit cost coefficient of conventional units; Indicates the actual output value of conventional generators; Indicates the number of the conventional generator; Indicates the number of conventional generators.

7. The method for optimizing power grid simulation environment scheduling based on reinforcement learning according to claim 1, characterized in that: According to the action space and state space of the power grid simulation environment, a constraint-based reward function is constructed, which is as follows: in, is a fixed reward value, For constraint-based rewards, to Represents the constraints of the new energy system.

8. The method for optimizing power grid simulation environment scheduling based on reinforcement learning according to claim 1, characterized in that: The weighted reward value is obtained according to the target-based reward function and the constraint-based reward function, as follows: in, is the weighted reward value, is the calculation result of the normalized reward value, is the weight coefficient of each reward item.

9. A power grid simulation environment dispatch optimization system based on reinforcement learning, characterized in that: include: A power grid parameter acquisition module, wherein the power grid parameter acquisition module is used to construct a state space of a power grid simulation environment and an action space of a power grid simulation environment; A reward function construction module, wherein the reward function construction module is used to construct a target-based reward function and a constraint-based reward function according to an action space of a power grid simulation environment and a state space of a power grid simulation environment; The scheduling optimization module is used to obtain weighted reward values ​​according to a target-based reward function and a constraint-based reward function, and to perform scheduling optimization in a power grid simulation environment using an Actor-Critic network structure combined with the weighted reward values.

10. The power grid simulation environment dispatch optimization system based on reinforcement learning according to claim 9, characterized in that: According to the action space and state space of the power grid simulation environment, a goal-based reward function is constructed, which is as follows: The target-based reward function includes the first dispatching reward, second dispatching reward and third dispatching reward of the new energy system. The first dispatching reward of the grid includes the power divergence reward, power control reward and voltage balance control reward; the second dispatching reward of the grid includes the abandoned wind and solar cost reward and the unit operation cost reward; the third dispatching reward of the grid includes the renewable energy output reward and the carbon emission reward; Among them, the constraints of the new energy system are: in, to Indicates constraints; represents the lower limit of active power output of each generator, represents the upper limit of active power output of each generator, represents the lower limit of reactive power output of each generator, Represents the upper limit of reactive power output of each generator; represents the lower voltage limit of each generator node, represents the voltage upper limit of each generator node, represents the lower voltage limit of each generator load node, Indicates the voltage upper limit of each generator load node; Is a node i The imbalance between active power, Is a node i The imbalance between reactive power, Is a node i The active power injected at Is a node i The reactive power injected at Is a node i The voltage amplitude at Is a node j The voltage amplitude at is the conductance of the branch, is the branch circuit susceptance; is the voltage phase angle difference; represents the active power of the generator, represents the active power of the load node, is the reactive power of the generator, represents the reactive power of the load node, Indicates normal values ​​close to 0; Indicates the generator number; Indicates the number of generators; Indicates the number of the load; Indicates the amount of load.

11. The power grid simulation environment dispatch optimization system based on reinforcement learning according to claim 10, characterized in that: The trend divergence reward r t 1 As shown below: The power control rewards r t 2 As shown below: The voltage balancing control rewards r t 3 As shown below: in, represents the power flow divergence reward function, represents the power control reward function, represents the voltage balance reward function; represents exponential normalization; Indicates the minimum reward during the operation of the system; represents the penalty imposed on the agent when power flow divergence occurs; Indicates transmission line w The current transmission power of Indicates transmission line w The rated maximum transmission power, Over the transmission line w Penalty factor for power flow limitation; As a default reward function that reflects the consistency of grid operation; w Indicates the number of the power line; represents the number of power lines, represents the active power of the generator, Indicates the reactive power of the generator.

12. The power grid simulation environment dispatch optimization system based on reinforcement learning according to claim 10, characterized in that: The abandoned wind and solar cost reward r t 4 As shown below: The unit operating cost reward r t 5 As shown below: in, represents the power generation cost reward function, represents the cost reward function for abandoned wind and solar power; represents the cost coefficient of the quadratic term, represents the cost coefficient of the linear term, represents the cost coefficient of the constant term; Indicates the active power of the generator; Indicates the unit cost coefficient of abandoned wind and solar power; Indicates the maximum output value of the new energy generator; Indicates the actual output value of the new energy generator; Indicates the number of the new energy generator; Indicates the number of renewable energy generators.

13. The power grid simulation environment dispatch optimization system based on reinforcement learning according to claim 10, characterized in that: Renewable Energy Output Incentive r t 6 As shown below: Carbon Emissions Rewards r t 7 As shown below: in, represents the renewable energy output reward function, represents the carbon emission reward function; It represents the unit cost coefficient of conventional units; Indicates the actual output value of conventional generators; Indicates the number of the conventional generator; Indicates the number of conventional generators.

14. The power grid simulation environment dispatch optimization system based on reinforcement learning according to claim 9, characterized in that: According to the action space and state space of the power grid simulation environment, a constraint-based reward function is constructed, which is as follows: in, is a fixed reward value, For constraint-based rewards, to Represents the constraints of the new energy system.

15. The power grid simulation environment dispatch optimization system based on reinforcement learning according to claim 9, characterized in that: The weighted reward value is obtained according to the target-based reward function and the constraint-based reward function, as follows: in, is the weighted reward value, is the calculation result of the normalized reward value, is the weight coefficient of each reward item.

16. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the power grid simulation environment scheduling optimization method based on reinforcement learning are implemented as described in any one of claims 1 to 8.

17. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, the steps of the power grid simulation environment scheduling optimization method based on reinforcement learning are implemented as described in any one of claims 1 to 8.

Images