Intra-day ideal scheduling intelligent decision-making method based on safe deep reinforcement learning

Abstract

The invention discloses an intra-day ideal scheduling intelligent decision-making method based on safety deep reinforcement learning, and belongs to the technical field of combination of an intelligent power grid and artificial intelligence, and the method comprises the steps: obtaining a day-ahead scheduling plan set, and obtaining an intra-day scheduling plan set according to intra-day new energy output data, load data and meteorological observation data; constructing a reward function and a cost function which are used by a decision-making agent according to intra-day operation requirements of a power grid; based on a safe deep reinforcement learning theory, training to obtain an intra-day ideal scheduling decision agent; and generating a power grid real-time scheduling strategy in a rolling manner by using the trained intra-day ideal decision-making agent. The method can realize on-line generation of the intra-day scheduling plan considering the safe and economic operation requirement of the power grid, thereby supporting the intra-day scheduling decision of the power grid.

Description

Technical Field

[0001] The present invention belongs to the technical field of the combination of smart grid and artificial intelligence, and specifically relates to an intelligent decision-making method for intraday ideal scheduling based on secure deep reinforcement learning. Background Art

[0002] In recent years, global climate and energy security issues have become increasingly severe, and countries around the world are actively seeking response strategies. New energy sources such as wind and light, with their clean and renewable advantages, are an important boost to achieving the "dual carbon" goals. Building a new power system with new energy sources such as wind and light as the main body has become an important means of low-carbon transformation in the power industry.

[0003] However, in the process of power system transformation, the uncertainty problem of power system has become increasingly prominent. On the energy side, a high proportion of new energy sources such as wind power and photovoltaics are connected to the power grid. Since their power generation capacity is greatly affected by weather conditions, the output shows strong uncertainty and randomness, which brings challenges to the stability of the power grid. On the load side, with the large-scale access of electric vehicles and the rapid development of distributed photovoltaic and energy storage equipment on the user side, the characteristics of power load have become more complex, showing strong randomness and uncertainty. Therefore, in the process of low-carbon transformation of the power system, how to deal with the uncertainty on both the source and load sides and ensure the safe, stable and economical operation of the power system has become a key issue that needs to be solved urgently.

[0004] In order to solve the uncertainty problems on both sides of the source and the load, the commonly used methods include fuzzy optimization, stochastic optimization, robust optimization, and distributed robust optimization. However, these uncertain optimization methods are computationally complex and take a long time to solve, and the final results are static optimization results. In practical applications, they still need to be dynamically adjusted according to the actual situation.

[0005] In order to solve the problems existing in these uncertain optimization methods, the scheduling plan generation method based on artificial intelligence has become one of the research hotspots. Currently, the commonly used artificial intelligence methods are deep learning and deep reinforcement learning. Deep learning requires a large number of training samples, but the AC power flow model is non-convex and it is difficult to generate a large number of high-quality training samples. Deep reinforcement learning can learn the optimal strategy without prior knowledge through continuous exploration, but there are problems such as long training time, low training efficiency, and poor interpretability. How to improve the training efficiency of deep reinforcement learning and the reliability of generating scheduling strategies is one of the main issues of concern in the field of smart grids.

[0006] Patent document CN114662798B discloses a dispatching method, device, and electronic equipment based on the economic operation domain of a power grid, including determining the uncertainty factors and dispatching targets that need to be considered in formulating a power grid dispatching plan; rolling depicting the economic operation domain of the power grid according to the uncertainty factors and dispatching targets, and selecting the form of the economic operation domain of the power grid according to actual engineering needs; and calculating the optimal dispatching plan in real time based on the economic operation domain of the power grid according to the form of the economic operation domain of the power grid. There may be some situations in the economic operation domain in this method that do not meet the safety constraints, and the final output action needs to be corrected. Summary of the invention

[0007] In view of the above-mentioned technical problems existing in the prior art, the purpose of the present invention is to provide an intelligent decision-making method for ideal intra-day scheduling based on safe deep reinforcement learning, which improves the reliability of intelligent body actions while improving the training efficiency of intelligent bodies.

[0008] To achieve the above-mentioned purpose of the invention, the embodiment provides an intelligent decision-making method for ideal intraday scheduling based on secure deep reinforcement learning, comprising the following steps:

[0009] The set of day-ahead dispatch plans of the power grid is obtained, and the day-ahead dispatch plan set is corrected in combination with the intra-day renewable energy output data, load data and meteorological observation data to obtain the intra-day dispatch plan set.

[0010] According to the requirements of daily operation of the power grid, the reward function and cost function for training the intelligent agent are constructed.

[0011] Based on the theory of safe deep reinforcement learning, the constructed reward function and cost function are used to train the intraday ideal scheduling decision-making agent.

[0012] The trained intraday ideal scheduling decision-making agent is used in combination with the intraday scheduling plan set to generate the intraday ideal scheduling strategy in a rolling manner.

[0013] In one embodiment, the method also includes: further narrowing the source-load uncertainty range based on intra-day renewable energy output data, load data and meteorological observation data, and correcting the day-ahead scheduling plan set in combination with physical constraints such as power balance constraints and flow constraints, eliminating scheduling plans that do not meet safety constraints, and obtaining an intra-day scheduling plan set.

[0014] In one embodiment, the method includes: starting from the perspectives of minimizing operating costs, minimizing carbon emissions, maximizing new energy consumption, etc., according to the actual needs of the daily operation of the power grid, constructing a suitable reward function for the training of an ideal scheduling decision-making agent within the day.

[0015] In one embodiment, the method includes: constructing a cost function based on the corrected intraday scheduling plan set, and providing a larger cost when the action performed by the agent is not within the scope of the scheduling plan set.

[0016] In one embodiment, the training of an ideal scheduling decision agent for the day based on the safe deep reinforcement learning theory includes: introducing a safe deep reinforcement learning method and using the agent's exploration ability to optimize power grid scheduling. This method takes into account physical safety constraints such as power flow restrictions, node voltage restrictions, and line transmission power restrictions while ensuring economic operation.

[0017] In one embodiment, the method includes: constructing a reward function and a cost function, training an intelligent agent, so that the intelligent agent can maximize the economic benefits of the power grid under the premise of satisfying safety constraints. Through multiple rounds of training, the intelligent agent can learn a safe and economical power grid dispatching strategy.

[0018] In one embodiment, the ideal scheduling result within a day is a scheduling plan that meets various safety constraints of the power grid, faces uncertainty on both the source and load sides, and maximizes economic benefits. The economic benefits include: minimizing operating costs, minimizing carbon emissions, maximizing new energy consumption, etc.

[0019] In one embodiment, the method for generating an ideal intraday scheduling result includes: using a trained ideal intraday scheduling decision agent and combining it with an intraday scheduling plan set to generate an ideal intraday scheduling strategy in a rolling manner.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] Based on the intraday dispatch plan set obtained by correcting the day-ahead dispatch plan set by combining the intraday renewable energy output data, load data, and meteorological observation data, the intraday ideal dispatch decision-making agent is trained based on the safe deep reinforcement learning theory. That is, a cost function is constructed based on the intraday dispatch plan set during the training process. This can improve the training efficiency of the agent and enable the agent to quickly generate ideal dispatch results to support the intraday dispatch decision-making of the power grid. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1It is a flow chart of an intraday ideal scheduling intelligent decision-making method based on secure deep reinforcement learning provided by an embodiment;

[0024] Figure 2 is a schematic diagram of a constrained Markov decision process provided by an embodiment;

[0025] Figure 3 It is a schematic diagram of the structure of an ideal intraday decision-making intelligent agent provided in an embodiment. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention more clear, the present invention is further described in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the protection scope of the present invention.

[0027] In view of the problems that traditional uncertain optimization methods take a long time to calculate ideal scheduling results online and have low reliability, this example provides an intelligent decision-making method for intraday ideal scheduling based on secure deep reinforcement learning, which can improve the reliability of the solution while increasing the speed of solution.

[0028] like Figure 1 As shown, the intraday ideal scheduling intelligent decision-making method based on secure deep reinforcement learning provided by the embodiment includes the following steps:

[0029] S110: Obtain a set of day-ahead dispatch plans of the power grid, and modify the day-ahead dispatch plan set in combination with the intra-day renewable energy output data, load data, and meteorological observation data to obtain an intra-day dispatch plan set.

[0030] In the embodiment, the day-ahead scheduling plan set used can be expressed in the following form:

[0031]

[0032] Among them, S t is the dispatching plan set of the power grid at time t; u t is the uncertainty variable of the power grid at time t, which is the uncertainty variable on both the source and load sides in this application; U t is the uncertainty variable u t The collection of x t is the power grid at time t, and the uncertainty variable is u t The economic dispatch result of power grid at this time; t is the economic dispatch model of the power grid at time t.

[0033] Furthermore, in one embodiment, the day-ahead scheduling plan set may be described in the form of a domain, and the day-ahead scheduling plan set may be characterized by a half-space representation:

[0034] ψEOR,t ={x G,t |cf t x G,t +df t ≤0}

[0035] Among them, ψ EOR,t That is the day-ahead scheduling plan set; cf t is a coefficient matrix whose number of rows is equal to the number of hyperplanes constituting the domain, and whose number of columns is equal to the number of dispatchable units; df t is a column vector with the same number of rows as cf t Same; x G,t It is a column vector consisting of the active and reactive outputs of all dispatchable units at time t.

[0036] In one implementation, the day-ahead scheduling plan may be modified in the following manner to obtain a set of intra-day scheduling plans:

[0037] First, construct the following series of physical constraints:

[0038] Based on the daily renewable energy output data, load forecast data and meteorological observation data, the source-load bilateral uncertainty interval is constructed in the following form:

[0039]

[0040] in, For the contribution of new energy w; is the power of load d; are the minimum and maximum output of the new energy w respectively; are the minimum and maximum power of load d respectively.

[0041] Other physical constraints include unit active output constraints, unit reactive output constraints, node voltage amplitude constraints, node voltage phase angle constraints, line transmission power capacity constraints, active power flow constraints, and reactive power flow constraints:

[0042]

[0043] Among them, P gi , Q gi are respectively the active output and reactive output of the i-th generator, and the generators include dispatchable units and new energy units; are the upper and lower limits of the active output of the i-th generator respectively; are the upper and lower limits of the reactive power output of the i-th generator; V i is the voltage amplitude at node i; are the upper and lower limits of the node voltage amplitude respectively; δ i is the voltage phase angle at node i; are the upper and lower limits of the node voltage phase angle respectively; |S l | is the transmission power of branch l; is the upper limit of the transmission power of branch l; P di , Q di The active power and reactive power of load i respectively; G ij , B ij , δ ij are the conductance, susceptance, and voltage phase angle difference of the path between node i and node j, δ ij =δ i -δ j .

[0044] Based on the above constraints, the safe operation domain of the power grid during the day can be obtained as shown in the following formula:

[0045] ψ safe,t ={x G,t |Satisfy the above constraints}

[0046] Finally, combined with the power grid day-ahead dispatch plan set ψ EOR,t The safe operation domain of the power grid during the day safe,t The grid daily dispatch plan set ψ can be obtained t =ψ EOR,t ∩ψ safe,t .

[0047] S120: Construct a reward function and a cost function for training the intelligent agent according to the requirements of the daily operation of the power grid.

[0048] In one embodiment, a reward function for training an agent may be constructed based on the cost of electricity generation.

[0049] The power generation cost of the power grid can be expressed as follows:

[0050]

[0051] Where cost is the power generation cost of the power grid; N G is the total number of dispatchable units; is the active output of dispatchable unit i; a i 、b i 、c i They are respectively the quadratic coefficient, linear coefficient and constant term of the power generation cost of dispatchable unit i.

[0052] Since the learning goal of deep reinforcement learning is to maximize the reward, and the goal of ideal scheduling is to minimize the power generation cost, the reward function r t Negatively correlated with the power generation cost function:

[0053] rt =-cost / 100

[0054] In one embodiment, a cost function for training an agent can be constructed based on a set of daily scheduling plans. When the agent outputs an action within the set of daily scheduling plans, the cost is 0; when the agent outputs an action outside the set of daily scheduling plans, the cost function will generate a larger cost.

[0055]

[0056] S130: Based on the theory of safe deep reinforcement learning, the constructed reward function and cost function are used to train the intraday ideal scheduling decision-making agent.

[0057] In one embodiment, based on the theory of secure deep reinforcement learning, a structure such as Figure 2 The constrained Markov decision process shown is used to train the intraday ideal scheduling decision agent.

[0058] The main elements of a constrained Markov process include environment, agent, state, action, reward and cost. The special thing is that the cost needs to be minimized to meet specific constraints during training and testing. t Execute action a t , after interacting with the environment, we get immediate rewards and costs, which are calculated by the reward function and the cost function.

[0059] Through the secure deep reinforcement learning algorithm, the agent will learn a secure strategy to maximize the cumulative reward while the power grid is operating safely. Specifically, in one embodiment, the elements of the constrained Markov decision process are as follows:

[0060] Intelligent agent: Dispatcher, who controls the active and reactive output of each dispatchable unit.

[0061] Environment: The current operating environment of the power grid.

[0062] Status: active and reactive power demand of load, new energy output data, and daily dispatch plan collection.

[0063] Action: The active output and voltage of the unit can be adjusted.

[0064] Reward: A function designed based on the cost of electricity generation.

[0065] Cost: A function designed based on a set of intraday scheduling plans.

[0066] In one embodiment, the intraday ideal scheduling decision agent can be constructed by an actor-critic algorithm based on primal-dual optimization.

[0067] The constrained Markov decision process can be expressed as the following optimization problem:

[0068]

[0069] Among them, γ t is the discount factor; d is the safety limit; ρ π is the round distribution determined by the strategy π.

[0070] By performing Lagrangian relaxation on the above model, we can obtain:

[0071]

[0072] in, λ=(λ1,...,λ m ) is the Lagrange multiplier.

[0073] The original constrained optimization problem can be transformed into the following unconstrained optimization problem:

[0074]

[0075] In the executor-critic algorithm, the executor is a policy function responsible for executing actions and updating parameters through policy gradient ascent based on the feedback from the critic; the critic is a value function that evaluates the value of the action based on the actions taken by the executor and the observed rewards, and updates the parameters through gradient descent.

[0076] In the primal-dual optimization-based actor-critic algorithm, the actor (policy network) is updated as follows (taking the kth iteration as an example):

[0077] Fixed λ = λ (k) , perform policy gradient ascent:

[0078] Fixed π = π k , perform dual gradient ascent:

[0079] Among them, α k , β k is the update step size; [x] + =max{0,x}

[0080] In the actor-critic algorithm based on primal-dual optimization, the update method of the critic (value network) is as follows (taking the kth iteration as an example):

[0081]

[0082] Among them, η kis the update step size; V(st) is the current state value estimated by the value critic.

[0083] Through the above-mentioned policy network and value network parameter updating method, after multiple iterations, the obtained policy network is the ideal intraday scheduling decision-making intelligent agent.

[0084] S140: Utilize the trained intraday ideal scheduling decision agent and combine it with the intraday scheduling plan set to generate the intraday ideal scheduling strategy in a rolling manner.

[0085] In one embodiment, the state includes active and reactive power requirements of the load, new energy output data, and a set of intraday dispatch plans. Figure 3 As shown in the figure, the state is input into the intraday ideal scheduling decision-making agent, and the agent will give the intraday ideal scheduling strategy, including the active output and voltage of the dispatchable units, to support the intraday scheduling decision of the power grid.

Claims

1. An intelligent decision-making method for ideal intraday scheduling based on secure deep reinforcement learning, characterized in that: The following steps are involved: Step 1: Obtain a set of day-ahead dispatch plans of the power grid, and modify the day-ahead dispatch plan set in combination with the intra-day renewable energy output data, load data and meteorological observation data to obtain an intra-day dispatch plan set; Step 2: Based on the requirements of the daily operation of the power grid, construct the reward function and cost function for training the intelligent agent; Step 3: Based on secure deep reinforcement learning, use the constructed reward function and cost function to train the intraday ideal scheduling decision-making agent; Step 4: Use the trained intraday ideal scheduling decision-making agent and the intraday scheduling plan set to generate the intraday ideal scheduling strategy in a rolling manner.

2. The intelligent decision-making method for ideal scheduling within a day based on the economic operation domain of a power grid according to claim 1 is characterized in that: In step 1, the day-ahead dispatch plan set is modified according to physical constraints such as power balance constraints and flow constraints, and the dispatch plans that do not meet the safety constraints are eliminated to obtain the modified day-ahead dispatch plan set.

3. The intelligent decision-making method for ideal scheduling within a day based on the economic operation domain of a power grid according to claim 2 is characterized in that: The cost function is constructed based on the modified intraday scheduling plan set. When the action performed by the agent is not within the scope of the scheduling plan set, a larger cost is given.

4. The intelligent decision-making method for ideal scheduling within a day based on the economic operation domain of a power grid according to claim 1 is characterized in that: In step 2, the reward function is constructed based on minimizing operating costs, minimizing carbon emissions, maximizing new energy consumption, etc.

5. The intelligent decision-making method for ideal scheduling within a day based on the economic operation domain of a power grid according to claim 1 is characterized in that: In step three, a secure deep reinforcement learning method is introduced to utilize the exploration capability of intelligent agents to optimize grid dispatch, while ensuring economic operation, combined with physical safety constraints such as power flow restrictions, node voltage restrictions, and line transmission power restrictions.

6. The intelligent decision-making method for ideal scheduling within a day based on the economic operation domain of a power grid according to claim 1 is characterized in that: In step three, the intelligent agent is trained so that it can maximize the economic benefits of the power grid while meeting safety constraints. Through multiple rounds of training, the intelligent agent learns a safe and economical power grid dispatching strategy.

7. The intelligent decision-making method for ideal scheduling within a day based on the economic operation domain of a power grid according to claim 1 is characterized in that: The ideal intraday dispatching strategy described in step 4 is a dispatching plan that meets the uncertainties on both the source and load sides and maximizes economic benefits while satisfying various safety constraints of the power grid; the economic benefits include: minimizing operating costs, minimizing carbon emissions, and maximizing new energy consumption.

Images