Energy storage and multi-resource optimization scheduling method and system, electronic equipment and storage medium

Abstract

The invention discloses an energy storage and multi-resource optimization scheduling method and system, electronic equipment and a storage medium, and the method comprises the steps: obtaining the historical operation data of a power system, and constructing an energy storage and multi-resource optimization scheduling model; defining an energy storage and multi-resource optimization scheduling model as a Markov decision process, and giving a state space, an action space and a reward function; packaging the defined state space, action space and reward function of the Markov decision process into a reinforcement learning environment; based on historical operation data of the power system, interactive learning is carried out through a reinforcement learning environment and a reinforcement learning algorithm, agent actions are executed, an environment state is updated, a reward value is calculated, and an agent is trained; and inputting the trained intelligent agent into the energy storage and multi-resource optimization scheduling model to obtain an optimal scheduling scheme. According to the method, the adaptive capacity to the uncertainty of new energy can be improved, and the optimal scheduling scheme can be quickly given online.

Description

Technical Field

[0001] The present invention relates to the technical field of power systems, and in particular to an energy storage and multi-resource optimization scheduling method, system, electronic equipment and storage medium. Background Art

[0002] With the large-scale access of new energy to the power system, the volatility and intermittency of new energy output make the operation of the power system increasingly complex. Due to the existence of various uncertain factors on different time scales, the performance and behavior of the system in different time periods cannot be fully predicted and determined, which brings new challenges to the operation of the power system. The dispatch flexibility provided by traditional generators is difficult to meet the dispatch needs of new power systems. With the continuous development of energy storage, its rapid adjustment and forward and reverse charging and discharging capabilities make it have broad application prospects in the optimization and dispatch of power systems.

[0003] The traditional dispatching method is deterministic dispatching, which regards the load forecast value and the new energy forecast value as accurate values. However, with the increasing number of uncertainties in the power system, the traditional deterministic optimization dispatching method can no longer meet the dispatching needs of the new power system. Improving the adaptability of the dispatching method to the uncertainty of new energy is a problem that must be solved in the current optimization dispatching. Summary of the invention

[0004] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0005] In view of the above existing problems, the present invention is proposed.

[0006] Therefore, the present invention provides an energy storage and multi-resource optimization scheduling method, system, electronic device and storage medium to solve the problem that traditional scheduling methods have poor adaptability to the uncertainty of new energy.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] In a first aspect, the present invention provides an energy storage and multi-resource optimization scheduling method, comprising:

[0009] Obtain historical operation data of the power system and build energy storage and multi-resource optimization dispatch models;

[0010] The energy storage and multi-resource optimization scheduling model is defined as a Markov decision process, and the state space, action space and reward function are given;

[0011] Encapsulate the state space, action space and reward function of the defined Markov decision process into a reinforcement learning environment;

[0012] Based on the historical operation data of the power system, the agent is trained by interactive learning between the reinforcement learning environment and the reinforcement learning algorithm, executing the agent action, updating the environment state, calculating the reward value, and training the agent.

[0013] The trained intelligent agent is input into the energy storage and multi-resource optimization scheduling model to obtain the optimal scheduling plan.

[0014] As a preferred solution of the energy storage and multi-resource optimization scheduling method described in the present invention, wherein:

[0015] The method of interactively learning with the reinforcement learning environment and the reinforcement learning algorithm, executing the agent action, updating the environment state, calculating the reward value, and training the agent includes the following steps:

[0016] Define the state space dimensions, action space dimensions and parameters of each energy storage, thermal power unit, hydropower unit and DC in the power system;

[0017] Receive action instructions from the agent, update the environment state according to the action instructions, and calculate the reward value based on the new state;

[0018] Reset the reinforcement learning environment, initialize all methods and variables, and randomly read the power grid operation data of a certain day, and use the initial section as the initial state of the environment for the current training round;

[0019] Get the current environment state and pass it to the agent;

[0020] Determine whether to end the interaction with the agent in the current training round;

[0021] If the power flow calculation does not converge, the round ends;

[0022] If the number of interactions reaches the preset maximum number of interactions, the round ends.

[0023] As a preferred solution of the energy storage and multi-resource optimization scheduling method described in the present invention, wherein:

[0024] The construction of the energy storage and multi-resource optimization scheduling model includes determining the objective function and constraints of the energy storage and multi-resource optimization scheduling model. The objective function of the energy storage and multi-resource optimization scheduling model is expressed as:

[0025]

[0026] Among them, T is the total number of optimized scheduling periods, F G,t is the operating cost of thermal power units, F hp,tis the operating cost of the hydropower unit, F tl,t is the DC operating cost, F E,t The operating cost of energy storage.

[0027] The constraints of the energy storage and multi-resource optimization scheduling model include power flow constraints, thermal power unit output constraints, thermal power unit climbing constraints, thermal power unit start and shutdown constraints, energy storage output upper and lower limit constraints, energy storage charge state constraints, hydropower unit output constraints, hydropower unit climbing constraints, DC transmission power constraints, DC transmission power adjustment rate constraints, power constraints that cannot be adjusted in the opposite direction during adjacent time periods, DC adjustment number limit constraints, power balance constraints, system standby constraints, system voltage safe operation constraints and line transmission capacity constraints.

[0028] As a preferred solution of the energy storage and multi-resource optimization scheduling method described in the present invention, wherein:

[0029] The state space representation of the Markov decision process is:

[0030]

[0031] in, The output power of each energy storage in the current period t, The output power of each thermal power unit in the current period t, is the output power of each hydropower unit in the current period t, is the DC transmission power of each period in the current t period, is the upward spinning reserve provided by the system during period t, is the downward spinning reserve provided by the system during period t, and They are respectively the prediction sequences of load size and wind power output for the next multiple time periods.

[0032] As a preferred solution of the energy storage and multi-resource optimization scheduling method described in the present invention, wherein:

[0033] The action space of the Markov decision process is expressed as:

[0034]

[0035] in, The change in output of each energy storage during the current dispatch period compared with the previous dispatch period; The change in output of each thermal power unit during the current dispatch period compared with the previous dispatch period; The change in output of each hydropower unit during the current dispatch period compared with the previous dispatch period; It is the change in power delivered during the current DC scheduling period compared with the previous scheduling period.

[0036] As a preferred solution of the energy storage and multi-resource optimization scheduling method described in the present invention, wherein:

[0037] The reward function of the Markov decision process is expressed as:

[0038] r t =a1r1+a2r2+a3r3+a4r4+a5r5+a6r6

[0039] Among them, r1 is the dispatching operation cost reward item, r2 is the node voltage over-limit reward item, r3 is the line over-limit reward item, r4 is the balancing machine power over-limit reward item, r5 is the reactive power over-limit reward item, r6 is the system standby reward item, and a i are the coefficients of each reward item, a1 is 3, a2 is 1, a3 is 1, a4 is 1, a5 is 1, and a6 is 2;

[0040] The system standby reward item is expressed as:

[0041]

[0042] in, is the upward spinning reserve demand of the system during period t, It is the downward spinning reserve demand of the system during period t.

[0043] In a second aspect, the present invention provides an energy storage and multi-resource optimization scheduling system, comprising:

[0044] Build a model module to obtain historical operation data of the power system and build energy storage and multi-resource optimization scheduling models;

[0045] Definition module, used to define energy storage and multi-resource optimization scheduling model as Markov decision process, giving state space, action space and reward function;

[0046] The encapsulation module is used to encapsulate the state space, action space and reward function of the defined Markov decision process into a reinforcement learning environment;

[0047] The training module is used to perform interactive learning between the reinforcement learning environment and the reinforcement learning algorithm based on the historical operation data of the power system, execute the agent action, update the environment state, calculate the reward value, and train the agent;

[0048] The input module is used to input the trained intelligent agent into the energy storage and multi-resource optimization scheduling model to obtain the optimal scheduling plan.

[0049] As a preferred solution of the energy storage and multi-resource optimization scheduling system described in the present invention, wherein:

[0050] Initialization module, used to define the state space dimension, action space dimension and parameters of each energy storage, thermal power unit, hydropower unit and DC in the power system;

[0051] The execution module is used to receive the action instructions issued by the agent, update the environment state according to the action instructions, and calculate the reward value according to the new state;

[0052] The reset module is used to reset the reinforcement learning environment, initialize all methods and variables, and randomly read the power grid operation data of a certain day, and use the initial section as the initial state of the environment for the current training round;

[0053] The state observation module is used to obtain the current environment state and pass it to the agent;

[0054] The termination module is used to determine whether to end the interaction with the agent in the current training round;

[0055] If the power flow calculation does not converge, the round ends;

[0056] If the number of interactions reaches the preset maximum number of interactions, the round ends.

[0057] In a third aspect, the present invention provides a computing device, comprising:

[0058] Memory, used to store programs;

[0059] A processor is used to execute the computer executable instructions, which, when executed by the processor, implement the steps of the energy storage and multi-resource optimization scheduling method.

[0060] In a fourth aspect, the present invention provides a computer-readable storage medium, comprising: when the program is executed by a processor, the steps of implementing the energy storage and multi-resource optimization scheduling method are implemented.

[0061] Beneficial effects of the present invention: The present invention improves the adaptability to the uncertainty of new energy through the coordinated operation of energy storage power stations and multiple resources, and provides the optimal scheduling plan online and quickly. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0063] Figure 1 A basic flow chart of an energy storage and multi-resource optimization scheduling method provided by an embodiment of the present invention;

[0064] Figure 2 A logic diagram of each module of a reinforcement learning environment for an energy storage and multi-resource optimization scheduling method provided by an embodiment of the present invention;

[0065] Figure 3 A schematic diagram of the convergence of reward values ​​of an energy storage and multi-resource optimization scheduling method provided by an embodiment of the present invention;

[0066] Figure 4 A schematic diagram of the scheduling results of a certain day of an energy storage and multi-resource optimization scheduling method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0067] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0068] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0069] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0070] The present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0071] At the same time, in the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "upper, lower, inner and outer" are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first, second or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0072] In the present invention, unless otherwise clearly specified and limited, the terms "install, connect, connect" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integral connection; it can also be a mechanical connection, an electrical connection or a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0073] Example 1

[0074] Reference Figure 1-2 , is an embodiment of the present invention, which provides a method for optimizing energy storage and multi-resource scheduling, such as Figure 1 As shown, the following steps are included:

[0075] S1: Obtain historical operation data of the power system and build energy storage and multi-resource optimization dispatching models;

[0076] In the embodiment of the present application, the historical operation data of the power system includes:

[0077] 1) Actual load demand, actual wind power output, load forecast demand and wind power forecast output of an actual power grid at each time section over 30 days;

[0078] 2) Number of grid nodes N sub ;

[0079] 3) Maximum output P of each thermal power unit G,max , minimum output P G,min , Maximum upward climbing rate U G , Maximum downward climbing rate D G , minimum startup time T on and minimum downtime T off ;

[0080] 4) Maximum charging power P of each energy storage E,cha,max , Maximum discharge power P E,dis,max , Maximum State of Charge and minimum state of charge

[0081] 5) Maximum output P of each hydropower unit H,max , minimum output P H,min , Maximum upward climbing rate U H and the maximum downward climbing rate D H ;

[0082] 6) Each DC transmission power upper limit P D,max , transmission power lower limit P D,min , Maximum power adjustment rate Δ tland the maximum number of adjustments X max .

[0083] In an embodiment of the present application, constructing an energy storage and multi-resource optimization scheduling model includes determining an objective function and constraints of the energy storage and multi-resource optimization scheduling model. The objective function of the energy storage and multi-resource optimization scheduling model is expressed as:

[0084]

[0085] Among them, T is the total number of optimized scheduling periods, F G,t is the operating cost of thermal power units, F hp,t is the operating cost of the hydropower unit, F tl,t is the DC operating cost, F E,t is the energy storage operation cost, N is the number of thermal power units in the system; C n is the unit power generation cost of thermal power unit n; is the power generation of thermal power unit n in period t; and are the startup cost and shutdown cost of thermal power unit n, and are the single start-up and shutdown costs of thermal power unit n; η n,t is the startup state variable of thermal power unit n, which is 1 when it is started and 0 when it is stopped; γ n,t is the shutdown state variable of thermal power unit n, which is 1 when shut down and 0 when started; C hp is the unit power generation cost of the hydropower unit hp; is the hp output of the hydropower unit in period t; C tl is the unit transmission power cost of DC tl; B is the power delivered by DC tl during period t; m is the unit power generation cost of energy storage m, The energy storage m outputs power during period t.

[0086] In the embodiment of the present application, the constraints of the energy storage and multi-resource optimization scheduling model include power flow constraints, thermal power unit output constraints, thermal power unit climbing constraints, thermal power unit start and shutdown constraints, energy storage output upper and lower limit constraints, energy storage charge state constraints, hydropower unit output constraints, hydropower unit climbing constraints, DC transmission power constraints, DC transmission power adjustment rate constraints, power constraints that cannot be adjusted in the opposite direction during adjacent time periods, DC adjustment number limit constraints, power balance constraints, system standby constraints, system voltage safe operation constraints and line transmission capacity constraints.

[0087] (1) Power flow constraints

[0088]

[0089] Among them, Nsub represents the number of grid nodes, P i (t) and Q i (t) represents the injected active and reactive power of node i in period t; U i (t) and U j (t) represents the voltage amplitude of nodes i and j in time period t respectively; G ij and B ij Respectively represent the conductance and susceptance of the branch; θ ij (t) represents the voltage phase difference between nodes i and j during time period t.

[0090] (2) Output constraints of thermal power units

[0091]

[0092] in, They are respectively the upper and lower limits of the output of thermal power unit n.

[0093] (3) Climbing constraints of thermal power units

[0094]

[0095] in, and are the outputs of thermal power unit n at t and t-1 respectively, and is the maximum upward and downward climbing rate of thermal power unit n.

[0096] (4) Constraints on the start and shutdown of thermal power units

[0097]

[0098] Among them, u nt is the start and stop status of unit n in time period t (start is 1, stop is 0), and The minimum start-up and shutdown time allowed for unit n; and It is the continuous startup and shutdown time of thermal power unit n at the end of period t-1.

[0099] (5) Upper and lower limits of energy storage output

[0100]

[0101] (6) Energy storage state of charge constraints

[0102]

[0103]

[0104] Among them, Soc,m (t) and S oc,m (t+1) is the SOC of energy storage m at time periods t and t+1; and are the charging efficiency, discharging efficiency and effective capacity of the energy storage m. and are the minimum and maximum states of charge of the energy storage m, respectively.

[0105] (7) Hydropower unit output constraints

[0106]

[0107] in, They respectively represent the upper and lower limits of the hp output of the hydropower unit.

[0108] (8) Climbing constraints of hydropower units

[0109]

[0110] in, and are the hp output of the hydropower unit at t and t-1 respectively, and It is the maximum upward and downward climbing rate of the hp of the hydropower unit.

[0111] (9) DC transmission power constraints

[0112]

[0113] in, They represent the upper and lower limits of DC tl transmission power respectively.

[0114] (10) DC transmission power adjustment rate constraints

[0115]

[0116] Among them, Δ tl Indicates the maximum value of DC tl power adjustment rate, They represent the power upward adjustment state and downward adjustment state of DC tl at time t respectively. Indicates upward adjustment. Indicates downward adjustment.

[0117] (11) Power constraints cannot be adjusted in the opposite direction during adjacent time periods

[0118]

[0119] (12) DC adjustment times limit constraint

[0120]

[0121] Among them, X max Indicates the maximum number of DC adjustment times.

[0122] (13) Power balance constraints

[0123]

[0124] in, and are the actual values ​​of wind power output and load in period t respectively; and are the predicted values ​​of wind power output and load in period t respectively; and are the prediction errors of wind power output and load in period t respectively.

[0125] (14) System backup constraints

[0126]

[0127] in, is the upward spinning reserve provided by the system during period t, is the upward spinning reserve demand of the system during period t, is the downward spinning reserve provided by the system during period t, It is the downward spinning reserve demand of the system during period t.

[0128] (15) System voltage safe operation constraints

[0129]

[0130] in, is the voltage lower limit of the jth node; is the voltage upper limit of the jth node; is the voltage amplitude of the jth node in the base state; is the voltage amplitude of the jth node under the mth “N-1” fault event; J is the number of nodes in the system; N0 is the number of “N-1” fault events in the system, N sub is the number of grid nodes;

[0131] (16) Line transmission capacity constraints

[0132]

[0133] Among them, P ij (t) and Q ij (t) is the active and reactive transmission power of the line between nodes i and j in period t; S ij is the line transmission capacity.

[0134] S2: Define the energy storage and multi-resource optimization scheduling model as a Markov decision process, and give the state space, action space and reward function;

[0135] In the embodiment of the present application, the state space of the Markov decision process in S3 is:

[0136]

[0137] in, and They are respectively the prediction sequences of load size and wind power output for the next multiple time periods.

[0138] It should be noted that the state space selection of reinforcement learning will directly affect the convergence performance of the algorithm, and the optimal scheduling needs to determine the scheduling arrangement based on the load size and wind power output at future moments. To this end, it is considered to incorporate the load size and wind power output of multiple time periods in the future into the state space to improve the adaptability of the scheduling scheme to uncertainty and improve the convergence ability of the algorithm.

[0139] In the embodiment of the present application, the action space of the Markov decision process is expressed as:

[0140]

[0141] in, The change in output of each energy storage during the current dispatch period compared with the previous dispatch period; The change in output of each thermal power unit during the current dispatch period compared with the previous dispatch period; The change in output of each hydropower unit during the current dispatch period compared with the previous dispatch period; It is the change in power delivered during the current DC scheduling period compared with the previous scheduling period.

[0142] It should be noted that, since the present invention is energy storage and multi-resource optimization scheduling, the control objects involve energy storage, thermal power units, hydropower units and DC, so the adjustment amounts of these four control objects are selected as the action space of reinforcement learning.

[0143] In the embodiment of the present application, the reward function of the Markov decision process in S2 is:

[0144] r t =a1r1+a2r2+a3r3+a4r4+a5r5+a6r6

[0145] Among them, r1 is the dispatching operation cost reward item, r2 is the node voltage over-limit reward item, r3 is the line over-limit reward item, r4 is the balancing machine power over-limit reward item, r5 is the reactive power over-limit reward item, r6 is the system standby reward item, and a iare the coefficients of each reward item, a1 is 3, a2 is 1, a3 is 1, a4 is 1, a5 is 1, and a6 is 2;

[0146] It should be noted that the design of the reward function is crucial to the convergence performance of the reinforcement learning algorithm. To this end, the present invention designs the reward function in combination with the economy, abundance and some constraints of the power system operation to improve the algorithm performance.

[0147] (1) Dispatch operation cost reward item r1

[0148] r1=-F t

[0149] (2) Node voltage over-limit reward item r2

[0150]

[0151] Among them, N sub is the number of grid nodes; v i is the voltage value of node i; is the voltage upper limit of node i; is the lower voltage limit of node i.

[0152] (3) Line over-limit reward item r3

[0153]

[0154] Among them, N line is the number of power grid branches; I i and T are the current and thermal limits of branch i respectively; β is a constant.

[0155] (4) Balance machine power over-limit reward item r4

[0156]

[0157] Among them, N balanced is the number of balancing machines; p i The actual active power output of the balancing machine i; is the upper limit of the output of balancing machine i; is the lower limit of the output of balancing machine i.

[0158] (5) Reactive power output over-limit reward item r5

[0159]

[0160] Among them, N gen is the total number of units; q i is the actual reactive power output of unit i; is the upper limit of reactive power output of unit i, is the lower limit of reactive power output of unit i.

[0161] (6) System reserve reward item r6

[0162]

[0163] It should be noted that the operation of the power system needs to meet the adequacy constraints, otherwise the system reserve cannot meet the net load prediction error caused by the wind power output and load size prediction errors, which may lead to load shedding or wind abandonment. For this reason, the reward function r6 is designed to guide the intelligent agent to find the best solution, so that the optimized scheduling plan can meet the adequacy requirements of the power system.

[0164] S3: Encapsulate the state space, action space and reward function of the defined Markov decision process into a reinforcement learning environment;

[0165] S4: Based on the historical operation data of the power system, the agent is trained by interactive learning between the reinforcement learning environment and the reinforcement learning algorithm, executing the agent action, updating the environment state, calculating the reward value, and training the agent;

[0166] In the embodiment of the present application, interactive learning is performed by the reinforcement learning environment and the reinforcement learning algorithm, the agent action is executed, the environment state is updated, the reward value is calculated, and the agent is trained, including the following steps:

[0167] Define the state space dimensions, action space dimensions and parameters of each energy storage, thermal power unit, hydropower unit and DC in the power system;

[0168] Receive action instructions from the agent, update the environment state according to the action instructions, and calculate the reward value based on the new state;

[0169] Reset the reinforcement learning environment, initialize all methods and variables, and randomly read the power grid operation data of a certain day, and use the initial section as the initial state of the environment for the current training round;

[0170] Get the current environment state and pass it to the agent;

[0171] Determine whether to end the interaction with the agent in the current training round;

[0172] If the power flow calculation does not converge, the round ends;

[0173] If the number of interactions reaches the preset maximum number of interactions, the round ends.

[0174] S5: Input the trained intelligent agent into the energy storage and multi-resource optimization scheduling model to obtain the optimal scheduling plan.

[0175] In an embodiment of the present application, the trained intelligent agent is input into the energy storage and multi-resource optimization scheduling model. The intelligent agent generates the optimal actions according to the current state, and the model generates the optimal scheduling plan according to these actions to assist the power system operators in making decisions.

[0176] It should be noted that reinforcement learning has powerful autonomous search, learning and decision-making capabilities. Using it for optimal scheduling of power systems can improve the ability to adapt to the uncertainty of new energy and provide the optimal scheduling plan online and quickly.

[0177] This embodiment also provides an energy storage and multi-resource optimization scheduling system, including:

[0178] Build a model module to obtain historical operation data of the power system and build energy storage and multi-resource optimization scheduling models;

[0179] Definition module, used to define energy storage and multi-resource optimization scheduling model as Markov decision process, giving state space, action space and reward function;

[0180] The encapsulation module is used to encapsulate the state space, action space and reward function of the defined Markov decision process into a reinforcement learning environment;

[0181] The training module is used to perform interactive learning between the reinforcement learning environment and the reinforcement learning algorithm based on the historical operation data of the power system, execute the agent action, update the environment state, calculate the reward value, and train the agent;

[0182] The input module is used to input the trained intelligent agent into the energy storage and multi-resource optimization scheduling model to obtain the optimal scheduling plan.

[0183] Furthermore, it also includes:

[0184] Initialization module, used to define the state space dimension, action space dimension and parameters of each energy storage, thermal power unit, hydropower unit and DC in the power system;

[0185] The execution module is used to receive the action instructions issued by the agent, update the environment state according to the action instructions, and calculate the reward value according to the new state;

[0186] The reset module is used to reset the reinforcement learning environment, initialize all methods and variables, and randomly read the power grid operation data of a certain day, and use the initial section as the initial state of the environment for the current training round;

[0187] The state observation module is used to obtain the current environment state and pass it to the agent;

[0188] The termination module is used to determine whether to end the interaction with the agent in the current training round;

[0189] If the power flow calculation does not converge, the round ends;

[0190] If the number of interactions reaches the preset maximum number of interactions, the round ends.

[0191] Furthermore, it also includes:

[0192] The reinforcement learning environment includes an initialization module, an execution module, a reset module, a state observation module and a termination module.

[0193] The logic of each module of the reinforcement learning environment is as follows Figure 2 As shown in the figure, the state space dimension, action space dimension and parameters of each energy storage, thermal power unit, hydropower unit and DC in the power system are defined through the initialization module (init module); the agent resets the reinforcement learning environment by calling the reset module (reset module), initializes all methods and variables, and randomly reads the power grid operation data of a certain day, and takes the initial section as the initial state of the environment in the current training round; the agent obtains the system state S by calling the state observation module (get_ob module) t And give action a t ; Call the execution module (step module) to execute action a t , get the reward value r t+1 ; Then call the state observation module (get_ob module) to obtain the system state S t+1 , and r t+1 and S t+1 Pass it to the agent for training; call the terminal module to end one round of training.

[0194] Furthermore, it also includes:

[0195] Memory, used to store programs;

[0196] A processor is used to load the program to execute the energy storage and multi-resource optimization scheduling method.

[0197] This embodiment also provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the energy storage and multi-resource optimization scheduling method.

[0198] The storage medium proposed in this embodiment and the energy storage and multi-resource optimization scheduling method proposed in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0199] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ReadOnly, Memory, ROM), random access memory (RandomAccess Memory, RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform the methods of various embodiments of the present invention.

[0200] Example 2

[0201] Reference Figure 3-4 , which is an embodiment of the present invention, provides an energy storage and multi-resource optimization scheduling method. In order to verify its beneficial effects, scientific demonstration is carried out through specific implementation methods and implementation effects.

[0202] In this embodiment, the power system used is an improved IEEE39 node system, which includes 2 energy storage power stations, 8 thermal power units, 1 hydropower unit, 1 DC line, and 3 wind farms. The load and wind power data are obtained from the actual power grid. During the training process, the convergence of the reward value is as follows Figure 3 As shown in the figure, the whole training process is trained 10,000 times. After 4,000 times of training, the reward value reaches the maximum value and gradually stabilizes, indicating that at this time the intelligent agent of the reinforcement learning algorithm has learned the optimal decision-making in different situations through interaction with the environment, and can give an optimized scheduling plan. Figure 4 It represents the dispatching plan given by the intelligent agent under the load and wind power forecast data of a certain day, including energy storage output, thermal power output, hydropower output and DC transmission power.

[0203] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for optimizing energy storage and multi-resource scheduling, characterized in that: include: Obtain historical operation data of the power system and build energy storage and multi-resource optimization dispatch models; The energy storage and multi-resource optimization scheduling model is defined as a Markov decision process, and the state space, action space and reward function are given; Encapsulate the state space, action space and reward function of the defined Markov decision process into a reinforcement learning environment; Based on the historical operation data of the power system, the agent is trained by interactive learning between the reinforcement learning environment and the reinforcement learning algorithm, executing the agent action, updating the environment state, calculating the reward value, and training the agent. The trained intelligent agent is input into the energy storage and multi-resource optimization scheduling model to obtain the optimal scheduling plan.

2. The energy storage and multi-resource optimization scheduling method according to claim 1, characterized in that: The method of interactively learning with the reinforcement learning environment and the reinforcement learning algorithm, executing the agent action, updating the environment state, calculating the reward value, and training the agent includes the following steps: Define the state space dimensions, action space dimensions and parameters of each energy storage, thermal power unit, hydropower unit and DC in the power system; Receive action instructions from the agent, update the environment state according to the action instructions, and calculate the reward value based on the new state; Reset the reinforcement learning environment, initialize all methods and variables, and randomly read the power grid operation data of a certain day, and use the initial section as the initial state of the environment for the current training round; Get the current environment state and pass it to the agent; Determine whether to end the interaction with the agent in the current training round; If the power grid flow calculation does not converge, the round ends; If the number of interactions reaches the preset maximum number of interactions, the round ends.

3. The energy storage and multi-resource optimization scheduling method according to claim 1 or 2, characterized in that: The construction of the energy storage and multi-resource optimization scheduling model includes determining the objective function and constraints of the energy storage and multi-resource optimization scheduling model. The objective function of the energy storage and multi-resource optimization scheduling model is expressed as: Among them, T is the total number of optimized scheduling periods, F G,t is the operating cost of thermal power units, F hp,t is the operating cost of the hydropower unit, F tl,t is the DC operating cost, F E,t The operating cost of energy storage. The constraints of the energy storage and multi-resource optimization scheduling model include power flow constraints, thermal power unit output constraints, thermal power unit climbing constraints, thermal power unit start and shutdown constraints, energy storage output upper and lower limit constraints, energy storage charge state constraints, hydropower unit output constraints, hydropower unit climbing constraints, DC transmission power constraints, DC transmission power adjustment rate constraints, power constraints that cannot be adjusted in the opposite direction during adjacent time periods, DC adjustment number limit constraints, power balance constraints, system standby constraints, system voltage safe operation constraints and line transmission capacity constraints.

4. The energy storage and multi-resource optimization scheduling method according to claim 3, characterized in that: The state space representation of the Markov decision process is: in, The output power of each energy storage in the current period t, The output power of each thermal power unit in the current period t, is the output power of each hydropower unit in the current period t, is the DC transmission power of each period in the current t period, is the upward spinning reserve provided by the system during period t, is the downward spinning reserve provided by the system during period t, and They are respectively the prediction sequences of load size and wind power output for the next multiple time periods.

5. The energy storage and multi-resource optimization scheduling method according to claim 4, characterized in that: The action space of the Markov decision process is expressed as: in, The change in output of each energy storage during the current dispatch period compared with the previous dispatch period; The change in output of each thermal power unit during the current dispatch period compared with the previous dispatch period; The change in output of each hydropower unit during the current dispatch period compared with the previous dispatch period; It is the change in power delivered during the current DC scheduling period compared with the previous scheduling period.

6. The energy storage and multi-resource optimization scheduling method according to claim 5, characterized in that: The reward function of the Markov decision process is expressed as: <h2 style=";text-align:left;direction:ltr">r<h2 style=";text-align:left;direction:ltr"> t <h2 style=";text-align:left;direction:ltr"> =a1r1+a2r2+a3r3+a4r4+a5r5+a6r6 Among them, r1 is the dispatching operation cost reward item, r2 is the node voltage over-limit reward item, r3 is the line over-limit reward item, r4 is the balancing machine power over-limit reward item, r5 is the reactive power over-limit reward item, r6 is the system standby reward item, and a i are the coefficients of each reward item, a1 is 3, a2 is 1, a3 is 1, a4 is 1, a5 is 1, and a6 is 2; The system standby reward item is expressed as: in, is the upward spinning reserve demand of the system during period t, It is the downward spinning reserve demand of the system during period t.

7. A system for implementing energy storage and multi-resource optimization scheduling method, characterized in that: Build a model module to obtain historical operation data of the power system and build energy storage and multi-resource optimization scheduling models; Definition module, used to define energy storage and multi-resource optimization scheduling model as Markov decision process, giving state space, action space and reward function; The encapsulation module is used to encapsulate the state space, action space and reward function of the defined Markov decision process into a reinforcement learning environment; The training module is used to perform interactive learning between the reinforcement learning environment and the reinforcement learning algorithm based on the historical operation data of the power system, execute the agent action, update the environment state, calculate the reward value, and train the agent; The input module is used to input the trained intelligent agent into the energy storage and multi-resource optimization scheduling model to obtain the optimal scheduling plan.

8. The energy storage and multi-resource optimization scheduling system according to claim 7, characterized in that: Initialization module, used to define the state space dimension, action space dimension and parameters of each energy storage, thermal power unit, hydropower unit and DC in the power system; The execution module is used to receive the action instructions issued by the agent, update the environment state according to the action instructions, and calculate the reward value according to the new state; The reset module is used to reset the reinforcement learning environment, initialize all methods and variables, and randomly read the power grid operation data of a certain day, and use the initial section as the initial state of the environment for the current training round; The state observation module is used to obtain the current environment state and pass it to the agent; The termination module is used to determine whether to end the interaction with the agent in the current training round; If the power grid flow calculation does not converge, the round ends; If the number of interactions reaches the preset maximum number of interactions, the round ends.

9. An electronic device, characterized in that: include: Memory, used to store programs; A processor, used to load the program to execute the steps of the energy storage and multi-resource optimization scheduling method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the steps of the energy storage and multi-resource optimization scheduling method as described in any one of claims 1 to 7 are implemented.

Images