Multisource power real-time scheduling optimization method and system based on reinforcement learning

Abstract

The invention relates to the technical field of power dispatching optimization, in particular to a multi-source power real-time dispatching optimization method and system based on reinforcement learning. The method comprises the following steps: acquiring required load information data of power generation equipment in different time periods; constructing an optimal scheduling strategy model of hybrid wind-fire distributed power generation; adopting a Monte Carlo method and a double-Q learning algorithm to optimize the optimal scheduling strategy model; and performing multi-source power real-time scheduling optimization by using the optimal scheduling strategy model. The multi-source electric power dispatching method is optimized in real time through the reinforcement learning method, the optimal electric power dispatching strategy is searched, compared with a traditional method, the electric power cost is reduced by 15% or above, meanwhile, automatic dispatching can be achieved, and manual intervention and experience dependence are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of power dispatch optimization, and in particular to a multi-source power real-time dispatch optimization method and system based on reinforcement learning. Background Art

[0002] Smart grid systems are an important part of modern power infrastructure. They improve the efficiency, reliability and sustainability of power transmission and distribution by integrating advanced communication technologies, data analysis and automated control. Traditional power grids are unable to cope with the challenges of increasingly complex power demand and renewable energy access due to the lack of real-time data monitoring and flexible control methods. With the transformation of the global energy structure, the role of distributed energy in smart grids has become increasingly prominent. Distributed energy includes solar photovoltaic, wind power, small hydropower, biomass power generation and energy storage systems for electric vehicles. They are usually distributed on the side of power users and are characterized by small scale, decentralization and flexibility. The access of these energy resources has profoundly changed the operation mode of the power grid. The traditional one-way power flow has been replaced by two-way or even multi-directional power flow, which has posed new challenges to the stability and dispatching capabilities of the power grid. A typical example is hybrid wind-fire distributed energy. The demand load of distributed energy is expected to be met by wind power generation as much as possible. However, due to the intermittent nature of wind power generation, it is necessary to integrate the system with traditional thermal generators. Thermal generators have the ability to quickly rise or fall, which can significantly improve the flexibility and reliability of distributed energy.

[0003] Traditional control strategies usually allocate as much demand load as possible to wind generators in the first stage. Thermal generators make up for the unmet demand in the second stage. But this strategy lacks flexibility. Wind turbines are usually designed and operated at a fixed startup speed (cut-in point). When wind speed fluctuates around the cut-in point, wind turbines may start and shut down repeatedly and frequently, resulting in startup costs. Excessive stress caused by frequent startup and shutdown may increase the likelihood of component failure and negatively affect the availability of wind turbines. This strategy may lead to higher operating and maintenance costs. Summary of the invention

[0004] In order to solve the above-mentioned problems, the present invention provides a multi-source power real-time scheduling optimization method and system based on reinforcement learning.

[0005] In the first aspect, the present invention provides a multi-source power real-time dispatch optimization method based on reinforcement learning, which adopts the following technical solution: A multi-source power real-time dispatch optimization method based on reinforcement learning, comprising: Obtain the power generation equipment demand load information data by time period; Construct an optimal dispatch strategy model for hybrid wind-thermal distributed generation; Monte Carlo method and double-Q learning algorithm are used to optimize the optimal scheduling strategy model; Use the optimal dispatch strategy model to optimize multi-source power real-time dispatch.

[0006] Furthermore, the construction of the optimal dispatching strategy model for hybrid wind-thermal distributed generation includes performing real-time dispatching mathematical modeling, wherein the dispatching strategy of distributed power is defined as To the action space The mapping of As a feasible scheduling strategy, Indicated in Actions to be taken in the state , the goal becomes to find the optimal scheduling strategy ; Define the function , means according to Strategy Action The income for each time period , when the system changes state, the cost The change in cost reflects the correctness of the decision, and the profit function under the current state Defined as: in, is the discount parameter, is Mathematical expectation under state transition probability, the scheduling optimization goal is to find the optimal strategy Optimal, that is .

[0007] Furthermore, the optimal dispatching strategy model for hybrid wind-thermal distributed generation is constructed, including performing real-time power demand load loading, wherein the real-time demand load is assumed to be a random process, following the Gaussian distribution given in the following formula: in is the predicted demand load.

[0008] Furthermore, the optimal dispatching strategy model for hybrid wind-thermal distributed generation includes power generation cost modeling, wherein the startup cost generated by the wind turbine when starting is used as the wind power cost, and the wind power cost The calculation method is: in It is The startup cost of a wind power generation equipment, the operating cost of a thermal power unit includes production cost, startup cost and shutdown cost; the cost of thermal power generation Designated as equipment power generation The quadratic function of in, is the power generation capacity of the jth thermal power generation equipment, are three hyperparameters; The total state transition cost is expressed as: .

[0009] Furthermore, the optimal dispatching strategy model for hybrid wind-thermal distributed generation is constructed, including power generation modeling, wherein the thermal power generation power is regarded as a fixed value, and the output of the wind turbine generator depends on the wind speed, which is expressed as: in, is the scale factor of the i-th power generation equipment, are wind turbine parameters.

[0010] Furthermore, the use of the double-Q learning algorithm to optimize the optimal scheduling strategy model includes defining a state-action value function Q, that is, Indicates in status Next, execute the action cost.

[0011] Furthermore, the use of the double-Q learning algorithm to optimize the optimal scheduling strategy model also includes introducing a double-Q learning mechanism, using another Q value to evaluate the next state when calculating the target Q value, and iteratively optimizing by alternately updating the two Q values ​​to ultimately obtain the average of the two Q values.

[0012] In the second aspect, a multi-source power real-time dispatch optimization system based on reinforcement learning includes: The data acquisition module is configured to acquire the power generation equipment demand load information data in different time periods; The model building module is configured to build an optimal dispatch strategy model for hybrid wind-thermal distributed generation The model optimization module is configured to optimize the optimal scheduling strategy model using a double-Q learning algorithm; The dispatching module is configured to perform real-time dispatch optimization of multi-source power using an optimal dispatching strategy model.

[0013] In a third aspect, the present invention provides a computer-readable storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device to implement a multi-source power real-time scheduling optimization method based on reinforcement learning.

[0014] In a fourth aspect, the present invention provides a terminal device comprising a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded by the processor and executing the multi-source power real-time scheduling optimization method based on reinforcement learning.

[0015] In summary, the present invention has the following beneficial technical effects: The present invention optimizes the dispatching method of multi-source electricity in real time through reinforcement learning method and finds the optimal electricity dispatching strategy. Compared with the traditional method, it can reduce the electricity cost by more than 15%. At the same time, it can realize automatic dispatching and reduce manual intervention and experience dependence. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of a multi-source power real-time dispatch optimization method based on reinforcement learning according to Example 1 of the present invention. DETAILED DESCRIPTION

[0017] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0018] Example 1 Reference Figure 1 , a multi-source power real-time dispatch optimization method based on reinforcement learning in this embodiment includes: Obtain the power generation equipment demand load information data by time period; Construct an optimal dispatch strategy model for hybrid wind-thermal distributed generation; Monte Carlo method and double-Q learning algorithm are used to optimize the optimal scheduling strategy model; Use the optimal dispatch strategy model to optimize multi-source power real-time dispatch.

[0019] Specifically, the following steps are included: S1. Calculate the total demand load information data of power generation equipment in different time periods; Assume that the real-time demand load is a random process, and the total demand load of the power generation equipment is follows a Gaussian distribution given by: in is the demand load in period t-1, It is a Gaussian distribution used to adjust for real-time load changes.

[0020] S2. Construct an optimal dispatch strategy model for hybrid wind-thermal distributed generation; These include: 1. Real-time scheduling mathematical modeling, which regards the scheduling process as a random multi-stage decision-making problem. , the system is based on the current state , take action , is a three-dimensional variable including wind state , demand load And the status of all power generation equipment in the previous period , Indicates the switch status of wind power and thermal power generation equipment, specifically expressed as: in, Indicates the switch status of the power generation equipment. represents the power generation of the power generation equipment, i and j represent the indexes of wind power and thermal power generation equipment respectively. The scheduling process is as follows: From the current state and Action Decision, that is .

[0021] Since wind power generation is affected by wind conditions, is it necessary to consider whether the current power generation allocation status Convert to Affected by probability, this probability is expressed as: .

[0022] Therefore, the system evolves into a Markov process in the state space S, and its state transition probability depends on the control action. The dispatch strategy of distributed power is defined as To the action space The mapping of . , is a feasible scheduling strategy. Indicated in Actions to be taken in the state Therefore, the goal becomes to find the optimal scheduling strategy . Define the function , means according to Strategy Action For each time period , when the system changes state, the cost The change in cost reflects the correctness of the decision. Therefore, the profit function in the current state is Defined as: in, is the discount parameter, is Mathematical expectation under state transition probability, the scheduling optimization goal is to find the optimal strategy Optimal, that is .

[0023] 2. Modeling of power generation costs. As a renewable energy source, wind turbines are considered to have no power generation costs, but they may incur startup costs when they are started. The calculation method is: in It is The startup cost of a wind power plant is the cost of starting up, and the operating cost of a thermal power plant includes production cost, startup cost and shutdown cost. Designated as equipment power generation The quadratic function of in, is the power generation capacity of the jth thermal power generation equipment, are three hyperparameters.

[0024] The total state transition cost is expressed as: .

[0025] 3. Power generation modeling: Thermal power generation is not affected by external factors and is therefore a fixed value. The output of wind turbines mainly depends on wind speed. Wind power generation can be expressed as: in, is the scale factor of the i-th power generation equipment, is the wind turbine generator set parameter. The total power generation should be equal to the real-time power demand load ,Right now .

[0026] The total power generation in the total state transition cost function should be equal to the real-time power demand load, so the total state transition cost function is updated as: .

[0027] S3. Build a double-Q learning algorithm to find the optimal scheduling strategy ,Q learning guides the agent to choose the optimal action in different states by learning an action value function, namely the Q value function, which is constructed by the above modeling. , Here it is regarded as Q value, and Q learning is used to find the optimal strategy Optimization .

[0028] Define the state-action value function Q, that is Indicates in status Next, execute the action If we want to optimize the strategy , only need to optimize Right now .

[0029] In order to obtain the optimal Q value, it needs to be iterated. The specific calculation method of the iteration is: in, is in state Due to this maximization operation, a bias may be introduced, causing the Q value to be systematically overestimated. This overestimation may lead to policy instability or performance degradation in some environments, especially in complex or random environments. Therefore, the dual Q learning mechanism is introduced to initialize two independent Q values. , in the state Next, use Select strategy ,Right now .

[0030] Used when calculating the target Q value Evaluate the next state and update Right now .

[0031] Use when re-evaluating Evaluate, update Right now Alternate Update and .

[0032] The specific iterative process of the algorithm is as follows: (1) Initialize two independent Q value functions , , initialize the maximum number of iterations N.

[0033] (2) In the current state Next, use alternately Select Action .

[0034] (3) Execute action , observe the cost and the next state .

[0035] (4) Randomly select a Q-value function and use it to choose the best action for the next state.

[0036] (5) Update another Q-value function.

[0037] (6) Iterate the above steps until the last iteration, and finally output The average value of .

[0038] S4. Multi-source power real-time dispatch optimization step; (1) Setting algorithm hyperparameters, including the number of wind and thermal power generation facilities , set the scale factor of the wind turbine and unit parameters , set the hyper parameters of the firepower device etc. hyperparameters.

[0039] (2) The smart grid system collects the demand load of the previous day and uses the formula , calculate the daily demand load .

[0040] (3) For each interval t, new hyperparameter information is collected and the cost function is reinitialized .

[0041] (4) Initialize the cost function Input into the reinforcement learning network for iteration to obtain the optimal strategy.

[0042] (5) Based on the optimal strategy , readjust the power of the power generation equipment.

[0043] Example 2 This embodiment provides a multi-source power real-time dispatch optimization system based on reinforcement learning, including: The data acquisition module is configured to acquire the power generation equipment demand load information data in different time periods; The model building module is configured to build an optimal dispatch strategy model for hybrid wind-thermal distributed generation The model optimization module is configured to optimize the optimal scheduling strategy model using a double-Q learning algorithm; The dispatching module is configured to perform real-time dispatch optimization of multi-source power using an optimal dispatching strategy model.

[0044] A computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device to implement a multi-source power real-time scheduling optimization method based on reinforcement learning.

[0045] A terminal device includes a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded by the processor and executing the multi-source power real-time scheduling optimization method based on reinforcement learning.

[0046] The above are all preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-source power real-time dispatch optimization method based on reinforcement learning, characterized in that: include: Obtain the power generation equipment demand load information data by time period; Construct an optimal dispatch strategy model for hybrid wind-thermal distributed generation; Monte Carlo method and double-Q learning algorithm are used to optimize the optimal scheduling strategy model; Use the optimal dispatch strategy model to optimize multi-source power real-time dispatch.

2. According to the reinforcement learning-based multi-source power real-time dispatch optimization method of claim 1, it is characterized in that: The optimal dispatching strategy model for hybrid wind-thermal distributed generation is constructed, including real-time dispatching mathematical modeling, wherein the dispatching strategy of distributed power is defined as To the action space The mapping of As a feasible scheduling strategy, Indicated in Actions to be taken in the state , the goal becomes to find the optimal scheduling strategy ; Define the function , means according to Strategy Action The income for each time period , when the system changes state, the cost The change in cost reflects the correctness of the decision, and the profit function under the current state Defined as: in, is the discount parameter, is Mathematical expectation under state transition probability, the scheduling optimization goal is to find the optimal strategy Optimal, that is 。 3. The multi-source power real-time dispatch optimization method based on reinforcement learning according to claim 2 is characterized in that: The optimal dispatching strategy model for hybrid wind-thermal distributed generation is constructed, including real-time power demand load loading, wherein the real-time demand load is assumed to be a random process, following the Gaussian distribution given in the following formula: in is the predicted demand load.

4. The multi-source power real-time dispatch optimization method based on reinforcement learning according to claim 3 is characterized in that: The optimal dispatching strategy model for hybrid wind-thermal distributed generation includes power generation cost modeling, wherein the startup cost generated by the wind turbine when starting is used as the wind power cost. The calculation method is: in It is The startup cost of a wind power generation equipment, the operating cost of a thermal power unit includes production cost, startup cost and shutdown cost; the cost of thermal power generation Designated as equipment power generation The quadratic function of in, is the power generation capacity of the jth thermal power generation equipment, are three hyperparameters; The total state transition cost is expressed as: 。 5. The multi-source power real-time dispatch optimization method based on reinforcement learning according to claim 4 is characterized in that: The optimal dispatching strategy model for hybrid wind-thermal distributed generation is constructed, including power generation modeling, wherein the thermal power generation power is regarded as a fixed value, and the output of the wind generator depends on the wind speed, which is expressed as: in, is the scale factor of the i-th power generation equipment, are wind turbine parameters.

6. The multi-source power real-time dispatch optimization method based on reinforcement learning according to claim 5 is characterized in that: The double-Q learning algorithm is used to optimize the optimal scheduling strategy model, including defining the state-action value function Q, that is, Indicates in status Next, execute the action cost.

7. The multi-source power real-time dispatch optimization method based on reinforcement learning according to claim 6 is characterized in that: The method of optimizing the optimal scheduling strategy model using the double-Q learning algorithm also includes introducing a double-Q learning mechanism, using another Q value to evaluate the next state when calculating the target Q value, and iteratively optimizing by alternately updating the two Q values ​​to finally obtain the average value of the two Q values.

8. A multi-source power real-time dispatch optimization system based on reinforcement learning, comprising: The data acquisition module is configured to acquire the power generation equipment demand load information data in different time periods; The model building module is configured to build an optimal dispatch strategy model for hybrid wind-thermal distributed generation The model optimization module is configured to optimize the optimal scheduling strategy model using a double-Q learning algorithm; The dispatching module is configured to perform real-time dispatch optimization of multi-source power using an optimal dispatching strategy model.

9. A computer-readable storage medium storing a plurality of instructions, characterized in that: The instructions are suitable for being loaded by a processor of a terminal device and executing the method according to claim 1 .

10. A terminal device, comprising a processor and a computer-readable storage medium, wherein the processor is used to implement each instruction; and the computer-readable storage medium is used to store multiple instructions, characterized in that: The instructions are suitable for being loaded by a processor and executing the method as claimed in claim 1 .

Images