Two-stage low-carbon economic dispatching method based on deep reinforcement learning

Abstract

The invention relates to the field of power systems, in particular to a two-stage low-carbon economic dispatching method based on deep reinforcement learning. The invention provides a two-stage low-carbon economic dispatching method based on deep reinforcement learning in order to solve the problem that potential safety hazards are brought to a transmission line between a micro-grid and a main power grid due to uncertainty of renewable energy sources which cannot be solved by existing two-stage low-carbon economic dispatching. In the day-ahead scheduling stage, the transmission power limit between the micro-grid and the main power grid is used as a constraint condition, a day-ahead scheduling strategy is generated through the first stage, and meanwhile, the weight of each factor in a reward function is determined. In the intra-day scheduling stage, two methods of out-of-limit penalty weight adjustment and conservative safety limit constraint are used to deal with the power transmission line out-of-limit problem caused by renewable energy uncertainty and prediction errors.

Description

Technical Field

[0001] The present invention relates to the field of power system management, and in particular to a two-stage low-carbon economic dispatching method based on deep reinforcement learning. Background Art

[0002] Today, human society is facing the problem of global warming caused by a large amount of carbon emissions, and abnormal climate has posed a threat to mankind. One of the main sources of carbon emissions is the power industry, which is a pillar industry for national economic development. As countries around the world realize that high-quality economic development requires a lot of energy support, environmentally friendly and sufficient renewable energy has been added to the traditional power supply. Renewable energy generation has gradually replaced some traditional power generation methods with high carbon emissions. In recent years, the utilization rate of renewable energy has been increasing, among which wind power and solar energy account for a large proportion. Although the use of renewable energy can meet the needs of both carbon-free emissions and providing electricity, its own characteristics bring some problems. There is a lot of uncertainty in the influence of renewable energy on natural factors. In addition, the distribution of renewable energy in nature is uneven, and the output of renewable energy cannot be fully controlled. This situation will bring safety problems to the power system. Therefore, at this stage, it is still impossible to rely solely on renewable energy to achieve full power supply, and traditional power generation methods still occupy a large proportion in the power system.

[0003] For the power system, the extensive use of renewable energy can reduce the proportion of traditional thermal power generation in the energy supply and thus reduce carbon emissions, but the problems caused by uncertainty cannot be ignored. Although renewable energy cannot completely control its power generation period according to human will, the impact of renewable energy uncertainty can be mitigated through energy storage. When renewable energy is sufficient, the power generation may exceed the actual load demand. At this time, the excess power is stored through energy storage batteries or pumped storage. The stored energy is released when the load demand cannot be met to make up for the power gap. Adding energy storage measures can indeed improve the utilization rate of renewable energy and make the power supply more flexible, but this requires the energy storage system to be able to respond in time, store or release a large amount of energy in time, and reduce energy loss during the storage process. If only relying on energy storage to solve the uncertainty problem of renewable energy will be too costly. Mathematical methods can be used to model the uncertainty of renewable energy, and then use uncertainty optimization tools to solve it. However, the modeling process is relatively complicated and difficult to solve.

[0004] One of the unavoidable problems in the existing traditional power generation methods is the existence of a large amount of carbon emissions. In order to reduce the greenhouse gases discharged into the atmosphere during the power generation process, many scholars have conducted relevant research. In the literature, the author uses carbon capture equipment to capture the carbon emissions generated by thermal power generation equipment, and combines the carbon capture equipment with power generation equipment to form a carbon capture power plant. This can reduce carbon emissions on the power generation side. Since the operation of carbon capture equipment also requires electricity supply, this part of the supply can be obtained from traditional power generation methods or from renewable energy generation. Adding carbon capture equipment to the power generation side has become one of the widely used means of reducing carbon emissions, but it also makes the energy system more complicated.

[0005] Today's power system has become more complex after adding modules such as renewable energy generation, energy storage systems and carbon capture systems. It has evolved into an integrated energy system. It includes the generation and consumption of electricity, heat and other energy on both the supply and demand sides. The specific components of the integrated energy system will vary according to the actual application scenarios. Different integrated energy systems require an economic dispatch strategy suitable for themselves to achieve the expected goals of the system. Many scholars have studied the economic dispatch strategies of various integrated energy systems and proposed solutions to deal with the uncertainty and low-carbon problems of renewable energy. In the literature, the author proposed a low-carbon economic dispatch model for an integrated energy system containing electricity and thermal energy to enable the system to operate in a low-carbon and safe manner. The model takes into account the reflection of the real-time state of the cogeneration unit, constructs dynamic constraints, and uses the finite element orthogonal configuration method to improve the solution speed when solving nonlinear programming problems. In the literature, the author proposed a low-carbon economic dispatch model for an integrated energy system containing electricity, natural gas, thermal energy and electric vehicles. The model uses vehicle-to-grid technology and demand response technology to achieve the low carbon and economic efficiency of the system. In response to the uncertainty of renewable energy, the author chose to sacrifice some economic benefits to achieve stable system operation. In the multi-objective economic dispatch task, it is impossible to make every goal optimal. In the integrated energy system, combining new trading mechanisms and energy conversion technologies to formulate corresponding dispatch strategies can meet the low-carbon goals and deal with the uncertainty of renewable energy.

[0006] The above-mentioned related studies have many environmental and economic considerations, but for the safety and stability of the entire integrated energy system, the safety of power transmission should also be taken into account in the dispatching strategy. In the literature, the author proposed an economic dispatch model that considers frequency stability constraints to optimize the power generation cost under operation constraints and frequency stability constraints. In the literature, the author solves the problems of load loss and transmission line overload by predefined confidence. The integrated energy system that takes into account many factors and combines various modules requires a more intelligent method to obtain a suitable dispatching strategy so that the value of each part can be reflected. Moreover, multiple safety constraints must be considered at the same time in solving the constructed low-carbon economic dispatch optimization problem, which places higher requirements on traditional calculation methods. When using traditional methods to solve system planning problems, the effect and speed of the solution are often limited by the modeling of the problem. Complex scenarios and real-time changes in scenarios make the modeling and solution process difficult and time-consuming. Some researchers use related machine learning technologies to try to obtain new solutions. In the literature, the authors used a deep neural network (DNN) structure to reduce the problem size of safety constrained economic dispatch (SCED), and proposed a data-driven method to identify the active constraints of the SCED problem. The training efficiency was improved by the designed recognition model and the corresponding learning strategy, and transfer learning was used to quickly adjust the DNN parameters. Ultimately, the computational efficiency was significantly improved. However, as the operating state continues to change, the strategy generated according to the original optimization problem is no longer a scheduling strategy, and the optimization objective and constraint equations need to be readjusted according to the current state.

[0007] Therefore, the existing two-stage low-carbon economic dispatch fails to solve the problem of safety hazards in the transmission lines between microgrids and the main grid caused by the uncertainty of renewable energy. Summary of the invention

[0008] The purpose of the present invention is to solve the problem that the existing two-stage low-carbon economic dispatch fails to solve the problem that the uncertainty of renewable energy brings safety hazards to the transmission line between the microgrid and the main grid.

[0009] A two-stage low-carbon economic dispatch method based on deep reinforcement learning, including:

[0010] S1: Construct dataset;

[0011] S2: Construct a two-stage low-carbon economic dispatch model based on deep reinforcement learning;

[0012] S3: Convert the two-stage low-carbon economic dispatch model based on deep reinforcement learning into a Markov decision model;

[0013] S4: Solve the Markov decision model of the two-stage low-carbon economic dispatch model based on deep reinforcement learning and obtain the power dispatch strategy.

[0014] The beneficial effects of the present invention are: to solve the low-carbon economic dispatch problem of ensuring transmission power safety, a two-stage low-carbon economic dispatch model based on deep reinforcement learning is proposed. In the first stage (day-ahead dispatch stage), the transmission power limit between the microgrid and the main grid is used as a constraint condition, and the day-ahead dispatch strategy is generated through the first stage, and the weights of each factor in the reward function are determined at the same time. In the second stage (intraday dispatch stage), the present invention uses two methods, namely, over-limit penalty weight adjustment and conservative safety limit constraint, to deal with the problem of power transmission line over-limit caused by renewable energy uncertainty and prediction errors. The present invention solves the two-stage low-carbon economic dispatch model based on deep reinforcement learning based on three deep reinforcement learning algorithms of the Actor-Critic structure. And the effectiveness of the two-stage low-carbon economic dispatch model based on deep reinforcement learning proposed by the present invention in reducing the number of transmission line over-limits, reducing carbon emissions and costs is verified through experiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the two-stage model structure of the present invention;

[0017] Figure 2 This is a schematic diagram of the scheduling network of the present invention.

[0018] Figure 3 is a schematic diagram of the cost, carbon emission and transmission power curves under different weights of the present invention;

[0019] Figure 4 It is a schematic diagram of the day-ahead scheduling result of the present invention;

[0020] Figure 5 It is a schematic diagram of the day-ahead forecast data and the intraday short-term forecast data of the present invention;

[0021] Figure 6 It is a schematic diagram comparing the working conditions of the network transmission line after the second stage optimization and after the second stage optimization of the present invention;

[0022] Figure 7 It is a schematic diagram of carbon emission results and operating cost results before the intraday optimization of the present invention and after using two intraday optimization methods;

[0023] Figure 8 It is a schematic diagram of the scheduling results after optimization by the two methods of the present invention. DETAILED DESCRIPTION

[0024] Specific implementation method 1: Combination Figure 1 Description of the present invention

[0025] S1: Construct dataset;

[0026] S2: Construct a two-stage low-carbon economic dispatch model based on deep reinforcement learning;

[0027] S3: Convert the two-stage low-carbon economic dispatch model based on deep reinforcement learning into a Markov decision model;

[0028] S4: Solve the Markov decision model of the two-stage low-carbon economic dispatch model based on deep reinforcement learning and obtain the two-stage low-carbon economic dispatch.

[0029] Specific implementation method 2: The difference between this implementation method and specific implementation method 1 is that:

[0030] The data set constructed in S1 includes: historical wind power generation data, historical photovoltaic power generation data and historical power load data;

[0031] The historical electricity load data includes: time-of-use electricity prices, loads of microgrid power systems, energy storage percentages of energy storage systems, output power of thermal power units, and carbon emissions.

[0032] The other steps and parameters are the same as those in the first specific implementation.

[0033] Specific implementation method 3: This implementation method is different from specific implementation methods 1 to 2 in that:

[0034] In S2, a two-stage low-carbon economic dispatch model based on deep reinforcement learning is constructed; the specific process is as follows:

[0035] S2.1: Constructing the objective function of the two-stage low-carbon economic dispatch model based on deep reinforcement learning; the objective function of the two-stage low-carbon economic dispatch model based on deep reinforcement learning includes: energy supply cost C ES , the cost of using energy storage equipment C B , CO2 emission penalty cost C penal and CO2 storage and transmission costs C ST ;

[0036] S2.2: Construct constraints for a two-stage low-carbon economic dispatch model based on deep reinforcement learning; the constraints of the two-stage low-carbon economic dispatch model based on deep reinforcement learning include: supply and demand balance constraints, transmission power limit constraints, thermal power unit constraints, carbon capture system constraints and energy storage equipment constraints.

[0037] The other steps and parameters are the same as those in the first to second embodiments.

[0038] Specific implementation method 4: This implementation method is different from specific implementation methods 1 to 4 in that:

[0039] The specific process of constructing the objective function of the two-stage low-carbon economic dispatch model based on deep reinforcement learning in S2.1 is:

[0040] The objective function formula is as follows:

[0041] min(C ES +C ST +C penal +C B ) (1)

[0042]

[0043]

[0044]

[0045]

[0046] Among them, P t,a,ex is the transmission power between the main grid and the microgrid at time step t, p t,grid is the time-of-use electricity price, d c The unit cost of storing and transporting captured carbon for the cth carbon capture device, k b is the unit cost of charging and discharging of the bth energy storage device, h c The penalty cost for each unit of carbon emissions not captured by the cth carbon capture device, is the output of the g-th thermal power unit at time step t, a g ,b g ,c g is the power generation cost coefficient of the g-th thermal power unit, G represents the thermal power unit set, C represents the carbon capture equipment set, B represents the energy storage equipment set, T represents the time step set, represents the output of the bth energy storage device at time step t, represents the carbon capture amount of the cth carbon capture device at time step t, represents the amount of carbon emissions not captured at time step t. The other steps and parameters are the same as those in one of the first to third embodiments.

[0047] Specific implementation method 5: This implementation method is different from specific implementation methods 1 to 4 in that:

[0048] The constraints of the two-stage low-carbon economic dispatch model based on deep reinforcement learning are constructed in S2.2. The specific process is as follows:

[0049] Supply and demand balance constraints:

[0050]

[0051]

[0052] in, represents the load demand of microgrid a at time step t, represents the energy required for the cth carbon capture device at time step t, represents the output of the s-th solar power supply device at time step t, It represents the output of the w-th wind power supply equipment at time step t, and S represents the set of solar power supply equipment. When scheduling, it is necessary to consider the balance between net output and load. Renewable energy here considers photovoltaic and wind power. The present invention directly uses renewable energy output data to participate in power supply without detailed calculation.

[0053] Transmission power limit constraints:

[0054] P t,a,ex ≤P transMax (8)

[0055] Among them, P transMax Indicates the maximum transmission power of the transmission line; the transmission safety limit is considered during system operation, and this limit is used to constrain power exchange with the main power grid. Exceeding the limit will put the transmission line in an unsafe state;

[0056] Constraints on thermal power units:

[0057]

[0058]

[0059] in, RU g ,RD g respectively Indicates the lower limit of the output of thermal power units. Indicates the upper limit of the output of thermal power units, RU g Indicates the upper limit of thermal power unit climbing, RD g Indicates the upper limit of the downward climbing of thermal power units; the output of different units cannot exceed their own upper and lower limits, and the output change cannot exceed the climbing limit.

[0060] Carbon capture system constraints:

[0061]

[0062]

[0063]

[0064]

[0065] 0≤γ c ≤0.9(15)

[0066] in, represents the carbon capture amount of the cth carbon capture device at time step t, Q t,g represents the emission of the g-th thermal power unit at time step t, represents the maximum carbon capture capacity of the cth carbon capture device at time step t, γ c represents the carbon capture rate of the cth carbon capture device, u c represents the unit power consumption of the cth carbon capture device, α g represents the carbon emission intensity of the g-th thermal power unit; QR represents the downward ramp limit of carbon capture and Indicates the upward climbing limit of carbon capture; the carbon emission intensity of thermal power units is different, and the corresponding carbon capture rate is also different. The maximum carbon capture rate has an upper limit constraint, and the capture amount has a climbing constraint, and the capture amount is affected by the carbon production and capture rate of the thermal power unit. The power consumption of carbon capture is controlled by the carbon capture amount and the unit energy consumption of capture. The final cost is the power consumption and the storage and transmission cost of the captured carbon. The power consumption of the carbon capture system is directly provided by the thermal power unit, so the power consumption of the carbon capture equipment is transferred to the thermal power unit for calculation.

[0067] Energy storage equipment constraints:

[0068]

[0069]

[0070]

[0071] in represents the storage capacity percentage of energy storage device b at time step t, represents the minimum percentage of energy storage capacity of energy storage device b at time step t, Indicates the maximum percentage of power storage of energy storage device b at time step t; Indicates the maximum capacity of the energy storage device, Indicates the lower limit of charge and discharge of energy storage equipment and It indicates the upper limit of charging and discharging of the energy storage device. Other steps and parameters are the same as those in the first to fourth embodiments.

[0072] Specific implementation method 6: This implementation method is different from specific implementation methods 1 to 5 in that:

[0073] In S3, the two-stage low-carbon economic dispatch model based on deep reinforcement learning is converted into a corresponding Markov decision model; the specific process is:

[0074] The two-stage low-carbon economic dispatch based on deep reinforcement learning includes day-ahead dispatch and intraday dispatch.

[0075] The two-stage low-carbon economic dispatch model based on deep reinforcement learning is replaced by a Markov decision model for day-ahead dispatch and a Markov decision model for intraday dispatch.

[0076] The decision process of the Markov decision model for the day-ahead stage scheduling and the intraday stage scheduling is:

[0077] At each time step t, the Markov decision model receives a state s t , and select an action a according to the current policy π t , and the action a t Applied in the actual power system environment, the reward and punishment information r and the next state s fed back from the actual power system environment are obtained. t+1 , repeat this process until the entire scheduling cycle is completed;

[0078] The Markov decision model is mainly composed of a five-tuple data (S, A, P, R, λ), including: state set S, action set A, reward function R, state transition probability P and discount factor λ; the other steps and parameters are the same as those of specific implementation methods one to five.

[0079] Specific implementation method 7: This implementation method is different from specific implementation methods 1 to 6 in that:

[0080] In S4, the Markov decision model of the two-stage low-carbon economic dispatch model based on deep reinforcement learning is solved to obtain the two-stage low-carbon economic dispatch; the specific process is:

[0081] S4.1: Solve the Markov decision model for day-ahead scheduling; obtain the day-ahead scheduling decision; the specific process is:

[0082] The Markov decision model for the day-ahead scheduling is:

[0083] State set S:

[0084] For each day-ahead state time step t 1 , using 8 variables to describe the state of the day-ahead scheduling phase The formula is as follows:

[0085]

[0086] The state information of the environment provides an important basis for the intelligent agent to generate a scheduling strategy. The present invention defines the state as equation (19), where p is the time-of-use electricity price, l is the load of the microgrid power system, w is the wind power output, pv represents the photovoltaic output, soc is the energy storage percentage of the energy storage system, g is the output power of the thermal power unit, c emi for carbon emissions;

[0087] Action set A:

[0088] The Markov decision model receives the state of the day-ahead scheduling phase According to the day-ahead scheduling strategy π 1 Produce corresponding actions And the action Applied in actual power system environment, action It consists of three variables, expressed as:

[0089]

[0090] Among them, the parameter g change Respectively represent the percentage change of thermal power unit output, ccs change Indicates the percentage change in carbon capture, soc change Energy storage system energy storage change percentage;

[0091] The reward function should guide the RL agent to take actions to minimize the operating cost while enforcing the power balance constraint and the transmission line power limit constraint;

[0092] Reward function R:

[0093] When action A t1 After being applied in the environment, rewards R are fed back based on changes in the environment t1 , the formula is:

[0094]

[0095] in is the action reward, penal overline is the penalty for exceeding the power exchange limit; 1 ,σ 2 is the weight;

[0096] State transition probability P: set to 1;

[0097] The model of the Markov process of the day-ahead scheduling is solved by the proximal strategy optimization algorithm based on deep reinforcement learning, and the day-ahead scheduling decision π is obtained. 1 * ;

[0098] S4.2: The day-ahead scheduling decision π obtained in S4.1 1 * As the initial decision for intraday stage scheduling; solve the Markov decision model for intraday stage scheduling; obtain the intraday scheduling decision; the specific process is:

[0099] The Markov decision model for intraday stage scheduling is:

[0100] State set S:

[0101] For each intra-day state time step t 2 , using 8 variables to describe the status of the intraday scheduling phase The formula is as follows:

[0102]

[0103] The state information of the environment provides an important basis for the intelligent agent to generate a scheduling strategy. The present invention defines the state as equation (22), where p is the time-of-use electricity price, l is the load of the microgrid power system, w is the wind power output, pv represents the photovoltaic output, soc is the energy storage percentage of the energy storage system, g is the output power of the thermal power unit, c emi for carbon emissions;

[0104] Action set A:

[0105] The Markov decision model receives the status of the intraday scheduling phase According to the intraday scheduling strategy π 2 Produce corresponding actions And the action Applied in actual power system environment, action It consists of three variables, expressed as:

[0106]

[0107] Among them, the parameter g change Respectively represent the percentage change of thermal power unit output, ccs change Indicates the percentage change in carbon capture, soc change Energy storage system energy storage change percentage;

[0108] The reward function should guide the RL agent to take actions to minimize the operating cost while enforcing the power balance constraint and the transmission line power limit constraint;

[0109] Reward function R:

[0110] When action After being applied in the environment, rewards are fed back based on changes in the environment The formula is:

[0111]

[0112] in is the action reward, penal overline is the penalty for exceeding the power exchange limit, σ 1 ,σ 2 is the weight;

[0113] State transition probability P: set to 1;

[0114] The model of the Markov process of intraday stage scheduling is solved using the proximal strategy optimization algorithm based on deep reinforcement learning to obtain the intraday scheduling decision π 2 * , that is, a two-stage low-carbon economic dispatch strategy.

[0115] The solution process of the proximal strategy optimization algorithm based on deep reinforcement learning is:

[0116] The present invention uses a deep reinforcement learning algorithm based on the Actor-Critic (AC) structure to generate scheduling strategies. The characteristics of the Actor-Critic type of deep reinforcement learning algorithm are that it has the advantages of both policy-based and value-based deep reinforcement learning algorithms. It can not only select appropriate actions in high-dimensional continuous states, but also realize single-step or multi-step updates without going through a complete round.

[0117] The original deep reinforcement learning algorithm with AC structure is the Actor-Critic algorithm. This algorithm uses a neural network that fits the policy function as the Actor to generate actions and interact with the environment, and uses a neural network that fits the value function as the Critic to evaluate the Actor's performance and guide the Actor's next output action. Therefore, the Actor-Critic algorithm needs to approximate two parts. The first part is the approximation of the policy function as shown in formula (28), and the second part is the approximation of the value function. The value function can be simply divided into a state value function and an action value function as shown in formula (29):

[0118] π θ (s,a)=P(a|s,θ)≈π(a|s) (28)

[0119]

[0120] Where θ,w are neural network parameters, They are the state value function and the action value function respectively.

[0121] The policy update formula of the Actor-Critic algorithm will change according to the evaluation method adopted by the Critic due to the addition of the value-based Critic part. The specific process is:

[0122] Step 1: Critic uses the Q network to output the state value. The Actor iteratively updates the network parameters according to the value given by the Critic and then selects the next action to obtain the reward and the next state.

[0123] Step 2: Critic updates the Q network parameters w or θ based on the reward and the next state using the mean square error loss.

[0124] Step 3: Use the new parameters w or θ to evaluate the Actor's performance and generate the state value v t or action value,

[0125] The action value uses Temporal-Difference (TD) error or advantage function as the action value evaluation indicator of Critic.

[0126] Step 4: The Actor update strategy formula when using state value as the Critic evaluation indicator can be expressed as follows:

[0127]

[0128] Where α is the gradient update learning rate.

[0129] Although the Actor-Critic algorithm is an algorithm framework that combines multiple advantages, the actual application effect is not ideal. The neural networks of both the Actor and Critic need to update the gradient and are interdependent, making it difficult to converge during training. Although the original Actor-Critic algorithm is not very effective, its algorithmic ideas and structure are very valuable for reference, so many deep reinforcement learning algorithms based on AC have been generated.

[0130] In order to solve the problem of too large a difference between the new and old strategies after the AC algorithm is updated, the Actor-Critic Proximal Policy Optimization (PPO) algorithm was proposed. The PPO algorithm is an improvement on the AC algorithm, which can solve the problem of too large a policy update and control the difference between the new and old strategies within a smaller range after each update. The PPO algorithm can handle problems with continuous action space. The core idea of ​​the algorithm is to limit the policy update by clipping the probability ratio of the new and old strategies. The formula for the probability ratio of the new and old strategies and the clipping function are defined as follows:

[0131]

[0132]

[0133]

[0134] Where θ is the policy network parameter, ε is a hyperparameter, generally set to 0.2, and the advantage function The value of is generally estimated using the generalization advantage estimate (GAE)

[0135] The generalized optimization estimate (GAE) is actually a version of the input-return method applied to the estimated advantage function. The estimators of N advantage functions can be listed according to the idea of ​​using n-step return values ​​in the introduction of the input-return method.

[0136]

[0137] Formula (33) is used to trim the strategy probability ratio to prevent the gap between the new and old strategies from being too large. The evaluation index of the Critic part is to obtain the state value by inputting the next state, and use this value to evaluate the quality of the Actor's action selection and guide the Actor to choose the next action. The network parameter update of the Critic is the mean square error between the accumulation of the discounted reward and the state value.

[0138] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0139] Specific implementation eight: This implementation differs from specific implementations one to seven in that:

[0140] According to the action in S4.2 After being applied in the environment, rewards are fed back based on changes in the environment In the intraday scheduling stage, the present invention uses two methods, namely, over-limit penalty weight adjustment and conservative safety limit constraint, to deal with the over-limit problem of power transmission lines caused by renewable energy uncertainty and prediction errors. Specifically, the penalty for over-limit power exchange is overline Determined according to the transmission power between the main grid and the microgrid, the specific process is:

[0141] When the transmission power between the main grid and the microgrid is below the safety margin limit, the penalty for power exchange exceeding the limit is overline The formula is:

[0142]

[0143] When the transmission power between the main grid and the microgrid is within the buffer area, the penalty for exceeding the power exchange limit is overline The formula is:

[0144]

[0145] When the transmission power between the main grid and the microgrid is above the safety limit in the day-ahead dispatching phase, the penalty for power exchange exceeding the limit is overline The formula is:

[0146]

[0147] Among them, α 1 , α2 To induce the hyperparameter, P t,transMax represents the maximum transmission power of the transmission line at time step t, P t,a,ex is the transmission power between the main grid and the microgrid at time step t.

[0148] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.

[0149] Specific implementation method 9: This implementation method is different from specific implementation methods 1 to 8 in that:

[0150] In S4.1, the model of the Markov process of the day-ahead scheduling is solved using the Soft Actor-Critic algorithm based on deep reinforcement learning to obtain the day-ahead scheduling decision π 1 * ,

[0151] In S4.2, the model of the Markov process for day-ahead scheduling is solved using the Soft Actor-Critic algorithm based on deep reinforcement learning to obtain the intraday scheduling decision π 2 * , that is, a two-stage low-carbon economic dispatch strategy;

[0152] The method of solving the SoftActor-Critic (SAC) algorithm based on AC structure

[0153] The algorithm adopts a random distributed strategy function in the Actor part, similar to the PPO algorithm, and maximizes the entropy of the strategy while optimizing the strategy. The introduction of entropy can make the strategy as random as possible, prevent the strategy from converging to the local optimum too early, explore more feasible strategies, and improve the anti-interference ability. The target strategy of the deep reinforcement learning algorithm with maximum entropy is defined as follows:

[0154]

[0155] where ρ π It represents the distribution of state-action pairs generated by the agent under the control of strategy π, β is the hyperparameter that controls the proportion of entropy value, and finally H function is the entropy of the strategy.

[0156] The SAC algorithm is also a deep reinforcement learning algorithm based on the AC framework. The Actor and Critic parts are fitted by two neural networks to fit the value function and strategy respectively. In the value function part, you can choose to maintain only one value function Q, or maintain both value functions V and Q at the same time. Maintaining two value functions without introducing the method of automatically adjusting β can make training more stable. The value function iteration formula and strategy optimization formula are defined as follows:

[0157]

[0158]

[0159]

[0160] Here Π represents a set of optional strategies, which is actually a set of Gaussian distributions with parameters, and s' represents the next state.

[0161] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0162] Specific implementation method ten: The difference between this implementation method and specific implementation methods one to nine is that in S4.1, the model of the Markov process of the day-ahead stage scheduling is solved using the Twin Delayed Deep Deterministic policy gradient algorithm based on deep reinforcement learning to obtain the day-ahead scheduling decision π 1 * ,

[0163] In S4.2, the model of the Markov process for day-ahead scheduling is solved using the Twin Delayed Deep Deterministic Policy Gradient algorithm based on deep reinforcement learning to obtain the intraday scheduling decision π 2 * , that is, a two-stage low-carbon economic dispatch strategy.

[0164] The TwinDelayed DeepDeterministic policy gradient (TD3) algorithm has made improvements to the overestimation of the value function and the variance problem in the AC structure. The algorithm estimates the value of the state-action pairs generated by the actor by using two critic networks, and selects the smaller of the two estimated values ​​to limit the critic's overestimation of the state-action value. At the same time, in order to solve the noisy gradient caused by the variance to the policy update, the algorithm also uses the delayed update method to reduce the error of the policy update, so that the update frequency of the valuation network is higher than that of the policy network to reduce the estimation error first. In order to stabilize the training, the target network is added to the actor network and the two critic networks in the algorithm, and soft updates are used to update the target network. The TD3 algorithm uses a regularization method to smooth the target policy in solving the problem of overfitting of the valuation function. The specific implementation is to add a clipped small variance noise that follows a normal distribution to the target policy, and use the mean of a small batch to approximate the action expectation. The specific formula is as follows:

[0165]

[0166] Wherein θ' is the parameter of the Critic target network, φ' is the parameter of the Actor target network. Other steps and parameters are the same as those in the first to eighth embodiments.

[0167] This embodiment is described with reference to specific embodiments 1 to 10.

[0168] In order to verify the effectiveness of the two-stage model proposed in this invention in reducing carbon emissions, lowering operating costs, and ensuring power transmission safety, a simplified power system simulation scenario was established. Figure 2 The simulation system includes a wind farm, a photovoltaic power station, an energy storage system, and two carbon capture power plants equipped with three thermal power units. In order to simplify the problem and exclude the impact of the start and stop time of the thermal power units, these three thermal power units will not be shut down as long as they are started within a scheduling cycle.

[0169] The whole system is divided into two parts, one is the microgrid containing the power load, and the other is the main grid. The two grids exchange power through transmission lines. Wind power and photovoltaic power generation are the total power generation of wind farms and photovoltaic power stations respectively. Wind power data, photovoltaic data and power load data are taken from Data platform–open power system data. https: / / data.open-power-system-data.org / . Other system component related parameters are based on the literature Hou Shengren, Edgar Mauricio Salazar, Pedro P Vergara, and Peter Palensky. Performance comparison of deep rl algorithms for energy systems optimal scheduling. In 2022 IEEE PES Innovative Smart Grid Technologies Conference Europe (ISGT-Europe), pages 1–6. IEEE, 2022.

[0170] Yue Xiang, Gang Wu, Xiaodong Shen, Yuhang Ma, Jing Gou, Weiting Xu, and Junyong Liu. Low-carbon economic dispatch of electricity-gas systems. Energy, 226: 120-267, July 2021

[0171] SuyangZhou,ZijianHu,Wei Gu,MengJiang,Meng Chen,QitengHong,andCampbellBooth.Combined heat and power systemintelligent economic dispatch:A deep reinforcement learning approach.International journal of electrical power & energy systems,120:106016,2020 modified,

[0172] See Tables 1, 2, and 3 for details. Figure 2 As shown,

[0173] Table 1. Thermal power unit parameters

[0174]

[0175] Table 2. Energy storage and transmission parameters

[0176]

[0177] Table 3. Carbon capture and training parameters

[0178]

[0179]

[0180] The results of the training today:

[0181] Tables 4 and 5 show the impact of different weights of the penalty for exceeding the transmission limit on the scheduling results of different algorithms (the results are obtained from the average of the data on the date used for training). As shown in Table 4, the probability of exceeding the transmission power limit decreases with the increase of the penalty weight. When the penalty weight is greater than 0.5, the scheduling strategies obtained by the three algorithms will be more inclined to transmission safety and reduce transmission exceeding the limit. As shown in Table 5, the initial row in the table corresponds to different penalty weight values. Each column shows the operating cost, carbon emissions and transmission power of the results obtained by different algorithms. These values ​​are obtained by summing and averaging the corresponding data of each day of training. As shown in Table 5, when the penalty weight for exceeding the limit is greater than 0.5 (when the penalty weight for exceeding the limit is equal to 1, the carbon emissions generated by the microgrid increase significantly), the carbon emissions generated by the microgrid remain basically unchanged. Also as shown in Table 5, the operating cost of the microgrid increases with the increase of the penalty weight and increases with the decrease of the power on the transmission line. This means that the increase in the penalty weight leads to a decrease in electricity sales, increases the operating cost of the microgrid, and enhances the safety of the transmission line. Therefore, a penalty weight of 0.5 is selected to reduce the operating costs and carbon emissions of the system while ensuring the safe operation of the power system as much as possible.

[0182] Figure 3The effects of using different over-limit penalty weight parameters in day-ahead training using different algorithms on the changes in operating costs, carbon emissions, and transmission power are shown. Due to space limitations, the figure only shows the impact of one day (July 9). Fig.3(a), Fig.3(b), and Fig.3(c) are the changes in system operating costs, system carbon emissions, and transmission power obtained by the model using the PPO algorithm. The corresponding Fig.3(d), Fig.3(e), and Fig.3(f) are the results of using the SAC algorithm, and Fig.3(g), Fig.3(h), and Fig.3(i) are the results of using the TD3 algorithm. Fig.3(a), Fig.3(d), and Fig.3(g) show that as the value of the over-limit penalty weight parameter continues to increase, the operating cost continues to increase, and the strategy generated by the intelligent agent will pay more and more attention to the safety of power transmission. As the intelligent agent chooses a more conservative strategy for power transmission, the power transmission power between the microgrid and the main grid is also decreasing. From Fig.3(c), Fig.3(f), and Fig.3(i), we can see that as the over-limit penalty weight increases, the transmission power is decreasing. The change in carbon emissions can be seen from Fig.3(b), Fig.3(e), and Fig.3(h). As the over-limit penalty weight increases, the carbon emissions fluctuate slightly. Only in extreme cases (when the over-limit penalty weight is 1) do carbon emissions increase suddenly. Combined with Tables 4 and 5, when the penalty weight is set to 0.5, it can not only ensure the safe operation of the power system to a large extent, but also reduce the operating cost and carbon emissions of the power system. Therefore, different parameter combinations can be designed to formulate scheduling strategies to achieve different goals. The cost, carbon emissions, and transmission power curves under different weights are shown in Figure 3(b), Figure 3(e), and Figure 3(h). Figure 3 .

[0183] Table 4. Average values ​​under different over-limit penalty weights

[0184]

[0185]

[0186] Table 5. Percentage of time when the limit is exceeded

[0187]

[0188] Figure 4 The following are the day-ahead dispatch results of the three algorithms on July 9 when the over-limit penalty weight parameter is 0.5. It can be seen from the figure that the load demand is met and renewable energy is consumed first. The strategy obtained by the PPO algorithm is more inclined to generate more power to obtain benefits from the main grid. SAC and TD3 also learned this strategy in a similar way, but the electricity sold is different from PPO, and the benefits obtained are not as high as PPO.

[0189] The results of the day-ahead training obtained weight parameters with good comprehensive effects, but this parameter was obtained based on the day-ahead forecast data. The data of renewable energy and load cannot be accurately predicted, and the weight parameters obtained the day-ahead may cause transmission line safety problems during intraday scheduling. Therefore, intraday data is needed to perform intraday optimization and obtain a new intraday scheduling strategy. Intraday optimization results:

[0190] The appropriate weight parameters were obtained in the first stage, and the transmission line was operated at a safe power level as much as possible while considering the operating cost. However, the uncertainty of renewable energy and the error of load forecasting may make the weights of the first stage invalid. Therefore, the second stage is required to use the weights obtained in the first stage for intraday optimization. The day-ahead forecast data and intraday short-term forecast data for July 9 are as follows: Figure 5 It can be seen that there is a certain error in the prediction, and the uncertainty of renewable energy can be clearly seen through the wind power data. The strategy obtained by directly using the weights obtained in the first stage for intraday training is likely to perform poorly or even fail in this case.

[0191] Figure 6 The three algorithms are compared on July 9th, without and after the second-stage optimization of the inter-network transmission line. It can be seen from the figure that the weight parameters of the TD3 algorithm still perform well and do not need to be optimized in the second stage. However, for the PPO algorithm and the SAC algorithm, the strategy scheduling results have exceeded the limit. The two algorithms are optimized intraday, and the over-limit penalty weight of PPO is increased to 0.55 or adjusted by adding conservative safety limit constraints. The over-limit penalty weight of SAC is increased to 0.6 or adjusted by conservative safety limit constraints. It can be seen from the figure that the two intraday adjustment methods proposed by the present invention can cope with the over-limit situation caused by the uncertainty of renewable energy and inaccurate load forecasting, so that the power of the transmission line can be optimized. PPO originally had an over-limit situation, but the over-limit situation disappeared after intraday optimization. The original three over-limit situations of SAC disappeared after the intraday optimization using weight adjustment, and the number and power of over-limits were reduced after the intraday optimization using conservative safety limit constraints.

[0192] Figure 7 The carbon emission results and operating cost results before and after the two intraday optimization methods are compared. It can be seen from the figure that after optimization, the intraday carbon emissions and total operating costs have not increased significantly. Figure 8 The optimized dispatching results of the two methods are shown, and the optimized results also meet the load demand. The above results show the effectiveness of the two intra-day optimization methods of the present invention for transmission power security.

[0193] The present invention proposes a two-stage optimization model using a deep reinforcement learning algorithm based on an AC framework for the energy system optimization scheduling problem considering low carbon and transmission line safety limits. Two intraday optimization methods are proposed to deal with the uncertainty of renewable energy and load forecasting errors. Experimental results show that a large amount of preliminary training and learning can be carried out through the two-stage model deep reinforcement learning algorithm of the present invention to obtain an intelligent agent with a limited number of training steps, through which scheduling plans can be generated almost in real time. The trained intelligent agent strategy has certain fault tolerance and flexibility, but it also requires intraday optimization and adjustment. Ultimately, the scheduling problem of the present invention can be better solved.

[0194] The above only describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the above specific implementation methods. Although the present invention has been disclosed as above in the preferred embodiments, it is not used to limit the present invention. Any technician familiar with the profession can make some changes or modify the technical contents disclosed above into equivalent embodiments with equivalent changes without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement made to the above embodiments without departing from the content of the technical solution of the present invention, based on the technical essence of the present invention, within the spirit and principles of the present invention, still fall within the protection scope of the technical solution of the present invention.

Claims

1. A two-stage low-carbon economic dispatch method based on deep reinforcement learning, characterized in that: The following steps are involved: S1: Construct dataset; S2: Construct a two-stage low-carbon economic dispatch model based on deep reinforcement learning; S3: Convert the two-stage low-carbon economic dispatch model based on deep reinforcement learning into a Markov decision model; S4: Solve the Markov decision model of the two-stage low-carbon economic dispatch model based on deep reinforcement learning and obtain the two-stage low-carbon economic dispatch.

2. The two-stage low-carbon economic dispatching method based on deep reinforcement learning according to claim 1 is characterized in that: The data set constructed in S1 includes: historical wind power generation data, historical photovoltaic power generation data and historical power load data; The historical power load data includes: time-of-use electricity price and load of the microgrid power system.

3. The two-stage low-carbon economic dispatching method based on deep reinforcement learning according to claim 2 is characterized in that: In S2, a two-stage low-carbon economic dispatch model based on deep reinforcement learning is constructed; the specific process is as follows: S2.1: Constructing the objective function of the two-stage low-carbon economic dispatch model based on deep reinforcement learning; the objective function of the two-stage low-carbon economic dispatch model based on deep reinforcement learning includes: energy supply cost C ES , the cost of using energy storage equipment C B , CO2 emission penalty cost C penal and CO2 storage and transmission costs C ST ; S2.2: Construct constraints for a two-stage low-carbon economic dispatch model based on deep reinforcement learning; the constraints of the two-stage low-carbon economic dispatch model based on deep reinforcement learning include: supply and demand balance constraints, transmission power limit constraints, thermal power unit constraints, carbon capture system constraints and energy storage equipment constraints.

4. The two-stage low-carbon economic dispatching method based on deep reinforcement learning according to claim 3 is characterized in that: The specific process of constructing the objective function of the two-stage low-carbon economic dispatch model based on deep reinforcement learning in S2.1 is: The objective function formula is as follows: min(C ES +C ST +C penal +C B ) (1) Among them, P t,a,ex is the transmission power between the main grid and the microgrid at time step t, p t,grid is the time-of-use electricity price, d c The unit cost of storing and transporting captured carbon for the cth carbon capture device, k b is the unit cost of charging and discharging of the bth energy storage device, h c The penalty cost for each unit of carbon emissions not captured by the cth carbon capture device, is the output of the g-th thermal power unit at time step t, a g ,b g ,c g is the power generation cost coefficient of the g-th thermal power unit, G represents the thermal power unit set, C represents the carbon capture equipment set, B represents the energy storage equipment set, T represents the time step set, represents the output of the bth energy storage device at time step t, represents the carbon capture amount of the cth carbon capture device at time step t, represents the amount of carbon emissions that are not captured at time step t.

5. The two-stage low-carbon economic dispatching method based on deep reinforcement learning according to claim 4 is characterized in that: The constraints of the two-stage low-carbon economic dispatch model based on deep reinforcement learning are constructed in S2.

2. The specific process is as follows: Supply and demand balance constraints: in, represents the load demand of microgrid a at time step t, represents the energy required for the cth carbon capture device at time step t, represents the output of the s-th solar power supply device at time step t, represents the output of the w-th wind power supply device at time step t, and S represents the set of solar power supply devices; Transmission power limit constraints: P t,a,ex ≤P transMax (8) Among them, P transMax Indicates the maximum transmission power of the transmission line; Constraints on thermal power units: in, RU g ,RD g respectively Indicates the lower limit of the output of thermal power units. Indicates the upper limit of the output of thermal power units, RU g Indicates the upper limit of thermal power unit climbing, RD g Indicates the upper limit of downward climbing of thermal power units; Carbon capture system constraints: 0≤γ c ≤0.9 (15) in, represents the carbon capture amount of the cth carbon capture device at time step t, Q t,g represents the emission of the g-th thermal power unit at time step t, represents the maximum carbon capture capacity of the cth carbon capture device at time step t, γ c represents the carbon capture rate of the cth carbon capture device, u c represents the unit power consumption of the cth carbon capture device, α g represents the carbon emission intensity of the g-th thermal power unit; QR represents the downward ramp limit of carbon capture and represents the upper limit of the upward slope of carbon capture; Energy storage equipment constraints: in represents the storage capacity percentage of energy storage device b at time step t, represents the minimum percentage of energy storage capacity of energy storage device b at time step t, Indicates the maximum percentage of power storage of energy storage device b at time step t; Indicates the maximum capacity of the energy storage device, Indicates the lower limit of charge and discharge of energy storage equipment and Indicates the upper limit of charging and discharging of energy storage equipment.

6. The two-stage low-carbon economic dispatching method based on deep reinforcement learning according to claim 5 is characterized in that: In S3, the two-stage low-carbon economic dispatch model based on deep reinforcement learning is converted into a corresponding Markov decision model; the specific process is: The two-stage low-carbon economic dispatch based on deep reinforcement learning includes day-ahead dispatch and intraday dispatch. The two-stage low-carbon economic dispatch model based on deep reinforcement learning is replaced by a Markov decision model for day-ahead dispatch and a Markov decision model for intraday dispatch. The decision process of the Markov decision model for the day-ahead stage scheduling and the intraday stage scheduling is: At each time step t, the Markov decision model receives a state s t , and select an action a according to the current policy π t , and the action a t Applied in the actual power system environment, the reward and punishment information r and the next state s fed back from the actual power system environment are obtained. t+1 , repeat this process until the entire scheduling cycle is completed; The Markov decision model is mainly composed of a five-tuple data (S, A, P, R, λ), including: state set S, action set A, reward function R, state transition probability P and discount factor λ.

7. The two-stage low-carbon economic dispatch method based on deep reinforcement learning according to claim 6 is characterized in that: In S4, the Markov decision model of the two-stage low-carbon economic dispatch model based on deep reinforcement learning is solved to obtain the two-stage low-carbon economic dispatch; the specific process is: S4.1: Solve the Markov decision model for day-ahead scheduling; obtain the day-ahead scheduling decision; the specific process is: The Markov decision model for the day-ahead scheduling is: State set S: For each day-ahead state at time step t1, eight variables are used to describe the state of the day-ahead scheduling phase. The formula is as follows: Among them, p is the time-of-use electricity price, l is the load of the microgrid power system, w is the wind power output, pv is the photovoltaic output, soc is the energy storage percentage of the energy storage system, g is the output power of the thermal power unit, c emi for carbon emissions; Action set A: The Markov decision model receives the state of the day-ahead scheduling phase Generate corresponding actions according to the day-ahead scheduling strategy π1 And the action Applied in actual power system environment, action It consists of three variables, expressed as: Among them, the parameter g change Respectively represent the percentage change of thermal power unit output, ccs change Indicates the percentage change in carbon capture, soc change Energy storage system energy storage change percentage; Reward function R: When the action After being applied in the environment, rewards are fed back based on changes in the environment The formula is: in is the action reward, penal overline is the penalty for exceeding the power exchange limit; σ1, σ2 are weights; State transition probability P: set to 1; the model of the Markov process of the day-ahead scheduling is solved using the proximal strategy optimization algorithm based on Actor-Critic to obtain the day-ahead scheduling decision π1 * ; S4.2: The day-ahead scheduling decision π1 obtained in S4.1 * As the initial decision for intraday stage scheduling; solve the Markov decision model for intraday stage scheduling; obtain the intraday scheduling decision; the specific process is: The Markov decision model for intraday stage scheduling is: State set S: For each intraday state time step t2, 8 variables are used to describe the state of the intraday scheduling phase The formula is as follows: Among them, p is the time-of-use electricity price, l is the load of the microgrid power system, w is the wind power output, pv is the photovoltaic output, soc is the energy storage percentage of the energy storage system, g is the output power of the thermal power unit, c emi for carbon emissions; Action set A: The Markov decision model receives the status of the intraday scheduling stage Generate corresponding actions according to the intraday scheduling strategy π2 And the action Applied in actual power system environment, action It consists of three variables, expressed as: Among them, the parameter g change Respectively represent the percentage change of thermal power unit output, ccs change Indicates the percentage change in carbon capture, soc change Energy storage system energy storage change percentage; Reward function R: When the action After being applied in the environment, rewards are fed back based on changes in the environment The formula is: in is the action reward, penal overline is the penalty for exceeding the power exchange limit, σ1, σ2 are weights; state transition probability P: set to 1; The model of the Markov process of intraday stage scheduling is solved using the proximal strategy optimization algorithm based on Actor-Critic to obtain the intraday scheduling decision π2 * , that is, a two-stage low-carbon economic dispatch strategy.

8. The two-stage low-carbon economic dispatching method based on deep reinforcement learning according to claim 7 is characterized in that: According to the action in S4.2 After being applied in the environment, rewards are fed back based on changes in the environment The penalty for exceeding the power exchange limit is penal overline Determined according to the transmission power between the main grid and the microgrid, the specific process is: When the transmission power between the main grid and the microgrid is below the safety margin limit, the penalty for power exchange exceeding the limit is overline The formula is: When the transmission power between the main grid and the microgrid is within the buffer area, the penalty for exceeding the power exchange limit is overline The formula is: When the transmission power between the main grid and the microgrid is above the safety limit in the day-ahead dispatching phase, the penalty for power exchange exceeding the limit is overline The formula is: Among them, α1, α2 are induced hyperparameters, P t,transMax represents the maximum transmission power of the transmission line at time step t, P t,a,ex is the transmission power between the main grid and the microgrid at time step t.

9. The two-stage low-carbon economic dispatch method based on deep reinforcement learning according to claim 8 is characterized in that: In S4.1, the model of the Markov process of the day-ahead scheduling is solved using the SoftActor-Critic algorithm based on Actor-Critic to obtain the day-ahead scheduling decision π1 * , In S4.2, the model of the Markov process for the day-ahead scheduling is solved using the SoftActor-Critic algorithm based on Actor-Critic to obtain the intraday scheduling decision π2 * , that is, a two-stage low-carbon economic dispatch strategy.

10. The two-stage low-carbon economic dispatch method based on deep reinforcement learning according to claim 8 is characterized in that: In S4.1, the model of the Markov process of the day-ahead scheduling is solved using the Actor-Critic based TwinDelayed Deep Deterministic policy gradient algorithm to obtain the day-ahead scheduling decision π1 * , In S4.2, the model of the Markov process for day-ahead scheduling is solved using the Actor-Critic based TwinDelayed Deep Deterministic policy gradient algorithm to obtain the intraday scheduling decision π2 * , that is, a two-stage low-carbon economic dispatch strategy.

Images