Scheduling optimization method for energy storage power generator in integrated energy system
By constructing an intelligent agent model for energy storage power generators and using reinforcement learning methods, the quantity and price coordinated scheduling strategy of energy storage power generators is optimized. This solves the problem of discrepancy between the scheduling instructions of energy storage power generators and the actual regulation capacity in traditional scheduling strategies, and realizes the efficient operation and market adaptability of energy storage power generators in integrated energy systems.
Patent Information
- Application Number
- CN202511673057.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-14
- Publication Date
- 2026-03-03
AI Technical Summary
In integrated energy systems with a high proportion of renewable energy, traditional dispatch strategies fail to fully utilize the rapid response capabilities of energy storage generators, resulting in discrepancies between dispatch instructions and actual adjustment capabilities. Furthermore, the imperfect dispatch methods in a market-oriented operating environment reduce the participation enthusiasm of energy storage generators.
A smart agent model for energy storage power generators is constructed, and iterative training is carried out using reinforcement learning methods to optimize the quantity and price coordinated scheduling strategy of energy storage power generators. By constructing a Markov decision process and a deep deterministic policy gradient algorithm, the operational efficiency of energy storage power generators is maximized. Joint training is carried out in the energy storage capacity sharing market to form the optimal scheduling strategy of the energy storage power generator alliance.
It improves the dispatch accuracy and operational efficiency of energy storage generators, solves the problem of deviation between dispatch instructions and actual regulation capacity, enhances the adaptability and robustness of energy storage generators in a market environment, and optimizes system regulation efficiency.
Smart Images
Figure CN121599346A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of power system dispatching technology, and more specifically, relates to a dispatching optimization method for energy storage generators in an integrated energy system. Background Technology
[0002] As the proportion of renewable energy sources such as wind and solar power in integrated energy systems continues to rise, the randomness and volatility of their output place higher demands on the real-time balance of the power grid. The infrastructure construction and capacity-sharing spot market for large-scale energy storage are gradually improving. Meanwhile, electrochemical energy storage devices in energy storage generators possess the important attribute of power generation, enabling precise power adjustment on a second-level timescale, which is a core technological approach to smoothing fluctuations and mitigating peak-valley differences. However, under the existing dispatch framework, energy storage often operates according to fixed plans or simple price difference rules, and its rapid response capability is not fully utilized, making it difficult to obtain continuous, reliable, and flexible support from the system side.
[0003] In power system dispatching and market-oriented operation scenarios, traditional energy storage dispatching models often position energy storage devices as passive price takers. They fail to consider the active response capability of energy storage to electricity price signals or quantify the marginal costs corresponding to different output ranges, easily leading to discrepancies between dispatching instructions and the actual adjustment capabilities of energy storage. While energy storage generators can improve their operating revenue through price difference adjustment and participation in ancillary service dispatching in a market-oriented environment, current dispatching methods and trading mechanisms for them in competitive electricity markets are still imperfect. This, to some extent, restricts the economic viability of energy storage generators participating in market-oriented dispatching and reduces their enthusiasm for participation. Against this backdrop, treating energy storage generators in the energy storage capacity-sharing market as independent decision-making entities and studying scientifically sound and reasonable quantity-price coordinated dispatching application strategies is crucial for promoting the marketization of energy storage resources and improving the efficiency of power system resource allocation. Summary of the Invention
[0004] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a scheduling optimization method for energy storage generators in integrated energy systems. Its purpose is to solve the problem of discrepancies between scheduling instructions and the actual adjustment capabilities of energy storage caused by insufficient quantity and price coordination in traditional scheduling strategies under high-proportion renewable energy access scenarios.
[0005] To achieve the above objectives, the present invention provides a scheduling optimization method for energy storage generators in an integrated energy system, comprising: Energy storage generator As an intelligent entity of energy storage power generation With energy storage power generation e-commerce smart body The optimization objective is to maximize operational efficiency, including intelligent systems for energy storage and power generation. Constraints of each scheduling bid capacity interval and their in the 1st Scheduling price parameters within the selected capacity range Under constraints including those related to energy storage and power generation, construct an intelligent energy storage and power generation system. The operating model; , , The total number of energy storage generators in the integrated energy system. This represents the total number of successful bid capacity intervals submitted for each scheduling period; The operational model is solved using reinforcement learning, including: Building an intelligent energy storage and power generation system During the current scheduling period state space and action space : , , and These represent the demand side of the integrated energy system during the previous and current scheduling periods, respectively. The energy storage demand, and These represent the intelligent entities of energy storage generators in the previous scheduling period. Marginal clearing price and allocated capacity; and These represent the current scheduling period. Energy storage generator smart body The endpoint values of each scheduling bid capacity interval and the corresponding scheduling bidding coefficient, and the scheduling price parameter This equals the unit actual operating cost within the corresponding scheduling bid capacity range and the scheduling bidding coefficient. The product; Using the aforementioned optimization objective as an immediate reward, the intelligent agent of the energy storage generator is... During iterative training, when the agent converges, a representation of the energy storage generator is obtained. The capacity range of each scheduling action and the aforementioned scheduling price parameters This is to optimize the scheduling of the corresponding energy storage generators.
[0006] Furthermore, it also includes: jointly training intelligent agents of energy storage and power generation companies belonging to the same interest group within the integrated energy system, wherein the action space of the joint training is... , This refers to the total number of smart energy storage generators belonging to the same interest group within an integrated energy system. Indicates the intelligent body of energy storage power generation The space of motion ; The state space of the joint training is: ; The instant reward is to maximize the benefits for those belonging to the same interest group. Overall operational efficiency of an energy storage power generation intelligent system.
[0007] Furthermore, the operating model is as follows:
[0008]
[0009]
[0010]
[0011]
[0012] in, Indicates the intelligent body of energy storage power generation operational efficiency , Intelligent bodies for energy storage generators in each scheduling period In the The scheduling bidding coefficient and unit actual operating cost within each scheduling bid capacity range. To set the upper limit of the scheduling bidding coefficient, The set of indices for the scheduled capacity range. ; Intelligent body for energy storage generators in each scheduling period The marginal clearing price; , They are respectively energy storage generator smart bodies The allocated capacity in each scheduling period and in the... The scheduling bid-winning capacity of each scheduling bid-winning capacity range; and Intelligent bodies for energy storage generators in each scheduling period In the The endpoint values of each scheduled capacity interval. hour, , hour, , For energy storage power generation e-commerce intelligent body The idle capacity range; For energy storage power generation e-commerce intelligent body The energy storage cost function.
[0013] Furthermore, the intelligent body of energy storage generators in each scheduling period In the The unit real operating cost within the allocated capacity range of each scheduling bid The calculation method is as follows:
[0014] In the formula, , They are respectively energy storage generator smart bodies In the The cost per kilowatt-hour and the unit loss cost of the rated capacity of energy storage within the selected dispatch capacity range; among which... The cost was calculated using the levelized cost of energy storage (LDC) method. The calculation method is as follows:
[0015] In the formula, Indicates the intelligent body of energy storage power generation Corresponding energy storage generator The total discounted cost of the configured energy storage device over its entire life cycle. , , , They are respectively energy storage generator smart bodies In the Total dispatchable energy storage capacity, self-discharge loss, cryogenic loss, and cycle loss within the dispatchable capacity range.
[0016] Furthermore, a deep deterministic strategy gradient algorithm is adopted for the intelligent agent of energy storage power generators. Perform iterative training, or perform the joint training described above; The intelligent agent of the energy storage power generator is built based on a multilayer sensor. .
[0017] This invention also provides a unified clearing method for a spot market for shared energy storage capacity in an integrated energy system, comprising: Based on the capacity range of each scheduling bid and the aforementioned scheduling price parameters A unified market clearing process will be implemented to obtain the necessary permits from energy storage power generators during each dispatch period. The scheduling bid capacity in each scheduling bid capacity interval ; Among them, the respective scheduling bid capacity intervals and the aforementioned scheduling price parameters The scheduling optimization method described above is used to determine the method.
[0018] Furthermore, a unified market clearing model is adopted for unified market clearing, wherein the unified market clearing model is as follows:
[0019]
[0020]
[0021]
[0022]
[0023] in, , , , ; This represents the set of indexes for energy storage generators. ; For the set of electricity demand-side nodes, As a demand-side node of the integrated energy system The energy storage demand, , Let be a Lagrange multiplier, representing the marginal price of the system; and They represent energy storage generators. With electricity demand-side nodes To the line The power transfer distribution factor, For the line Maximum permissible current, Represents a set of routes; and The lines are respectively Lagrange multipliers for the maximum positive and negative power flow constraints; and They are energy storage generators The minimum and maximum scheduling bid capacity.
[0024] The present invention also provides an electronic device, including a computer-readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is configured to read executable instructions stored in the computer-readable storage medium and execute any of the above-described scheduling optimization methods, or / and execute the above-described unified clearing method.
[0025] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the scheduling optimization method as described above, or / and the unified clearing method as described above.
[0026] The present invention also provides a computer program product, including a computer program that, when the computer program is run on a computer, causes the computer to execute any of the scheduling optimization methods described above, or / and execute the unified clearing method described above.
[0027] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects: (1) The scheduling optimization method for energy storage generators in integrated energy systems of the present invention realizes the coordinated optimization of the output scheduling plan (representation of the bidding capacity range) and the electricity price signal of the energy storage generator, which can improve the regulation accuracy and maximize the operating benefits of the energy storage generator. Specifically, the present invention constructs a scheduling optimization model with the goal of maximizing the operating benefits of the energy storage generator. The energy storage generator adjusts the quantity and price coordination strategy through reinforcement learning to achieve active response to the electricity price signal. At the same time, it divides the bidding capacity range into multiple segments and designs the scheduling price parameter as the product of the unit real operating cost of the corresponding range and the scheduling bidding coefficient to quantify the marginal cost difference of different output ranges. This invention models the scheduling process as a Markov decision process, where the agent only observes key information and does not involve the private information of other power generators, which fits the actual market scenario. By using reinforcement learning technology to learn the segmented output-cost characteristics of each energy storage power generator, a scheduling scheme that takes into account both system supply and demand balance and equipment economy is dynamically generated. This effectively solves the problem of discrepancies between scheduling instructions and the actual adjustment capacity and cost of energy storage caused by insufficient quantity and price coordination in traditional scheduling strategies under high-proportion renewable energy access scenarios, thereby improving scheduling gain.
[0028] (2) Furthermore, the present invention constructs a multi-energy storage generator scheduling training environment with a hybrid competition-cooperation relationship. By treating energy storage generators belonging to the same interest group as an intelligent agent alliance for joint training, the optimal quantity and price collaborative scheduling application strategy of the energy storage generator alliance is generated to maximize the overall operating efficiency of the same interest group. This can effectively solve the problem of information asymmetry interaction faced by energy storage generators in the existing energy storage capacity sharing spot market scheduling.
[0029] (3) As a preferred option, in the calculation of the intelligent agent of energy storage generator. In the Unit real operating cost under each energy storage capacity range The cost per kilowatt-hour of energy storage equipment and the capacity loss cost of energy storage in the integrated energy system are considered. The levelized cost of energy storage generators is calculated using the energy storage cost method, and a correction term for rated capacity loss of energy storage is introduced on this basis to more accurately reflect the real cost of energy storage devices in long-term operation.
[0030] Overall, this invention embeds a "power output-price" collaborative decision-making mechanism at the dispatching end, enabling energy storage to meet system regulation needs while also considering its own economic operation objectives. This significantly supports increasing energy storage penetration and optimizing system regulation efficiency. In the complex, hybrid cooperative-competitive market environment of integrated energy systems, this invention, based on a data-driven reinforcement learning framework, jointly models and optimizes the adjustability and response price of smart energy storage generators. This allows smart energy storage generators to adapt to complex and rapidly changing market environments, achieving automatic dispatching and coordinated economic operation control of energy storage generator alliances in the integrated energy system's electricity market. This invention significantly improves the dispatching revenue of smart energy storage generators, enhances the adaptability and robustness of their dispatching strategies in the energy storage capacity sharing spot market of integrated energy systems, effectively handles information asymmetric interaction scenarios, and optimizes complex market hybrid relationships, making the dispatching strategies of energy storage generators more feasible and engineering-significant. Attached Figure Description
[0031] Figure 1 This is a diagram illustrating the intelligent scheduling optimization method for energy storage power generation alliances in integrated energy systems provided in this embodiment of the invention. Figure 2 This is a basic framework diagram of the energy storage capacity sharing spot market in the embodiments of the present invention; Figure 3 A basic framework diagram of Markov decision process in energy storage capacity sharing market scheduling provided for embodiments of the present invention; Figure 4 This is a training structure diagram of the deep deterministic policy gradient algorithm provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the IEEE-57 node system topology and energy storage generator distribution in an embodiment of the present invention; Figure 6 The diagram shows the training and scheduling results of the DDPG algorithm under different scheduling methods in this embodiment of the invention. Figure 7 This is a diagram showing the training scheduling results of different reinforcement learning algorithms in this embodiment of the invention; Figure 8 This is a diagram showing the training and scheduling results of the DDPG algorithm for the energy storage power generation alliance under a hybrid relationship in this embodiment of the invention. Figure 9 This diagram illustrates the monthly revenue scheduling results implemented by various energy storage generators under the hybrid relationship in this embodiment of the invention. Detailed Implementation
[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.
[0033] Example 1 This invention provides a scheduling optimization method for energy storage generators in integrated energy systems, mainly including: 1) Construct a scheduling optimization model for energy storage generators in an integrated energy system and a clearing model for the energy storage capacity sharing spot market; 2) Establish a scheduling agent model for the energy storage alliance based on Markov decision processes; 3) Construct a scheduling training environment with a hybrid competition-cooperation relationship; 4) Train the intelligent energy storage generator alliance using a deep deterministic policy gradient algorithm; 5) Deploy the trained agent model in the automatic online scheduling execution task for energy storage generators participating in the energy storage capacity sharing spot market, such as... Figure 1 As shown.
[0034] 1) Construct a scheduling optimization model for energy storage generators and a clearing model for the energy storage capacity sharing spot market within an integrated energy system. Wind and solar power account for a significant portion of integrated energy systems. Due to the fluctuating output of new energy units and dynamic changes in load demand, it is necessary to adjust supply and demand discrepancies by utilizing installed energy storage devices, such as… Figure 2 As shown in the figure, in the energy storage capacity sharing spot market of this invention, the dispatching and trading mechanism is as follows: energy storage generators on the supply side of the integrated energy system need to simultaneously declare dispatching capacity and corresponding price parameters, while electricity demand-side nodes only need to declare demand, without declaring prices; the final clearing of market transactions is determined by the unified clearing method of the integrated energy storage capacity sharing market. Based on this mechanism, the dispatching optimization model of energy storage generators is clearly divided into two core parts: the market dispatching declaration optimization model of energy storage generators, and the unified clearing dispatching model of the energy storage capacity sharing spot market.
[0035] During each scheduling period (trading period) of the system, energy storage generators must declare their shared energy storage capacity and corresponding scheduling price parameters to the market manager. The energy storage capacity sharing market manager, based on the energy storage demand declared by the integrated energy system operator, uses a market clearing method to determine the actual winning scheduling capacity and corresponding clearing price for each energy storage generator. In this embodiment of the invention, the constructed unified market clearing model is as follows: (1) (2) (3) (4) (5) Formula (1) represents minimizing the total energy cost at the demand-side nodes of the integrated energy system. In Formula (1), This represents the index of the application range for the winning bid capacity of energy storage generators in each dispatching period during the current dispatching time. For indexed sets, , This indicates that a total of [number] applications were submitted during the current scheduling period. Segment capacity (electrical capacity); and They represent energy storage generators. In the current scheduling period, the [number]th The bid price parameters and the winning capacity of each dispatching bid range. This represents the set of indexes for energy storage generators. , Let be the total number of energy storage generators. The constraints include formulas (2)-(5), where formula (2) is the supply and demand balance constraint, and . For the set of electricity demand-side nodes, As a demand-side node of the integrated energy system The demand for energy storage power. Let be a Lagrange multiplier, representing the marginal price of the system. For energy storage generators The total actual scheduled capacity during the current scheduling period Formula (3) represents the power flow constraint for the line, where, and They represent energy storage generators. With electricity demand-side nodes To the line The power transfer distribution factor, For the line Maximum permissible current, This represents a set of routes. and The lines are respectively The Lagrange multipliers of the maximum positive and negative power flow constraints. Equation (4) represents the energy storage generator. The actual scheduling bid capacity range constraint. and They are energy storage generators The minimum and maximum scheduling bid capacity. Formula (5) represents the minimum and maximum scheduling bid capacity of the energy storage generator. In the Upper and lower bound constraints on the dispatchable bid capacity within each energy storage capacity range (dispatch bid capacity range), and They are energy storage generators In the The minimum and maximum scheduled bid capacity for each scheduled bid capacity interval.
[0036] Based on the market clearing and scheduling results of energy storage capacity sharing in formula (1), each energy storage power generator... Marginal clearing price Represented as: (6) Multiple intelligent agents for energy storage power generators are constructed. Each intelligent agent is used to determine the various energy storage capacity ranges and dispatch price parameters declared by its corresponding energy storage power generator during the current dispatch period. In this embodiment of the invention, the intelligent agents for energy storage power generators... With energy storage generators Correspondingly, the intelligent body of energy storage power generation Used to determine its corresponding energy storage generator The various energy storage capacity ranges declared and scheduling price parameters , At least one energy storage generator uses the energy storage generator intelligent agent designed in this embodiment to determine the capacity and scheduling price parameters for each segment. When all energy storage generators use the energy storage generator intelligent agent designed in this invention to determine the capacity and scheduling price parameters for each segment, the total number of energy storage generator intelligent agents is the same as the total number of energy storage generators, and each energy storage generator intelligent agent corresponds to one energy storage generator. In this embodiment, the energy storage generator intelligent agent... In the The declared price for each energy storage capacity range Based on the actual cost of energy storage, multiplied by the dispatch bidding coefficient ,Right now , Indicates the intelligent body of energy storage power generation In the The unit real operating cost under each energy storage capacity range Corresponding to the above Energy storage power generation smart body By adjusting the declared energy storage capacity ranges (i.e. ) and scheduling bidding coefficient To participate in the energy storage capacity sharing market dispatch, the objective function of the energy storage power generator intelligent agent is to maximize its own operational efficiency. The operating model is as follows: (7) (8) (9) (10) (11) Formula (7) represents the maximum energy storage generator intelligent agent. The operational efficiency, in formula (7), For energy storage power generation e-commerce intelligent body operational efficiency For energy storage power generation e-commerce intelligent body The node (energy storage generator) The marginal clearing price of ), i.e. Equivalent to energy storage generator Marginal clearing price , For energy storage power generation e-commerce intelligent body The energy storage cost function, For energy storage power generation e-commerce intelligent body The allocated capacity during the current scheduling period; Formulas (8) and (9) represent the intelligent agent of the energy storage generator. Dispatch price parameters constraint, The upper limit of the dispatch bidding coefficient; formulas (10) and (11) represent the intelligent agent of the energy storage generator. Constraints on the actual dispatching capacity range of the assembled energy storage device. For energy storage power generation e-commerce intelligent body In the The dispatchable capacity of each energy storage capacity range. and As respectively and , and The clearing method of the energy storage capacity sharing spot market is used, that is, calculated by formulas (1) to (6). In the case of multi-stage scheduling application, a decision model needs to be constructed based on the scheduling application information submitted by the energy storage generator intelligent agent. Energy storage generator intelligent agent Based on their electricity consumption plan, the idle capacity range of energy storage is obtained. ,declare Segment capacity, i.e. ,in, , , For energy storage power generation e-commerce intelligent body The endpoints of the declared dispatch energy storage capacity range.
[0037] Energy storage generator smart body The energy storage cost function is: (12) In formula (12), Indicates the intelligent body of energy storage power generation In the The dispatchable capacity of each energy storage capacity range. , .
[0038] Compared to other forms of electrochemical energy storage such as lead-acid and nickel-cadmium batteries, lithium-ion battery energy storage has significant advantages such as high energy density, high operating voltage, and long cycle life, thus better meeting the dynamic dispatching needs of integrated energy systems. Considering the economic viability of energy storage generators throughout their entire lifecycle, this embodiment of the invention uses a levelized cost of energy storage (LCOE) method to calculate the LCOE, and introduces a correction term for rated capacity loss to more accurately reflect the true cost of the energy storage device during long-term operation. This represents the unit real operating cost of the integrated energy storage generator. The calculation is as follows: (13) in, For the cost per kilowatt-hour, The unit loss cost is the rated capacity of the energy storage.
[0039] The specific formula for calculating the cost per kilowatt-hour is as follows: (14) in, The period index indicates the discounting calculation and is used to represent the cost discounting and revenue discounting process at each stage of the energy storage generator's life cycle; To improve the overall efficiency of energy storage generators; The rated power of the energy storage device; The average number of charge-discharge cycles for an energy storage generator over its lifespan; The economic lifespan of energy storage generators; The discount rate; Investment costs for energy storage power generation construction; For operating and maintenance costs; and These are the unit energy cost and the unit power cost, respectively.
[0040] Furthermore, the rated capacity loss cost of the energy storage generator considered in this invention consists of self-discharge loss. Low temperature loss and cyclic loss The energy storage generator consists of three parts, and its capacity loss cost is expressed as follows: (15) (16) (17) (18) in, This represents the total discounted cost of the energy storage devices configured by the energy storage generator over their entire life cycle. This represents the total dispatchable energy storage capacity. (Including self-discharge losses) middle, for Time-storage self-discharge current, This represents the absolute temperature value of the energy storage environment. for corresponding absolute temperature , It is a constant with dimensions of absolute temperature, usually taken as... , interval This is the time interval for energy storage to settle. Indicates time. Loss at low temperatures. middle, The specific loss coefficient for energy storage generators can be identified through curve fitting. In the cycle loss... middle, This represents the cyclic decay rate.
[0041] 2) Establish a Markov decision process-based energy storage alliance scheduling agent model to solve the energy storage generator agent problem. The operating model, namely the above and , Specifically, this includes the design of a Markov game model and an agent structure for the energy storage capacity sharing spot market scheduling environment (hereinafter referred to as the "environment").
[0042] In this invention, the energy storage capacity sharing spot market clearing model is transformed into a reinforcement learning scheduling environment. In this environment, the energy storage generator's intelligent agent can continuously learn and master the market scheduling operation rules through repeated interactions with the environment, thereby generating scheduling strategies that fit actual market scheduling needs. Markov Decision Process (MDP) is a fundamental framework for reinforcement learning, mainly consisting of... Composition, in which Representing the state space, Represents the action space, Represents the state transition probability. This represents the reward function. The energy storage generator's intelligent agent is in a state... At that time, according to the quantity and price coordinated scheduling strategy Select Action And receive rewards by interacting with the environment. The subsequent state Based on state transition probability Become Repeat the above steps until the endpoint. Finally, return to the reward. It is a discount reward, where the parameters are... It is used to weigh the importance of current and future rewards. The goal of the agent is to maximize the cumulative reward within a finite time step.
[0043] Transform the energy storage capacity sharing market dispatch and clearing model into an MDP, such as Figure 3 As shown.
[0044] The energy storage capacity sharing market serves as the external environment, corresponding to market clearing. Energy storage generator intelligent agents implement actions (scheduling behaviors) according to a quantity-price collaborative scheduling strategy, obtaining rewards (scheduling benefits) and the next state (clearing result) through interaction with the environment (clearing settlement). It should be noted that energy storage capacity sharing market scheduling involves complex multi-participant interactions. Since energy storage generator intelligent agents cannot obtain scheduling parameter information from other participants, their state space, action space, and reward function need to be specifically configured, as follows: (1) State space: Intelligent agent of energy storage generator The state space contains known energy storage capacity sharing market scheduling information and feedback market clearing results; therefore, its state variables consist of four parts, namely... .in, and These represent the previous scheduling period. and the current scheduling period The total energy storage demand of the electricity demand-side nodes of the integrated energy system, to meet ; Indicates the previous scheduling period Energy storage generator smart body The marginal clearing price at the node, Indicates the previous scheduling period Energy storage generator smart body The energy storage sharing capacity that was won in the bid is the actual capacity that was won in the dispatch.
[0045] (2) Action Space: The intelligent agent of the energy storage generator needs to realize the coordinated declaration of quantity and price during the scheduling process. Therefore, the intelligent agent of the energy storage generator is set for each scheduling period. The action is That is, intelligent energy storage and power generation systems. The declared multi-stage energy storage dispatch capacity With corresponding scheduling bidding coefficient The combination of [various factors]. During each scheduling period, the agent selects its own scheduling strategy based on the observed market state.
[0046] (3) Reward function: The setting of the reward function will have a significant effect on the construction of the agent's behavior strategy and the convergence effect of the algorithm. In this embodiment of the invention, the objective function value of the energy storage generator agent scheduling decision model (i.e., the value corresponding to formula (7)) is used as the agent's immediate reward, while the cumulative reward is the sum of all the immediate rewards obtained by the agent in each transaction period.
[0047] In this embodiment of the invention, the intelligent agent of the energy storage generator can adjust according to market demand ( and ) and price changes ( Adjust its own volume and price coordinated scheduling and reporting strategy. Compared to simply adjusting price parameters, the action space is more complex and the solution is more difficult. In the Markov Decision Process (MDP) constructed above, the information that the energy storage generator agent can observe only includes the winning bid energy storage capacity, capacity price, and market demand for the two periods in the previous trading session, without covering the private scheduling information of other energy storage generators, which is consistent with the actual market scheduling interaction scenario.
[0048] 3) Constructing a hybrid competition-cooperation scheduling training environment: In actual construction and operation, power generation nodes (energy storage generators) should have a competitive-cooperative relationship. Some energy storage generators belong to the same interest group, and the scheduling behavior information of competing generators is not explicitly disclosed. Therefore, as a further design of this invention, by constructing a hybrid competition-cooperation scheduling training environment, the energy storage generator alliance participating in the energy storage capacity sharing market is regarded as a scheduling agent. This scheduling agent corresponds to the scheduling decision of the energy storage generator agent in the optimization scheduling model. Considering the hybrid competition-cooperation relationship, energy storage generators (energy storage generator alliances) with the same interest group are included in the same scheduling agent for joint training. Similarly, the system has... If there is an alliance of interests among energy storage and power generators, then the scope for joint action is as follows: Soon The operational space of each energy storage generator is pieced together. That is, for the first... An energy storage generator The declared energy storage dispatch capacity for each segment and the corresponding scheduling bidding coefficient The state space still consists of market scheduling information and the feedback of market clearing results. The immediate reward function for the scheduling agent is: That is, to maximize the overall operational efficiency belonging to the same interest group.
[0049] 4) The Deep Deterministic Policy Gradient (DDPG) algorithm is used to train the smart energy storage power generation alliance, such as... Figure 4 As shown. Alternatively, the agent in this invention can be built based on a Multi-Layer Perceptron (MLP). DDPG is a type of deep reinforcement learning algorithm, characterized by directly outputting continuous action values using a trained neural network, thus making it more suitable for technical problems involving complex action and state spaces, such as power market dispatching in integrated energy systems. In other embodiments, other commonly used algorithms can also be used for training.
[0050] The DDPG algorithm is based on an actor-critic framework, where the actor network is responsible for formulating the scheduling strategy, i.e., selecting the optimal action based on the current state. It achieves the agent's goal by mapping the state of the environment to specific actions; in this embodiment, it maximizes the profit of the energy storage power generation consortium through the scheduling strategy. The critic network is mainly used to evaluate the value of the selected actions. It measures the contribution of the current action to the overall goal by calculating the value function of the current action and provides feedback to the actor network based on this evaluation result. The main role of the critic network is to help the actor network optimize the scheduling strategy by providing feedback on the long-term value of actions, enabling the agent to choose more appropriate actions in subsequent decisions, thereby continuously improving its scheduling strategy. The Bellman equation is used for value function estimation. The estimation process of the value function is dynamic; the energy storage power generation consortium agent continuously learns and adjusts to approximate the optimal value function, enabling it to make the best scheduling decisions in complex electricity market environments. The value function formula is as follows: (19) in, It is the optimal state-action value function; It's an instant reward; It is a discount factor used to control the trade-off between immediate rewards and future rewards; This represents the state-action pair for the next time step. DDPG employs several key mechanisms to improve training stability, sample efficiency, and policy optimization in the continuous action space. These mechanisms include a target network for stable updates, empirical replay for efficient use of data, and a deterministic policy gradient for optimizing action selection.
[0051] The design of the target network is introduced. The target network is updated using a soft update rule, which involves updating the parameters of the current network to the target network with small step sizes. This avoids rapid and drastic parameter changes, ensuring the stability of the training process. The soft update rule adopts existing methods.
[0052] During training, the energy storage generator's intelligent system will learn from each round of scheduling interactions. The data is stored in an experience replay buffer. By randomly sampling subsets of these experiences (i.e., mini-batch data), the agent can break down correlations between data points, avoid potential biases during training, and thus improve sample utilization efficiency. This method accelerates the learning process of scheduling policies and avoids over-reliance on the latest scheduling experience, making training more stable and efficient.
[0053] The actor network is trained using a deterministic policy gradient, which optimizes the scheduling policy to maximize the expected reward. During this process, the actor network adjusts the scheduling policy parameters... This allows the action selection in a given state to maximize long-term rewards. The scheduling policy gradient update is calculated using the commonly used deterministic policy gradient formula. The core function of the critic network is to evaluate the value of scheduling actions by minimizing the Mean-Square Bellman Error (MSBE). MSBE measures the difference between the predicted Q-value of the scheduling action chosen by the agent in the current state and the target Q-value, which is the expected value obtained through environmental feedback and the target network. By minimizing this error, the critic network can continuously improve its value evaluation accuracy, thereby providing more accurate feedback to the actor network and helping it optimize the scheduling policy.
[0054] After the training of the intelligent agent of the energy storage power generator alliance is completed, the final scheduling scheme of the intelligent agent of the energy storage power generator in the hybrid relationship environment of the integrated energy system will be formed based on the training results.
[0055] 5) The trained alliance intelligent agent model is used in the automatic online scheduling execution task of energy storage power generators participating in the capacity sharing spot market. After the training of the energy storage power generator alliance intelligent agent is completed, the execution phase begins. The energy storage power generator alliance intelligent agent can automatically generate the final scheduling scheme in the hybrid relationship environment of the integrated energy system according to the market environment, that is... And quickly participate in the unified clearing of the spot market for energy storage capacity sharing in integrated energy systems. Specifically, For energy storage power generation e-commerce intelligent body During the current scheduling period The first internal declaration One energy storage capacity range (the capacity range for which the dispatching bid was won). In the current scheduling period The first internal declaration Energy storage capacity range The scheduling bidding coefficient is below. and Characterizes the intelligent body of energy storage power generator (Energy storage generator) The scheduling behavior of energy storage generators was optimized, thus realizing the scheduling optimization of energy storage generators.
[0056] based on Calculate according to formula (8) ,Will As ,Will and As in formula (5) and Solve the market clearing model shown in formulas (1)-(6) to obtain the current scheduling period. Energy storage power generator In the Segment capacity range scheduling winning capacity This will enable market clearing.
[0057] To make the objectives, technical solutions, and advantages of the present invention clearer, the methods in the above embodiments of the present invention will be further described in detail below with reference to examples.
[0058] A typical IEEE-57 node system is used to conduct a computational example analysis of the model and corresponding scheduling solution method proposed in the above embodiments of this invention. In this embodiment, a real-time market is used as the scenario, and the market clearing cycle is set to 1 hour. Each energy storage generator submits a scheduling application and participates in clearing once per hour, thus completing a total of 24 scheduling applications and clearings per day. The strategic hybrid competition-cooperation behavior of seven energy storage generators (G1-G7) in the electricity market is set, and they are deployed on the 7 standard generator nodes of the IEEE-57 node system. The system topology and the specific distribution of energy storage generators are as follows. Figure 5 As shown. The cost coefficients and market behaviors of each energy storage generator affect their market share and pricing power, thus playing a significant role in overall market price formation and resource allocation. Data from the entire year of 2021, publicly available on the PJM website, were used, with test cases using data from a specific month in 2022. Table 1 shows the operation and scheduling parameters of the energy storage generators. Table 2 shows the parameters of the DDPG algorithm.
[0059] Table 1 Operation and Scheduling Parameters for Energy Storage Power Generators
[0060] Table 2 DDPG Algorithm Parameters
[0061] According to the method in this embodiment of the invention, the superiority of the quantity-price coordinated scheduling application method proposed in this embodiment of the invention is first verified. Energy storage generator 6 is designated as a smart energy storage generator. For the scheduling application method based solely on price parameters, the energy storage generator only applies for the scheduling price parameters and does not apply for the actual capacity application action; for the quantity-price coordinated scheduling application method, it applies for segmented price-capacity information simultaneously. After 5000 training iterations, the training process is as follows... Figure 6 As shown in Table 3, during the initial training phase, the energy storage generator frequently adjusts the scheduling price parameters, resulting in significant fluctuations in the reward curve. As the training rounds increase, the agent accumulates more scheduling experience during the learning process, leading to a gradual increase and stabilization of the learned reward value, and the agent's scheduling strategy gradually converges. The trained strategy was then tested for one month, comparing the performance of the energy storage generator's agent strategy under the two scheduling application methods. Table 3 shows that the quantity-price coordinated scheduling application strategy outperforms the price-only scheduling strategy. For the price-only scheduling strategy, the average daily revenue was 79,257 yuan, and the total revenue was 2,377,727 yuan. For the quantity-price coordinated scheduling strategy, the average daily revenue increased to 119,251 yuan, and the total revenue was 3,577,541 yuan. The results demonstrate that compared to the price-only scheduling strategy, the quantity-price coordinated scheduling application strategy brought an approximately 50.46% increase in revenue during the testing period, exhibiting stronger market adaptability and profit-generating capabilities.
[0062] Table 3. Comparison of Dispatch and Operation Benefits of Smart Agents for Energy Storage Power Generators under Different Bidding Types
[0063] To verify the applicability of the DDPG algorithm used in this embodiment of the invention in a competitive and complex electricity market, energy storage generator 6 was selected as the smart energy storage generator. 5000 training comparison experiments were conducted using the traditional Actor-Critic (AC) algorithm, the Proximal Policy Optimization (PPO) algorithm, and the DDPG algorithm used in this embodiment of the invention. The performance changes of the three algorithms are as follows: Figure 7 As shown in Table 4, the trained scheduling strategy was then tested, and the returns after one month of testing are shown in Table 4. The results indicate that, under the same market conditions, the DDPG algorithm is more suitable than the AC and PPO algorithms for training market-oriented energy storage generator agents, exhibiting better economic performance and scheduling strategy robustness.
[0064] Table 4. Comparison of scheduling and operation benefits of different reinforcement learning algorithms for intelligent agents of energy storage power generators.
[0065] Next, similarly, the scheduling strategy of this invention under a real-world hybrid cooperative-competitive environment was trained and tested. Energy storage generators 5, 6, and 7 formed a smart energy storage generator alliance, and 5000 training iterations were conducted on the alliance's scheduling strategy. The training process is as follows: Figure 8 As shown in Table 5, the test results for the revenue of each energy storage generator are presented, and the detailed daily revenue of each energy storage generator is as follows. Figure 9 As shown.
[0066] The "competition-cooperation" scheduling logic of energy storage generators in the capacity sharing market proposed in this embodiment of the invention enables the alliance intelligent agent to ensure the maximum interests of each member while sharing energy storage resources, effectively achieving a balance between system-level and individual-level scheduling objectives.
[0067] Table 5. Dispatch and operation benefits of each energy storage generator under a hybrid competition-cooperation relationship.
[0068] Example 2 This invention provides an electronic device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the scheduling optimization method for energy storage generators in the integrated energy system described in Embodiment 1 above, or executes the steps of the unified clearing method described above.
[0069] The electronic device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The memory can be used to store computer programs and / or modules. The processor performs various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory.
[0070] The relevant technical solutions are the same as above, and will not be repeated here.
[0071] Example 3 This invention provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the scheduling optimization method for energy storage generators in the integrated energy system described in Embodiment 1 above, or executes the steps of the unified clearing method described above.
[0072] Specifically, the memory may include high-speed random access memory, as well as non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital (SD) cards, flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.
[0073] The relevant technical solutions are the same as above, and will not be repeated here.
[0074] Example 4 This invention provides a computer program product, including a computer program that, when run on a computer, causes the computer to execute the steps of the scheduling optimization method for energy storage generators in the integrated energy system described in Embodiment 1 above, or to execute the steps of the unified clearing method described above.
[0075] The relevant technical solutions are the same as above, and will not be repeated here.
[0076] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A scheduling optimization method for energy storage generators in an integrated energy system, characterized in that, include: Energy storage generator As an intelligent entity of energy storage power generation With energy storage power generation e-commerce smart body The optimization objective is to maximize operational efficiency, including intelligent systems for energy storage and power generation. Constraints of each scheduling bid capacity interval and their in the first... Scheduling price parameters within the selected capacity range Under constraints including those related to energy storage and power generation, construct an intelligent energy storage and power generation system. The operating model; , , The total number of energy storage generators in the integrated energy system. This represents the total number of successful bid capacity intervals submitted for each scheduling period; The operational model is solved using reinforcement learning, including: Building an intelligent energy storage and power generation system During the current scheduling period state space and action space : , , and These represent the demand side of the integrated energy system during the previous and current scheduling periods, respectively. The energy storage demand, and These represent the intelligent entities of energy storage generators in the previous scheduling period. Marginal clearing price and allocated capacity; and These represent the current scheduling period. Energy storage generator smart body The endpoint values of each scheduling bid capacity interval and the corresponding scheduling bidding coefficient, and the scheduling price parameter This equals the unit actual operating cost within the corresponding scheduling bid capacity range and the scheduling bidding coefficient. The product; Using the aforementioned optimization objective as an immediate reward, the intelligent agent of the energy storage generator is... During iterative training, when the agent converges, a representation of the energy storage generator is obtained. The capacity range of each scheduling action and the aforementioned scheduling price parameters This is to optimize the scheduling of the corresponding energy storage generators.
2. The scheduling optimization method according to claim 1, characterized in that, Also includes: The intelligent agents of energy storage and power generators belonging to the same interest group in an integrated energy system are jointly trained, and the action space of the joint training is: , This refers to the total number of smart energy storage generators belonging to the same interest group within an integrated energy system. Indicates the intelligent body of energy storage power generation The space of motion ; The state space of the joint training is: ; The instant reward is to maximize the benefits for those belonging to the same interest group. Overall operational efficiency of an energy storage power generation intelligent system.
3. The scheduling optimization method according to claim 1 or 2, characterized in that, The operating model is as follows: in, Indicates the intelligent body of energy storage power generation operational efficiency , Intelligent bodies for energy storage generators in each scheduling period In the The scheduling bidding coefficient and unit actual operating cost within each scheduling bid capacity range. To set the upper limit of the scheduling bidding coefficient, The set of indices for the scheduled capacity range. ; Intelligent body for energy storage generators in each scheduling period The marginal clearing price; , They are respectively energy storage generator smart bodies The allocated capacity in each scheduling period and in the... The scheduling bid-winning capacity of each scheduling bid-winning capacity range; and Intelligent bodies for energy storage generators in each scheduling period In the The endpoint values of each scheduled capacity interval. hour, , hour, , For energy storage power generation e-commerce intelligent body The idle capacity range; For energy storage power generation e-commerce intelligent body The energy storage cost function.
4. The scheduling optimization method according to claim 3, characterized in that, Intelligent system for energy storage generators during each scheduling period In the The unit real operating cost within the allocated capacity range of each scheduling bid The calculation method is as follows: In the formula, , They are respectively energy storage generator smart bodies In the The cost per kilowatt-hour and the unit loss cost of the rated capacity of energy storage within the selected dispatch capacity range; among which... The cost was calculated using the levelized cost of energy storage (LDC) method. The calculation method is as follows: In the formula, Indicates the intelligent body of energy storage power generation Corresponding energy storage generator The total discounted cost of the configured energy storage device over its entire life cycle. , , , They are respectively energy storage generator smart bodies In the Total dispatchable energy storage capacity, self-discharge loss, cryogenic loss, and cycle loss within the dispatchable capacity range.
5. The scheduling optimization method according to claim 2, characterized in that, Deep deterministic strategy gradient algorithm is used to study the intelligent agent of energy storage power generator. Perform iterative training, or perform the joint training described above; The intelligent agent of the energy storage power generator is built based on a multilayer sensor. .
6. A unified clearing method for a spot market for shared energy storage capacity in an integrated energy system, characterized in that, include: Based on the capacity range of each scheduling bid and the aforementioned scheduling price parameters A unified market clearing process will be implemented to obtain the necessary permits from energy storage power generators during each dispatch period. The scheduling bid capacity in each scheduling bid capacity interval ; Among them, the respective scheduling bid capacity intervals and the aforementioned scheduling price parameters The scheduling optimization method described in any one of claims 1-5 is used to determine the method.
7. The unified clearing method according to claim 6, characterized in that, A unified market clearing model is used for unified market clearing, and the unified market clearing model is as follows: in, , , , ; This represents the set of indexes for energy storage generators. ; For the set of electricity demand-side nodes, As a demand-side node of the integrated energy system The energy storage demand, , Let be a Lagrange multiplier, representing the marginal price of the system; and They represent energy storage generators. With electricity demand-side nodes To the line The power transfer distribution factor, For the line Maximum permissible current, Represents a set of routes; and The lines are respectively Lagrange multipliers for the maximum positive and negative power flow constraints; and They are energy storage generators The minimum and maximum scheduling bid capacity.
8. An electronic device, characterized in that, Includes computer-readable storage media and processors; The computer-readable storage medium is used to store executable instructions; The processor is configured to read executable instructions stored in the computer-readable storage medium to execute the scheduling optimization method according to any one of claims 1-5, or / and execute the unified clearing method according to claims 6-7.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the scheduling optimization method as described in any one of claims 1-5, or / and the unified clearing method as described in claims 6-7.
10. A computer program product, characterized in that, Includes a computer program that, when run on a computer, causes the computer to perform the scheduling optimization method according to any one of claims 1-5, or / and, the unified clearing method according to claims 6-7.