Bidding and incentive combined strategy optimization method, system and device for source-load double-side peak shaving auxiliary service market and storage medium
By constructing a multi-task, multi-agent reinforcement learning framework for the source-load dual-side peak shaving ancillary service market, and optimizing the bidding and incentive strategies for load aggregators and thermal power units, the problem of the failure to effectively consider the joint decision-making of the source and load sides in the existing technology is solved, and efficient market response and revenue improvement are achieved.
Patent Information
- Application Number
- CN202511362511.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-23
- Publication Date
- 2025-10-31
- Estimated Expiration
- 2045-09-23
AI Technical Summary
Existing technologies in the electricity market fail to effectively consider the game behavior among multiple market participants in the source-load dual-side peak-shaving ancillary service market and the joint decision-making of demand-side market entities, resulting in poor strategy optimization effects, insufficient learning efficiency and stability, and failure to effectively explore the inherent coupling relationship between the information level and the strategy space.
A multi-task, multi-agent reinforcement learning training framework based on Markov decision processes is constructed. By asynchronously training a multi-task, multi-agent flexible action-evaluation algorithm, the bidding and incentive strategies of load aggregators and thermal power units are optimized to form a joint optimization model, taking into account the game behavior and joint decision-making of both the source and load sides.
It achieves joint optimization of both source and load sides, improves overall system benefits, shortens policy training time, improves multi-agent training efficiency, and significantly enhances market response quality.
Smart Images

Figure CN120876055A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of electricity market trading technology, specifically relating to a method, system, device, and storage medium for optimizing the bidding and incentive joint strategy of the source-load dual-side peak-shaving ancillary service market. Background Technology
[0002] With the increasing penetration rate of renewable energy, the peak-shaving mechanism of source-load linkage is becoming a key support for power system operation. Load aggregators, as important market players on the demand side, play an increasingly important role in the ancillary services market and demand response market by integrating small and medium-sized adjustable load resources. On the one hand, during peak hours, insufficient renewable energy output necessitates load-side resource coordination for peak shaving to ensure supply-demand balance; on the other hand, during off-peak hours, abundant renewable energy output requires proactive load response to promote efficient energy utilization. Simultaneously, thermal power units on the source side, as traditional regulating power sources, are continuously promoting deep peak-shaving operation, flexibly adjusting output to match real-time load changes in the system, further enhancing the system's adaptability to renewable energy integration. Against this backdrop, the market behavior of load aggregators is becoming increasingly complex, facing a dual task in the electricity market: first, participating in the peak-shaving ancillary services market and developing strategic bids to increase the probability of winning bids and market share; second, designing reasonable incentive mechanisms to guide end-users to proactively respond to the characteristics of renewable energy output, reducing system pressure during renewable energy off-peak periods and improving absorption capacity during peak periods, thereby achieving effective coordination of the system's response to renewable energy fluctuations. Meanwhile, thermal power units, as source-side peak-shaving resources, also need to engage in strategic bidding during deep peak shaving to achieve optimal resource scheduling and cost recovery. Since the peak-shaving behaviors of the load side and the source side are highly coupled in time and space, the synergistic optimization of bidding strategies and incentive mechanisms becomes a core issue in maximizing market efficiency and the profits of stakeholders. In the context of a combined source-load peak-shaving ancillary service market, researching the strategic bidding behavior of thermal power units, as well as the optimal joint decision-making method for load aggregators' bidding and demand response incentives, will not only help improve the market competitiveness of load aggregators but also effectively support the flexible and efficient operation of the power system under the background of high-proportion renewable energy access, which is of great significance and value.
[0003] Domestic and international scholars have conducted extensive research on the optimal bidding strategies or optimal pricing strategies of electricity market participants such as power generation companies, electricity retailers, and load aggregators. This research primarily includes studies based on experimental economics, game theory, and reinforcement learning paradigms. However, existing bidding strategy optimization methods based on reinforcement learning paradigms mostly assume that counterparties are price takers, neglecting their potential strategic bidding behavior. Furthermore, existing electricity ancillary service market environments typically only consider one-sided bidding, thus ignoring the game between source and load market entities. Moreover, existing decision optimization methods for demand-side market participants often separate the bidding behavior in the upper-level electricity spot market or ancillary service market from the pricing / incentive behavior in the lower-level electricity retail market or demand response market, failing to effectively explore the inherent coupling relationship between the two at the information level and in the strategy space. This task-separated modeling approach not only weakens the overall integrity of the optimization results but also limits the collaborative optimization capability of strategies, thereby reducing overall decision-making efficiency and market response quality. More importantly, existing strategy optimization methods based on reinforcement learning paradigms still have significant shortcomings in terms of learning efficiency, learning stability, and optimization effectiveness. Summary of the Invention
[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, system, device and storage medium for optimizing the joint strategy of bidding and incentive in the source-load dual-side peak shaving ancillary service market. It takes into account the game behavior among multiple market participants and the joint decision-making behavior of demand-side market entities in the dual market, and is widely applicable to multi-task intelligent agent collaborative optimization problems in the power market.
[0005] This invention provides the following technical solution:
[0006] Firstly, a method for optimizing the bidding and incentive strategies of the peak-shaving ancillary service market on both the source and load sides is provided, including: constructing a dual-market joint optimization model for load aggregators and thermal power units based on the parameter data of the peak-shaving ancillary service market and the incentive-based demand response market obtained in advance;
[0007] The dual-market joint optimization model is formalized as a Markov decision process.
[0008] Based on the Markov decision process, a multi-task multi-agent reinforcement learning training framework is constructed, which includes a load aggregator bidding quantity agent, a load aggregator bidding price agent, an incentive pricing agent, and a thermal power unit bidding quantity agent and a thermal power unit bidding price agent.
[0009] The multi-task multi-agent reinforcement learning training framework is solved by asynchronously training a multi-task multi-agent flexible action-evaluation algorithm, which yields the optimal joint strategy of bidding and incentive for load aggregators in the peak-shaving ancillary service market and the incentive-based demand response market, as well as the optimal bidding quantity and pricing strategy for thermal power units.
[0010] As an optional technical solution of the present invention, the dual-market joint optimization model includes a peak-shaving ancillary service market clearing model and a thermal power unit bidding model;
[0011] The peak-shaving ancillary service market clearing model includes a first objective function, expressed as:
[0012] ;
[0013] in, This indicates taking the minimum value. Indicates the first Individual load aggregator, Represents the set of load aggregators. express Time period The bidding volume of each load aggregator. express The bidding price of the i-th load aggregator in the time period. express Time period Is the load aggregator selected? Indicates the first One thermal power unit, This represents the set of thermal power units participating in peak shaving. express Time period The bidding volume of each thermal power unit in the peak-shaving market. express Time period The bidding price of individual thermal power units in the peak-shaving market express Time period A thermal power unit successfully won the bid, when ,express Time period If a thermal power unit is successfully bid, the bid will be rejected.
[0014] The peak-shaving ancillary service market clearing model also includes a first constraint, which includes power balance constraint, load demand constraint, network flow constraint and bidding constraint;
[0015] The power balance constraint is expressed as:
[0016] ;
[0017] in, Indicates peak shaving demand. express Time period Peak shaving capacity provided by individual load aggregators Indicates the total number of load aggregators;
[0018] The load demand constraint is expressed as follows:
[0019] ;
[0020] in, and They represent the first The lower and upper limits of the adjustable bidding volume after aggregating and signing users of each load aggregator;
[0021] The network flow constraint is represented as follows:
[0022] ;
[0023] in, Indicates the line The upper limit of the trend, express Time period nodes The injected node power, Indicates the total number of nodes. Indicates the line With nodes The power transfer distribution factor between them;
[0024] The bidding constraint is expressed as follows:
[0025] ;
[0026] in, Indicates the first The upper limit of the bid price for each load aggregator;
[0027] The bidding model for thermal power units is a bidding mode in which thermal power units submit their peak-shaving capacity and compensation prices in stages based on their marginal costs, expressed as:
[0028] ;
[0029] ;
[0030] in, express Time period Peak-shaving compensation cost for each thermal power unit. express Time period The unit peak-shaving cost of a thermal power unit express Time period The planned output of each thermal power unit is as follows. Indicates the first Minimum output of a thermal power unit during deep peak shaving;
[0031] The bidding model for thermal power units includes a second constraint, which includes a bidding range constraint, a power constraint, and a ramp constraint.
[0032] The unit bidding range constraint is expressed as follows:
[0033] ;
[0034] in, and Let $\mathbf{j}$ and $\mathbf{j}$ represent the lower limit and upper limit of the bidding price for the $j$-th thermal power unit in the peak-shaving market, respectively.
[0035] The power constraint of the thermal power unit is expressed as follows:
[0036] ;
[0037] Among them, when =1 means Time period Each thermal power unit is in operation; otherwise, it is not in operation. Indicates the first The maximum technical output of each thermal power unit Indicates the first Minimum technical output of each thermal power unit;
[0038] The unit ramping constraint is expressed as follows:
[0039] ;
[0040] in, Indicates the first The rate of ascent for each thermal power unit. Indicates the first Landslide rate of each thermal power unit express Time period The technological contribution of each thermal power unit in the peak-shaving market express Time period The technological contribution of each thermal power unit in the peak-shaving market Indicates the first The maximum output of each thermal power unit express Time period Each thermal power unit is in operation; otherwise, it is not in operation.
[0041] As an optional technical solution of the present invention, the dual-market joint optimization model further includes a load aggregator revenue model and a contracted user response model:
[0042] The load aggregator revenue model includes a second objective function, expressed as:
[0043] ;
[0044] in, express Time period The strategy bidding price factor of individual load aggregators express Time period The incentive price for individual load aggregators express Time period Strategic bidding price factors for individual thermal power units Represents the mathematical expectation. This indicates the number of times grid operators initiate peak-shaving ancillary services market. , This represents the total number of times a grid operator initiates the peak-shaving ancillary services market within a complete optimized week. This represents the percentage of future earnings discounted to the present time. express Time period Peak shaving revenue of individual load aggregators express Time period The loss of electricity sales revenue for individual load aggregators express Time period Demand response incentive costs for individual load aggregators express Time period Deviation penalty for each load aggregator express Time period The output cost of a single thermal power unit;
[0045] Among them, the Time period Peak shaving revenue of individual load aggregators Represented as:
[0046] ;
[0047] in, express Clearing price of the peak-shaving ancillary services market express Time period The winning capacity of each load aggregator express Time period The actual load reduction of the load aggregator, β represents the deviation tolerance coefficient;
[0048] The Time period Electricity sales revenue loss for individual load aggregators Represented as:
[0049] ;
[0050] in, express Time period The retail price agreed upon in the contract between the load aggregator and the contracted user. express The wholesale electricity price at which time-of-use load aggregators purchase electricity from the day-ahead market;
[0051] The Time period Demand response incentive cost for individual load aggregators Represented as:
[0052] ;
[0053] The Time period Deviation penalty for individual load aggregators Represented as:
[0054] ;
[0055] ;
[0056] in, express Economic penalty per unit deviation over a given time period;
[0057] The Time period The output cost of a single thermal power unit Represented as:
[0058] ;
[0059] in, Indicates the first Each thermal power unit Power output during a given time period All of these represent cost constants for thermal power units;
[0060] The load aggregator revenue model also includes a bidding price function, expressed as:
[0061] ;
[0062] ;
[0063] in, express Time period The theoretical bidding price of each load aggregator. express Time period The original load of the aggregated load of the load aggregator. and Represents constant coefficients. express Time period The strategic bidding price of each load aggregator;
[0064] The signed user response model includes a third objective function. When the current time period is for peak shaving, the third objective function of the signed user response model is expressed as:
[0065] ;
[0066] When the current period is during valley filling, the third objective function of the signed user response model is expressed as:
[0067] ;
[0068] in, express Time period The total response volume for each contracted customer, including load shedding and load transfer, where η represents the percentage of load shedding out of the total response volume. express Time period The response cost for each signed customer, where max represents the maximum value. express Time period The retail price agreed upon in the contract between the load aggregator and the contracted user;
[0069] The contracted user response model also includes a third constraint, expressed as follows:
[0070] ;
[0071] in, express Time period The original load before responding to the needs of a signed customer. This indicates the percentage of the maximum response relative to the original load.
[0072] As an optional technical solution of the present invention, formalizing the dual-market joint optimization model into a Markov decision process includes:
[0073] The Markov decision process of the dual-market joint optimization model includes the Markov decision process for the joint strategy optimization problem of bidding and incentive of load aggregators and the Markov decision process for the bidding strategy optimization problem of thermal power units.
[0074] The state-space representation of the Markov decision process for the joint bidding and incentive strategy optimization problem of the load aggregator is as follows: , represents the set of observable states of the load aggregator agent across all time periods, where the load aggregator agent is in observable state during the period Represented as ,include Historical observation trajectory of intelligent agents in the bidding volume of load aggregators during the time period. Historical observation trajectory of the intelligent agent for the bidding price of load aggregators during the time period and Historical observation trajectory of incentive pricing agents under different time periods;
[0075] The The historical observation trajectory of the bidding volume of the intelligent agent during the load aggregator in a given time period is represented as follows:
[0076] ;
[0077] in, express Historical observation trajectory of intelligent agents' bidding volume of load aggregators during specific time periods. Indicates from what has already happened Time's up Historical data on bidding volume for a given period. Indicates from what has already happened Time's up The aggregated original load of the load aggregator for the time period. Indicates from what has already happened Time's up Peak-shaving capacity of thermal power units during a given period;
[0078] The The historical observation trajectory of the bidding price intelligence agent of the load aggregator during the time period is represented as follows:
[0079] ;
[0080] in, express Historical observation trajectory of the bidding price of intelligent agents for load aggregators during a given time period. Indicates from what has already happened Time's up Historical data on the strategy bidding coefficient for the specified time period. Indicates from what has already happened Time's up Historical data on the retail prices agreed upon in contracts between load aggregators and contracted users during specific time periods. Indicates that something has already happened. Time's up Historical data on wholesale electricity prices for load aggregators purchasing electricity from the day-ahead market during specific time periods;
[0081] The The historical observation trajectory of the incentive pricing agent during the specified time period is represented as follows:
[0082] ;
[0083] in, express Historical observation trajectory of incentive pricing agents under different time periods Indicates from what has already happened Time's up Historical data on the incentive prices offered to contracted users by load aggregators during specific time periods. Indicates from what has already happened Time's up Historical data on the actual load reduction of load aggregators during a given time period;
[0084] The action space of the Markov decision process for the joint bidding and incentive strategy optimization problem of the load aggregator is represented as follows: It includes the continuous strategy bidding volume of the load aggregator bidding volume agent within a range, the continuous strategy bidding coefficient of the load aggregator bidding price agent within a range, and the continuous incentive pricing of the incentive pricing agent within a range.
[0085] The continuous policy reporting volume of the load aggregator bidding agent within a certain interval is represented as:
[0086] ;
[0087] in, This represents the continuous policy reporting volume of the agent within a certain interval, representing the bidding volume of the load aggregator. This indicates the upper limit of the bid volume for load aggregators;
[0088] The continuous strategy bidding coefficients of the load aggregator bidding price agent within a certain interval are expressed as:
[0089] ;
[0090] in, This represents the continuous strategy bidding coefficient of the agent within a certain interval, representing the bidding price of the load aggregator. and These represent the lower and upper limits of the bidding price factor for the load aggregator strategy, respectively;
[0091] The incentive pricing agent's continuous incentive pricing within a certain interval is expressed as:
[0092] ;
[0093] in, This indicates that the incentive pricing agent continuously sets incentive prices within a certain range. and These represent the lower and upper limits of the incentive price, respectively.
[0094] The mapping from the state space to the action space of the Markov decision process for the joint bidding and incentive strategy optimization problem of the load aggregator is expressed as:
[0095] ;
[0096] in, , , as well as These represent the load aggregator bidding volume strategy, load aggregator bidding price strategy, incentive pricing strategy, and load aggregator joint strategy, respectively. Let the state space of the Markov decision process representing the joint strategy optimization problem of bidding and incentives for load aggregators be denoted. The action space represents the Markov decision process for optimizing the joint bidding and incentive strategy of load aggregators;
[0097] The reward function of the Markov decision process for the joint strategy optimization problem of bidding and incentives for the load aggregator is expressed as:
[0098] ;
[0099] in, express Time period The immediate profit obtained by each load aggregator Indicates in Time period Peak shaving revenue for load aggregators Indicates in Time period The loss of electricity sales revenue for individual load aggregators Indicates in Time period Demand response incentive costs for individual load aggregators Indicates in Time period Deviation penalty for each load aggregator.
[0100] As an optional technical solution of the present invention, the state space representation of the Markov decision process for the bidding strategy optimization problem of thermal power units is as follows: , representing the set of observable states of the thermal power unit agent over all time periods, where the thermal power unit agent is in observable state during the period Represented as It includes the historical observation trajectory of the bidding volume intelligence agent for thermal power units and the historical observation trajectory of the bidding price intelligence agent for thermal power units;
[0101] The historical observation trajectory of the intelligent agent for bidding volume of thermal power units is represented as follows:
[0102] ;
[0103] in, express Historical observation trajectory of bidding volume for thermal power units under different time periods. Indicates from what has already happened Time's up Historical adjustment capacity submissions for a given period Indicates from what has already happened Time's up Historical data on system peak-shaving demand during specific time periods. Indicates from what has already happened Time's up The remaining regulating capacity of the generating units during the time period;
[0104] The historical observation trajectory of the intelligent agent for bidding prices of thermal power units is represented as follows:
[0105] ;
[0106] in, express Historical observation trajectory of bidding prices for thermal power units during a given period. Indicates from what has already happened Time's up Historical data on the bidding coefficient of thermal power strategy for different time periods. Indicates from what has already happened Time's up Historical market electricity price data for the period Indicates from what has already happened Time's up The actual electricity price fluctuation during the period;
[0107] The action space representation of the Markov decision process for the bidding strategy optimization problem of thermal power units is as follows: It includes the continuous strategy bidding volume of the thermal power unit bidding agent within a range and the continuous strategy bidding coefficient of the thermal power unit bidding price agent within a range.
[0108] The continuous policy reporting of the bidding agent for thermal power units within a certain interval is represented as:
[0109] ;
[0110] in, This represents the continuous strategy reporting of the bidding agent for thermal power units within a certain interval. This indicates the upper limit of the number of thermal power units that can be bid on;
[0111] The continuous strategy bidding coefficient of the bidding agent for thermal power units within a certain interval is expressed as:
[0112] ;
[0113] in, This represents the continuous strategy bidding coefficient of the intelligent agent for bidding on thermal power units within a certain range. and These represent the lower and upper limits of the strategic bidding price factor for thermal power units, respectively;
[0114] The mapping from the state space to the action space of the Markov decision process in the joint strategy optimization problem of bidding and incentive for thermal power units is expressed as:
[0115] ;
[0116] in, , as well as These represent the bidding volume strategy, bidding price strategy, and joint bidding strategy for thermal power units, respectively. Let the state space of the Markov decision process be used to address the bidding strategy optimization problem for thermal power units. The action space of a Markov decision process representing the optimization problem of bidding strategy for thermal power units;
[0117] The reward function of the Markov decision process for the bidding strategy optimization problem of the thermal power unit is expressed as:
[0118] ;
[0119] in, Indicates in Time period The immediate profit obtained by each thermal power unit Indicates in Time period Peak-shaving revenue of each thermal power unit. Indicates in Time period Deviation penalty for each thermal power unit Indicates in Time period The cost of deep peak shaving for each thermal power unit.
[0120] As an optional technical solution of the present invention, the construction of a multi-task, multi-agent reinforcement learning training framework based on the Markov decision process includes:
[0121] The architecture employs a fully cooperative action-comment framework, comprising action networks for bidding quantities of thermal power units, bidding prices of thermal power units, bidding quantities of load aggregators, bidding prices of load aggregators, and incentive pricing. Each action network corresponds to several decision-making tasks for one agent. All agents share a centralized global comment network, which receives the states of all agents and outputs a joint Q-value. , represented as:
[0122] ;
[0123] in, , This represents the intelligent agent that aggregates loads. This refers to the intelligent agent of a thermal power unit. Represents the mathematical expectation. and These represent the network weights of the main network and the target network of the global comment network, respectively. Indicates the discount factor. Indicates the temperature coefficient. This represents the Q-value of the main network output of the global comment network for the load aggregator agent or thermal power unit agent. This represents the Q-value of the target network output of the global evaluation network for the load aggregator agent or thermal power unit agent. express The total profit of the time-of-use load aggregator or thermal power unit when ent = Gen. Indicates in The observable status of the intelligent agent of the thermal power unit during the specified time period. Indicates in The combined action of the intelligent agent outputting the bidding quantity of thermal power units and the intelligent agent outputting the bidding price of thermal power units during the same time period. Indicates in The strategy reporting volume of the intelligent agent for bidding volume of thermal power units during the same period. Indicates in The strategy bidding coefficient of the intelligent agent for bidding prices of thermal power units during the specified time period. This indicates the combined strategy of thermal power units; when ent=LA, Indicates in The observable state of the load aggregator agent during the specified time period. Indicates in The combined actions output by the load aggregator bidding volume agent, the load aggregator bidding price agent, and the incentive pricing agent during the specified time period. This indicates a joint strategy by load aggregators;
[0124] A multi-task joint loss function based on a shared recurrent neural network is constructed to dynamically weight and optimize the bidding quantity loss function, bidding quantity prediction loss function, and bidding price loss function of thermal power units. It also dynamically weights and optimizes the bidding quantity loss function, bidding quantity prediction loss function, bidding price strategy loss function, and incentive pricing loss function of load aggregators.
[0125] The process of constructing the multi-task joint loss function includes:
[0126] The historical bidding volume and historical winning bid volume of the thermal power units are input into a shared recurrent neural network to finally obtain the predicted bidding volume of the thermal power units.
[0127] The historical bidding volume and historical winning volume of the thermal power units are processed to obtain the historical data layer of the intelligent agent neural network of the thermal power units, represented as follows:
[0128] ;
[0129] in, This represents the historical data layer of the neural network of the intelligent agent in a thermal power unit. and These represent the historical number of bids and the historical number of successful bids for thermal power units, respectively. This indicates a splicing operation. The parameter matrix representing the historical data layer of the neural network of a thermal power unit intelligent agent;
[0130] The historical bid volume, historical winning bid volume, and historical user response load of the load aggregator are input into a shared recurrent neural network to finally obtain the predicted bid volume of the load aggregator.
[0131] The historical bidding volume, historical winning volume, and historical user response load of the load aggregator are processed to obtain the historical data layer of the load aggregator's intelligent agent neural network, represented as follows:
[0132] ;
[0133] in, This represents the historical data layer of the neural network of the load aggregator intelligent agent. These represent historical bidding volume, historical successful bidding volume, and historical user response load, respectively. This indicates a splicing operation. This represents the parameter matrix of the historical data layer of the neural network of the load aggregator intelligent agent;
[0134] Historical data layer based on neural network of intelligent agent of thermal power unit Or load aggregator intelligent agent neural network historical data layer Perform linear normalization, which is expressed as:
[0135] ;
[0136] in, This represents the output of the linear normalization layer. , This represents the historical maximum data. Represents the lowest historical data;
[0137] The shared recurrent neural network outputs the linear normalization layer. The operation is represented as:
[0138] ;
[0139] in, This represents the predicted bid amount from the output of the shared recurrent neural network. This represents the weight matrix of a shared recurrent neural network. This represents the bias term in a shared recurrent neural network.
[0140] The bidding agent action network predicts the bidding volume from the output of the shared recurrent neural network. The operation is represented as:
[0141] ;
[0142] in, This represents the bidding action output by the bidding agent action network after passing through a shared recurrent neural network. This represents the weight matrix of the bidding agent action network. Represents the bias term of the bidding agent action network; ReLU and Tanh are activation functions.
[0143] The mean squared error between the predicted bid volume and the winning bid volume in the current period is used as the bid volume prediction loss function. , represented as:
[0144] ;
[0145] in, Indicates in The winning capacity of thermal power units or load aggregators during the time period; the predicted bidding volume includes the predicted bidding volume of thermal power units and the predicted bidding volume of load aggregators.
[0146] Using KL divergence as the bid loss function of the bid agent action network. , represented as:
[0147] ;
[0148] in, This indicates the bidding volume strategy of thermal power units or load aggregators. This represents the network weight of the intelligent agent action network representing the bidding volume of thermal power units or load aggregators. This represents the bidding quantity action output by the intelligent agent action network for bidding quantities of thermal power units or load aggregators. The allocation function representing the intelligent agent of thermal power units or load aggregators;
[0149] Loss function for predicting bid volume With bid quantity loss function The first joint loss function is obtained by weighting. , represented as:
[0150] ;
[0151] in, and All represent weighting coefficients;
[0152] Bidding price loss function of bidding price action network Represented as:
[0153] ;
[0154] in, This indicates the bidding price strategy of the intelligent entity that manages thermal power units or load aggregators. This represents the network weight of the intelligent agent action network, indicating the bidding price of thermal power units or load aggregators. This indicates the bidding price action output by the intelligent agent action network for bidding prices of thermal power units or load aggregators.
[0155] Incentive pricing loss function of incentive pricing action network Represented as:
[0156] ;
[0157] in, Indicates an incentive pricing strategy. This represents the network weights of the action network of the incentive pricing agent. express The incentive price action output by the time-period incentive price agent action network;
[0158] The bidding loss function of the intelligent agent action network for bidding quantity of the thermal power unit. The bidding price loss function of the intelligent agent action network for bidding prices of thermal power units. The second joint loss function of thermal power units is obtained by weighting. , represented as:
[0159] ;
[0160] in, and Indicates the weighting coefficient;
[0161] The bidding loss function of the bidding quantity agent action network of the load aggregator bidding quantity The bidding price loss function of the intelligent agent action network for load aggregator bidding price. Incentive pricing loss function of incentive pricing agent action network The second joint loss function of the load aggregator is obtained by weighting. , represented as:
[0162] ;
[0163] in, , and All of these represent weighting coefficients.
[0164] As an optional technical solution of the present invention, the shared recurrent neural network adopts an LSTM structure. The LSTM structure receives a dataset sampled from a priority experience replay buffer and outputs extracted abstract features and sampling probabilities. Represented as:
[0165] ;
[0166] in, This indicates the size of the priority experience replay buffer. Indicates hyperparameters, This indicates the priority of sample v after passing through hyperparameters. The adjusted value, The priority of sample ζ is determined by the hyperparameter. The adjusted value;
[0167] The centralized global comment network, bidding quantity action network, bidding price action network, and incentive pricing action network of the thermal power unit or load aggregator intelligent agent are configured with corresponding target networks. All target networks are updated using a soft update method, as shown below:
[0168] ;
[0169] ;
[0170] in, This represents the network weights of the target network in the centralized global evaluation network of the thermal power unit or load aggregator intelligent agent, when ent=Gen. , This represents the network weight of the target network in the intelligent agent action network for bidding quantity of thermal power units or the intelligent agent action network for bidding price of thermal power units; when ent=LA. , This represents the network weights of the target network, which is the agent action network representing the bid volume of the load aggregator, the agent action network representing the bid price of the load aggregator, or the agent action network representing the incentive pricing. Indicates soft update parameters, This represents the network weight of the main network in the centralized global evaluation network of the thermal power unit or load aggregator intelligent agent. This represents the network weight of the main network of the action network of the thermal power unit or load aggregator intelligent agent. This indicates the actions of the bidding agent. The action of the intelligent agent indicates the bidding price. This indicates the actions of the incentive pricing agent;
[0171] The action-evaluation architecture employs an asynchronously trained, multi-task, multi-agent flexible action-evaluation algorithm. The bidding quantity agent, bidding price agent, load aggregator bidding quantity agent, load aggregator bidding price agent, and incentive pricing agent respectively output the distribution functions of the bidding strategy and incentive pricing strategy. After action sampling, these actions—bidding quantity action, bidding price action, and incentive pricing action—can directly interact with the peak-shaving auxiliary service market environment. Each action network of the thermal power unit agent outputs an action distribution function, expressed as:
[0172] ;
[0173] in, Indicates that the intelligent agent of the thermal power unit is in The observable state of a given time period Indicates a Gaussian distribution. and This represents the mean and standard deviation of the output of the intelligent agent action network of thermal power units. This represents the bidding quantity or bidding price action of thermal power units sampled from the action distribution. This indicates the bidding quantity strategy or bidding price strategy for thermal power units.
[0174] The action distribution function output by each action network of the load aggregator agent is expressed as:
[0175] ;
[0176] in, This indicates that the load aggregator intelligent agent is in The observable state of a given time period and This represents the mean and standard deviation of the load aggregator agent action network output. This represents the load aggregator bidding quantity action, bidding price action, or incentive pricing action sampled from the action distribution. This indicates the load aggregator's bidding volume strategy, bidding price strategy, or incentive pricing strategy;
[0177] The centralized global review network of the thermal power unit intelligent agent or load aggregator intelligent agent uses joint Q-values. Updated; the intelligent agent action network for bidding quantity of thermal power units and the intelligent agent action network for bidding price of thermal power units are updated through the first joint loss function of thermal power units. and the second joint loss function of thermal power units Update; the load aggregator bidding quantity agent action network, the load aggregator bidding price agent action network, and the incentive pricing agent action network are updated using the first joint loss function of the load aggregator. Second joint loss function of load aggregator Update.
[0178] Secondly, a device for optimizing a market bidding and incentive strategy for dual-side peak shaving auxiliary services is provided, comprising:
[0179] The model building module is used to construct a dual-market joint optimization model for load aggregators and thermal power units based on the pre-acquired parameter data of the peak-shaving ancillary service market and the incentive-based demand response market.
[0180] A formalization module is used to formalize the dual-market joint optimization model into a Markov decision process;
[0181] The training framework construction module is used to construct a multi-task multi-agent reinforcement learning training framework based on the Markov decision process. The multi-task multi-agent reinforcement learning training framework includes a load aggregator bidding quantity agent, a load aggregator bidding price agent, an incentive pricing agent, and a thermal power unit bidding quantity agent and a thermal power unit bidding price agent.
[0182] The solution module is used to solve the multi-task multi-agent reinforcement learning training framework by asynchronously training a multi-task multi-agent flexible action-evaluation algorithm, so as to obtain the optimal joint strategy of bidding and incentive for load aggregators in the peak-shaving auxiliary service market and the incentive-based demand response market, as well as the optimal bidding quantity and pricing strategy for thermal power units.
[0183] Thirdly, a device for optimizing the bidding and incentive joint strategy of the source-load dual-side peak shaving auxiliary service market is provided, characterized in that it includes a processor and a storage medium;
[0184] The storage medium is used to store instructions;
[0185] The processor is configured to operate according to the instructions to execute the steps of the source-load dual-side peak shaving auxiliary service market bidding and incentive joint strategy optimization method described in the first aspect.
[0186] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the steps of the method described in the first aspect.
[0187] Compared with the prior art, the beneficial effects of the present invention are:
[0188] This invention provides a joint strategy optimization method for bidding and incentives in the source-load dual-side peak shaving ancillary service market. Through a multi-task multi-agent reinforcement learning training framework, it can simultaneously consider the game behavior between multiple market participants and the coupling of bidding and incentive strategies of individual market participants, achieving joint optimization on both the source and load sides and significantly improving the overall system benefits. The asynchronous multi-task multi-agent training mechanism can be widely applied to multi-task agent collaborative optimization problems in the power market. The asynchronous training multi-task multi-agent flexible action-evaluation algorithm can significantly reduce training resource conflicts, shorten strategy training time, and improve the training efficiency of multi-agents. Attached Figure Description
[0189] Figure 1 This is a flowchart of the method for optimizing the bidding and incentive joint strategy of the source-load dual-side peak shaving auxiliary service market in an embodiment of the present invention;
[0190] Figure 2 This is a schematic diagram illustrating the operating mechanism of load aggregators and thermal power units in the ancillary services and demand response dual markets in this embodiment of the invention;
[0191] Figure 3 This is a schematic diagram of the asynchronous multi-task multi-agent training mechanism for load aggregator in an embodiment of the present invention;
[0192] Figure 4 This is a schematic diagram of the asynchronous multi-task multi-agent training mechanism for thermal power units in an embodiment of the present invention;
[0193] Figure 5 This is a schematic diagram of the framework of the load aggregator PA-MTMASAC algorithm in an embodiment of the present invention;
[0194] Figure 6 This is a schematic diagram of the framework of the PA-MTMASAC algorithm for thermal power units in an embodiment of the present invention;
[0195] Figure 7 This is a flowchart of the PA-MTMASAC algorithm in an embodiment of the present invention. Detailed Implementation
[0196] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.
[0197] Example 1
[0198] This embodiment provides a joint strategy optimization method for bidding and incentives in the source-load dual-side peak-shaving ancillary service market. It considers both the game-theoretic behavior among multiple market participants and the joint decision-making behavior of demand-side market entities in both markets, making it widely applicable to multi-task agent-based collaborative optimization problems in the power market. For example... Figure 1As shown, the specific steps include the following:
[0199] Step 1: Based on the parameter data of the pre-acquired peak-shaving ancillary service market and incentive-based demand response market, construct a dual-market joint optimization model for load aggregators and thermal power units.
[0200] The parameter data for the peak-shaving ancillary services market and the incentive-based demand response market include day-ahead spot market wholesale electricity price data, retail market electricity price data, peak-shaving (including peak shaving and valley filling) capacity demand curves based on source-load forecasts, and historical operational data on market clearing and execution results.
[0201] like Figure 2 The diagram illustrates the operational mechanisms of load aggregators and thermal power units in the ancillary services and demand response markets. This indicates the total number of thermal power units. Indicates the first One end user, This represents the total number of end users. (See reference) Figure 2 A dual-market joint optimization model is constructed for load aggregators and thermal power units. This model includes a peak-shaving ancillary service market clearing model, a thermal power unit bidding model, a load aggregator revenue model, and a contracted user response model. Specifically:
[0202] (1) The peak-shaving ancillary service market clearing model includes a first objective function, expressed as:
[0203] ;
[0204] in, This indicates taking the minimum value. Indicates the first Individual load aggregator, Represents the set of load aggregators. express Time period The bidding volume of each load aggregator. express The bidding price of the i-th load aggregator in the time period. express Time period Is the load aggregator selected? Indicates the first One thermal power unit, This represents the set of thermal power units participating in peak shaving. express Time period The bidding volume of each thermal power unit in the peak-shaving market. express Time period The bidding price of individual thermal power units in the peak-shaving market express Time period A thermal power unit successfully won the bid, when ,express Time period If a thermal power unit is successfully bid, the bid will be rejected.
[0205] The peak-shaving ancillary service market clearing model also includes a first constraint, which includes power balance constraint, load demand constraint, network flow constraint and bidding constraint;
[0206] The power balance constraint is expressed as:
[0207] ;
[0208] in, Indicates peak shaving demand. express Time period Peak shaving capacity provided by individual load aggregators Indicates the total number of load aggregators;
[0209] The load demand constraint is expressed as follows:
[0210] ;
[0211] in, and They represent the first The lower and upper limits of the adjustable bidding volume after aggregating and signing users of each load aggregator;
[0212] The network flow constraint is represented as follows:
[0213] ;
[0214] in, Indicates the line The upper limit of the trend, express Time period nodes The injected node power, Indicates the total number of nodes. Indicates the line With nodes The power transfer distribution factor between them;
[0215] The bidding constraint is expressed as follows:
[0216] ;
[0217] in, Indicates the first The upper limit of the bid price for each load aggregator.
[0218] (2) The bidding model for thermal power units is a bidding mode in which thermal power units submit their peak-shaving capacity and compensation price in stages based on their marginal cost, as expressed as:
[0219] ;
[0220] ;
[0221] in, express Time period Peak-shaving compensation cost for each thermal power unit. express Time period The unit peak-shaving cost of a thermal power unit express Time period The planned output of each thermal power unit is as follows. Indicates the first The minimum output of a thermal power unit during deep peak shaving.
[0222] The bidding model for thermal power units includes a second constraint, which includes a bidding range constraint, a power constraint, and a ramp constraint.
[0223] The unit bidding range constraint is expressed as follows:
[0224] ;
[0225] in, and Let $\begin{batch}$ and $\begin{batch}$ represent the lower and upper limits of the bidding price for the $j$-th thermal power unit in the peak-shaving market, respectively. Let $\begin{batch}$ and $\begin{batch}$ represent the unit peak-shaving cost of the $j$-th thermal power unit under different load factors. If so, the thermal power unit will not offer a price quote based on its peak-shaving capacity.
[0226] When thermal power units participate in deep peak shaving, the lower limit of their output after winning the bid for peak shaving capacity must take into account the minimum output constraint for deep peak shaving and the upper limit power constraint for unit output. The power constraint of the thermal power unit is expressed as follows:
[0227] ;
[0228] Among them, when =1 means Time period Each thermal power unit is in operation; otherwise, it is not in operation. Indicates the first The maximum technical output of each thermal power unit Indicates the first The minimum technical output of a thermal power unit.
[0229] The unit ramping constraint is expressed as follows:
[0230] ;
[0231] in, Indicates the first The rate of ascent for each thermal power unit. Indicates the first Landslide rate of each thermal power unit express Time period The technological contribution of each thermal power unit in the peak-shaving market express Time period The technological contribution of each thermal power unit in the peak-shaving market Indicates the first The maximum output of each thermal power unit express Time period Each thermal power unit is in operation; otherwise, it is not in operation.
[0232] (3) The load aggregator revenue model includes a second objective function, expressed as:
[0233] ;
[0234] in, express Time period The strategy bidding price factor of individual load aggregators express Time period The incentive price for individual load aggregators express Time period Strategic bidding price factors for individual thermal power units Represents the mathematical expectation. This indicates the number of times grid operators initiate peak-shaving ancillary services market. , This represents the total number of times a grid operator initiates the peak-shaving ancillary services market within a complete optimized week. This represents the percentage of future earnings discounted to the present time. express Time period Peak shaving revenue of individual load aggregators express Time period The loss of electricity sales revenue for individual load aggregators express Time period Demand response incentive costs for individual load aggregators express Time period Deviation penalty for each load aggregator express Time period The output cost of a single thermal power unit;
[0235] Where the actual load reduction of the load aggregator exceeds the bid-winning capacity, the excess portion will not be settled. If the actual load reduction is lower than the bid-winning capacity but within an acceptable deviation range, compensation will be calculated based on the actual load reduction. Conversely, if the actual load reduction is lower than the bid-winning capacity and the deviation exceeds the allowable range, no peak-shaving compensation will be provided. Time period Peak shaving revenue of individual load aggregators Represented as:
[0236] ;
[0237] in, express Clearing price of the peak-shaving ancillary services market express Time period The winning capacity of each load aggregator express Time period The actual load reduction of the load aggregator, β represents the deviation tolerance coefficient;
[0238] The Time period Electricity sales revenue loss for individual load aggregators Represented as:
[0239] ;
[0240] in, express Time period The retail price agreed upon in the contract between the load aggregator and the contracted user. express The wholesale electricity price at which time-of-use load aggregators purchase electricity from the day-ahead market;
[0241] The Time period Demand response incentive cost for individual load aggregators Represented as:
[0242] ;
[0243] The Time period Deviation penalty for individual load aggregators Represented as:
[0244] ;
[0245] ;
[0246] in, express Economic penalty per unit deviation over a given time period;
[0247] The Time period The output cost of a single thermal power unit Represented as:
[0248] ;
[0249] in, Indicates the first Each thermal power unit Power output during a given time period All of these represent cost constants for thermal power units;
[0250] The load aggregator revenue model also includes a bidding price function, expressed as:
[0251] ;
[0252] ;
[0253] in, express Time period The theoretical bidding price of each load aggregator. express Time period The original load of the aggregated load of the load aggregator. and Represents constant coefficients. express Time period The strategic bidding price of each load aggregator.
[0254] (4) The signed user response model includes a third objective function. When the current time period is for peak shaving, the third objective function of the signed user response model is expressed as:
[0255] ;
[0256] When the current period is during valley filling, the third objective function of the signed user response model is expressed as:
[0257] ;
[0258] in, express Time period The total response volume for each contracted customer, including load shedding and load transfer, where η represents the percentage of load shedding out of the total response volume. express Time period The response cost for each signed customer, where max represents the maximum value. express Time period The retail price agreed upon in the contract between the load aggregator and the contracted user;
[0259] The contracted user response model also includes a third constraint, expressed as follows:
[0260] ;
[0261] in, express Time period The original load before responding to the needs of a signed customer. This indicates the percentage of the maximum response relative to the original load.
[0262] Step 2: Formalize the dual-market joint optimization model into a Markov decision process.
[0263] The Markov decision process of the dual-market joint optimization model includes the Markov decision process for the joint strategy optimization problem of bidding and incentives for load aggregators, and the Markov decision process for the bidding strategy optimization problem of thermal power units. Specifically:
[0264] (1) The state-space representation of the Markov decision process for the joint strategy optimization problem of bidding and incentives for the load aggregator is as follows: , represents the set of observable states of the load aggregator agent across all time periods, where the load aggregator agent is in observable state during the period Represented as ,include Historical observation trajectory of intelligent agents in the bidding volume of load aggregators during the time period. Historical observation trajectory of the intelligent agent for the bidding price of load aggregators during the time period and The historical observation trajectory of the incentive pricing agent during the specified time period. Among them, the load aggregator bidding volume agent and the load aggregator bidding price agent are responsible for simulating the strategic bidding behavior of price-influencing load aggregators in the peak-shaving ancillary service market, while the incentive pricing agent is responsible for simulating the real-time incentive pricing behavior of price-influencing load aggregators in the demand response market. The three agents are fully cooperative to ensure the most efficient allocation of resources in the peak-shaving ancillary service market.
[0265] The The historical observation trajectory of the bidding volume of the intelligent agent during the load aggregator in a given time period is represented as follows:
[0266] ;
[0267] in, express Historical observation trajectory of intelligent agents' bidding volume of load aggregators during specific time periods. Indicates from what has already happened Time's up Historical data on bidding volume for a given period. Indicates from what has already happened Time's up The aggregated original load of the load aggregator for the time period. Indicates from what has already happened Time's up Peak-shaving capacity of thermal power units during a given period;
[0268] The The historical observation trajectory of the bidding price intelligence agent of the load aggregator during the time period is represented as follows:
[0269] ;
[0270] in, express Historical observation trajectory of the bidding price of intelligent agents for load aggregators during a given time period. Indicates from what has already happened Time's up Historical data on the strategy bidding coefficient for the specified time period. Indicates from what has already happened Time's up Historical data on the retail prices agreed upon in contracts between load aggregators and contracted users during specific time periods. Indicates that something has already happened. Time's up Historical data on wholesale electricity prices for load aggregators purchasing electricity from the day-ahead market during specific time periods;
[0271] The The historical observation trajectory of the incentive pricing agent during the specified time period is represented as follows:
[0272] ;
[0273] in, express Historical observation trajectory of incentive pricing agents under different time periods Indicates from what has already happened Time's up Historical data on the incentive prices offered to contracted users by load aggregators during specific time periods. Indicates from what has already happened Time's up Historical data on the actual load reduction of load aggregators during a given time period.
[0274] The action space of the Markov decision process for the joint bidding and incentive strategy optimization problem of the load aggregator is represented as follows: It includes the continuous strategy bidding volume of the load aggregator bidding volume agent within a range, the continuous strategy bidding price coefficient of the load aggregator bidding price agent within a range, and the continuous incentive pricing of the incentive pricing agent within a range.
[0275] The continuous policy reporting volume of the load aggregator bidding agent within a certain interval is represented as:
[0276] ;
[0277] in, This represents the continuous policy reporting volume of the agent within a certain interval, representing the bidding volume of the load aggregator. This indicates the upper limit of the bid volume for load aggregators.
[0278] The continuous strategy bidding coefficients of the load aggregator bidding price agent within a certain interval are expressed as:
[0279] ;
[0280] in, This represents the continuous strategy bidding coefficient of the agent within a certain interval, representing the bidding price of the load aggregator. and These represent the lower and upper limits of the bidding price factor for the load aggregator strategy, respectively.
[0281] The incentive pricing agent's continuous incentive pricing within a certain interval is expressed as:
[0282] ;
[0283] in, This indicates that the incentive pricing agent continuously sets incentive prices within a certain range. and These represent the lower and upper limits of the incentive price, respectively.
[0284] The mapping from the state space to the action space of the Markov decision process for the joint bidding and incentive strategy optimization problem of the load aggregator is expressed as:
[0285] ;
[0286] in, , , as well as These represent the load aggregator bidding volume strategy, load aggregator bidding price strategy, incentive pricing strategy, and load aggregator joint strategy, respectively. Let the state space of the Markov decision process representing the joint strategy optimization problem of bidding and incentives for load aggregators be denoted. The action space represents the Markov decision process for optimizing the joint bidding and incentive strategy of load aggregators.
[0287] The reward function of the Markov decision process for the joint strategy optimization problem of bidding and incentives for the load aggregator is expressed as:
[0288] ;
[0289] in, express Time period The immediate profit obtained by each load aggregator Indicates in Time period Peak shaving revenue for load aggregators Indicates in Time period The loss of electricity sales revenue for individual load aggregators Indicates in Time period Demand response incentive costs for individual load aggregators Indicates in Time period Deviation penalty for each load aggregator.
[0290] (2) The state-space representation of the Markov decision process for the bidding strategy optimization problem of the thermal power unit is as follows: , representing the set of observable states of the thermal power unit agent over all time periods, where the thermal power unit agent is in observable state during the period Represented as It includes the historical observation trajectory of the bidding volume intelligence agent for thermal power units and the historical observation trajectory of the bidding price intelligence agent for thermal power units;
[0291] The historical observation trajectory of the intelligent agent for bidding volume of thermal power units is represented as follows:
[0292] ;
[0293] in, express Historical observation trajectory of bidding volume for thermal power units under different time periods. Indicates from what has already happened Time's up Historical adjustment capacity submissions for a given period Indicates from what has already happened Time's up Historical data on system peak-shaving demand during specific time periods. Indicates from what has already happened Time's up The remaining regulating capacity of the generating units during the time period;
[0294] The historical observation trajectory of the intelligent agent for bidding prices of thermal power units is represented as follows:
[0295] ;
[0296] in, express Historical observation trajectory of bidding prices for thermal power units during a given period. Indicates from what has already happened Time's up Historical data on the bidding coefficient of thermal power strategy for different time periods. Indicates from what has already happened Time's up Historical market electricity price data for the period Indicates from what has already happened Time's up The actual electricity price fluctuates during the period.
[0297] The action space representation of the Markov decision process for the bidding strategy optimization problem of thermal power units is as follows: It includes the continuous strategy bidding quantity of the thermal power unit bidding agent within a range and the continuous strategy bidding coefficient of the thermal power unit bidding price agent within a range.
[0298] The continuous policy reporting of the bidding agent for thermal power units within a certain interval is represented as:
[0299] ;
[0300] in, This represents the continuous strategy reporting of the bidding agent for thermal power units within a certain interval. This indicates the upper limit of the number of thermal power units that can be bid on.
[0301] The continuous strategy bidding coefficient of the bidding agent for thermal power units within a certain interval is expressed as:
[0302] ;
[0303] in, This represents the continuous strategy bidding coefficient of the intelligent agent for bidding on thermal power units within a certain range. and These represent the lower and upper limits of the strategic bidding price factor for thermal power units, respectively.
[0304] The mapping from the state space to the action space of the Markov decision process in the joint strategy optimization problem of bidding and incentive for thermal power units is expressed as:
[0305] ;
[0306] in, , as well as These represent the bidding volume strategy, bidding price strategy, and joint bidding strategy for thermal power units, respectively. Let the state space of the Markov decision process be used to address the bidding strategy optimization problem for thermal power units. The action space represents the Markov decision process for optimizing the bidding strategy of thermal power units.
[0307] The reward function of the Markov decision process for the bidding strategy optimization problem of the thermal power unit is expressed as:
[0308] ;
[0309] in, Indicates in Time period The immediate profit obtained by each thermal power unit Indicates in Time period Peak-shaving revenue of each thermal power unit. Indicates in Time period Deviation penalty for each thermal power unit Indicates in Time period The cost of deep peak shaving for each thermal power unit.
[0310] Step 3: Based on the Markov decision process, construct a multi-task multi-agent reinforcement learning training framework, which includes a load aggregator bidding quantity agent, a load aggregator bidding price agent, an incentive pricing agent, and a thermal power unit bidding quantity agent and a thermal power unit bidding price agent.
[0311] The architecture employs a fully cooperative action-critic (AC) architecture, comprising action networks for bidding quantities of thermal power units, bidding prices of thermal power units, bidding quantities of load aggregators, bidding prices of load aggregators, and incentive pricing. Each action network corresponds to several decision-making tasks for one agent. All agents share a centralized global critique network, which receives the states of all agents and outputs a joint Q-value. , represented as:
[0312] ;
[0313] in, , This represents the intelligent agent that aggregates loads. This refers to the intelligent agent of a thermal power unit. Represents the mathematical expectation. and These represent the network weights of the main network and the target network of the global comment network, respectively. Indicates the discount factor. Indicates the temperature coefficient. This represents the Q-value of the main network output of the global comment network for the load aggregator agent or thermal power unit agent. This represents the Q-value of the target network output of the global evaluation network for the load aggregator agent or thermal power unit agent. express The total profit of the time-of-use load aggregator or thermal power unit when ent = Gen. Indicates in The observable status of the intelligent agent of the thermal power unit during the specified time period. Indicates in The combined action of the intelligent agent outputting the bidding quantity of thermal power units and the intelligent agent outputting the bidding price of thermal power units during the same time period. Indicates in The strategy reporting volume of the intelligent agent for bidding volume of thermal power units during the same period. Indicates in The strategy bidding coefficient of the intelligent agent for bidding prices of thermal power units during the specified time period. This indicates the combined strategy of thermal power units; when ent=LA, Indicates in The observable state of the load aggregator agent during the specified time period. Indicates in The combined actions output by the load aggregator bidding volume agent, the load aggregator bidding price agent, and the incentive pricing agent during the specified time period. This indicates the joint strategy of load aggregators.
[0314] A multi-task joint loss function based on a shared recurrent neural network is constructed to dynamically weight and optimize the bidding quantity loss function, bidding quantity prediction loss function, and bidding price loss function of thermal power units. It also dynamically weights and optimizes the bidding quantity loss function, bidding quantity prediction loss function, bidding price strategy loss function, and incentive pricing loss function of load aggregators.
[0315] The construction process of the multi-task joint loss function includes:
[0316] The historical bidding volume and historical winning bid volume of the thermal power units are input into a shared recurrent neural network to obtain the predicted bidding volume of the thermal power units. Specifically:
[0317] (1) The historical bidding volume and historical winning volume of the thermal power unit are processed to obtain the historical data layer of the intelligent agent neural network of the thermal power unit, which is represented as:
[0318] ;
[0319] in, This represents the historical data layer of the neural network of the intelligent agent in a thermal power unit. and These represent the historical number of bids and the historical number of successful bids for thermal power units, respectively. This indicates a splicing operation. This represents the parameter matrix of the historical data layer of the neural network of the thermal power unit's intelligent agent.
[0320] The historical bid volume, historical winning bid volume, and historical user response load of the load aggregator are input into a shared recurrent neural network to finally obtain the predicted bid volume of the load aggregator.
[0321] The historical bidding volume, historical winning volume, and historical user response load of the load aggregator are processed to obtain the historical data layer of the load aggregator's intelligent agent neural network, represented as follows:
[0322] ;
[0323] in, This represents the historical data layer of the neural network of the load aggregator intelligent agent. These represent historical bidding volume, historical successful bidding volume, and historical user response load, respectively. This indicates a splicing operation. This represents the parameter matrix of the historical data layer of the neural network of the load aggregator intelligent agent.
[0324] (2) Historical data layer based on the neural network of thermal power unit intelligent agent Or load aggregator intelligent agent neural network historical data layer Perform linear normalization, which is expressed as:
[0325] ;
[0326] in, This represents the output of the linear normalization layer. , This represents the historical maximum data. This represents the lowest historical data.
[0327] (3) The output of the linear normalization layer of the shared recurrent neural network The operation is represented as:
[0328] ;
[0329] in, This represents the predicted bid amount from the output of the shared recurrent neural network. This represents the weight matrix of a shared recurrent neural network. This represents the bias term of a shared recurrent neural network.
[0330] (4) The predicted bid quantity of the bidding agent action network on the output of the shared recurrent neural network. The operation is represented as:
[0331] ;
[0332] in, This represents the bidding action output by the bidding agent action network after passing through a shared recurrent neural network. This represents the weight matrix of the bidding agent action network. This represents the bias term of the bidding agent action network, where ReLU and Tanh are activation functions.
[0333] (5) The mean square error between the predicted bid volume and the winning bid volume in the current period is used as the bid volume prediction loss function. , represented as:
[0334] ;
[0335] in, Indicates in The winning capacity of thermal power units or load aggregators during the time period; the predicted bidding volume includes the predicted bidding volume of thermal power units and the predicted bidding volume of load aggregators.
[0336] (6) Use KL divergence as the bid loss function of the bid agent action network. , represented as:
[0337] ;
[0338] in, This indicates the bidding volume strategy of thermal power units or load aggregators. This represents the network weight of the intelligent agent action network representing the bidding volume of thermal power units or load aggregators. This represents the bidding quantity action output by the intelligent agent action network for bidding quantities of thermal power units or load aggregators. The allocation function represents the distribution function of a thermal power unit or load aggregator intelligent agent.
[0339] Loss function for predicting bid volume With bid quantity loss function The first joint loss function is obtained by weighting. , represented as:
[0340] ;
[0341] in, and All represent weighting coefficients;
[0342] Bidding price loss function of bidding price action network Represented as:
[0343] ;
[0344] in, This indicates the bidding price strategy of the intelligent entity that manages thermal power units or load aggregators. This represents the network weight of the intelligent agent action network, indicating the bidding price of thermal power units or load aggregators. This indicates the bidding price action output by the intelligent agent action network for bidding prices of thermal power units or load aggregators.
[0345] Incentive pricing loss function of incentive pricing action network Represented as:
[0346] ;
[0347] in, Indicates an incentive pricing strategy. This represents the network weights of the action network of the incentive pricing agent. express The incentive price action output by the time-period incentive price agent action network.
[0348] The bidding loss function of the intelligent agent action network for bidding quantity of the thermal power unit. The bidding price loss function of the intelligent agent action network for bidding prices of thermal power units. The second joint loss function of thermal power units is obtained by weighting. , represented as:
[0349] ;
[0350] in, and This represents the weighting coefficient.
[0351] The bidding loss function of the bidding quantity agent action network of the load aggregator bidding quantity The bidding price loss function of the intelligent agent action network for load aggregator bidding price. Incentive pricing loss function of incentive pricing agent action network The second joint loss function of the load aggregator is obtained by weighting. , represented as:
[0352] ;
[0353] in, , and All of these represent weighting coefficients.
[0354] The asynchronous multi-task multi-agent training mechanism is as follows: Figure 3 and 4 As shown, it includes:
[0355] Initialize the neural network parameters of each agent. In the initial stage, each agent uses a pre-set random exploration strategy to generate its own actions and interact with the two-layer market environment.
[0356] After the initial phase, at the beginning of each training iteration, the first joint loss function is first used. Update the bidding agent action network and shared recurrent neural network;
[0357] The bidding quantity agent, bidding price agent, and incentive pricing agent generate their actions for the current moment based on the current input state and interact with the environment.
[0358] Based on the rewards obtained from environmental interactions, the bid quantity loss function is derived. Bidding price loss function and incentive pricing loss function Using the second joint loss function of thermal power units Or load aggregator second joint loss function Update all neural networks (including the already updated bid agent action network and shared recurrent neural network).
[0359] Step 4: Solve the multi-task multi-agent reinforcement learning training framework by asynchronously training the multi-task multi-agent flexible action-evaluation algorithm (PA-MTMASAC) to obtain the optimal joint strategy of bidding and incentive for load aggregators in the peak-shaving ancillary service market and the incentive-based demand response market, as well as the optimal bidding quantity and pricing strategy for thermal power units.
[0360] like Figure 5 and Figure 6 The diagram shows the framework of the PA-MTMASAC algorithm for load aggregator and thermal power unit, respectively.
[0361] The shared recurrent neural network employs an LSTM structure. The LSTM structure receives a dataset sampled from a priority experience replay buffer and outputs extracted abstract features and sampling probabilities. Represented as:
[0362] ;
[0363] in, This indicates the size of the priority experience replay buffer. Indicates hyperparameters, This indicates the priority of sample v after passing through hyperparameters. The adjusted value, The priority of sample ζ is determined by the hyperparameter. The adjusted value.
[0364] The centralized global comment network, bidding quantity action network, bidding price action network, and incentive pricing action network of the thermal power unit or load aggregator intelligent agent are configured with corresponding target networks. All target networks are updated using a soft update method, as shown below:
[0365] ;
[0366] ;
[0367] in, This represents the network weights of the target network in the centralized global evaluation network of the thermal power unit or load aggregator intelligent agent, when ent=Gen. , This represents the network weight of the target network in the intelligent agent action network for bidding quantity of thermal power units or the intelligent agent action network for bidding price of thermal power units; when ent=LA. , This represents the network weights of the target network, which is the agent action network representing the bid volume of the load aggregator, the agent action network representing the bid price of the load aggregator, or the agent action network representing the incentive pricing. Indicates soft update parameters, This represents the network weight of the main network in the centralized global evaluation network of the thermal power unit or load aggregator intelligent agent. This represents the network weight of the main network of the action network of the thermal power unit or load aggregator intelligent agent. This indicates the actions of the bidding agent. The action of the intelligent agent indicates the bidding price. This indicates the actions of the incentive pricing agent.
[0368] The action-evaluation architecture employs an asynchronously trained, multi-task, multi-agent flexible action-evaluation algorithm. The thermal power unit bidding quantity agent, thermal power unit bidding price agent, load aggregator bidding quantity agent, load aggregator bidding price agent, and incentive pricing agent output the distribution functions of the bidding strategy and incentive pricing strategy, respectively. After action sampling, the bidding quantity action, bidding price action, and incentive pricing action are obtained, which can directly interact with the peak-shaving auxiliary service market environment.
[0369] The action distribution function output by each action network of the thermal power unit's intelligent agent is expressed as:
[0370] ;
[0371] in, Indicates that the intelligent agent of the thermal power unit is in The observable state of a given time period Indicates a Gaussian distribution. and This represents the mean and standard deviation of the output of the intelligent agent action network of thermal power units. This represents the bidding quantity or bidding price action of thermal power units sampled from the action distribution. This indicates the bidding quantity strategy or bidding price strategy for thermal power units.
[0372] The action distribution function output by each action network of the load aggregator agent is expressed as:
[0373] ;
[0374] in, This indicates that the load aggregator intelligent agent is in The observable state of a given time period and This represents the mean and standard deviation of the load aggregator agent action network output. This represents the load aggregator bidding quantity action, bidding price action, or incentive pricing action sampled from the action distribution. This indicates the load aggregator's bidding volume strategy, bidding price strategy, or incentive pricing strategy.
[0375] The centralized global review network of the thermal power unit intelligent agent or load aggregator intelligent agent uses joint Q-values. Updated; the intelligent agent action network for bidding quantity of thermal power units and the intelligent agent action network for bidding price of thermal power units are updated through the first joint loss function of thermal power units. and the second joint loss function of thermal power units Update; the load aggregator bidding quantity agent action network, the load aggregator bidding price agent action network, and the incentive pricing agent action network are updated using the first joint loss function of the load aggregator. Second joint loss function of load aggregator Update.
[0376] like Figure 7 As shown, the process of training the joint strategy of thermal power units and load aggregators using the PA-MTMASAC algorithm includes:
[0377] (1) Initialize the weight parameters and priority experience replay cache of all neural networks of the thermal power unit or load aggregator intelligent agent;
[0378] (2) The thermal power unit / load aggregator intelligent agents respectively obtain their respective initial observable states or update the current state. observable state at time And based on the current state, obtain the current action from the learned policy by sampling;
[0379] (3) Input the current action into the virtual environment and interact with the ancillary services market and the demand response market. At this time, the virtual environment will call the clearing model of the ancillary services market and the end-user response model of the demand response market, and feed back the clearing result and the actual user response to the thermal power unit / load aggregator intelligent agent.
[0380] (4) Randomly select a batch of samples from the priority experience replay buffer, and apply the joint loss function proposed above. The gradient backpropagation algorithm is used to update the shared deep recurrent neural network, global comment network, and bidding action network of the thermal power unit / load aggregator agent, respectively.
[0381] (5) The thermal power unit / load aggregator intelligent agent observes the new state after the environmental update. Calculate the immediate reward based on the reward function proposed above. ;
[0382] (6) Randomly select a batch of samples from the priority experience replay buffer, and apply the joint loss function proposed above. And use the gradient backpropagation algorithm to update all neural networks of the thermal power unit / load aggregator agent;
[0383] (7) Determine if the last moment of the day has been reached. If yes, proceed to step (8); otherwise, return to step (2).
[0384] (8) Determine whether the number of training iterations has reached the preset maximum value. If yes, proceed to step (9); otherwise, return to step (2).
[0385] (9) Obtain the trained action network of the thermal power unit / load aggregator intelligent agent, i.e. the trained strategy.
[0386] In this embodiment, based on the established Markov decision process and the PA-MTMASAC algorithm, the bidding volume, bidding price, and incentive pricing multi-agent of the load aggregator are trained in parallel. Through repeated iterative training until the learning curves of the multi-agent converge and stabilize, a well-trained price-influencer type load aggregator agent is obtained. Furthermore, the trained load aggregator agent is used to make decisions on bidding volume, bidding price, and incentive pricing in the peak-shaving ancillary service market with corresponding parameter settings. Specifically, a real-time peak-shaving ancillary service market clearing model and an end-user response model are used to complete clearing simulation and user response load simulation, thereby simulating the trading behavior of the price-influencer type load aggregator within a preset time period.
[0387] Example 2
[0388] This embodiment provides a device for optimizing a joint bidding and incentive strategy for a dual-side peak-shaving auxiliary service market, including:
[0389] The model building module is used to construct a dual-market joint optimization model for load aggregators and thermal power units based on the pre-acquired parameter data of the peak-shaving ancillary service market and the incentive-based demand response market.
[0390] A formalization module is used to formalize the dual-market joint optimization model into a Markov decision process;
[0391] The training framework construction module is used to construct a multi-task multi-agent reinforcement learning training framework based on the Markov decision process. The multi-task multi-agent reinforcement learning training framework includes a load aggregator bidding quantity agent, a load aggregator bidding price agent, an incentive pricing agent, and a thermal power unit bidding quantity agent and a thermal power unit bidding price agent.
[0392] The solution module is used to solve the multi-task multi-agent reinforcement learning training framework by asynchronously training a multi-task multi-agent flexible action-evaluation algorithm, so as to obtain the optimal joint strategy of bidding and incentive for load aggregators in the peak-shaving auxiliary service market and the incentive-based demand response market, as well as the optimal bidding quantity and pricing strategy for thermal power units.
[0393] Example 3
[0394] This embodiment provides a device for optimizing the bidding and incentive joint strategy of the source-load dual-side peak shaving auxiliary service market, including a processor and a storage medium;
[0395] The storage medium is used to store instructions;
[0396] The processor is configured to operate according to the instructions to execute the steps of the source-load dual-side peak shaving auxiliary service market bidding and incentive joint strategy optimization method described in Embodiment 1.
[0397] Example 4
[0398] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the source-load dual-side peak shaving auxiliary service market bidding and incentive joint strategy optimization method described in Embodiment 1.
[0399] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0400] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0401] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0402] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0403] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A method for optimizing a market bidding and incentive strategy for dual-side peak shaving auxiliary services, characterized in that, include: Based on the pre-acquired parameter data of the peak-shaving ancillary service market and the incentive-based demand response market, a dual-market joint optimization model for load aggregators and thermal power units is constructed. The dual-market joint optimization model is formalized as a Markov decision process. Based on the Markov decision process, a multi-task multi-agent reinforcement learning training framework is constructed, which includes a load aggregator bidding quantity agent, a load aggregator bidding price agent, an incentive pricing agent, and a thermal power unit bidding quantity agent and a thermal power unit bidding price agent. The multi-task multi-agent reinforcement learning training framework is solved by asynchronously training a multi-task multi-agent flexible action-evaluation algorithm, which yields the optimal joint strategy of bidding and incentive for load aggregators in the peak-shaving ancillary service market and the incentive-based demand response market, as well as the optimal bidding quantity and pricing strategy for thermal power units.
2. The method for optimizing the bidding and incentive joint strategy of the source-load dual-side peak shaving ancillary service market according to claim 1, characterized in that, The dual-market joint optimization model includes a peak-shaving ancillary service market clearing model and a thermal power unit bidding model. The peak-shaving ancillary service market clearing model includes a first objective function, expressed as: ; in, This indicates taking the minimum value. Indicates the first Individual load aggregator, Represents the set of load aggregators. express Time period The bidding volume of each load aggregator. express The bidding price of the i-th load aggregator in the time period. express Time period Is the load aggregator selected? Indicates the first One thermal power unit, This represents the set of thermal power units participating in peak shaving. express Time period The bidding volume of each thermal power unit in the peak-shaving market. express Time period The bidding price of individual thermal power units in the peak-shaving market express Time period A thermal power unit successfully won the bid, when ,express Time period If a thermal power unit is successfully bid, the bid will be rejected. The peak-shaving ancillary service market clearing model also includes a first constraint, which includes power balance constraint, load demand constraint, network flow constraint and bidding constraint; The power balance constraint is expressed as: ; in, Indicates peak shaving demand. express Time period Peak shaving capacity provided by individual load aggregators Indicates the total number of load aggregators; The load demand constraint is expressed as follows: ; in, and They represent the first The lower and upper limits of the adjustable bidding volume after aggregating and signing users of each load aggregator; The network flow constraint is represented as follows: ; in, Indicates the line The upper limit of the trend, express Time period nodes The injected node power, Indicates the total number of nodes. Indicates the line With nodes The power transfer distribution factor between them; The bidding constraint is expressed as follows: ; in, Indicates the first The upper limit of the bid price for each load aggregator; The bidding model for thermal power units is a bidding mode in which thermal power units submit their peak-shaving capacity and compensation prices in stages based on their marginal costs, expressed as: ; ; in, express Time period Peak-shaving compensation cost for each thermal power unit. express Time period The unit peak-shaving cost of a thermal power unit express Time period The planned output of each thermal power unit is as follows. Indicates the first Minimum output of a thermal power unit during deep peak shaving; The bidding model for thermal power units includes a second constraint, which includes a bidding range constraint, a power constraint, and a ramp constraint. The unit bidding range constraint is expressed as follows: ; in, and Let $\mathbf{j}$ and $\mathbf{j}$ represent the lower limit and upper limit of the bidding price for the $j$-th thermal power unit in the peak-shaving market, respectively. The power constraint of the thermal power unit is expressed as follows: ; Among them, when =1 means Time period Each thermal power unit is in operation; otherwise, it is not in operation. Indicates the first The maximum technical output of each thermal power unit Indicates the first Minimum technical output of each thermal power unit; The unit ramping constraint is expressed as follows: ; in, Indicates the first The rate of ascent for each thermal power unit. Indicates the first Landslide rate of each thermal power unit express Time period The technological contribution of each thermal power unit in the peak-shaving market express Time period The technological contribution of each thermal power unit in the peak-shaving market Indicates the first The maximum output of each thermal power unit express Time period Each thermal power unit is in operation; otherwise, it is not in operation.
3. The method for optimizing the bidding and incentive joint strategy of the source-load dual-side peak shaving ancillary service market according to claim 2, characterized in that, The dual-market joint optimization model also includes a load aggregator revenue model and a contracted user response model. The load aggregator revenue model includes a second objective function, expressed as: ; in, express Time period The strategy bidding price factor of individual load aggregators express Time period The incentive price for individual load aggregators express Time period Strategic bidding price factors for individual thermal power units Represents the mathematical expectation. This indicates the number of times grid operators initiate peak-shaving ancillary services market. , This represents the total number of times a grid operator initiates the peak-shaving ancillary services market within a complete optimized week. This represents the percentage of future earnings discounted to the present time. express Time period Peak shaving revenue of individual load aggregators express Time period The loss of electricity sales revenue for individual load aggregators express Time period Demand response incentive costs for individual load aggregators express Time period Deviation penalty for each load aggregator express Time period The output cost of a single thermal power unit; Among them, the Time period Peak shaving revenue of individual load aggregators Represented as: ; in, express Clearing price of the peak-shaving ancillary services market express Time period The winning capacity of each load aggregator express Time period The actual load reduction of the load aggregator, β represents the deviation tolerance coefficient; The Time period Electricity sales revenue loss for individual load aggregators Represented as: ; in, express Time period The retail price agreed upon in the contract between the load aggregator and the contracted user. express The wholesale electricity price at which time-of-use load aggregators purchase electricity from the day-ahead market; The Time period Demand response incentive cost for individual load aggregators Represented as: ; The Time period Deviation penalty for individual load aggregators Represented as: ; ; in, express Economic penalty per unit deviation over a given time period; The Time period The output cost of a single thermal power unit Represented as: ; in, Indicates the first Each thermal power unit Power output during a given time period All of these represent cost constants for thermal power units; The load aggregator revenue model also includes a bidding price function, expressed as: ; ; in, express Time period The theoretical bidding price of each load aggregator. express Time period The original load of the aggregated load of the load aggregator. and Represents constant coefficients. express Time period The strategic bidding price of each load aggregator; The signed user response model includes a third objective function. When the current time period is for peak shaving, the third objective function of the signed user response model is expressed as: ; When the current period is during valley filling, the third objective function of the signed user response model is expressed as: ; in, express Time period The total response volume for each contracted customer, including load shedding and load transfer, where η represents the percentage of load shedding out of the total response volume. express Time period The response cost for each signed customer, where max represents the maximum value. express Time period The retail price agreed upon in the contract between the load aggregator and the contracted user; The contracted user response model also includes a third constraint, expressed as follows: ; in, express Time period The original load before responding to the needs of a signed customer. This indicates the percentage of the maximum response relative to the original load.
4. The method for optimizing the bidding and incentive joint strategy of the source-load dual-side peak shaving ancillary service market according to claim 1, characterized in that, The formalization of the dual-market joint optimization model into a Markov decision process includes: The Markov decision process of the dual-market joint optimization model includes the Markov decision process for the joint strategy optimization problem of bidding and incentive of load aggregators and the Markov decision process for the bidding strategy optimization problem of thermal power units. The state-space representation of the Markov decision process for the joint bidding and incentive strategy optimization problem of the load aggregator is as follows: , represents the set of observable states of the load aggregator agent across all time periods, where the load aggregator agent is in observable state during the period Represented as ,include Historical observation trajectory of intelligent agents in the bidding volume of load aggregators during the time period. Historical observation trajectory of the intelligent agent for the bidding price of load aggregators during the time period and Historical observation trajectory of incentive pricing agents under different time periods; The The historical observation trajectory of the bidding volume of the intelligent agent during the load aggregator in a given time period is represented as follows: ; in, express Historical observation trajectory of intelligent agents' bidding volume of load aggregators during specific time periods. Indicates from what has already happened Time's up Historical data on bidding volume for a given period. Indicates from what has already happened Time's up The aggregated original load of the load aggregator for the time period. Indicates from what has already happened Time's up Peak-shaving capacity of thermal power units during a given period; The The historical observation trajectory of the bidding price intelligence agent of the load aggregator during the time period is represented as follows: ; in, express Historical observation trajectory of the bidding price of intelligent agents for load aggregators during a given time period. Indicates from what has already happened Time's up Historical data on the strategy bidding coefficient for the specified time period. Indicates from what has already happened Time's up Historical data on the retail prices agreed upon in contracts between load aggregators and contracted users during specific time periods. Indicates that something has already happened. Time's up Historical data on wholesale electricity prices for load aggregators purchasing electricity from the day-ahead market during specific time periods; The The historical observation trajectory of the incentive pricing agent during the specified time period is represented as follows: ; in, express Historical observation trajectory of incentive pricing agents under different time periods Indicates from what has already happened Time's up Historical data on the incentive prices offered to contracted users by load aggregators during specific time periods. Indicates from what has already happened Time's up Historical data on the actual load reduction of load aggregators during a given time period; The action space of the Markov decision process for the joint bidding and incentive strategy optimization problem of the load aggregator is represented as follows: It includes the continuous strategy bidding volume of the load aggregator bidding volume agent within a range, the continuous strategy bidding coefficient of the load aggregator bidding price agent within a range, and the continuous incentive pricing of the incentive pricing agent within a range. The continuous policy reporting volume of the load aggregator bidding agent within a certain interval is represented as: ; in, This represents the continuous policy reporting volume of the agent within a certain interval, representing the bidding volume of the load aggregator. This indicates the upper limit of the bid volume for load aggregators; The continuous strategy bidding coefficients of the load aggregator bidding price agent within a certain interval are expressed as: ; in, This represents the continuous strategy bidding coefficient of the agent within a certain interval, representing the bidding price of the load aggregator. and These represent the lower and upper limits of the bidding price factor for the load aggregator strategy, respectively; The incentive pricing agent's continuous incentive pricing within a certain interval is expressed as: ; in, This indicates that the incentive pricing agent continuously sets incentive prices within a certain range. and These represent the lower and upper limits of the incentive price, respectively. The mapping from the state space to the action space of the Markov decision process for the joint bidding and incentive strategy optimization problem of the load aggregator is expressed as: ; in, , , as well as These represent the load aggregator bidding volume strategy, load aggregator bidding price strategy, incentive pricing strategy, and load aggregator joint strategy, respectively. Let the state space of the Markov decision process representing the joint strategy optimization problem of bidding and incentives for load aggregators be denoted. The action space represents the Markov decision process for optimizing the joint bidding and incentive strategy of load aggregators; The reward function of the Markov decision process for the joint strategy optimization problem of bidding and incentives for the load aggregator is expressed as: ; in, express Time period The immediate profit obtained by each load aggregator Indicates in Time period Peak shaving revenue for load aggregators Indicates in Time period The loss of electricity sales revenue for individual load aggregators Indicates in Time period Demand response incentive costs for individual load aggregators Indicates in Time period Deviation penalty for each load aggregator.
5. The method for optimizing the bidding and incentive joint strategy of the source-load dual-side peak shaving ancillary service market according to claim 4, characterized in that, The state-space representation of the Markov decision process for the bidding strategy optimization problem of thermal power units is as follows: , representing the set of observable states of the thermal power unit agent over all time periods, where the thermal power unit agent is in observable state during the period Represented as It includes the historical observation trajectory of the bidding volume intelligence agent for thermal power units and the historical observation trajectory of the bidding price intelligence agent for thermal power units; The historical observation trajectory of the intelligent agent for bidding volume of thermal power units is represented as follows: ; in, express Historical observation trajectory of bidding volume for thermal power units under different time periods. Indicates from what has already happened Time's up Historical adjustment capacity submissions for a given period Indicates from what has already happened Time's up Historical data on system peak-shaving demand during specific time periods. Indicates from what has already happened Time's up The remaining regulating capacity of the generating units during the time period; The historical observation trajectory of the intelligent agent for bidding prices of thermal power units is represented as follows: ; in, express Historical observation trajectory of bidding prices for thermal power units during a given period. Indicates from what has already happened Time's up Historical data on the bidding coefficient of thermal power strategy for different time periods. Indicates from what has already happened Time's up Historical market electricity price data for the period Indicates from what has already happened Time's up The actual electricity price fluctuation during the period; The action space representation of the Markov decision process for the bidding strategy optimization problem of thermal power units is as follows: It includes the continuous strategy bidding volume of the thermal power unit bidding agent within a range and the continuous strategy bidding coefficient of the thermal power unit bidding price agent within a range. The continuous policy reporting of the bidding agent for thermal power units within a certain interval is represented as: ; in, This represents the continuous strategy reporting of the bidding agent for thermal power units within a certain interval. This indicates the upper limit of the number of thermal power units that can be bid on; The continuous strategy bidding coefficient of the bidding agent for thermal power units within a certain interval is expressed as: ; in, This represents the continuous strategy bidding coefficient of the intelligent agent for bidding on thermal power units within a certain range. and These represent the lower and upper limits of the strategic bidding price factor for thermal power units, respectively; The mapping from the state space to the action space of the Markov decision process in the joint strategy optimization problem of bidding and incentive for thermal power units is expressed as: ; in, , as well as These represent the bidding volume strategy, bidding price strategy, and joint bidding strategy for thermal power units, respectively. Let the state space of the Markov decision process be used to address the bidding strategy optimization problem for thermal power units. The action space of a Markov decision process representing the optimization problem of bidding strategy for thermal power units; The reward function of the Markov decision process for the bidding strategy optimization problem of the thermal power unit is expressed as: ; in, Indicates in Time period The immediate profit obtained by each thermal power unit Indicates in Time period Peak-shaving revenue of each thermal power unit. Indicates in Time period Deviation penalty for each thermal power unit Indicates in Time period The cost of deep peak shaving for each thermal power unit.
6. The method for optimizing the bidding and incentive joint strategy of the source-load dual-side peak shaving ancillary service market according to claim 5, characterized in that: The multi-task, multi-agent reinforcement learning training framework constructed based on the Markov decision process includes: The architecture employs a fully cooperative action-comment framework, comprising action networks for bidding quantities of thermal power units, bidding prices of thermal power units, bidding quantities of load aggregators, bidding prices of load aggregators, and incentive pricing. Each action network corresponds to several decision-making tasks for one agent. All agents share a centralized global comment network, which receives the states of all agents and outputs a joint Q-value. , represented as: ; in, , This represents the intelligent agent that aggregates loads. This refers to the intelligent agent of a thermal power unit. Represents the mathematical expectation. and These represent the network weights of the main network and the target network of the global comment network, respectively. Indicates the discount factor. Indicates the temperature coefficient. This represents the Q-value of the main network output of the global comment network for the load aggregator agent or thermal power unit agent. This represents the Q-value of the target network output of the global evaluation network for the load aggregator agent or thermal power unit agent. express The total profit of the time-of-use load aggregator or thermal power unit when ent = Gen. Indicates in The observable status of the intelligent agent of the thermal power unit during the specified time period. Indicates in The combined action of the intelligent agent outputting the bidding quantity of thermal power units and the intelligent agent outputting the bidding price of thermal power units during the same time period. Indicates in The strategy reporting volume of the intelligent agent for bidding volume of thermal power units during the same period. Indicates in The strategy bidding coefficient of the intelligent agent for bidding prices of thermal power units during the specified time period. This indicates the combined strategy of thermal power units; when ent=LA, Indicates in The observable state of the load aggregator agent during the specified time period. Indicates in The combined actions output by the load aggregator bidding volume agent, the load aggregator bidding price agent, and the incentive pricing agent during the specified time period. This indicates a joint strategy by load aggregators; A multi-task joint loss function based on a shared recurrent neural network is constructed to dynamically weight and optimize the bidding quantity loss function, bidding quantity prediction loss function, and bidding price loss function of thermal power units. It also dynamically weights and optimizes the bidding quantity loss function, bidding quantity prediction loss function, bidding price strategy loss function, and incentive pricing loss function of load aggregators. The process of constructing the multi-task joint loss function includes: The historical bidding volume and historical winning bid volume of the thermal power units are input into a shared recurrent neural network to finally obtain the predicted bidding volume of the thermal power units. The historical bidding volume and historical winning volume of the thermal power units are processed to obtain the historical data layer of the intelligent agent neural network of the thermal power units, represented as follows: ; in, This represents the historical data layer of the neural network of the intelligent agent in a thermal power unit. and These represent the historical number of bids and the historical number of successful bids for thermal power units, respectively. This indicates a splicing operation. The parameter matrix representing the historical data layer of the neural network of a thermal power unit intelligent agent; The historical bid volume, historical winning bid volume, and historical user response load of the load aggregator are input into a shared recurrent neural network to finally obtain the predicted bid volume of the load aggregator. The historical bidding volume, historical winning volume, and historical user response load of the load aggregator are processed to obtain the historical data layer of the load aggregator's intelligent agent neural network, represented as follows: ; in, This represents the historical data layer of the neural network of the load aggregator intelligent agent. These represent historical bidding volume, historical successful bidding volume, and historical user response load, respectively. This indicates a splicing operation. This represents the parameter matrix of the historical data layer of the neural network of the load aggregator intelligent agent; Historical data layer based on neural network of intelligent agent of thermal power unit Or load aggregator intelligent agent neural network historical data layer Perform linear normalization, which is expressed as: ; in, This represents the output of the linear normalization layer. , This represents the historical maximum data. Represents the lowest historical data; The shared recurrent neural network outputs the linear normalization layer. The operation is represented as: ; in, This represents the predicted bid amount from the output of the shared recurrent neural network. This represents the weight matrix of a shared recurrent neural network. This represents the bias term in a shared recurrent neural network. The bidding agent action network predicts the bidding volume from the output of the shared recurrent neural network. The operation is represented as: ; in, This represents the bidding action output by the bidding agent action network after passing through a shared recurrent neural network. This represents the weight matrix of the bidding agent action network. Represents the bias term of the bidding agent action network; ReLU and Tanh are activation functions. The mean squared error between the predicted bid volume and the winning bid volume in the current period is used as the bid volume prediction loss function. , represented as: ; in, Indicates in The winning capacity of thermal power units or load aggregators during the time period; the predicted bidding volume includes the predicted bidding volume of thermal power units and the predicted bidding volume of load aggregators. Using KL divergence as the bid loss function of the bid agent action network. , represented as: ; in, This indicates the bidding volume strategy of thermal power units or load aggregators. This represents the network weight of the intelligent agent action network representing the bidding volume of thermal power units or load aggregators. This represents the bidding quantity action output by the intelligent agent action network for bidding quantities of thermal power units or load aggregators. The allocation function representing the intelligent agent of thermal power units or load aggregators; Loss function for predicting bid volume With bid quantity loss function The first joint loss function is obtained by weighting. , represented as: ; in, and All represent weighting coefficients; Bidding price loss function of bidding price action network Represented as: ; in, This indicates the bidding price strategy of the intelligent entity that manages thermal power units or load aggregators. This represents the network weight of the intelligent agent action network, indicating the bidding price of thermal power units or load aggregators. This indicates the bidding price action output by the intelligent agent action network for bidding prices of thermal power units or load aggregators. Incentive pricing loss function of incentive pricing action network Represented as: ; in, Indicates an incentive pricing strategy. This represents the network weights of the action network of the incentive pricing agent. express The incentive price action output by the time-period incentive price agent action network; The bidding loss function of the intelligent agent action network for bidding quantity of the thermal power unit. The bidding price loss function of the intelligent agent action network for bidding prices of thermal power units. The second joint loss function of thermal power units is obtained by weighting. , represented as: ; in, and Indicates the weighting coefficient; The bidding loss function of the bidding quantity agent action network of the load aggregator bidding quantity The bidding price loss function of the intelligent agent action network for load aggregator bidding price. Incentive pricing loss function of incentive pricing agent action network The second joint loss function of the load aggregator is obtained by weighting. , represented as: ; in, , and All of these represent weighting coefficients.
7. The method for optimizing the bidding and incentive joint strategy of the source-load dual-side peak shaving ancillary service market according to claim 6, characterized in that, The step of solving the multi-task multi-agent reinforcement learning training framework by asynchronously training a multi-task multi-agent flexible action-evaluation algorithm includes: The shared recurrent neural network employs an LSTM structure. The LSTM structure receives a dataset sampled from a priority experience replay buffer and outputs extracted abstract features and sampling probabilities. Represented as: ; in, This indicates the size of the priority experience replay buffer. Indicates hyperparameters, This indicates the priority of sample v after passing through hyperparameters. The adjusted value, The priority of sample ζ is determined by the hyperparameter. The adjusted value; The centralized global comment network, bidding quantity action network, bidding price action network, and incentive pricing action network of the thermal power unit or load aggregator intelligent agent are configured with corresponding target networks. All target networks are updated using a soft update method, as shown below: ; ; in, This represents the network weights of the target network in the centralized global evaluation network of the thermal power unit or load aggregator intelligent agent, when ent=Gen. , This represents the network weight of the target network in the intelligent agent action network for bidding quantity of thermal power units or the intelligent agent action network for bidding price of thermal power units; when ent=LA. , This represents the network weights of the target network, which is the agent action network representing the bid volume of the load aggregator, the agent action network representing the bid price of the load aggregator, or the agent action network representing the incentive pricing. Indicates soft update parameters, This represents the network weight of the main network in the centralized global evaluation network of the thermal power unit or load aggregator intelligent agent. This represents the network weight of the main network of the action network of the thermal power unit or load aggregator intelligent agent. This indicates the actions of the bidding agent. The action of the intelligent agent indicates the bidding price. This indicates the actions of the incentive pricing agent; The action-evaluation architecture employs an asynchronously trained, multi-task, multi-agent flexible action-evaluation algorithm. The bidding quantity agent, bidding price agent, load aggregator bidding quantity agent, load aggregator bidding price agent, and incentive pricing agent respectively output the distribution functions of the bidding strategy and incentive pricing strategy. After action sampling, these actions—bidding quantity action, bidding price action, and incentive pricing action—can directly interact with the peak-shaving auxiliary service market environment. Each action network of the thermal power unit agent outputs an action distribution function, expressed as: ; in, Indicates that the intelligent agent of the thermal power unit is in The observable state of a given time period Indicates a Gaussian distribution. and This represents the mean and standard deviation of the output of the intelligent agent action network of thermal power units. This represents the bidding quantity or bidding price action of thermal power units sampled from the action distribution. This indicates the bidding quantity strategy or bidding price strategy for thermal power units. The action distribution function output by each action network of the load aggregator agent is expressed as: ; in, This indicates that the load aggregator intelligent agent is in The observable state of a given time period and This represents the mean and standard deviation of the load aggregator agent action network output. This represents the load aggregator bidding quantity action, bidding price action, or incentive pricing action sampled from the action distribution. This indicates the load aggregator's bidding volume strategy, bidding price strategy, or incentive pricing strategy; The centralized global review network of the thermal power unit intelligent agent or load aggregator intelligent agent uses joint Q-values. Updated; the intelligent agent action network for bidding quantity of thermal power units and the intelligent agent action network for bidding price of thermal power units are updated through the first joint loss function of thermal power units. and the second joint loss function of thermal power units Update; the load aggregator bidding quantity agent action network, the load aggregator bidding price agent action network, and the incentive pricing agent action network are updated using the first joint loss function of the load aggregator. Second joint loss function of load aggregator Update.
8. A joint strategy optimization system for bidding and incentives in the market for dual-side peak shaving auxiliary services, characterized in that, include: The model building module is used to construct a dual-market joint optimization model for load aggregators and thermal power units based on the pre-acquired parameter data of the peak-shaving ancillary service market and the incentive-based demand response market. A formalization module is used to formalize the dual-market joint optimization model into a Markov decision process; The training framework construction module is used to construct a multi-task multi-agent reinforcement learning training framework based on the Markov decision process. The multi-task multi-agent reinforcement learning training framework includes a load aggregator bidding quantity agent, a load aggregator bidding price agent, an incentive pricing agent, and a thermal power unit bidding quantity agent and a thermal power unit bidding price agent. The solution module is used to solve the multi-task multi-agent reinforcement learning training framework by asynchronously training a multi-task multi-agent flexible action-evaluation algorithm, so as to obtain the optimal joint strategy of bidding and incentive for load aggregators in the peak-shaving auxiliary service market and the incentive-based demand response market, as well as the optimal bidding quantity and pricing strategy for thermal power units.
9. A device for optimizing a market bidding and incentive strategy for dual-side peak shaving auxiliary services, characterized in that, Including processor and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the source-load dual-side peak shaving auxiliary service market bidding and incentive joint strategy optimization method as described in any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the joint strategy optimization method for bidding and incentive in the source-load dual-side peak shaving auxiliary service market as described in any one of claims 1 to 7.
Citation Information
Patent Citations
Virtual power plant multi-object optimizing scheduling method in consideration of load side and power supply side
CN106300336A
Dual-objective optimization method for load aggregator participating in declaration quantity and price of electric power peak regulation market
CN114331760A
User and new energy bilateral transaction system peak regulation operation method and storage medium
CN115036937A
Spot peak regulation market joint clearing method considering load side non-convex quotation
CN116882658A
Planning and solving method for optimal scheme of flexible load participating in peak shaving auxiliary service
CN117713110A
Cited By
Thermal power plant operation control optimization method and system based on artificial intelligence
CN121300048A
Microgrid P2P electric energy transaction bidding and matching optimization method
CN121660769A
Demand response user side multi-agent collaborative decision-making method based on switchable preference double-graph reinforcement learning
CN121684539A
