Generator quotation method based on bidding agent
By adopting the power quotation method based on bidding agents and two-stage deep reinforcement learning algorithm in the power system, the scheduling challenges caused by the volatility of new energy and market self-interest in the power system are solved, and a more efficient power market scheduling and supply balance is achieved.
Patent Information
- Application Number
- CN202411883309.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-19
- Publication Date
- 2025-04-25
AI Technical Summary
Before new energy gradually developed into adjustable and controllable power supplies, the power system faced volatility and uncertain energy supply, resulting in huge challenges in the balance of power system supply and clean energy consumption. At the same time, the self-interest of various stakeholders in the market environment will lead to blockage of power transmission and make it difficult to effectively dispatch.
Using the power generator quotation method based on bidding agents, by establishing a bidding model for free joint power generators, and combining the two-stage deep reinforcement learning algorithm D3QN-SAC, the bidding and clearing process of power generators is simulated, the object of conspiracy is selected and the bidding strategy is formed to solve the problem of discrete-continuous decision-making in the two-stage.
It effectively solves the physical blockage caused by insufficient transmission line capacity and the "false" blockage caused by market entities' self-interested behavior, and improves the scheduling efficiency and supply balance of the power market.
Smart Images
Figure SMS_1 
Figure SMS_2 
Figure SMS_6
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of electric energy dispatching, and in particular to a power generator quotation method based on a bidding intelligent body. Background Art
[0002] my country's power system is transforming into a new power system with low-carbon attributes. By 2030, wind and solar new energy will account for 48% of my country's installed power generation capacity, and traditional thermal power (coal and gas) will gradually drop to 32.5%. Before new energy gradually develops into an adjustable and controllable power source, the power system faces a scenario of energy supply with volatility and uncertainty, and there are huge challenges in ensuring the balance of power system supply and the absorption of clean energy.
[0003] In a market environment, the self-interest of various stakeholders will affect the transmission process of electricity. Electric energy dispatching needs to fully consider the decentralization, autonomy and interest motivation of the investment planning of stakeholders to avoid the blockage of power transmission due to the market behavior of market members. Therefore, when considering the transmission planning problem in the context of the power market, it is necessary not only to consider the physical blockage of the transmission section due to insufficient transmission line capacity, but also to distinguish the "false" blockage of the transmission section caused by the selfish behavior of market players. From the perspective of the solution algorithm, the intelligent agent in reinforcement learning gradually learns how to improve the strategy through repeated interactions with the market clearing process, which is more in line with the needs of actual scenarios. However, when reinforcement learning is used to solve problems with multiple decision variables, there is a problem of training difficulties, and it is even more difficult to solve two-stage continuous decision problems. Summary of the invention
[0004] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art, provide a bidding agent-based power producer quotation method, construct an optimization model for bidding considering the free association of power producers, and propose a two-stage deep reinforcement learning algorithm framework, combining two deep reinforcement learning algorithms to solve the two-stage discrete-continuous decision-making problem, which is specifically implemented by the following technical solutions:
[0005] The power generator quotation method based on the bidding intelligent agent includes:
[0006] Establish a bidding model for free association of power producers, including: a revenue model for power producers that coordinate energy status across days and a clearing revenue model for power producers that perform day-ahead self-dispatch;
[0007] Based on the two-stage deep reinforcement learning algorithm D3QN-SAC, a bidding agent is constructed to simulate the bidding and clearing process of free association of power producers. In the first stage, the bidding agent takes the highest expected profit of power producers as the goal based on the bidding model of free association of power producers, and takes the daily energy change trend of energy storage as the decision variable to select the collusion object; in the second stage, the bidding agent takes the bidding coefficient as the decision variable, and forms a bidding strategy according to the collusion object in the first stage and the market clearing results of the previous round when the collusion is successful, and bids are made according to the bidding strategy.
[0008] The further design of the power producer quotation method based on the bidding agent is to construct a power producer profit model according to formula (1):
[0009]
[0010] stk min ≤k i ≤k max
[0011] F≤F0
[0012] minf(1) In formula (1), f1 is the bidding profit of market entity g; k max and k min represents the upper and lower limits of various bidding coefficients; F and F0 represent various operating constraints and the boundaries of the constraints; f is the objective function of the market operator to minimize the power generation cost, a r is the first-order term of the marginal power generation cost of the new energy generator r, p r To contribute to the new energy power generators that need to make decisions, g is the power generation of the traditional power generator g, a g , b g are the quadratic and linear coefficients of the marginal generation cost of the traditional power generator g, and λ represents the day-ahead electricity market clearing price.
[0013] The further design of the bidding agent-based power producer quotation method is to construct the objective function of the power producer clearing profit model according to formula (2):
[0014]
[0015] In formula (2), f2 represents the social welfare maximization of day-ahead power market transactions, a g , b g are the quadratic and linear coefficients of the marginal power generation cost of the traditional power generator g; p g is the power generation of the traditional power generator g; k g is the quotation coefficient of the traditional power generator g; a l and b lare the quotes of load l in the day-ahead power market; k r is the price quoted by the new energy power generator r in the day-ahead power market; p r To contribute to the new energy power generators that need to make decisions, l is the reported amount of load l in the electricity market, a r is the first-order term of the marginal power generation cost of the new energy generator r, For the collection of thermal power generators, is the set of loads, A collection of new energy power generators.
[0016] The further design of the bidding agent-based power producer quotation method is that the constraints of the power producer clearing revenue model include:
[0017] Power balance constraints:
[0018]
[0019] λ t represents the day-ahead electricity market clearing price, is the set of loads, A collection of new energy power generators. A collection of thermal power generators;
[0020] Minimum and maximum output constraints:
[0021]
[0022]
[0023] k g ,a l ,b l ,p g ,p r ,p l ≥0
[0024] In the formula, capping the output of traditional power generators; capping the output of new energy generators; k is the upper limit of load reporting g is the quotation coefficient of the traditional power generator g; a l and b l are the quotes of load l in the day-ahead power market; k r is the price quoted by the new energy power generator r in the day-ahead power market; p g is the power generation of the traditional power generator g; p r To contribute to the new energy power generators that need to make decisions, l is the reported amount of load l in the electricity market, a ris the first-order term of the marginal power generation cost of the new energy generator r, F i min is the lower capacity limit of line i, F i max is the upper capacity limit of line i.
[0025] The further design of the bidding agent-based power producer quotation method is that the first stage of the bidding agent is constructed based on the D3QN algorithm and includes:
[0026] D3QN algorithm network building module, used to build a deep Q network consisting of Current Q Network and Target Q Network and an experience replay library;
[0027] The action conversion module selects discrete actions based on the Q network using the ε-greedy strategy and converts the actions into the output of the first stage, i.e., the target vector;
[0028] The target vector conversion module converts the target vector output from the first stage into a specific comprehensive utility function as a reward function as the state of the generator clearing revenue model;
[0029] The algorithm training module trains the Q network through historical data obtained through interaction to improve the performance of the algorithm.
[0030] The further design of the power generator quotation method based on the bidding agent is that the action conversion module selects discrete actions. The specific selection method is: in the early stage of training, the agent randomly selects actions to explore feasible action space. When the number of training times increases to a certain value, it is reduced as the number of training times increases. In each training process, a random number is generated. If the random number is greater than this number, the greedy method is used to select the action, otherwise the action is selected randomly.
[0031] The power generator bidding method based on the bidding agent is further designed in that the second stage of the bidding agent is constructed based on the SAC algorithm and includes:
[0032] SAC algorithm network construction module, which builds the Actor network for outputting continuous actions and the Critic network for evaluating the output actions of the Actor network;
[0033] The bidding action conversion module selects collusion actions using the ε-greedy strategy through the Actor network and converts the actions into the output of the second stage, namely the node electricity price, clearing electricity and bidding coefficient plan;
[0034] Algorithm training module, the historical data obtained through interaction is used to train the Actor and Critic networks.
[0035] A further design of the bidding agent-based power producer quotation method is that the bidding action conversion module selects collusion actions specifically as follows: random actions are initially adopted, but when the number of training times increases to a set value, greedy actions are selected.
[0036] The present invention also provides an electronic device, comprising a memory, a processor and a computer program, wherein the computer program is stored in the memory and is configured to be executed by the processor to implement the power generator quotation method based on the bidding intelligent agent.
[0037] The present invention also provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor to implement the power generation company quotation method based on the bidding agent. The advantages of the present invention are as follows:
[0038] The power producer quotation method based on the bidding agent of the present invention establishes a bidding agent construction method of the bidding model of the free association of power producers, thereby calculating the solution to the market bidding game problem; according to the actual market operation situation, the technical output constraints are taken into consideration to avoid the blockage of power transmission due to the market behavior of market members, solve the physical blockage of the transmission section caused by insufficient transmission line capacity, and distinguish the problem of "false" blockage of the transmission section caused by the selfish behavior of market entities. DETAILED DESCRIPTION
[0039] The technical solution of the present invention is described in detail below.
[0040] Embodiment 1
[0041] The bidding agent-based bidding method for power generators of this embodiment includes: establishing a bidding model for free association of power generators that includes a power generator profit model for cross-day energy state coordination and a power generator clearing profit model for day-ahead self-dispatching.
[0042] Based on the two-stage deep reinforcement learning algorithm D3QN-SAC, a bidding agent is constructed to simulate the bidding and clearing process of free association of power producers. In the first stage, the bidding agent takes the highest expected profit of power producers as the goal based on the bidding model of free association of power producers, and takes the daily energy change trend of energy storage as the decision variable to select the collusion object; in the second stage, the bidding agent takes the bidding coefficient as the decision variable, and forms a bidding strategy according to the collusion object in the first stage and the market clearing results of the previous round when the collusion is successful, and bids are made according to the bidding strategy.
[0043] According to formula (1), the power generator profit model is constructed.
[0044]
[0045] stk min ≤k i ≤k max
[0046] F≤F0
[0047] minf(1) In formula (1), f1 is the bidding profit of market entity g; k max and k min represents the upper and lower limits of various bidding coefficients; F and F0 represent various operating constraints and the boundaries of the constraints; f is the objective function of the market operator to minimize the power generation cost, a r is the first-order term of the marginal power generation cost of the new energy generator r, p r To contribute to the new energy power generators that need to make decisions, g is the power generation of the traditional power generator g, a g , b g are the quadratic and linear coefficients of the marginal generation cost of the traditional power generator g, and λ represents the day-ahead electricity market clearing price.
[0048] According to formula (2), the objective function of the generator clearing profit model is constructed:
[0049]
[0050] In formula (2), f2 represents the social welfare maximization of day-ahead power market transactions, a g , b g are the quadratic and linear coefficients of the marginal power generation cost of the traditional power generator g; p g is the power generation of the traditional power generator g; k g is the quotation coefficient of the traditional power generator g; a l and b l are the quotes of load l in the day-ahead power market; k r is the price quoted by the new energy power generator r in the day-ahead power market; p r To contribute to the new energy power generators that need to make decisions, l is the reported amount of load l in the electricity market, a r is the first-order term of the marginal power generation cost of the new energy generator r, For the collection of thermal power generators, is the set of loads, A collection of new energy power generators.
[0051] The constraints of the generator clearing revenue model include:
[0052] Power balance constraints:
[0053]
[0054] λ represents the day-ahead electricity market clearing price, is the set of loads, A collection of new energy power generators. A collection of thermal power generators;
[0055] Minimum and maximum output constraints:
[0056]
[0057]
[0058] k g ,a l ,b l ,p g ,p r ,p l ≥0
[0059] In the formula, capping the output of traditional power generators; capping the output of new energy generators; k is the upper limit of load reporting g is the quotation coefficient of the traditional power generator g; a l and b l are the quotes of load l in the day-ahead power market; k r is the price quoted by the new energy power generator r in the day-ahead power market; p g is the power generation of the traditional power generator g; p r To contribute to the new energy power generators that need to make decisions, l is the reported amount of load l in the electricity market, a r is the first-order term of the marginal power generation cost of the new energy generator r, F i min is the lower capacity limit of line i, F i max is the upper capacity limit of line i.
[0060] The first stage of the bidding agent is based on the D3QN algorithm and includes:
[0061] D3QN algorithm network building module, used to build a deep Q network consisting of Current Q Network and Target Q Network and an experience replay library;
[0062] The action conversion module selects discrete actions based on the Q network using the ε-greedy strategy and converts the actions into the output of the first stage, i.e., the target vector;
[0063] The target vector conversion module converts the target vector output from the first stage into a specific comprehensive utility function as a reward function as the state of the generator clearing revenue model;
[0064] The algorithm training module trains the Q network through historical data obtained through interaction to improve the performance of the algorithm.
[0065] The action conversion module selects discrete actions. The specific selection method is as follows: in the early stage of training, the agent randomly selects actions to explore the feasible action space. When the number of training times increases to a certain value, it is reduced as the number of training times increases. In each training process, a random number is generated. If the random number is greater than this number, the greedy method is used to select the action, otherwise the action is selected randomly.
[0066] The second stage of the bidding agent is based on the SAC algorithm and includes:
[0067] SAC algorithm network construction module, which builds the Actor network for outputting continuous actions and the Critic network for evaluating the output actions of the Actor network;
[0068] The bidding action conversion module selects collusion actions using the ε-greedy strategy through the Actor network and converts the actions into the output of the second stage, namely the node electricity price, clearing electricity and bidding coefficient plan;
[0069] Algorithm training module, the historical data obtained through interaction is used to train the Actor and Critic networks.
[0070] The bidding action conversion module selects collusion actions as follows: random actions are used at the beginning, but when the number of training times increases to the set value, greedy actions are selected.
[0071] The bidding agent in this embodiment is specifically:
[0072] 1) State input of the D3QN algorithm
[0073] The first-stage algorithm needs to obtain the current power price in the zone when making decisions on the free association of power generators. Consider the market clearing results of the previous round and the collusion vector dK given by the first-stage algorithm of this round i , forming a comprehensive input state, as follows:
[0074] S d =[λ t ,p g ,s dk ]
[0075] In the formula, λ t is the node marginal electricity price; p g To clear the electricity;dk is the collusion result, which is calculated based on the collusion vector given by generator g in the previous round.
[0076] 2) D3QN algorithm action output
[0077] The output action of the algorithm is the collusion vector dK of the generator i The algorithm estimates the benefits of collusion among different power generators based on the network, and then selects the power generator with the highest benefit as the collusion target based on the ε-greedy strategy, and then transforms the selection result into a collusion vector, namely:
[0078] a D =dK i
[0079] In the formula, a D is the decision action output by the D3QN algorithm; dK i is the collusion vector of the generator.
[0080] 3) State input of SAC algorithm
[0081] The clearing results of the generators in the previous round and the collusion vector dK given in this round are i , forming an input state, namely:
[0082] S d =[λ t ,p g ,s dk ,dK i ]
[0083] 4) Action selection of SAC algorithm
[0084] The action of the SAC algorithm is the bid coefficient k of the generator i , the specific form is as follows:
[0085] a s =k i
[0086] 5) Reward function of SAC algorithm
[0087] R=R(k i ,dK i )+εR -i
[0088] In the formula, it is necessary to consider whether the power generator collusion is successful, and then obtain the reward function.
[0089] Embodiment 2
[0090] This embodiment provides an electronic device, including a memory, a processor, and a computer program, wherein the computer program is stored in the memory and is configured to be executed by the processor to implement the power supplier quotation method based on the bidding intelligent agent.
[0091] Embodiment 3
[0092] This embodiment provides a computer-readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the power generator quotation method based on the bidding intelligent agent.
[0093] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. A bidding agent-based power generation bidding method, characterized in that include: Establish a bidding model for free association of power producers, including: a revenue model for power producers that coordinate energy status across days and a clearing revenue model for power producers that perform day-ahead self-dispatch; Based on the two-stage deep reinforcement learning algorithm D3QN-SAC, a bidding agent is constructed to simulate the bidding and clearing process of free association of power producers. In the first stage, the bidding agent takes the highest expected profit of power producers as the goal based on the bidding model of free association of power producers, and takes the daily energy change trend of energy storage as the decision variable to select the collusion object; in the second stage, the bidding agent takes the bidding coefficient as the decision variable, and forms a bidding strategy according to the collusion object in the first stage and the market clearing results of the previous round when the collusion is successful, and bids are made according to the bidding strategy.
2. The power generation company quotation method based on bidding agent according to claim 1 is characterized in that According to formula (1), the power generator profit model is constructed. s.t.k min ≤k i ≤k max F≤F0 minf(1) In formula (1), f1 is the bidding profit of market entity g; k max and k min represents the upper and lower limits of various bidding coefficients; F and F0 represent various operating constraints and the boundaries of the constraints; f is the objective function of the market operator to minimize the power generation cost, a r is the first-order term of the marginal power generation cost of the new energy generator r, p r To contribute to the new energy power generators that need to make decisions, g is the power generation of the traditional power generator g, a g , b g are the quadratic and linear coefficients of the marginal generation cost of the traditional power generator g, and λ represents the day-ahead electricity market clearing price.
3. The power generation company quotation method based on bidding agent according to claim 1 is characterized in that According to formula (2), the objective function of the generator clearing profit model is constructed: In formula (2), f2 represents the social welfare maximization of day-ahead power market transactions, a g , b g are the quadratic and linear coefficients of the marginal power generation cost of the traditional power generator g; p g is the power generation of the traditional power generator g; k g is the quotation coefficient of the traditional power generator g; a l and b l are the quotes of load l in the day-ahead power market; k r is the price quoted by the new energy power generator r in the day-ahead power market; r To contribute to the new energy power generators that need to make decisions, l is the reported amount of load l in the electricity market, a r is the first-order term of the marginal power generation cost of the new energy generator r, For the collection of thermal power generators, is the set of loads, A collection of new energy power generators.
4. The power generation company quotation method based on bidding agent according to claim 3 is characterized in that The constraints of the generator clearing revenue model include: Power balance constraints: λ represents the day-ahead electricity market clearing price, is the set of loads, A collection of new energy power generators. A collection of thermal power generators; Minimum and maximum output constraints: k g ,a l ,b l ,p g ,p r ,p l ≥0 In the formula, capping the output of traditional power generators; capping the output of renewable energy generators; k is the upper limit of the load reporting amount; g is the quotation coefficient of the traditional power generator g; a l and b l are the quotes of load l in the day-ahead power market; k r is the price quoted by the new energy power generator r in the day-ahead power market; g is the power generation of the traditional power generator g; p r To contribute to the new energy power generators that need to make decisions, l is the reported amount of load l in the electricity market, a r is the first-order term of the marginal power generation cost of the new energy generator r, F i min is the lower capacity limit of line i, F i max is the upper capacity limit of line i.
5. The power generation company quotation method based on bidding agent according to claim 1 is characterized in that The first stage of the bidding agent is based on the D3QN algorithm and includes: D3QN algorithm network building module, used to build a deep Q network consisting of Current Q Network and Target Q Network and an experience replay library; The action conversion module selects discrete actions based on the Q network using the ε-greedy strategy and converts the actions into the output of the first stage, i.e., the target vector; The target vector conversion module converts the target vector output from the first stage into a specific comprehensive utility function as a reward function as the state of the generator clearing revenue model; The algorithm training module trains the Q network through historical data obtained through interaction to improve the performance of the algorithm.
6. The power generation company quotation method based on bidding agent according to claim 5 is characterized in that The action conversion module selects discrete actions. The specific selection method is as follows: in the early stage of training, the agent randomly selects actions to explore the feasible action space. When the number of training times increases to a certain value, it is reduced as the number of training times increases. In each training process, a random number is generated. If the random number is greater than this number, the greedy method is used to select the action, otherwise the action is randomly selected.
7. The power generation company quotation method based on bidding agent according to claim 5 is characterized in that The second stage of the bidding agent is based on the SAC algorithm and includes: SAC algorithm network construction module, which builds the Actor network for outputting continuous actions and the Critic network for evaluating the output actions of the Actor network; The bidding action conversion module selects collusion actions using the ε-greedy strategy through the Actor network and converts the actions into the output of the second stage, namely the node electricity price, clearing electricity and bidding coefficient plan; Algorithm training module, the historical data obtained through interaction is used to train the Actor and Critic networks.
8. The power generation company quotation method based on bidding agent according to claim 7 is characterized in that The bidding action conversion module selects the collusion action specifically as follows: initially a random action is adopted, but when the number of training times increases to a set value, a greedy action is started to be selected.
9. An electronic device, characterized in that: It includes a memory, a processor and a computer program, wherein the computer program is stored in the memory and is configured to be executed by the processor to implement the power producer bidding method based on the bidding agent as described in any one of claims 1-8.
10. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and the computer program is executed by a processor to implement the power generator quotation method based on a bidding intelligent agent as described in any one of claims 1-8.