A virtual power plant collaborative optimization control method and device based on multi-agent game reinforcement learning

Abstract

The present application relates to power distribution network engineering application technical field, specifically provides a kind of virtual power plant collaborative optimization control method and device based on multi-agent game reinforcement learning, comprising: based on the virtual power plant of each virtual power plant Charlip value solution pre-constructed virtual power plant optimization model, and determine virtual power plant alliance in each virtual power plant;The pre-constructed virtual power plant aggregator optimization model corresponding to the virtual power plant alliance is solved using multi-agent reinforcement learning method, and the action strategy of each virtual power plant in virtual power plant alliance participating demand response and day-ahead peak shaving is obtained;Wherein, the action strategy includes: output plan and electricity price.This scheme, on the one hand, stimulates virtual power plant aggregator to provide high-quality service to help virtual power plant to enter the market smoothly, on the other hand, improves the enthusiasm of virtual power plant to participate in system regulation, expands its profit space, and the business model also explores the adjustable capacity of load side at a lower cost, provides protection for the safe, stable and economic operation of power system, realizes the win-win of all parties.

Description

Technical Field

[0001] This invention relates to the field of power distribution network engineering application technology, specifically to a virtual power plant collaborative optimization control method and device based on multi-agent game reinforcement learning. Background Technology With the development and improvement of the spot market and ancillary services market, the development of virtual power plants is in a transitional stage from invitation-based to market-based. Currently, multiple new market players are forming alliances to participate in the market in the form of VPPs, which can improve their competitiveness and economic benefits.

[0002] However, due to the limitation of a single profit model, virtual power plants are difficult to operate continuously in a normalized manner, and their annual utilization hours are low. Therefore, it is urgent to guide virtual power plants to actively participate in system regulation through market-based means. It is necessary to conduct research on the profit model and control methods of virtual power plants from different time scales and different trading products. Summary of the Invention

[0003] To overcome the above-mentioned shortcomings, this invention proposes a virtual power plant collaborative optimization control method and device based on multi-agent game reinforcement learning.

[0004] Firstly, a collaborative optimization control method for virtual power plants based on multi-agent game-theoretic reinforcement learning is provided, wherein the collaborative optimization control method for virtual power plants based on multi-agent game-theoretic reinforcement learning includes: The pre-built virtual power plant optimization model is solved based on the Sharpe values ​​of each virtual power plant, and a virtual power plant alliance is determined among the virtual power plants. The pre-built virtual power plant aggregator optimization model corresponding to the virtual power plant alliance is solved by using multi-agent reinforcement learning method, so as to obtain the action strategies of each virtual power plant in the virtual power plant alliance for participating in demand response and day-ahead peak shaving. The collaborative optimization control of each virtual power plant in the virtual power plant alliance is carried out by utilizing the action strategies of each virtual power plant in demand response and day-ahead peak shaving. The action strategy includes output plan and electricity price. During the execution of the multi-agent reinforcement learning method, the state space includes the observation information of each virtual power plant.

[0005] Preferably, the pre-constructed virtual power plant optimization model includes: a first objective function aimed at minimizing the total operating cost of the virtual power plant and its corresponding first constraint conditions.

[0006] Furthermore, the process of solving the pre-built virtual power plant optimization model based on the Sharpe ratio of each virtual power plant, and determining the virtual power plant alliance among the virtual power plants, includes: The Q-value method is used to solve the pre-built virtual power plant optimization model, wherein the Q-value is the Sharpe ratio of the virtual power plant and the reward function is the first objective function.

[0007] Furthermore, the Sharpe ratio of the virtual power plant is as follows:

[0008] In the above formula, Let be the Sharpe value of the i-th virtual power plant. For the virtual power plant alliance, The number of virtual power plants. For the Virtual Power Plant Alliance After adding the i-th virtual power plant, the virtual power plant alliance set is formed. The revenue derived from demand response and day-ahead peak shaving. For the Virtual Power Plant Alliance The revenue derived from completing demand response and day-ahead peak shaving.

[0009] Furthermore, the virtual power plant alliance collection The revenue derived from completing demand response and day-ahead peak shaving is as follows:

[0010] In the above formula, The revenue sharing ratio for virtual power plant aggregators participating in demand response. For the revenue generated by Japanese virtual power plant aggregators participating in demand response, The revenue sharing ratio for virtual power plant aggregators participating in the peak-shaving market. This is for the revenue generated by Japanese virtual power plant aggregators participating in the peak-shaving market.

[0011] Furthermore, the first objective function is as follows:

[0012] In the above formula, The value of the first objective function. These are, respectively, the equipment operation and maintenance cost of the i-th virtual power plant, the electricity purchase cost paid to the virtual power plant aggregator, the electricity sales cost, the allocated operating cost of the virtual power plant aggregator, and the Sharpe value.

[0013] Furthermore, the equipment operation and maintenance costs of the i-th virtual power plant, the electricity purchase costs paid to the virtual power plant aggregator, the electricity sales costs, and the allocated operating costs of the virtual power plant aggregator are as follows:

[0014]

[0015]

[0016]

[0017] In the above formula, For the scheduling period, Let be the number of distributed power sources in the i-th virtual power plant. The installation cost of the j-th distributed power unit in the i-th virtual power plant is... Let the rated power of the j-th distributed power source in the i-th virtual power plant be . This refers to the annual utilization hours of the generating unit. For interest, For the expected service life of the unit, Let be the rated output of the j-th distributed power source in the i-th virtual power plant at time t. Let be the operating coefficient of the j-th distributed power source in the i-th virtual power plant. For time intervals, For the i-th virtual power plant at time t, the power purchase and sale status is as follows: The power purchase and sale plan submitted by the i-th virtual power plant at time t. The day-ahead electricity purchase price published by the virtual power plant aggregator to the virtual power plant at time t. To provide real-time output for the i-th virtual power plant at time t, The verification status of the i-th virtual power plant at time t. This refers to the daily power purchase price ceiling issued by the virtual power plant aggregator to the virtual power plants. The day-ahead electricity price published by the virtual power plant aggregator to the virtual power plant at time t. This refers to the lower limit of the day-ahead electricity sales price issued by virtual power plant aggregators to virtual power plants. The unit price for internet access fees. The weighted average of deviation charges paid by virtual power plant aggregators. This represents the actual daily electricity purchase and sale volume of the virtual power plant.

[0018] Furthermore, the first constraint condition is as follows:

[0019]

[0020] In the above formula, , Let $\frac{i}{j}$ be the minimum and maximum rated output values ​​of the $i$-th virtual power plant and the $j$-th distributed power source, respectively. , These are the minimum and maximum rated output values ​​of the j-th controllable load in the i-th virtual power plant, respectively. The rated output of the j-th controllable load of the i-th virtual power plant.

[0021] Furthermore, in the process of solving the pre-constructed virtual power plant optimization model using the Q-value method, the constraints satisfied after merging the two virtual power plant alliance sets include:

[0022]

[0023] In the above formula, For the Virtual Power Plant Alliance The revenue derived from demand response and day-ahead peak shaving. For the Virtual Power Plant Alliance The revenue derived from demand response and day-ahead peak shaving. For the Virtual Power Plant Alliance With the Virtual Power Plant Alliance The merged virtual power plant consortium will share the revenue from demand response and day-ahead peak shaving. For the Virtual Power Plant Alliance With the Virtual Power Plant Alliance The Sharpe ratio of the i-th virtual power plant in the merged virtual power plant consortium. For the Virtual Power Plant Alliance The Sharpe ratio of the i-th virtual power plant. For the Virtual Power Plant Alliance The Sharpe ratio of the i-th virtual power plant. For the Virtual Power Plant Alliance With the Virtual Power Plant Alliance The Sharpe ratio of the j-th virtual power plant in the merged virtual power plant consortium. For the Virtual Power Plant Alliance The Sharpe value of the j-th virtual power plant. For the Virtual Power Plant Alliance The Sharpe value of the j-th virtual power plant.

[0024] Furthermore, the pre-constructed virtual power plant aggregator optimization model includes a second objective function aimed at maximizing the virtual power plant aggregator's own revenue, and its corresponding second constraint conditions.

[0025] Furthermore, the second objective function is as follows:

[0026] In the above formula, For the revenue of virtual power plant aggregators on the settlement date, The revenue earned by virtual power plant aggregators from electricity trading on the settlement date. The revenue generated by virtual power plant aggregators participating in demand response during the settlement period. This refers to the revenue generated by virtual power plant aggregators participating in day-ahead peak shaving during the settlement period.

[0027] Furthermore, the revenue obtained by the virtual power plant aggregator from electricity trading is as follows:

[0028] In the above formula, , , These are the electricity purchase fees, electricity sales fees, and service fees charged by the virtual power plant aggregator to the virtual power plant. , These are the electricity purchase fees and market operation allocation fees paid by virtual power plant aggregators to the electricity spot market.

[0029] Furthermore, the electricity purchase fees, electricity sales fees, and service fees charged by the virtual power plant aggregator to the virtual power plant are as follows:

[0030]

[0031]

[0032] in: ,

[0033]

[0034] In the above formula, Let be the power purchase and sale plan value of the i-th virtual power plant at time t. , The weighting factor for the weighted value of the deviation electricity charges paid by virtual power plant aggregators. Settle deviation fees for virtual power plant aggregators in real time.

[0035] Furthermore, the electricity purchase fees and market operation sharing fees paid by the virtual power plant aggregator to the electricity spot market are as follows:

[0036]

[0037] In the above formula, The verification value for the plans recently submitted by virtual power plant aggregators. The current market-wide unified electricity price. This is the market operation cost allocation coefficient for virtual power plant aggregators, where:

[0038]

[0039] In the above formula, To provide real-time market clearing prices.

[0040] Furthermore, the benefits for virtual power plant aggregators participating in demand response during the settlement period are as follows:

[0041] in:

[0042]

[0043] In the above formula, Commission rate based on demand response , The revenue generated is generated by virtual power plant aggregators organizing their virtual power plants to participate in peak shaving and valley filling responses. for The subsidy coefficient for virtual power plant aggregators participating in peak shaving response. for The status of virtual power plant aggregators' bids for peak shaving response. express Virtual power plant aggregators are instructed to participate in peak shaving response. express The virtual power plant aggregator did not participate in the peak shaving response instructions. for Virtual power plant aggregators are constantly reducing their planned output values. for Real-time progress of virtual power plant aggregators' participation in peak shaving response completion. express Virtual power plant aggregators are reducing their output. The virtual power plant aggregator indicated that it had not reduced its output at any given time. The subsidy unit price for virtual power plant aggregators participating in peak shaving response. , The start and end times of virtual power plant aggregators' participation in peak shaving response bidding. for The subsidy coefficient for virtual power plant aggregators participating in valley-filling response. for The status of virtual power plant aggregators participating in valley filling response bidding. express Virtual power plant aggregators have instructions to participate in valley filling responses. express The virtual power plant aggregator did not participate in the valley filling response instructions. for Virtual power plant aggregators are constantly adjusting their planned output values. for Real-time progress of virtual power plant aggregator participation in valley filling response. express Virtual power plant aggregators are reducing their output. The virtual power plant aggregator indicated that it had not reduced its output at any given time. The unit price for subsidies for virtual power plant aggregators participating in valley filling responses. , The start and end times of virtual power plant aggregators' participation in valley filling response bidding.

[0044] Furthermore, the aforementioned The completion status of peak shaving response by virtual power plant aggregators is as follows:

[0045] The The subsidy coefficients for virtual power plant aggregators participating in peak shaving response are as follows:

[0046] The The completion status of virtual power plant aggregators' participation in valley filling response is as follows:

[0047] The The subsidy coefficients for virtual power plant aggregators participating in valley filling responses are as follows:

[0048] In the above formula, for Real-time virtual power plant aggregator baseline load value, for Real-time virtual power plant aggregator baseline load value.

[0049] Furthermore, the revenue of virtual power plant aggregators participating in day-ahead peak shaving during the settlement period is as follows:

[0050] in:

[0051]

[0052]

[0053]

[0054] In the above formula, To enable virtual power plant aggregators to participate in the positive peak-shaving market and generate revenue. To enable virtual power plant aggregators to participate in the negative peak-shaving market and generate revenue. This is the percentage of revenue collected for peak shaving activities in advance. This indicates the winning bid status of virtual power plant aggregators participating in positive peak shaving. This indicates that virtual power plant aggregators have received messages regarding participation in positive peak shaving. The virtual power plant aggregator indicated that it had not received any notification regarding participation in positive peak shaving. This shows the completion status of virtual power plant's positive peak shaving during peak shaving periods. This indicates that the virtual power plant aggregator's electricity consumption increased during peak-shaving periods but did not exceed the aggregator's baseline load maximum. These are the start and end times of the peak-shaving period. This indicates that the virtual power plant aggregator failed to achieve peak shaving during the peak shaving period. The subsidy unit price for virtual power plant aggregators participating in positive peak shaving. for Real-time virtual power plant aggregator baseline load value, This refers to the winning bid status of virtual power plant aggregators participating in negative peak shaving. This indicates that virtual power plant aggregators have received messages regarding participation in negative peak shaving. The virtual power plant aggregator indicated that it had not received any notification to participate in negative peak shaving. This indicates the completion status of negative peak shaving by virtual power plants during peak shaving periods. This indicates that virtual power plant aggregators consume less electricity during peak-shaving periods. This indicates that the virtual power plant aggregator failed to achieve negative peak shaving during the peak shaving period. The subsidy unit price for virtual power plant aggregators participating in negative peak shaving. To determine the threshold for negative peak modulation, The threshold for determining whether positive peak modulation is valid.

[0055] Furthermore, the second constraint is as follows:

[0056]

[0057]

[0058]

[0059]

[0060]

[0061] ,

[0062]

[0063] ,

[0064]

[0065] In the above formula, The continuous response time of the entities participating in peak shaving demand response. The continuous response time of the objects participating in the valley filling demand response. The minimum continuous response time for objects participating in peak shaving demand response. for Real-time progress of virtual power plant aggregators' participation in peak shaving response completion. for Real-time progress of virtual power plant aggregators' participation in peak shaving response completion. The minimum continuous response time for objects participating in valley filling demand response. for Real-time progress of virtual power plant aggregator participation in valley filling response. for Real-time progress of virtual power plant aggregator participation in valley filling response. The number of times a virtual power plant aggregator participates in demand response per day. The daily limit for participating in demand response is set. , These are the upper limits of the subsidy unit price for day-ahead peak shaving and valley filling responses in demand response. , These are the upper limits of the subsidy unit price for positive peak shaving and negative peak shaving, respectively. This refers to the subsidy unit price for day-ahead valley filling response in the demand response. This refers to the unit price of subsidies for day-ahead peak shaving responses in demand response. for Real-time progress of virtual power plant aggregators' participation in peak shaving response completion. for Real-time progress of virtual power plant aggregators' participation in valley filling response.

[0066] Secondly, a virtual power plant collaborative optimization control device based on multi-agent game-theoretic reinforcement learning is provided, the virtual power plant collaborative optimization control device based on multi-agent game-theoretic reinforcement learning includes: The first analysis module is used to solve the pre-built virtual power plant optimization model based on the Sharpe value of each virtual power plant, and to determine the virtual power plant alliance among each virtual power plant. The second analysis module is used to solve the pre-built virtual power plant aggregator optimization model corresponding to the virtual power plant alliance using multi-agent reinforcement learning method, so as to obtain the action strategies of each virtual power plant in the virtual power plant alliance for participating in demand response and day-ahead peak shaving. The third analysis module is used to perform collaborative optimization control on each virtual power plant in the virtual power plant alliance by utilizing the action strategies of each virtual power plant in the virtual power plant alliance in demand response and day-ahead peak shaving. The action strategy includes output plan and electricity price. During the execution of the multi-agent reinforcement learning method, the state space includes the observation information of each virtual power plant.

[0067] Thirdly, a computer device is provided, comprising: one or more processors; The processor is used to execute one or more programs; When the one or more programs are executed by the one or more processors, the virtual power plant collaborative optimization control method based on multi-agent game reinforcement learning is implemented.

[0068] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, wherein when the computer program is executed, the virtual power plant collaborative optimization control method based on multi-agent game reinforcement learning is implemented.

[0069] The above-described technical solutions of the present invention have at least one or more of the following beneficial effects: This invention provides a method and apparatus for collaborative optimization control of virtual power plants based on multi-agent game-theoretic reinforcement learning, comprising: solving a pre-constructed virtual power plant optimization model based on the Sharpe ratio of each virtual power plant, and determining a virtual power plant alliance among the virtual power plants; using multi-agent reinforcement learning to solve a pre-constructed virtual power plant aggregator optimization model corresponding to the virtual power plant alliance, obtaining the action strategies of each virtual power plant in the virtual power plant alliance for participating in demand response and day-ahead peak shaving; and using the action strategies of each virtual power plant in the virtual power plant alliance for participating in demand response and day-ahead peak shaving to perform collaborative optimization control of each virtual power plant in the virtual power plant alliance; wherein, the action strategies include: output plan and electricity price, and during the execution of the multi-agent reinforcement learning method, the state space includes the observation information of each virtual power plant. The technical solution provided by this invention, on the one hand, motivates virtual power plant aggregators to provide high-quality services to help virtual power plants enter the market smoothly, and on the other hand, enhances the enthusiasm of virtual power plants to participate in system regulation and expand their profit margins. At the same time, this business model also taps into the considerable adjustability of the load side at a lower cost, providing a guarantee for the safe, stable and economical operation of the power system and achieving a win-win situation for all parties. Attached Figure Description

[0070] Figure 1 This is a schematic diagram of the main steps of the virtual power plant collaborative optimization control method based on multi-agent game reinforcement learning in an embodiment of the present invention. Detailed Implementation

[0071] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0072] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0073] Example 1 See appendix Figure 1 , Figure 1 This is a schematic flowchart illustrating the main steps of a virtual power plant collaborative optimization control method based on multi-agent game-theoretic reinforcement learning, according to an embodiment of the present invention. Figure 1 As shown, the virtual power plant cooperative optimization control method based on multi-agent game reinforcement learning in this embodiment of the invention mainly includes the following steps: Step S101: Solve the pre-built virtual power plant optimization model based on the Sharpe values ​​of each virtual power plant, and determine the virtual power plant alliance among each virtual power plant; Step S102: Use multi-agent reinforcement learning to solve the pre-built virtual power plant aggregator optimization model corresponding to the virtual power plant alliance, and obtain the action strategies of each virtual power plant in the virtual power plant alliance for participating in demand response and day-ahead peak shaving. Step S103 utilizes the action strategies of each virtual power plant in the virtual power plant alliance to participate in demand response and day-ahead peak shaving to perform collaborative optimization control on each virtual power plant in the virtual power plant alliance; The action strategy includes output plan and electricity price. During the execution of the multi-agent reinforcement learning method, the state space includes the observation information of each virtual power plant.

[0074] In this embodiment, the observation information of the virtual power plant consists of the winning bid amount and winning bid price of each component of the virtual power plant and its operating status.

[0075] In this embodiment, the pre-constructed virtual power plant optimization model includes: a first objective function aimed at minimizing the total operating cost of the virtual power plant and its corresponding first constraint conditions.

[0076] In one implementation, the step of solving a pre-built virtual power plant optimization model based on the Sharpe ratio of each virtual power plant, and determining a virtual power plant alliance among the virtual power plants, includes: The Q-value method is used to solve the pre-built virtual power plant optimization model, wherein the Q-value is the Sharpe ratio of the virtual power plant and the reward function is the first objective function.

[0077] In one implementation, the Sharpe ratio of the virtual power plant is as follows:

[0078] In the above formula, Let be the Sharpe value of the i-th virtual power plant. For the virtual power plant alliance, The number of virtual power plants. For the Virtual Power Plant Alliance After adding the i-th virtual power plant, the virtual power plant alliance set is formed. The revenue derived from demand response and day-ahead peak shaving. For the Virtual Power Plant Alliance The revenue derived from completing demand response and day-ahead peak shaving.

[0079] In one implementation, the virtual power plant alliance collection The revenue derived from completing demand response and day-ahead peak shaving is as follows:

[0080] In the above formula, The revenue sharing ratio for virtual power plant aggregators participating in demand response. For the revenue generated by Japanese virtual power plant aggregators participating in demand response, The revenue sharing ratio for virtual power plant aggregators participating in the peak-shaving market. This is for the revenue generated by Japanese virtual power plant aggregators participating in the peak-shaving market.

[0081] In one implementation, the first objective function is as follows:

[0082] In the above formula, The value of the first objective function. These are, respectively, the equipment operation and maintenance cost of the i-th virtual power plant, the electricity purchase cost paid to the virtual power plant aggregator, the electricity sales cost, the allocated operating cost of the virtual power plant aggregator, and the Sharpe value.

[0083] In one implementation, the equipment operation and maintenance cost of the i-th virtual power plant is as follows:

[0084] The electricity purchase costs paid to virtual power plant aggregators are as follows:

[0085] The electricity sales costs are as follows:

[0086] The allocated operating costs for virtual power plant aggregators are as follows:

[0087] In the above formula, For the scheduling period, Let be the number of distributed power sources in the i-th virtual power plant. The installation cost of the j-th distributed power unit in the i-th virtual power plant is... Let the rated power of the j-th distributed power source in the i-th virtual power plant be . This refers to the annual utilization hours of the generating unit. For interest, For the expected service life of the unit, Let be the rated output of the j-th distributed power source in the i-th virtual power plant at time t. Let be the operating coefficient of the j-th distributed power source in the i-th virtual power plant. For time intervals, For the i-th virtual power plant at time t, the power purchase and sale status is as follows: The power purchase and sale plan submitted by the i-th virtual power plant at time t. The day-ahead electricity purchase price published by the virtual power plant aggregator to the virtual power plant at time t. To provide real-time output for the i-th virtual power plant at time t, The verification status of the i-th virtual power plant at time t. This refers to the daily power purchase price ceiling issued by the virtual power plant aggregator to the virtual power plants. The day-ahead electricity price published by the virtual power plant aggregator to the virtual power plant at time t. This refers to the lower limit of the day-ahead electricity sales price issued by virtual power plant aggregators to virtual power plants. The unit price for internet access fees. The weighted average of deviation charges paid by virtual power plant aggregators. This represents the actual daily electricity purchase and sale volume of the virtual power plant.

[0088] In one implementation, the first constraint condition is as follows:

[0089]

[0090] In the above formula, , Let $\frac{i}{j}$ be the minimum and maximum rated output values ​​of the $i$-th virtual power plant and the $j$-th distributed power source, respectively. , These are the minimum and maximum rated output values ​​of the j-th controllable load in the i-th virtual power plant, respectively. The rated output of the j-th controllable load of the i-th virtual power plant.

[0091] In one implementation, during the process of solving the pre-built virtual power plant optimization model using the Q-value method, the constraints satisfied after merging the two virtual power plant alliance sets include:

[0092]

[0093] In the above formula, For the Virtual Power Plant Alliance The revenue derived from demand response and day-ahead peak shaving. For the Virtual Power Plant Alliance The revenue derived from demand response and day-ahead peak shaving. For the Virtual Power Plant Alliance With the Virtual Power Plant Alliance The merged virtual power plant consortium will share the revenue from demand response and day-ahead peak shaving. For the Virtual Power Plant Alliance With the Virtual Power Plant Alliance The Sharpe ratio of the i-th virtual power plant in the merged virtual power plant consortium. For the Virtual Power Plant Alliance The Sharpe ratio of the i-th virtual power plant. For the Virtual Power Plant Alliance The Sharpe ratio of the i-th virtual power plant. For the Virtual Power Plant Alliance With the Virtual Power Plant Alliance The Sharpe ratio of the j-th virtual power plant in the merged virtual power plant consortium. For the Virtual Power Plant Alliance The Sharpe value of the j-th virtual power plant. For the Virtual Power Plant Alliance The Sharpe value of the j-th virtual power plant.

[0094] In one implementation, the pre-built virtual power plant aggregator optimization model includes a second objective function aimed at maximizing the revenue of the virtual power plant aggregator and its corresponding second constraint conditions.

[0095] In one implementation, the second objective function is as follows:

[0096] In the above formula, For the revenue of virtual power plant aggregators on the settlement date, The revenue earned by virtual power plant aggregators from electricity trading on the settlement date. The revenue generated by virtual power plant aggregators participating in demand response during the settlement period. This refers to the revenue generated by virtual power plant aggregators participating in day-ahead peak shaving during the settlement period.

[0097] In one implementation, the revenue obtained by the virtual power plant aggregator from electricity trading is as follows:

[0098] In the above formula, , , These are the electricity purchase fees, electricity sales fees, and service fees charged by the virtual power plant aggregator to the virtual power plant. , These are the electricity purchase fees and market operation allocation fees paid by virtual power plant aggregators to the electricity spot market.

[0099] In one implementation, the virtual power plant aggregator charges the virtual power plant the following fees for electricity purchase, electricity sales, and services:

[0100]

[0101]

[0102] in: ,

[0103]

[0104] In the above formula, Let be the power purchase and sale plan value of the i-th virtual power plant at time t. , The weighting factor for the weighted value of the deviation electricity charges paid by virtual power plant aggregators. Settle deviation fees for virtual power plant aggregators in real time.

[0105] In one implementation, the virtual power plant aggregator pays the following fees to the electricity spot market for electricity purchases and market operation allocations:

[0106]

[0107] In the above formula, The verification value for the plans recently submitted by virtual power plant aggregators. The current market-wide unified electricity price. This is the market operation cost allocation coefficient for virtual power plant aggregators, where:

[0108]

[0109] In the above formula, To provide real-time market clearing prices.

[0110] In one implementation, the benefits of virtual power plant aggregator organizations participating in demand response during the settlement period are as follows:

[0111] in:

[0112]

[0113] In the above formula, Commission rate based on demand response , The revenue generated is generated by virtual power plant aggregators organizing their virtual power plants to participate in peak shaving and valley filling responses. for The subsidy coefficient for virtual power plant aggregators participating in peak shaving response. for The status of virtual power plant aggregators' bids for peak shaving response. express Virtual power plant aggregators are instructed to participate in peak shaving response. express The virtual power plant aggregator did not participate in the peak shaving response instructions. for Virtual power plant aggregators are constantly reducing their planned output values. for Real-time progress of virtual power plant aggregators' participation in peak shaving response completion. express Virtual power plant aggregators are reducing their output. The virtual power plant aggregator indicated that it had not reduced its output at any given time. The subsidy unit price for virtual power plant aggregators participating in peak shaving response. , The start and end times of virtual power plant aggregators' participation in peak shaving response bidding. for The subsidy coefficient for virtual power plant aggregators participating in valley-filling response. for The status of virtual power plant aggregators participating in valley filling response bidding. express Virtual power plant aggregators have instructions to participate in valley filling responses. express The virtual power plant aggregator did not participate in the valley filling response instructions. for Virtual power plant aggregators are constantly adjusting their planned output values. for Real-time progress of virtual power plant aggregator participation in valley filling response. express Virtual power plant aggregators are reducing their output. The virtual power plant aggregator indicated that it had not reduced its output at any given time. The unit price for subsidies for virtual power plant aggregators participating in valley filling responses. , The start and end times of virtual power plant aggregators' participation in valley filling response bidding.

[0114] In one implementation, the The completion status of peak shaving response by virtual power plant aggregators is as follows:

[0115] The The subsidy coefficients for virtual power plant aggregators participating in peak shaving response are as follows:

[0116] The The completion status of virtual power plant aggregators' participation in valley filling response is as follows:

[0117] The The subsidy coefficients for virtual power plant aggregators participating in valley filling responses are as follows:

[0118] In the above formula, for Real-time virtual power plant aggregator baseline load value, for Real-time virtual power plant aggregator baseline load value.

[0119] In one implementation, the revenue of the virtual power plant aggregator organization participating in day-ahead peak shaving during the settlement period is as follows:

[0120] in:

[0121]

[0122]

[0123]

[0124] In the above formula, To enable virtual power plant aggregators to participate in the positive peak-shaving market and generate revenue. To enable virtual power plant aggregators to participate in the negative peak-shaving market and generate revenue. This is the percentage of revenue collected for peak shaving activities in advance. This indicates the winning bid status of virtual power plant aggregators participating in positive peak shaving. This indicates that virtual power plant aggregators have received messages regarding participation in positive peak shaving. The virtual power plant aggregator indicated that it had not received any notification regarding participation in positive peak shaving. This shows the completion status of virtual power plant's positive peak shaving during peak shaving periods. This indicates that the virtual power plant aggregator's electricity consumption increased during peak-shaving periods but did not exceed the aggregator's baseline load maximum. These are the start and end times of the peak-shaving period. This indicates that the virtual power plant aggregator failed to achieve peak shaving during the peak shaving period. The subsidy unit price for virtual power plant aggregators participating in positive peak shaving. for Real-time virtual power plant aggregator baseline load value, This refers to the winning bid status of virtual power plant aggregators participating in negative peak shaving. This indicates that virtual power plant aggregators have received messages regarding participation in negative peak shaving. The virtual power plant aggregator indicated that it had not received any notification to participate in negative peak shaving. This indicates the completion status of negative peak shaving by virtual power plants during peak shaving periods. This indicates that virtual power plant aggregators consume less electricity during peak-shaving periods. This indicates that the virtual power plant aggregator failed to achieve negative peak shaving during the peak shaving period. The subsidy unit price for virtual power plant aggregators participating in negative peak shaving. To determine the threshold for negative peak modulation, The threshold for determining whether positive peak modulation is valid.

[0125] In one implementation, the second constraint condition is as follows:

[0126]

[0127]

[0128]

[0129]

[0130]

[0131] ,

[0132]

[0133] ,

[0134]

[0135] In the above formula, The continuous response time of the entities participating in peak shaving demand response. The continuous response time of the objects participating in the valley filling demand response. The minimum continuous response time for objects participating in peak shaving demand response. for Real-time progress of virtual power plant aggregators' participation in peak shaving response completion. for Real-time progress of virtual power plant aggregators' participation in peak shaving response completion. The minimum continuous response time for objects participating in valley filling demand response. for Real-time progress of virtual power plant aggregator participation in valley filling response. for Real-time progress of virtual power plant aggregator participation in valley filling response. The number of times a virtual power plant aggregator participates in demand response per day. The daily limit for participating in demand response is set. , These are the upper limits of the subsidy unit price for day-ahead peak shaving and valley filling responses in demand response. , These are the upper limits of the subsidy unit price for positive peak shaving and negative peak shaving, respectively. This refers to the subsidy unit price for day-ahead valley filling response in the demand response. This refers to the unit price of subsidies for day-ahead peak shaving responses in demand response. for Real-time progress of virtual power plant aggregators' participation in peak shaving response completion. for Real-time progress of virtual power plant aggregators' participation in valley filling response.

[0136] Example 2 Based on the same inventive concept, this invention also provides a virtual power plant collaborative optimization control device based on multi-agent game reinforcement learning, wherein the virtual power plant collaborative optimization control device based on multi-agent game reinforcement learning includes: The first analysis module is used to solve the pre-built virtual power plant optimization model based on the Sharpe value of each virtual power plant, and to determine the virtual power plant alliance among each virtual power plant. The second analysis module is used to solve the pre-built virtual power plant aggregator optimization model corresponding to the virtual power plant alliance using multi-agent reinforcement learning method, so as to obtain the action strategies of each virtual power plant in the virtual power plant alliance for participating in demand response and day-ahead peak shaving. The third analysis module is used to perform collaborative optimization control on each virtual power plant in the virtual power plant alliance by utilizing the action strategies of each virtual power plant in the virtual power plant alliance in demand response and day-ahead peak shaving. The action strategy includes output plan and electricity price. During the execution of the multi-agent reinforcement learning method, the state space includes the observation information of each virtual power plant.

[0137] Preferably, the pre-constructed virtual power plant optimization model includes: a first objective function aimed at minimizing the total operating cost of the virtual power plant and its corresponding first constraint conditions.

[0138] Furthermore, the process of solving the pre-built virtual power plant optimization model based on the Sharpe ratio of each virtual power plant, and determining the virtual power plant alliance among the virtual power plants, includes: The Q-value method is used to solve the pre-built virtual power plant optimization model, wherein the Q-value is the Sharpe ratio of the virtual power plant and the reward function is the first objective function.

[0139] Furthermore, the Sharpe ratio of the virtual power plant is as follows:

[0140] In the above formula, Let be the Sharpe value of the i-th virtual power plant. For the virtual power plant alliance, The number of virtual power plants. For the Virtual Power Plant Alliance After adding the i-th virtual power plant, the virtual power plant alliance set is formed. The revenue derived from demand response and day-ahead peak shaving. For the Virtual Power Plant Alliance The revenue derived from completing demand response and day-ahead peak shaving.

[0141] Furthermore, the virtual power plant alliance collection The revenue derived from completing demand response and day-ahead peak shaving is as follows:

[0142] In the above formula, The revenue sharing ratio for virtual power plant aggregators participating in demand response. For the revenue generated by Japanese virtual power plant aggregators participating in demand response, The revenue sharing ratio for virtual power plant aggregators participating in the peak-shaving market. This is for the revenue generated by Japanese virtual power plant aggregators participating in the peak-shaving market.

[0143] Furthermore, the first objective function is as follows:

[0144] In the above formula, The value of the first objective function. These are, respectively, the equipment operation and maintenance cost of the i-th virtual power plant, the electricity purchase cost paid to the virtual power plant aggregator, the electricity sales cost, the allocated operating cost of the virtual power plant aggregator, and the Sharpe value.

[0145] Furthermore, the equipment operation and maintenance costs of the i-th virtual power plant, the electricity purchase costs paid to the virtual power plant aggregator, the electricity sales costs, and the allocated operating costs of the virtual power plant aggregator are as follows:

[0146]

[0147]

[0148]

[0149] In the above formula, For the scheduling period, Let be the number of distributed power sources in the i-th virtual power plant. The installation cost of the j-th distributed power unit in the i-th virtual power plant is... Let the rated power of the j-th distributed power source in the i-th virtual power plant be . This refers to the annual utilization hours of the generating unit. For interest, For the expected service life of the unit, Let be the rated output of the j-th distributed power source in the i-th virtual power plant at time t. Let be the operating coefficient of the j-th distributed power source in the i-th virtual power plant. For time intervals, For the i-th virtual power plant at time t, the power purchase and sale status is as follows: The power purchase and sale plan submitted by the i-th virtual power plant at time t. The day-ahead electricity purchase price published by the virtual power plant aggregator to the virtual power plant at time t. To provide real-time output for the i-th virtual power plant at time t, The verification status of the i-th virtual power plant at time t. This refers to the daily power purchase price ceiling issued by the virtual power plant aggregator to the virtual power plants. The day-ahead electricity price published by the virtual power plant aggregator to the virtual power plant at time t. This refers to the lower limit of the day-ahead electricity sales price issued by virtual power plant aggregators to virtual power plants. The unit price for internet access fees. The weighted average of deviation charges paid by virtual power plant aggregators. This represents the actual daily electricity purchase and sale volume of the virtual power plant.

[0150] Furthermore, the first constraint condition is as follows:

[0151]

[0152] In the above formula, , Let $\frac{i}{j}$ be the minimum and maximum rated output values ​​of the $i$-th virtual power plant and the $j$-th distributed power source, respectively. , These are the minimum and maximum rated output values ​​of the j-th controllable load in the i-th virtual power plant, respectively. The rated output of the j-th controllable load of the i-th virtual power plant.

[0153] Furthermore, in the process of solving the pre-constructed virtual power plant optimization model using the Q-value method, the constraints satisfied after merging the two virtual power plant alliance sets include:

[0154]

[0155] In the above formula, For the Virtual Power Plant Alliance The revenue derived from demand response and day-ahead peak shaving. For the Virtual Power Plant Alliance The revenue derived from demand response and day-ahead peak shaving. For the Virtual Power Plant Alliance With the Virtual Power Plant Alliance The merged virtual power plant consortium will share the revenue from demand response and day-ahead peak shaving. For the Virtual Power Plant Alliance With the Virtual Power Plant Alliance The Sharpe ratio of the i-th virtual power plant in the merged virtual power plant consortium. For the Virtual Power Plant Alliance The Sharpe ratio of the i-th virtual power plant. For the Virtual Power Plant Alliance The Sharpe ratio of the i-th virtual power plant. For the Virtual Power Plant Alliance With the Virtual Power Plant Alliance The Sharpe ratio of the j-th virtual power plant in the merged virtual power plant consortium. For the Virtual Power Plant Alliance The Sharpe value of the j-th virtual power plant. For the Virtual Power Plant Alliance The Sharpe value of the j-th virtual power plant.

[0156] Furthermore, the pre-constructed virtual power plant aggregator optimization model includes a second objective function aimed at maximizing the virtual power plant aggregator's own revenue, and its corresponding second constraint conditions.

[0157] Furthermore, the second objective function is as follows:

[0158] In the above formula, For the revenue of virtual power plant aggregators on the settlement date, The revenue earned by virtual power plant aggregators from electricity trading on the settlement date. The revenue generated by virtual power plant aggregators participating in demand response during the settlement period. This refers to the revenue generated by virtual power plant aggregators participating in day-ahead peak shaving during the settlement period.

[0159] Furthermore, the revenue obtained by the virtual power plant aggregator from electricity trading is as follows:

[0160] In the above formula, , , These are the electricity purchase fees, electricity sales fees, and service fees charged by the virtual power plant aggregator to the virtual power plant. , These are the electricity purchase fees and market operation allocation fees paid by virtual power plant aggregators to the electricity spot market.

[0161] Furthermore, the electricity purchase fees charged by the virtual power plant aggregator to the virtual power plant are as follows:

[0162] Electricity sales costs are as follows:

[0163] The service fees are as follows:

[0164] in: ,

[0165]

[0166] In the above formula, Let be the power purchase and sale plan value of the i-th virtual power plant at time t. , The weighting factor for the weighted value of the deviation electricity charges paid by virtual power plant aggregators. Settle deviation fees for virtual power plant aggregators in real time.

[0167] Furthermore, the electricity purchase fees and market operation sharing fees paid by the virtual power plant aggregator to the electricity spot market are as follows:

[0168]

[0169] In the above formula, The verification value for the plans recently submitted by virtual power plant aggregators. The current market-wide unified electricity price. This is the market operation cost allocation coefficient for virtual power plant aggregators, where:

[0170]

[0171] In the above formula, To provide real-time market clearing prices.

[0172] Furthermore, the benefits for virtual power plant aggregators participating in demand response during the settlement period are as follows:

[0173] in:

[0174]

[0175] In the above formula, Commission rate based on demand response , The revenue generated is generated by virtual power plant aggregators organizing their virtual power plants to participate in peak shaving and valley filling responses. for The subsidy coefficient for virtual power plant aggregators participating in peak shaving response. for The status of virtual power plant aggregators' bids for peak shaving response. express Virtual power plant aggregators are instructed to participate in peak shaving response. express The virtual power plant aggregator did not participate in the peak shaving response instructions. for Virtual power plant aggregators are constantly reducing their planned output values. for Real-time progress of virtual power plant aggregators' participation in peak shaving response completion. express Virtual power plant aggregators are reducing their output. The virtual power plant aggregator indicated that it had not reduced its output at any given time. The subsidy unit price for virtual power plant aggregators participating in peak shaving response. , The start and end times of virtual power plant aggregators' participation in peak shaving response bidding. for The subsidy coefficient for virtual power plant aggregators participating in valley-filling response. for The status of virtual power plant aggregators participating in valley filling response bidding. express Virtual power plant aggregators have instructions to participate in valley filling responses. express The virtual power plant aggregator did not participate in the valley filling response instructions. for Virtual power plant aggregators are constantly adjusting their planned output values. for Real-time progress of virtual power plant aggregator participation in valley filling response. express Virtual power plant aggregators are reducing their output. The virtual power plant aggregator indicated that it had not reduced its output at any given time. The unit price for subsidies for virtual power plant aggregators participating in valley filling responses. , The start and end times of virtual power plant aggregators' participation in valley filling response bidding.

[0176] Furthermore, the aforementioned The completion status of peak shaving response by virtual power plant aggregators is as follows:

[0177] The The subsidy coefficients for virtual power plant aggregators participating in peak shaving response are as follows:

[0178] The The completion status of virtual power plant aggregators' participation in valley filling response is as follows:

[0179] The The subsidy coefficients for virtual power plant aggregators participating in valley filling responses are as follows:

[0180] In the above formula, for Real-time virtual power plant aggregator baseline load value, for Real-time virtual power plant aggregator baseline load value.

[0181] Furthermore, the revenue of virtual power plant aggregators participating in day-ahead peak shaving during the settlement period is as follows:

[0182] in:

[0183]

[0184]

[0185]

[0186] In the above formula, To enable virtual power plant aggregators to participate in the positive peak-shaving market and generate revenue. To enable virtual power plant aggregators to participate in the negative peak-shaving market and generate revenue. This is the percentage of revenue collected for peak shaving activities in advance. This indicates the winning bid status of virtual power plant aggregators participating in positive peak shaving. This indicates that virtual power plant aggregators have received messages regarding participation in positive peak shaving. The virtual power plant aggregator indicated that it had not received any notification regarding participation in positive peak shaving. This shows the completion status of virtual power plant's positive peak shaving during peak shaving periods. This indicates that the virtual power plant aggregator's electricity consumption increased during peak-shaving periods but did not exceed the aggregator's baseline load maximum. These are the start and end times of the peak-shaving period. This indicates that the virtual power plant aggregator failed to achieve peak shaving during the peak shaving period. The subsidy unit price for virtual power plant aggregators participating in positive peak shaving. for Real-time virtual power plant aggregator baseline load value, This refers to the winning bid status of virtual power plant aggregators participating in negative peak shaving. This indicates that virtual power plant aggregators have received messages regarding participation in negative peak shaving. The virtual power plant aggregator indicated that it had not received any notification to participate in negative peak shaving. This indicates the completion status of negative peak shaving by virtual power plants during peak shaving periods. This indicates that virtual power plant aggregators consume less electricity during peak-shaving periods. This indicates that the virtual power plant aggregator failed to achieve negative peak shaving during the peak shaving period. The subsidy unit price for virtual power plant aggregators participating in negative peak shaving. To determine the threshold for negative peak modulation, The threshold for determining whether positive peak modulation is valid.

[0187] Furthermore, the second constraint is as follows:

[0188]

[0189]

[0190]

[0191]

[0192]

[0193] ,

[0194]

[0195] ,

[0196]

[0197] In the above formula, The continuous response time of the entities participating in peak shaving demand response. The continuous response time of the objects participating in the valley filling demand response. The minimum continuous response time for objects participating in peak shaving demand response. for Real-time progress of virtual power plant aggregators' participation in peak shaving response completion. for Real-time progress of virtual power plant aggregators' participation in peak shaving response completion. The minimum continuous response time for objects participating in valley filling demand response. for Real-time progress of virtual power plant aggregator participation in valley filling response. for Real-time progress of virtual power plant aggregator participation in valley filling response. The number of times a virtual power plant aggregator participates in demand response per day. The daily limit for participating in demand response is set. , These are the upper limits of the subsidy unit price for day-ahead peak shaving and valley filling responses in demand response. , These are the upper limits of the subsidy unit price for positive peak shaving and negative peak shaving, respectively. This refers to the subsidy unit price for day-ahead valley filling response in the demand response. This refers to the unit price of subsidies for day-ahead peak shaving responses in demand response. for Real-time progress of virtual power plant aggregators' participation in peak shaving response completion. for Real-time progress of virtual power plant aggregators' participation in valley filling response.

[0198] Example 3 Based on the same inventive concept, this invention also provides a computer device, which includes a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to implement corresponding method flows or corresponding functions, thereby realizing the steps of the virtual power plant collaborative optimization control method based on multi-agent game reinforcement learning in the above embodiments.

[0199] Example 4 Based on the same inventive concept, this invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the steps of the virtual power plant cooperative optimization control method based on multi-agent game-theoretic reinforcement learning in the above embodiments.

[0200] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0201] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0202] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0203] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0204] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A virtual power plant collaborative optimization control method based on multi-agent game-theoretic reinforcement learning, characterized in that, The method includes: The pre-built virtual power plant optimization model is solved based on the Sharpe values ​​of each virtual power plant, and a virtual power plant alliance is determined among the virtual power plants. The pre-built virtual power plant aggregator optimization model corresponding to the virtual power plant alliance is solved by using multi-agent reinforcement learning method, so as to obtain the action strategies of each virtual power plant in the virtual power plant alliance for participating in demand response and day-ahead peak shaving. The collaborative optimization control of each virtual power plant in the virtual power plant alliance is carried out by utilizing the action strategies of each virtual power plant in demand response and day-ahead peak shaving. The action strategy includes output plan and electricity price. During the execution of the multi-agent reinforcement learning method, the state space includes the observation information of each virtual power plant.

2. The method as described in claim 1, characterized in that, The pre-built virtual power plant optimization model includes: a first objective function aimed at minimizing the total operating cost of the virtual power plant and its corresponding first constraint conditions.

3. The method as described in claim 2, characterized in that, The process of solving the pre-built virtual power plant optimization model based on the Sharpe ratio of each virtual power plant, and determining the virtual power plant alliance among the virtual power plants, includes: The Q-value method is used to solve the pre-built virtual power plant optimization model, wherein the Q-value is the Sharpe ratio of the virtual power plant and the reward function is the first objective function.

4. The method as described in claim 3, characterized in that, The Sharpe ratio of the virtual power plant is as follows: In the above formula, Let be the Sharpe value of the i-th virtual power plant. For the virtual power plant alliance, The number of virtual power plants. For the Virtual Power Plant Alliance After adding the i-th virtual power plant, the virtual power plant alliance set is formed. The revenue derived from demand response and day-ahead peak shaving. For the Virtual Power Plant Alliance The revenue derived from completing demand response and day-ahead peak shaving.

5. The method as described in claim 4, characterized in that, The revenue shared by the virtual power plant consortium in fulfilling demand response and day-ahead peak shaving is as follows: In the above formula, The revenue sharing ratio for virtual power plant aggregators participating in demand response. For the revenue generated by Japanese virtual power plant aggregators participating in demand response, The revenue sharing ratio for virtual power plant aggregators participating in the peak-shaving market. This is for the revenue generated by Japanese virtual power plant aggregators participating in the peak-shaving market.

6. The method as described in claim 4, characterized in that, The first objective function is as follows: In the above formula, The first objective function value, These are, respectively, the equipment operation and maintenance cost of the i-th virtual power plant, the electricity purchase cost paid to the virtual power plant aggregator, the electricity sales cost, the allocated operating cost of the virtual power plant aggregator, and the Sharpe value.

7. The method as described in claim 6, characterized in that, The equipment operation and maintenance costs, electricity purchase costs, electricity sales costs, and allocated operating costs of the virtual power plant aggregator for the i-th virtual power plant are as follows: In the above formula, For the scheduling period, Let be the number of distributed power sources in the i-th virtual power plant. The installation cost of the j-th distributed power unit in the i-th virtual power plant is... Let the rated power of the j-th distributed power source in the i-th virtual power plant be . This refers to the annual utilization hours of the generating unit. For interest, For the expected service life of the unit, Let be the rated output of the j-th distributed power source in the i-th virtual power plant at time t. Let be the operating coefficient of the j-th distributed power source in the i-th virtual power plant. For time intervals, For the i-th virtual power plant, the power purchase and sale status at time t is... The power purchase and sale plan submitted by the i-th virtual power plant at time t. The day-ahead electricity purchase price published by the virtual power plant aggregator to the virtual power plant at time t. To provide real-time output for the i-th virtual power plant at time t, The verification status of the i-th virtual power plant at time t. This refers to the daily power purchase price ceiling issued by the virtual power plant aggregator to the virtual power plants. The day-ahead electricity price published by the virtual power plant aggregator to the virtual power plant at time t. This refers to the lower limit of the day-ahead electricity sales price issued by virtual power plant aggregators to virtual power plants. The unit price for internet access fees. The weighted average of deviation charges paid by virtual power plant aggregators. This represents the actual daily electricity purchase and sale volume of the virtual power plant.

8. The method as described in claim 7, characterized in that, The first constraint is as follows: ， In the above formula, , Let $\frac{i}{j}$ be the minimum and maximum rated output values ​​of the $i$-th virtual power plant and the $j$-th distributed power source, respectively. , These are the minimum and maximum rated output values ​​of the j-th controllable load in the i-th virtual power plant, respectively. The rated output of the j-th controllable load of the i-th virtual power plant.

9. The method as described in claim 8, characterized in that, In the process of solving the pre-built virtual power plant optimization model using the Q-value method, the constraints satisfied after merging the two virtual power plant alliance sets include: In the above formula, For the Virtual Power Plant Alliance The revenue derived from demand response and day-ahead peak shaving. For the Virtual Power Plant Alliance The revenue derived from demand response and day-ahead peak shaving. For the Virtual Power Plant Alliance With the Virtual Power Plant Alliance The merged virtual power plant consortium will share the revenue from demand response and day-ahead peak shaving. For the Virtual Power Plant Alliance With the Virtual Power Plant Alliance The Sharpe ratio of the i-th virtual power plant in the merged virtual power plant consortium. For the Virtual Power Plant Alliance The Sharpe ratio of the i-th virtual power plant. For the Virtual Power Plant Alliance The Sharpe ratio of the i-th virtual power plant. For the Virtual Power Plant Alliance With the Virtual Power Plant Alliance The Sharpe ratio of the j-th virtual power plant in the merged virtual power plant consortium. For the Virtual Power Plant Alliance The Sharpe value of the j-th virtual power plant. For the Virtual Power Plant Alliance The Sharpe value of the j-th virtual power plant.

10. The method as described in claim 9, characterized in that, The pre-constructed virtual power plant aggregator optimization model includes a second objective function aimed at maximizing the virtual power plant aggregator's own revenue, and its corresponding second constraint conditions.

11. The method as described in claim 10, characterized in that, The second objective function is as follows: In the above formula, For the revenue of virtual power plant aggregators on the settlement date, The revenue earned by virtual power plant aggregators from electricity trading on the settlement date. The revenue generated by virtual power plant aggregators participating in demand response during the settlement period. This refers to the revenue generated by virtual power plant aggregators participating in day-ahead peak shaving during the settlement period.

12. The method as described in claim 11, characterized in that, The revenue obtained by the virtual power plant aggregator from electricity trading is as follows: In the above formula, , , These are the electricity purchase fees, electricity sales fees, and service fees charged by the virtual power plant aggregator to the virtual power plant. , These are the electricity purchase fees and market operation allocation fees paid by virtual power plant aggregators to the electricity spot market.

13. The method as described in claim 12, characterized in that, The electricity purchase fees, electricity sales fees, and service fees charged by the virtual power plant aggregator to the virtual power plant are as follows: in: ， In the above formula, Let be the power purchase and sale plan value of the i-th virtual power plant at time t. , The weighting factor for the weighted value of the deviation electricity charges paid by virtual power plant aggregators. Settle deviation fees for virtual power plant aggregators in real time.

14. The method as described in claim 13, characterized in that, The virtual power plant aggregators pay the following fees to the electricity spot market for electricity purchases and market operation allocations: In the above formula, The verification value for the plans recently submitted by virtual power plant aggregators. The current market-wide unified electricity price is used for settlement. This is the market operation cost allocation coefficient for virtual power plant aggregators, where: In the above formula, To provide real-time market clearing prices.

15. The method as described in claim 14, characterized in that, The benefits for virtual power plant aggregators participating in demand response during the settlement period are as follows: in: In the above formula, Commission rate based on demand response , The revenue generated is generated by virtual power plant aggregators organizing their virtual power plants to participate in peak shaving and valley filling responses. for The subsidy coefficient for virtual power plant aggregators participating in peak shaving response. for The status of virtual power plant aggregators' bids for peak shaving response. express Virtual power plant aggregators are instructed to participate in peak shaving response. express The virtual power plant aggregator did not participate in the peak shaving response instructions. for Virtual power plant aggregators are constantly reducing their planned output values. for Real-time progress of virtual power plant aggregators' participation in peak shaving response completion. express Virtual power plant aggregators are reducing their output. The virtual power plant aggregator indicated that it had not reduced its output at any given time. The subsidy unit price for virtual power plant aggregators participating in peak shaving response. , The start and end times of virtual power plant aggregators' participation in peak shaving response bidding. for The subsidy coefficient for virtual power plant aggregators participating in valley-filling response. for The status of virtual power plant aggregators participating in valley filling response bidding. express Virtual power plant aggregators have instructions to participate in valley filling responses. express The virtual power plant aggregator did not participate in the valley filling response instructions. for Virtual power plant aggregators are constantly adjusting their planned output values. for Real-time progress of virtual power plant aggregator participation in valley filling response. express Virtual power plant aggregators are reducing their output. The virtual power plant aggregator indicated that it had not reduced its output at any given time. The unit price for subsidies for virtual power plant aggregators participating in valley filling responses. , The start and end times of virtual power plant aggregators' participation in valley filling response bidding.

16. The method as described in claim 15, characterized in that, The completion status of virtual power plant aggregators' participation in peak shaving response at the specified time is as follows: The The subsidy coefficients for virtual power plant aggregators participating in peak shaving response are as follows: The The completion status of virtual power plant aggregators' participation in valley filling response is as follows: The The subsidy coefficients for virtual power plant aggregators participating in valley filling responses are as follows: In the above formula, for Real-time virtual power plant aggregator baseline load value, for Real-time virtual power plant aggregator benchmark load value.

17. The method as described in claim 16, characterized in that, The revenue of virtual power plant aggregators participating in day-ahead peak shaving during the settlement period is as follows: in: In the above formula, To enable virtual power plant aggregators to participate in the positive peak-shaving market and generate revenue. To enable virtual power plant aggregators to participate in the negative peak-shaving market and generate revenue. This is the percentage of revenue collected for peak shaving activities in advance. This indicates the winning bid status of virtual power plant aggregators participating in positive peak shaving. This indicates that virtual power plant aggregators have received messages regarding participation in positive peak shaving. The virtual power plant aggregator indicated that it had not received any notification regarding participation in positive peak shaving. This shows the completion status of virtual power plant's positive peak shaving during peak shaving periods. This indicates that the virtual power plant aggregator's electricity consumption increased during peak-shaving periods but did not exceed the aggregator's baseline load maximum. These are the start and end times of the peak-shaving period. This indicates that the virtual power plant aggregator failed to achieve peak shaving during the peak shaving period. The subsidy unit price for virtual power plant aggregators participating in positive peak shaving. for Real-time virtual power plant aggregator baseline load value, This refers to the winning bid status of virtual power plant aggregators participating in negative peak shaving. This indicates that virtual power plant aggregators have received messages regarding participation in negative peak shaving. The virtual power plant aggregator indicated that it had not received any notification to participate in negative peak shaving. This indicates the completion status of negative peak shaving by virtual power plants during peak shaving periods. This indicates that virtual power plant aggregators consume less electricity during peak-shaving periods. This indicates that the virtual power plant aggregator failed to achieve negative peak shaving during the peak shaving period. The subsidy unit price for virtual power plant aggregators participating in negative peak shaving. To determine the threshold for negative peak modulation, The threshold for determining whether positive peak modulation is valid.

18. The method as described in claim 17, characterized in that, The second constraint is as follows: ， ， In the above formula, The continuous response time of the entities participating in peak shaving demand response. The continuous response time of the objects participating in the valley filling demand response. The minimum continuous response time for objects participating in peak shaving demand response. for Real-time progress of virtual power plant aggregators' participation in peak shaving response completion. for Real-time progress of virtual power plant aggregators' participation in peak shaving response completion. The minimum continuous response time for objects participating in valley filling demand response. for Real-time progress of virtual power plant aggregator participation in valley filling response. for Real-time progress of virtual power plant aggregator participation in valley filling response. The number of times a virtual power plant aggregator participates in demand response per day. The daily limit for participating in demand response is set. , These are the upper limits of the subsidy unit price for day-ahead peak shaving and valley filling responses in demand response. , These are the upper limits of the subsidy unit price for positive peak shaving and negative peak shaving, respectively. This refers to the subsidy unit price for day-ahead valley filling response in the demand response. This refers to the unit price of subsidies for day-ahead peak shaving responses in demand response. for Real-time progress of virtual power plant aggregators' participation in peak shaving response completion. for Real-time progress of virtual power plant aggregators' participation in valley filling response.

19. An apparatus based on the virtual power plant cooperative optimization control method based on multi-agent game-theoretic reinforcement learning as described in any one of claims 1-18, characterized in that, The device includes: The first analysis module is used to solve the pre-built virtual power plant optimization model based on the Sharpe value of each virtual power plant, and to determine the virtual power plant alliance among each virtual power plant. The second analysis module is used to solve the pre-built virtual power plant aggregator optimization model corresponding to the virtual power plant alliance using multi-agent reinforcement learning method, so as to obtain the action strategies of each virtual power plant in the virtual power plant alliance for participating in demand response and day-ahead peak shaving. The third analysis module is used to perform collaborative optimization control on each virtual power plant in the virtual power plant alliance by utilizing the action strategies of each virtual power plant in the virtual power plant alliance in demand response and day-ahead peak shaving. The action strategy includes output plan and electricity price. During the execution of the multi-agent reinforcement learning method, the state space includes the observation information of each virtual power plant.

20. A computer device, characterized in that, include: One or more processors; The processor is used to execute one or more programs; When the one or more programs are executed by the one or more processors, the virtual power plant collaborative optimization control method based on multi-agent game reinforcement learning as described in any one of claims 1 to 18 is implemented.

21. A computer-readable storage medium, characterized in that, It contains a computer program, which, when executed, implements the virtual power plant collaborative optimization control method based on multi-agent game reinforcement learning as described in any one of claims 1 to 18.

Images