Virtual power plant dynamic integration decision method based on reinforcement learning and related products

Abstract

The application discloses a kind of based on reinforcement learning's virtual power plant dynamic integration decision method and related product, belong to virtual power plant decision-making technical field.The application provides based on reinforcement learning's virtual power plant dynamic integration decision method, including based on historical market data to construct simulation environment and train multiple heterogeneous expert models, dynamic selection performance optimal champion model at each decision time to generate bid action, adapt to market changes by dynamically adjusting decision mechanism, select the model with the highest score of recent performance index as the champion model, and generate bid action according to the current real market state using the model;Receive immediate reward after execution, only update the sliding window performance tracker corresponding to the champion model and realize continuous adaptive decision.The method of the application can dynamically adapt to market style changes, improve decision robustness and stability, reduce bid income fluctuation, realize the advantages of long-term stable profit of virtual power plant in complex power environment.

Description

Technical Field

[0001] This invention relates to the field of virtual power plant decision-making technology, specifically to a dynamic integrated decision-making method for virtual power plants based on reinforcement learning and related products. Background Technology

[0002] With the rapid development of the electricity spot market, electricity prices are significantly affected by multiple factors, including fluctuations in renewable energy output, changes in supply and demand, and policy adjustments, exhibiting high-frequency, real-time, and non-linear volatility. This complex volatility poses a severe challenge to virtual power plants participating in market bidding, making it difficult to accurately grasp market dynamics and formulate stable profit strategies. Currently, virtual power plant bidding decisions mainly rely on manual traders or traditional mathematical optimization models. Manual trading strategies heavily depend on traders' personal experience and subjective judgment, resulting in slow response times and an inability to achieve 24 / 7 real-time monitoring and rapid response. Under high-pressure environments, they are prone to decision-making errors, leading to missed bidding opportunities and profit losses. Traditional mathematical optimization methods, such as model predictive control, heavily rely on accurate predictions of future electricity prices and renewable energy output. The model construction process is complex and cumbersome, with high computational costs. Furthermore, when the market encounters unexpected events (such as extreme weather or equipment failure), the prediction error expands sharply, and the optimization results deviate significantly from reality, indicating a significant lack of model adaptability and robustness.

[0003] In recent years, some studies have attempted to introduce single reinforcement learning algorithms (such as proximal policy optimization and soft actor-commentator algorithms) for bidding decisions. However, these single models are like "specialized" experts, with their performance highly limited by the degree of matching between market style and the algorithm's internal mechanisms. For example, in a stable market environment, the proximal policy optimization algorithm may perform stably, but in a highly volatile market, its performance is prone to a sharp drop due to unstable policy updates; the soft actor-commentator algorithm is efficient in continuous action spaces, but converges slowly in discrete decision-making scenarios. When the market style shifts (such as from a low-volatility period to a high-volatility period), previously high-performing models may quickly become ineffective, causing drastic fluctuations in bidding returns and bringing significant decision-making risks. The paper "Safe Reinforcement Learning for Strategic Bidding of Virtual PowerPlants in Day-Ahead Markets" elaborates on the limitations of applying a single reinforcement learning model in virtual power plant bidding, revealing its vulnerability in dynamic markets, and representing the baseline level of existing technology.

[0004] Therefore, how to solve the performance degradation problem of a single model when market style changes, and how to achieve long-term stable profitability of virtual power plants in complex power environments, has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide a dynamic integrated decision-making method for virtual power plants based on reinforcement learning and related products to overcome the problems mentioned in the background art. The method of this invention can dynamically evaluate model performance and adaptively select the optimal decision mechanism. It has the advantages of being able to dynamically adapt to changes in market style, improve decision robustness and stability, reduce fluctuations in bidding revenue, and achieve long-term stable profitability of virtual power plants in complex power environments.

[0006] The present invention solves the above-mentioned technical problems through the following technical solution: This invention provides a dynamic integrated decision-making method for virtual power plants based on reinforcement learning, comprising the following steps: S1: Construct a simulation environment based on historical market data, and train multiple heterogeneous expert models based on different reinforcement learning algorithms in parallel within the simulation environment. Initialize a sliding window performance tracker for each heterogeneous expert model to record recent performance metrics. S2: At each decision time t, obtain the current real market state and calculate the recent performance index score of each heterogeneous expert model based on the data in the sliding window performance tracker of each heterogeneous expert model. S3: Select the heterogeneous expert model with the highest recent performance index score as the champion model at the current decision moment. Use the champion model to generate bidding actions based on the current real market state and submit them to the real market environment for execution. S4: Receive the real-time reward from the actual market environment after the bidding action is performed, update the sliding window performance tracker corresponding to the champion model only with the real-time reward, return to step S2, and proceed to the next decision moment.

[0007] A further improvement of the present invention is that the reinforcement learning algorithm includes at least two of the following: Proximal Policy Optimization (PPO), Soft Actor-Commentator (SAC), Recurrent Proximal Policy Optimization (R-PPO), Dominant Actor-Commentator (A2C), and Dual-Delay Deep Deterministic Policy Gradient Algorithm (TD3).

[0008] A further improvement of this invention is that the recent performance metric is the immediate reward obtained by each heterogeneous expert model in recent decision-making, and the sliding window performance tracker is a first-in-first-out queue with a fixed sliding window length.

[0009] A further improvement of the present invention is that the sliding window performance tracker corresponding to the champion model is updated only using the instant reward, specifically by pushing the instant reward to the tail of the first-in-first-out queue corresponding to the champion model and removing the old reward value at the head of the first-in-first-out queue.

[0010] A further improvement of this invention is that the sliding window length is determined based on the market transaction cycle, the amount of historical data, and the decision response requirements, with the sliding window length ranging from 12 to 48.

[0011] A further improvement of the present invention is that the recent performance metric score of each heterogeneous expert model is calculated by: calculating the sum of the instantaneous rewards recorded by each heterogeneous expert model in its corresponding sliding window performance tracker.

[0012] This invention also provides a reinforcement learning-based dynamic integrated decision-making system for virtual power plants, comprising: The first module is used to build a simulation environment based on historical market data, and train multiple heterogeneous expert models based on different reinforcement learning algorithms in parallel in the simulation environment. A sliding window performance tracker is initialized for each heterogeneous expert model to record recent performance indicators. The second module is used to obtain the current real market state at each decision time t, and calculate the recent performance index score of each heterogeneous expert model based on the data in the sliding window performance tracker of each heterogeneous expert model. The third module is used to select the heterogeneous expert model with the highest recent performance index score as the champion model at the current decision moment. The champion model is used to generate bidding actions based on the current real market state and submit them to the real market environment for execution. The fourth module is used to receive the real-time reward feedback from the actual market environment after the bidding action is performed. The sliding window performance tracker corresponding to the champion model is updated only with the real-time reward, and then the process returns to step S2 to proceed to the next decision moment.

[0013] The present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described reinforcement learning-based virtual power plant dynamic integrated decision-making method.

[0014] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described reinforcement learning-based virtual power plant dynamic integrated decision-making method.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described reinforcement learning-based virtual power plant dynamic integrated decision-making method.

[0016] Compared with the prior art, the positive and progressive effects of the present invention are as follows: The present invention provides a dynamic integrated decision-making method for virtual power plants based on reinforcement learning. By dynamically integrating multiple heterogeneous reinforcement learning models and selecting the best model in real time based on their recent performance, the method effectively adapts to market fluctuations and improves the robustness and stability of decision-making. Specifically: A simulation environment is built based on historical market data, and multiple heterogeneous expert models are trained in parallel. A sliding window performance tracker is initialized, providing a diverse model pool and performance monitoring foundation for subsequent dynamic selection, avoiding the risk of a single model failing in market changes. At each decision point, the current real market state is acquired, and recent performance index scores are calculated based on the sliding window performance tracker data of each heterogeneous expert model, ensuring that model evaluation is based on actual performance rather than fixed rules and can capture dynamic market changes in a timely manner. The model with the highest recent performance index score is selected as the champion model, and this model is used to generate bidding actions based on the current real market state, realizing a dynamic integration mechanism. The optimal model is selected to execute the decision, reducing the uncertainty brought about by market style switching. Immediate rewards are received after execution, and only the sliding window performance tracker corresponding to the champion model is updated, efficiently maintaining performance data, focusing only on the feedback of the selected model, optimizing resource allocation, and returning to step S2 to achieve continuous adaptive decision-making. It has the advantages of dynamically adapting to changes in market style, improving decision robustness and stability, reducing bidding revenue fluctuations, and achieving long-term stable profitability of the virtual power plant in complex power environments. Attached Figure Description

[0017] The accompanying drawings are provided to further understand the invention and constitute a part of this invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0018] Figure 1 This is a flowchart illustrating the existing technology. Figure 2 This is a schematic diagram of the working principle of the system of the present invention; Figure 3 This diagram illustrates the performance evaluation results of our method and a single model in a virtual power plant bidding task. Figure 4 This is a flowchart illustrating the method of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] In the description of this invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0021] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0022] It should be understood that although terms such as first, second, third, etc., may be used in the embodiments of the present invention to describe the preset range, these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from one another. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0023] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0024] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. This is an explanation of the present invention and not a limitation thereof.

[0025] Existing technologies employ a fixed, single reinforcement learning algorithm (such as DDPG or PPO) as the core of decision-making. However, any single algorithm has its inherent mechanism bias, making it perform optimally only under specific market styles. For example, the robust PPO performs well in stable markets but may miss opportunities in highly volatile markets; while the exploratory SAC may do the opposite. When the volatility or regularity of the electricity market shifts, a previously high-performing single model may experience a sharp decline in performance, or even significant losses, because its strategy is incompatible with the current market environment. This inability to adapt to market style shifts is the fundamental reason for its insufficient decision-making robustness. Existing technologies typically follow a "one-size-fits-all" approach, selecting an algorithm, training it, and then deploying it directly. This approach lacks a mechanism for horizontal comparison and dynamic selection. In practical applications, operators cannot intuitively and quantitatively determine whether the currently used algorithm is the optimal choice, and there is a lack of a framework for automatically switching among multiple alternatives. This makes it difficult for the system to continuously utilize the most effective strategy for decision-making in long-term operation, thus failing to achieve global, long-term performance optimization. In summary, existing virtual power plant (VPP) pricing strategies often fail when market styles shift due to the "model-market mismatch" problem caused by a single fixed model.

[0026] In the academic paper "Safe Reinforcement Learning for Strategic Bidding of Virtual Power Plants in Day-Ahead Markets" (authors: Ognjen Stanojev, Lesia Mitridati, et al., 2023), the authors proposed a method using the DDPG algorithm to learn the competitive bidding strategy of virtual power plants (VPPs) in day-ahead markets. This paper demonstrates the learning capability of a single reinforcement learning agent under a specific market model. This approach has been explored in academic research; its core idea is to model the bidding problem of VPPs in the electricity market as a Markov Decision Process (MDP), and then select a specific deep reinforcement learning algorithm, such as Deep Deterministic Policy Gradient (DDPG) or Proximal Policy Optimization (PPO), as the decision-making agent. This agent learns an end-to-end mapping strategy from market states to optimal bidding actions through extensive trial-and-error interactions in a simulated market environment.

[0027] However, see Figure 1These schemes employing a single reinforcement learning algorithm are highly dependent on whether the market style matches the algorithm's inherent mechanism. For example, the robust PPO algorithm performs well in stable markets but may miss opportunities in highly volatile markets; conversely, the exploratory SAC algorithm may do the opposite. This risk of model-market mismatch leads to insufficient robustness of single-model decision-making, making it difficult to guarantee long-term stable profitability in volatile market environments. This is the core technical bottleneck that hinders the large-scale promotion of this existing technology in practical industrial applications.

[0028] If the aforementioned issues are not addressed, virtual power plants will face persistently high decision-making risks in dynamic market environments. With increasing market uncertainty, the system cannot guarantee the reliability of its bidding strategies, and frequent decision-making errors are expected to impact the economic returns and market participation capabilities of virtual power plants. In the long term, this problem will weaken the competitiveness of virtual power plants in the electricity spot market, hindering their sustainable operation and technological development.

[0029] To address this issue, this invention primarily employs a dynamically integrated decision-making framework that adaptively selects the optimal pricing strategy based on real-time market feedback. This overcomes the mismatch problem, thereby proactively mitigating decision-making risks and ensuring the long-term stable profitability of the VPP in a complex and volatile market environment. See also... Figure 4 A dynamic integrated decision-making method for virtual power plants based on reinforcement learning includes the following steps: S1: Construct a simulation environment based on historical market data, and train multiple heterogeneous expert models based on different reinforcement learning algorithms in parallel within the simulation environment. Initialize a sliding window performance tracker for each heterogeneous expert model to record recent performance metrics. S2: At each decision time t, obtain the current real market state and calculate the recent performance index score of each heterogeneous expert model based on the data in the sliding window performance tracker of each heterogeneous expert model. S3: Select the heterogeneous expert model with the highest recent performance index score as the champion model at the current decision moment. Use the champion model to generate bidding actions based on the current real market state and submit them to the real market environment for execution. S4: Receive the real-time reward from the actual market environment after the bidding action is performed, update the sliding window performance tracker corresponding to the champion model only with the real-time reward, return to step S2, and proceed to the next decision moment.

[0030] This invention effectively adapts to market fluctuations and improves the robustness and stability of decision-making by dynamically integrating multiple heterogeneous reinforcement learning models and selecting the best model in real time based on their recent performance. Specifically: a simulation environment is built based on historical market data, and multiple heterogeneous expert models are trained in parallel. A sliding window performance tracker is initialized, providing a diverse model pool and performance monitoring foundation for subsequent dynamic selection, avoiding the risk of a single model failing in market changes. At each decision moment, the current real market state is obtained, and the recent performance index score is calculated based on the sliding window performance tracker data of each heterogeneous expert model, ensuring that model evaluation is based on actual performance rather than fixed rules and can capture dynamic market changes in a timely manner. The model with the highest recent performance index score is selected as the champion model, and this model is used to generate bidding actions based on the current real market state, realizing a dynamic integration mechanism. The optimal model is selected to execute the decision, reducing the uncertainty brought about by market style switching. The immediate reward after execution is received, and only the sliding window performance tracker corresponding to the champion model is updated, which efficiently maintains performance data, focuses only on the feedback of the selected model, optimizes resource allocation, and returns to step S2 to achieve continuous adaptive decision-making.

[0031] This application constructs a diverse decision-making model pool by training multiple heterogeneous expert models based on different reinforcement learning algorithms in parallel and equipping each model with a sliding window performance tracker. At each decision-making moment, the system can acquire the real market state in real time and calculate the recent performance index score of each heterogeneous expert model based on its actual performance in recent decisions. This real-time performance-based evaluation mechanism enables the system to accurately capture the model's adaptability under current market conditions. By dynamically selecting the heterogeneous expert model with the highest recent performance index score as the champion model for decision-making, the "specialization" problem of a single model is effectively overcome. When the market style shifts from volatile to stable, if the performance of model M2 (good at volatile markets) begins to decline while the performance of model M1 (good at stable markets) gradually increases, this method can promptly identify this change and switch the champion model from M2 to M1. Thus, the virtual power plant can always use the experts most suitable for the current market environment for bidding, avoiding decision-making risks and decreased profitability caused by market style shifts.

[0032] A further improvement of the present invention is that the reinforcement learning algorithm includes at least two of the following: Proximal Policy Optimization (PPO), Soft Actor-Commentator (SAC), Recurrent Proximal Policy Optimization (R-PPO), Dominant Actor-Commentator (A2C), and Dual-Delay Deep Deterministic Policy Gradient Algorithm (TD3).

[0033] Reinforcement learning algorithms are used to construct multiple heterogeneous expert models, but the choice of algorithm type directly affects the complementarity and market adaptability of the models. By specifying at least two algorithms such as PPO, SAC, R-PPO, A2C, and TD3, each with its unique mechanisms and advantages—for example, PPO provides stability in policy optimization, SAC enhances efficiency when exploring unknown environments, R-PPO handles time-series dependencies, A2C simplifies policy gradient calculation, and TD3 addresses the overvaluation problem—this combination ensures the diversity of expert models, enabling the system to cope with high-frequency, non-linear market fluctuations. When market styles shift, different algorithm models can complement each other, avoiding performance degradation of a single model, thereby improving decision-making accuracy and long-term profitability. The specific characteristics of each algorithm contribute complementary advantages in solving technical problems, strengthening the overall effect of model ensemble.

[0034] A further improvement of this invention is that the recent performance metric is the immediate reward obtained by each heterogeneous expert model in recent decision-making, and the sliding window performance tracker is a first-in-first-out queue with a fixed sliding window length.

[0035] Recent performance metrics are defined as the immediate rewards obtained by each heterogeneous expert model in recent decisions. This is directly based on the actual feedback after decision execution as the evaluation standard, ensuring a close correlation between performance metrics and decision effectiveness. It avoids introducing irrelevant factors or subjective judgments, making model comparisons more objective and reliable. The sliding window performance tracker is defined as a fixed-length, first-in-first-out (FIFO) queue. This mechanism limits the data scale by a fixed length and manages data updates using the FIFO principle. New reward values ​​are added to the tail of the queue while old values ​​are removed from the head, ensuring the tracker efficiently processes real-time data, avoiding data staleness or accumulation, and ensuring that performance evaluation always reflects the latest market dynamics. This enhances the system's adaptability and decision stability, effectively solving the ambiguity and efficiency problems in performance evaluation, thereby improving the accuracy of model selection and system robustness.

[0036] A further improvement of the present invention is that the sliding window performance tracker corresponding to the champion model is updated only using the instant reward, specifically by pushing the instant reward to the tail of the first-in-first-out queue corresponding to the champion model and removing the old reward value at the head of the first-in-first-out queue.

[0037] Specifically, updating the sliding window performance tracker for the champion model using only the immediate reward ensures that updates are performed only on the champion model at the current decision moment. This avoids wasting computational resources by updating all heterogeneous expert models, reduces unnecessary operational overhead, and improves overall system efficiency. Pushing the immediate reward to the tail of the FIFO queue corresponding to the champion model ensures that newly acquired immediate rewards are added to the tail of the queue in a timely manner, making performance evaluation based on the latest data and enhancing the real-time performance and accuracy of the evaluation. Removing the old reward value from the head of the FIFO queue removes the oldest reward value in the queue, maintaining a fixed sliding window length and preventing the negative impact of outdated data on model performance evaluation, ensuring the stability and reliability of the evaluation results. These features work together to achieve efficient tracker maintenance, improve decision robustness and response speed, and solve the inefficiencies and data staleness problems that may occur when maintaining the sliding window performance tracker, thereby optimizing the performance evaluation process.

[0038] A further improvement of this invention is that the sliding window length is determined based on the market transaction cycle, the amount of historical data, and the decision response requirements, with the sliding window length ranging from 12 to 48.

[0039] The sliding window length is determined based on the market trading cycle, taking into account the cyclical fluctuations of the electricity market. This ensures the window length matches the actual trading rhythm, avoiding performance evaluation delays or overshoots due to cycle mismatches. It is also determined based on the amount of historical data, utilizing the available data scale to adjust the window size. This ensures that a more comprehensive performance trend can be captured when data is abundant, while avoiding overfitting when data is limited. Furthermore, it is determined based on decision response requirements, optimizing the window length for real-time decision-making, ensuring timely response in a rapidly changing market and reducing decision delays. Simultaneously, the sliding window length ranges from 12 to 48, providing reasonable boundary constraints to ensure the window is neither too short (avoiding noise interference and short-term fluctuations) nor too long (avoiding outdated historical data and sluggish response), achieving a balanced optimization of performance tracking in typical electricity market scenarios. By specifically defining the determination method and value range of the sliding window length, the problem of inappropriate sliding window length settings is solved, ensuring the performance tracker can dynamically adapt to changes in the market environment, improving the accuracy and adaptability of decision-making.

[0040] A further improvement of the present invention is that the recent performance metric score of each heterogeneous expert model is calculated by: calculating the sum of the instantaneous rewards recorded by each heterogeneous expert model in its corresponding sliding window performance tracker.

[0041] The purpose of calculating recent performance index scores is to quantify the performance of each heterogeneous expert model in recent decisions, thereby providing a reliable basis for selecting the champion model. Specifically, it calculates the sum of immediate rewards, which utilizes the immediate reward data recorded in the sliding window performance tracker. This ensures that the calculation is based on the latest accumulated information, avoids the computational burden brought by complex algorithms, improves the real-time performance and accuracy of the evaluation, and strengthens the stability of model selection.

[0042] This invention also provides a reinforcement learning-based dynamic integrated decision-making system for virtual power plants, comprising: The first module is used to build a simulation environment based on historical market data, and train multiple heterogeneous expert models based on different reinforcement learning algorithms in parallel in the simulation environment. A sliding window performance tracker is initialized for each heterogeneous expert model to record recent performance indicators. The second module is used to obtain the current real market state at each decision time t, and calculate the recent performance index score of each heterogeneous expert model based on the data in the sliding window performance tracker of each heterogeneous expert model. The third module is used to select the heterogeneous expert model with the highest recent performance index score as the champion model at the current decision moment. The champion model is used to generate bidding actions based on the current real market state and submit them to the real market environment for execution. The fourth module is used to receive the real-time reward feedback from the actual market environment after the bidding action is performed. The sliding window performance tracker corresponding to the champion model is updated only with the real-time reward, and then the process returns to step S2 to proceed to the next decision moment.

[0043] By dynamically integrating multiple heterogeneous expert models and selecting the optimal model based on real-time performance, the problem of poor robustness of a single model in a dynamic market environment is solved. Specifically: The first module constructs a simulation environment based on historical market data, allowing model training under safe conditions and reducing real-world market risks; it trains multiple heterogeneous expert models based on different algorithms in parallel, leveraging algorithmic diversity to cover a wide range of market scenarios and avoiding the limitations of a single model; it initializes a sliding window performance tracker for each model, laying the foundation for subsequent dynamic performance evaluation. The second module acquires the current real-world market state at each decision point, ensuring that decisions are based on the actual environment; it calculates model scores based on performance tracker data, capturing recent performance through a sliding window mechanism to adapt to short-term market changes and avoid relying on outdated data. The third module selects the model with the highest score as the champion model, prioritizing the use of the current best model to maximize decision quality; it uses the champion model to generate bidding actions based on the current real-world market state, matching decisions with the real-time environment and improving targeting and accuracy. The fourth module, upon receiving the immediate reward, only updates the tracker of the champion model, focusing resources on effective models and reducing computational overhead; it then returns to the next decision point, achieving cyclical adaptation and continuously optimizing the decision-making process to cope with market dynamics.

[0044] In a specific embodiment of the present invention, see Figure 2 A dynamic integrated decision-making method for virtual power plants based on reinforcement learning specifically includes the following steps: I. Offline Construction of Expert Model Library First, historical market data is collected and processed to construct a simulated market environment. Then, a set of heterogeneous reinforcement learning expert models based on different algorithms (such as PPO, SAC, etc.) are trained in parallel in this environment, and all trained models are stored to form a heterogeneous expert model library.

[0045] II. Initialization of the Window Performance Tracker The system loads all pre-trained expert models from a heterogeneous expert model library into the dynamic champion selector; simultaneously, for each expert model... Initialize a sliding window performance tracker with a fixed length of L. The tracker is a first-in-first-out queue used to record recent rewards.

[0046] III. Dynamic Champion Selection and Decision-Making At each decision time t, the dynamic champion selector obtains the market state from the real market environment. The selector calculates the recent performance score of each expert model based on the data in the sliding window performance tracker. For example, to calculate the total cumulative reward:

[0047] The model with the highest score will be selected as the champion of this round.

[0048] Ultimately, the champion model generates the bidding action. .

[0049] IV. Execution and Performance Feedback Updates Bidding action Once submitted to a real market environment for execution, the market returns an immediate reward. This reward value is fed back to the dynamic champion selector and is used only to update the tracking queue corresponding to the champion model in this round. (New rewards are added to the queue, old rewards are removed from the queue), and the system then proceeds to the next decision time t+1, and repeats steps three and four.

[0050] Preferably, enhancing the safety of a single reinforcement learning agent at the decision-making execution level requires no model library. Instead, it involves designing a Safety Shield module that checks in real-time whether the agent's bidding action violates preset physical or operational constraints after it generates a bid. If the action is risky, the shield corrects it, bringing it within the safety boundary, thus ensuring the system's stability and compliance in the final stage. Another alternative is to guide the agent to generate more robust policies at the learning source. This approach modifies the reward function by introducing a risk-related penalty term, a technique also known as reward shaping. When the agent's action approaches the safety boundary, a negative penalty is applied in addition to the reward, thus incentivizing the agent to autonomously learn an intrinsically safer bidding strategy during training, fundamentally improving the robustness of the decision.

[0051] This invention is described in detail using the day-ahead electricity market in New South Wales, Australia, as an application scenario. In step one, an expert model library containing five heterogeneous reinforcement learning models, including PPO, SAC, and TD3, is trained using historical data from the NSW market. In step two, the sliding window length L is set to 24. In a typical online decision-making scenario, assuming that at decision time t=100, the "dynamic champion selector" finds that the SAC model has the highest cumulative return over the past 24 decision periods. Therefore, in this round of decision-making, the SAC model generates a bidding action A100. After this action is executed, the market feedback return R100 = €50, which is then pushed into the performance tracking queue of the SAC model for evaluation in the next round. In this way, this invention can continuously and adaptively utilize the best-performing strategy for bidding, thereby maximizing the overall return.

[0052] See Figure 3 The performance evaluation results of the method of this invention and the single model in the virtual power plant bidding task show that the final net income is 9.55% higher than that of the best single model; the average daily income is higher than that of the single model; the avoidance ratio reaches 0.68, which is higher than that of the single model (0.65); the cumulative income curve is more stable and the risk resistance is stronger; the optimal strategy is dynamically selected, the market changes are quickly responded to, and intelligent bidding is achieved.

[0053] By introducing an innovative dynamic champion selection mechanism, significant technological advancements and economic benefits have been achieved in the day-ahead electricity market automatic bidding task of virtual power plants, demonstrating clear superiority over existing solutions that rely on a single reinforcement learning model.

[0054] Based on the same inventive concept, this application provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of a reinforcement learning-based dynamic integrated decision-making method for a virtual power plant. The memory may include main memory, such as high-speed random access memory, or it may also include non-volatile memory, such as at least one disk storage device. The processor, network interface, and memory are interconnected via an internal bus, which may be an industry-standard architecture bus, a peripheral component interconnection standard bus, an extended industry-standard architecture bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory stores the program; specifically, the program may include program code, which includes computer operation instructions. The memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0055] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the steps of the reinforcement learning-based virtual power plant dynamic integrated decision-making method. Specifically, the computer-readable storage medium includes, but is not limited to, volatile memory and / or non-volatile memory. The volatile memory may include RAM (Random Access Memory) and / or cache memory, etc. The non-volatile memory may include ROM (Read-Only Memory), hard disk, flash memory, optical disk, magnetic disk, etc.

[0056] Based on the same inventive concept, this application provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the above-described reinforcement learning-based virtual power plant dynamic integrated decision-making method.

[0057] Those skilled in the art will understand that embodiments of the present invention can be provided as methods or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM (Compact Disc Read-Only Memory), optical storage, etc.) containing computer-usable program code.

[0058] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer apparatus or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0059] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer device or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0060] These computer program instructions may also be loaded onto a computer device or other programmable data processing equipment to cause a series of operational steps to be performed on the computer device or other programmable equipment to produce a process implemented by the computer device, thereby providing instructions that execute on the computer device or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0061] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0062] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A dynamic integrated decision-making method for virtual power plants based on reinforcement learning, characterized in that, Includes the following steps: S1: Construct a simulation environment based on historical market data, and train multiple heterogeneous expert models based on different reinforcement learning algorithms in parallel within the simulation environment. Initialize a sliding window performance tracker for each heterogeneous expert model to record recent performance metrics. S2: At each decision time t, obtain the current real market state and calculate the recent performance index score of each heterogeneous expert model based on the data in the sliding window performance tracker of each heterogeneous expert model. S3: Select the heterogeneous expert model with the highest recent performance index score as the champion model at the current decision moment. Use the champion model to generate bidding actions based on the current real market state and submit them to the real market environment for execution. S4: Receive the real-time reward from the actual market environment after the bidding action is performed, update the sliding window performance tracker corresponding to the champion model only with the real-time reward, return to step S2, and proceed to the next decision moment.

2. The method for dynamic integrated decision-making in a virtual power plant based on reinforcement learning according to claim 1, characterized in that, The reinforcement learning algorithms include at least two of the following: Proximal Policy Optimization (PPO), Soft Actor-Commentator (SAC), Recurrent Proximal Policy Optimization (R-PPO), Dominant Actor-Commentator (A2C), and Double Delay Deep Deterministic Policy Gradient Algorithm (TD3).

3. The method for dynamic integrated decision-making in a virtual power plant based on reinforcement learning according to claim 1, characterized in that, The recent performance metric is the immediate reward obtained by each heterogeneous expert model in recent decisions, and the sliding window performance tracker is a first-in-first-out queue with a fixed sliding window length.

4. The method for dynamic integrated decision-making of a virtual power plant based on reinforcement learning according to claim 3, characterized in that, The sliding window performance tracker corresponding to the champion model is updated only using the instant reward, specifically by pushing the instant reward to the tail of the first-in-first-out queue corresponding to the champion model and removing the old reward value from the head of the first-in-first-out queue.

5. The method for dynamic integrated decision-making of a virtual power plant based on reinforcement learning according to claim 3, characterized in that, The sliding window length is determined based on the market trading cycle, the amount of historical data, and the decision-making response requirements. The sliding window length ranges from 12 to 48.

6. The method for dynamic integrated decision-making of a virtual power plant based on reinforcement learning according to claim 1, characterized in that, Calculate the recent performance metric score for each heterogeneous expert model by summing the instantaneous rewards recorded for each heterogeneous expert model within its corresponding sliding window performance tracker.

7. A virtual power plant dynamic integrated decision-making system based on reinforcement learning, characterized in that, include: The first module is used to build a simulation environment based on historical market data, and train multiple heterogeneous expert models based on different reinforcement learning algorithms in parallel in the simulation environment. A sliding window performance tracker is initialized for each heterogeneous expert model to record recent performance indicators. The second module is used to obtain the current real market state at each decision time t, and calculate the recent performance index score of each heterogeneous expert model based on the data in the sliding window performance tracker of each heterogeneous expert model. The third module is used to select the heterogeneous expert model with the highest recent performance index score as the champion model at the current decision moment. The champion model is used to generate bidding actions based on the current real market state and submit them to the real market environment for execution. The fourth module is used to receive the real-time reward feedback from the actual market environment after the bidding action is performed. The sliding window performance tracker corresponding to the champion model is updated only with the real-time reward, and then the process returns to step S2 to proceed to the next decision moment.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the reinforcement learning-based virtual power plant dynamic integrated decision-making method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the reinforcement learning-based virtual power plant dynamic integrated decision-making method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the reinforcement learning-based virtual power plant dynamic integrated decision-making method as described in any one of claims 1 to 6.

Images