Virtual power plant energy transaction self-optimization method and system based on reinforcement learning

Abstract

The invention discloses a reinforcement learning-based virtual power plant energy transaction self-optimization method and system, and the method comprises the steps: obtaining distributed energy operation state data, market price information and environment parameters, and constructing a state vector according to the obtained data; inputting the state vector into a shared feature network for feature extraction to obtain a feature vector; according to the feature vectors, discrete actions and continuous actions are obtained; combining the discrete action and the continuous action to form a combined action; receiving a market clearing result and equipment operation feedback information, and calculating a reward value according to the equipment operation feedback information; taking the state vector, the joint action, the reward value and the state vector at the next moment as experience samples to be stored in a distributed experience playback pool; obtaining global model parameters according to the sample data in the experience playback pool; and issuing the global model parameters to each edge terminal, and synchronously updating a local policy network to realize policy optimization.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary fields of intelligent power systems, artificial intelligence optimization control, power market mechanisms and distributed energy management, and particularly relates to a self-optimization method and system for virtual power plant energy trading based on reinforcement learning. Background Technology

[0002] With technological advancements, the power system is undergoing profound digitalization and a high-proportion transformation towards new energy sources. The proportion of photovoltaic and wind power in the power generation structure continues to increase, but their inherent volatility, intermittency, and uncertainty pose significant challenges to the safe and stable operation of the power grid. The rapid growth of distributed energy resources (DER) is driving the power grid to evolve from a traditional "centralized supply-passive consumption" model to a "distributed supply-active response" model.

[0003] Against this backdrop, virtual power plants (VPPs) are gradually becoming a key technological means for energy systems in various countries. Through information and communication systems, virtual power plants integrate a large number of dispersed photovoltaic, energy storage, electric vehicles, charging facilities, and controllable loads into a unified and dispatchable whole, achieving "coordinated control, revenue optimization, and grid friendliness".

[0004] However, virtual power plants still face numerous difficulties in actual operation:

[0005] 1. Uncertainties arising from a high proportion of new energy sources:

[0006] The output of new energy sources such as photovoltaics and wind power is significantly affected by weather, and forecasting errors are common, which increase with the duration of the forecast. Traditional dispatching mechanisms rely heavily on forecast accuracy; any forecasting deviation can cause optimized dispatching to deviate from the optimal result, affecting revenue and even causing grid stability problems.

[0007] 2. Distributed resources are massive in scale and complex in type:

[0008] When the number of resources connected to a virtual power plant reaches hundreds to tens of thousands of devices, the differences in device protocols, data structures, frequent state changes, and scheduling constraints make it difficult for any centralized optimization method to achieve real-time control.

[0009] 3. Electricity market reform requires virtual power plants to engage in highly intelligent bidding and behavioral decision-making:

[0010] Electricity spot market (real-time market) prices fluctuate wildly, with price jumps occurring every 15 minutes or even 5 minutes; demand-side response and ancillary services markets also require virtual power plants to make rapid and intelligent decisions. Traditional methods (LP / MILP / MPC) are ill-suited to the frequently changing prices and uncertain environment.

[0011] 4. The operating constraints of energy equipment are complex:

[0012] Energy storage batteries have the following limitations: SOC range limitation, charge and discharge rate limitation, temperature limitation, lifetime decay model, and safety protection mechanisms (such as overvoltage and undervoltage). These constraints make the optimization problem highly nonlinear and nonconvex, and traditional optimization methods often fail to obtain high-quality solutions.

[0013] Reinforcement learning (RL) can learn the optimal policy for an environment through trial and error, making it particularly suitable for scenarios involving market games, uncertainty, and multi-objective constraints. Its advantages include: it does not rely on predictive models, it can be directly optimized based on rewards, it can learn complex nonlinear policies, it can adapt to time-varying environments, and it can learn continuously over long periods.

[0014] However, traditional RL still faces the following problems in energy scenarios:

[0015] 1. Unstable training: Q-value is easily overestimated, policy gradient jitter is obvious, resulting in slow convergence.

[0016] 2. Difficulty in handling mixed motion spaces: DQN can only handle discrete motions; DDPG / SAC can only handle continuous motions.

[0017] 3. Limited computing power at the edge: Each edge terminal of the virtual power plant may be an ARM industrial control computer with limited computing power, which is not suitable for large model inference.

[0018] 4. Difficulty in ensuring policy consistency across multiple terminals: Independent learning by each terminal can lead to policy discrepancies and prevent the formation of a unified scheduling result.

[0019] To address the aforementioned technical challenges, there is an urgent need for a self-optimizing method and system for virtual power plant energy trading based on reinforcement learning. Summary of the Invention

[0020] To address the aforementioned technical issues, this invention proposes a self-optimization method and system for virtual power plant energy trading based on reinforcement learning. This method constructs a high-dimensional state vector containing photovoltaic power, energy storage status, electricity price data, and load information. It then utilizes a shared feature network to drive a DQN (Deep Q-Network) and Actor-Critic hybrid structure to generate discrete bidding strategies and continuous energy storage power, thereby achieving adaptive intelligent optimization of the virtual power plant under uncertain environments.

[0021] To achieve the above objectives, this invention provides a self-optimization method for virtual power plant energy trading based on reinforcement learning, comprising:

[0022] Acquire distributed energy operation status data, market price information, and environmental parameters, and construct a state vector based on the acquired data;

[0023] The state vector is input into a shared feature network for feature extraction to obtain a feature vector;

[0024] Based on the feature vector, discrete actions and continuous actions are obtained;

[0025] The discrete action is combined with the continuous action to form a joint action;

[0026] Receive market clearing results and equipment operation feedback information, and calculate the reward value based on the equipment operation feedback information;

[0027] The state vector, the joint action, the reward value, and the state vector at the next moment are stored as experience samples in a distributed experience replay pool.

[0028] The global model parameters are obtained based on the sample data in the experience replay pool.

[0029] The global model parameters are distributed to each edge terminal, and the local policy network is updated synchronously to achieve policy optimization.

[0030] Optionally, the state vector includes energy storage state of charge, photovoltaic output, load power, market electricity price sequence, carbon emission coefficient, weather characteristics, historical action sequence, and time characteristics.

[0031] Optionally, obtaining discrete and continuous actions based on the feature vector includes:

[0032] Based on the feature vector, discrete actions and their value functions are calculated using a deep Q-network, and continuous actions are generated using an Actor network. The discrete actions include market electricity price bidding tiers, which include conservative, neutral, and aggressive categories. The continuous actions include energy storage charging and discharging power, which is output through an activation function and mapped and scaled to the rated power range of the energy storage system.

[0033] Optionally, calculating the reward value based on the device operation feedback information includes:

[0034] ;

[0035] in, As a reward value, For energy trading revenue, For energy storage safety constraints, This is the fitted value for battery degradation. Weighting for reward penalties For security penalty weighting, This is to reduce the penalty weight.

[0036] Optionally, obtaining global model parameters based on sample data in the experience replay pool includes:

[0037] Based on the sample data in the experience replay pool, the temporal difference error is calculated using the Critic network, and the parameters of the deep Q-network, Actor network, and Critic network are updated using the temporal difference error; the network parameters of multiple edge terminals are then uploaded to the cloud.

[0038] The global model parameters are obtained by weighted aggregation based on the parameters of each terminal and the corresponding number of empirical samples.

[0039] Optionally, weighted aggregation can be performed based on the parameters of each terminal and the corresponding number of empirical samples, including:

[0040] ;

[0041] in, Let be the parameters of the i-th edge terminal. Its empirical sample size represents the global model parameters after aggregation, and N represents the number of edge terminals participating in this round of aggregation.

[0042] This invention also provides a self-optimizing system for virtual power plant energy trading based on reinforcement learning, comprising: a data acquisition module, a reinforcement learning decision-making module, and an optimization module;

[0043] The data acquisition module is used to acquire distributed energy operation status data, market price information and environmental parameters, and construct a state vector based on the acquired data;

[0044] The reinforcement learning decision module is used to input the state vector into a shared feature network for feature extraction to obtain a feature vector; obtain discrete actions and continuous actions based on the feature vector; combine the discrete actions and the continuous actions to form a joint action; receive market clearing results and equipment operation feedback information, and calculate a reward value based on the equipment operation feedback information; store the state vector, the joint action, the reward value, and the state vector at the next time step as experience samples in a distributed experience replay pool; and obtain global model parameters based on the sample data in the experience replay pool.

[0045] The optimization module is used to distribute the global model parameters to each edge terminal and synchronously update the local policy network to achieve policy optimization.

[0046] Compared with the prior art, the present invention has the following advantages and technical effects:

[0047] 1) Adaptive optimization of energy trading decisions: By using hybrid action space reinforcement learning to jointly optimize discrete bidding strategies and continuous energy storage power, virtual power plants can adaptively adjust trading and dispatch strategies under the conditions of electricity price fluctuations and new energy uncertainty, thereby improving overall trading returns and reducing the risk of strategy mismatch.

[0048] 2) Real-time performance and deployability: By using edge inference and end-edge-cloud collaboration mechanisms, the communication and computing wait caused by centralized cloud optimization is reduced, enabling policy generation to adapt to the market and control cycle requirements at the minute / sub-minute level.

[0049] 3) Hybrid action space support: DQN is used for discrete action estimation, Actor-Critic is used for continuous action generation, and feature reuse is achieved through a shared feature network, which can handle discrete and continuous decision variables that coexist in virtual power plant trading and scheduling.

[0050] 4) Safety and lifespan constraints: Safety penalty terms such as SOC, power, and temperature, as well as battery degradation penalty terms, are introduced into the reward function to ensure that the learned strategy satisfies the safety boundary of energy storage operation and takes into account the impact of lifespan while optimizing the returns.

[0051] 5) Multi-terminal consistency and scalability: By integrating the experience of multiple edge terminals through a cloud-based federated aggregation mechanism, the policy discrepancies caused by independent learning of multiple terminals are reduced, the consistency of collaborative behavior is improved, and the generalization ability across scenarios is enhanced.

[0052] 6) Improved training stability: Improved convergence stability and repeatability of reinforcement learning training through mechanisms such as target network, delayed update, experience replay and reward recalibration. Attached Figure Description

[0053] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0054] Figure 1 This is a flowchart of the self-optimization method for virtual power plant energy trading based on reinforcement learning, according to an embodiment of the present invention.

[0055] Figure 2 This is a system flow control diagram according to an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram of the reinforcement learning decision-making process according to an embodiment of the present invention. Detailed Implementation

[0057] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0058] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0059] This embodiment proposes a self-optimization method for virtual power plant energy trading based on reinforcement learning, such as... Figure 1 As shown, the specific steps include:

[0060] Acquire distributed energy operation status data, market price information, and environmental parameters, and construct a state vector based on the acquired data;

[0061] The state vector is input into a shared feature network for feature extraction to obtain a feature vector;

[0062] Based on the feature vector, discrete actions and continuous actions are obtained;

[0063] The discrete action is combined with the continuous action to form a joint action;

[0064] Receive market clearing results and equipment operation feedback information, and calculate the reward value based on the equipment operation feedback information;

[0065] The state vector, the joint action, the reward value, and the state vector at the next moment are stored as experience samples in a distributed experience replay pool.

[0066] The global model parameters are obtained based on the sample data in the experience replay pool.

[0067] The global model parameters are distributed to each edge terminal, and the local policy network is updated synchronously to achieve policy optimization.

[0068] Furthermore, the state vector includes energy storage state of charge, photovoltaic output, load power, market electricity price sequence, carbon emission coefficient, weather characteristics, historical action sequence, and time characteristics.

[0069] Furthermore, based on the feature vector, obtaining discrete actions and continuous actions includes:

[0070] Based on the feature vector, discrete actions and their value functions are calculated using a deep Q-network, and continuous actions are generated using an Actor network. The discrete actions include market electricity price bidding tiers, which include conservative, neutral, and aggressive categories. The continuous actions include energy storage charging and discharging power, which is output through an activation function and mapped and scaled to the rated power range of the energy storage system.

[0071] Furthermore, the global model parameters obtained based on the sample data in the experience replay pool include:

[0072] Based on the sample data in the experience replay pool, the temporal difference error is calculated using the Critic network, and the parameters of the deep Q network, Actor network, and Critic network are updated using the temporal difference error; the network parameters of multiple edge terminals are uploaded to the cloud.

[0073] The global model parameters are obtained by weighted aggregation based on the parameters of each terminal and the corresponding number of empirical samples.

[0074] Specifically, such as Figure 1 As shown, this embodiment specifically includes:

[0075] Step S1: Construct the state space state space Includes at least one or more of the following: photovoltaic output, energy storage state of charge (SOC), load power, market electricity price sequence (day-ahead, real-time), carbon emission coefficient, equipment operating temperature, weather characteristics (sunny, cloudy, irradiance, wind speed), historical actions used for Markov modeling, and time characteristics (hour, weekday / holiday).

[0076] The state dimension is 15–25.

[0077] Step S2: Constructing the Hybrid Action Space :

[0078] ;

[0079] Among them, discrete actions Market electricity price bidding tiers:

[0080] ;

[0081] Continuous Actions Energy storage charging and discharging power:

[0082] ;

[0083] Joint Action:

[0084] ;

[0085] This step is one of the key innovations of this invention.

[0086] Step S3, Shared Feature Extraction Network (SRN):

[0087] ;

[0088] Where W and b represent the network weights and biases, respectively, and the output activation function is ReLU.

[0089] Step S4: DQN is used for discrete action estimation, and a deep Q-network is used to calculate the Q-value for each bid level.

[0090]

[0091] Step S5, Actor-Critic is used for continuous action generation:

[0092] Actor:

[0093] Critic:

[0094] Step S6, Reward Engineering: This step involves designing a reward function that integrates market returns, system security, and equipment lifespan.

[0095] ;

[0096] in, For energy trading revenue; For energy storage safety constraints; This represents the fitted value for battery degradation.

[0097] Step S7, Edge Inference:

[0098] Inference is performed every 100ms at the edge to ensure millisecond-level response.

[0099] Step S8, experience replay pool update includes: (s, a, r, s′), edge local storage not less than Data entries.

[0100] Step S9, Cloud-based Aggregated Training (FederatedRL):

[0101] Cloud parameter updates use FedAvg, which can improve policy consistency across 20–100 endpoints simultaneously.

[0102] ;

[0103] in Let be the parameters of the i-th edge terminal. The number of its empirical samples.

[0104] Step S10, Update the End-to-Cloud Synchronization Strategy:

[0105] The latest parameters are sent from the cloud every 60 seconds. Communication method: 5G-RedCap ultra-reliable low-latency communication; latency: uplink and downlink <20ms.

[0106] The layers included in the core reinforcement learning algorithm are shown in Table 1: Table 1 The algorithm employs a hybrid structure: The DQN component is responsible for the value assessment of the discrete action space; The Actor-Critic part is used for policy output in the continuous action space; The target network and the delayed update mechanism (soft-update coefficient τ=0.01) ensure stable training. The reinforcement learning loss function is defined as:

[0107] ;

[0108] Policy updates are based on gradient ascent:

[0109]

[0110] Through online training and cloud aggregation, this model can continuously self-optimize, achieving rapid adaptation to different industrial parks and electricity pricing strategies. The decision-making process of reinforcement learning is as follows: Figure 3 As shown.

[0111] End-edge-cloud collaboration mechanism:

[0112] End layer (device acquisition layer): responsible for real-time sensing of energy consumption and electricity price changes;

[0113] Edge layer (decision control layer): performs reinforcement learning inference and generates control instructions;

[0114] Cloud layer (optimized management layer): Aggregates experience from multiple nodes, retrains the model, and distributes parameters;

[0115] The data transmission cycle between the three layers is shown in Table 2:

[0116] Table 2

[0117] direction content cycle End → Edge Collect data and status information 1 second Edge → Cloud Statistical characteristics and strategy parameters 5 seconds Cloud → Edge Model update and command issuance 1 minute edge → end Local control feedback Real-time (100ms)

[0118] This design achieves a dual optimization mechanism of local autonomy and cloud collaboration.

[0119] The following is an experimental verification and comparative analysis of this embodiment:

[0120] Experimental Environment Construction: The experimental platform of this invention consists of the following three parts:

[0121] (1) Simulation-side environment (cloud):

[0122] Operating System: Linux Ubuntu 22.04; CPU: Intel Xeon Silver; GPU: NVIDIA RTX 3090 (for initial model training); Memory: 128 GB RAM;

[0123] Software framework: Python 3.9, PyTorch 2.0, Ray RLlib (for multi-policy parallel training);

[0124] Electricity price data source: Day-ahead / real-time market data from a certain province, sampling period of 15 minutes, totaling 3 years of data;

[0125] Meteorological data: photovoltaic irradiance, temperature, humidity, wind speed, sampling period 1 minute;

[0126] (2) Edge computing terminal (device side):

[0127] Hardware configuration: CPU: Quad-core ARM Cortex-A55, Memory: 2GB LPDDR4, Storage: 32GB eMMC, Communication: 5G-RedCap module (Sub-6GHz), Interfaces: RS485, CAN, Ethernet, Power consumption: <10 W;

[0128] Software environment: Linux Embedded OS, C++ inference engine (ONNX Runtime), Python environment for data collection and logging.

[0129] The computational power of edge inference environments is relatively weak, so the inference network undergoes compression, quantization, and structural pruning.

[0130] The result guarantees that the inference latency at the edge is < 40 ms.

[0131] (3) Real-world environment on the equipment side (end side):

[0132] The experiment selected a zero-carbon demonstration park for semi-physical simulation, including: 2 MWp photovoltaic modules, 1MW / 2 MWh battery energy storage, 500 kW controllable load, and 30 electric vehicle charging piles;

[0133] All devices have physical units or equivalent digital twin models.

[0134] Comparison algorithm settings:

[0135] To fully evaluate the performance of this invention, it is compared with the following methods:

[0136] Compared to Method 1, Linear Programming (LP, the baseline method) is solved once per market cycle (15 min), with the objective of minimizing operating costs, and relies on load forecasting, photovoltaic forecasting, and electricity price forecasting.

[0137] Compared to method 2, model predictive control (MPC) uses LSTM load forecasting, has an optimization window of 1 hour, performs rolling optimization every 15 minutes, and can handle time series constraints better.

[0138] Compared to method 3, independent DQN only processes discrete bid levels and discretizes energy storage power (resulting in accuracy loss).

[0139] Compared to method 4, independent Actor-Critic: it only handles continuous energy storage power operations and cannot handle bidding tier strategies;

[0140] Compared to method 5, DDPG / SAC (Continuous Action Reinforcement Learning): it has strong continuous decision-making ability, but lacks discrete bidding part, which requires additional modules to supplement it, and has weaker convergence.

[0141] Compared to method 6, the present invention, Hybrid DRL (DQN + Actor-Critic), simultaneously handles discrete and continuous actions, supports end-edge-cloud collaboration, supports federated parameter synchronization, and possesses autonomous operation even when the network is down; this method is the solution of the present invention.

[0142] Test scenario construction:

[0143] To comprehensively evaluate the performance of this invention, the following five types of test scenarios are defined:

[0144] (1) Scenario 1, low sunlight in winter + large peak-valley electricity price fluctuation:

[0145] Features: Low and unstable photovoltaic output; high electricity prices in the morning and evening, and low electricity prices at noon, making it suitable for electricity price arbitrage;

[0146] (2) Scenario 2, Summer high temperature + high light intensity + high load:

[0147] Features: High photovoltaic output, high load, energy storage is more prone to overheating, and battery degradation risk is increased;

[0148] (3) Scenario 3, random disturbance electricity price scenario:

[0149] Electricity prices are generated using a stochastic process:

[0150] ;

[0151] Where ε is Gaussian noise. It is used to test the robustness of the strategy.

[0152] (4) Scenario 4, Communication Interruption Scenario:

[0153] Simulate network disconnection between the edge and the cloud: short-term network disconnection (<5 minutes), medium-term network disconnection (5–30 minutes), and long-term network disconnection (30–120 minutes); used to test autonomous capabilities.

[0154] (5) Scenario 5, Equipment malfunction scenario:

[0155] For example, PV inverter grid disconnection, battery temperature exceeding limits and derating, and sudden load changes are used to test the effectiveness of safety constraints.

[0156] Model parameter settings:

[0157] The core model of this invention uses the following parameters:

[0158] (1) Shared Feature Network SRN: Dense(128) + ReLU;

[0159] (2) DQN branches: Dense(128), Dense(64), output: 4 Q values, Double-DQN

[0160] Target update cycle: 10 steps, learning rate: 1e -4 ;

[0161] (3) Actor branches: Dense(128), Dense(64), tanh output, action clipping: [-1, 1] mapped to [-Pmax, Pmax];

[0162] (4) Critic branch: Input dimension = state dimension + action dimension, Dense(128), Dense(128), output Q value;

[0163] (5) Experience Replay Pool: Size: 1e 5 Sampling batch: 256, Priority experience replay (α=0.6).

[0164] Convergence experiment:

[0165] Multiple experiments have shown that MPC performs well in the early stages, but its long-term benefits are limited; DQN converges quickly but is unstable; AC converges slowly but is reasonably stable; SAC / DDPG converges slowly and is prone to oscillation; the Hybrid DRL of this invention converges the fastest and has the highest stability.

[0166] The profit comparison experiment (key data) shows the average daily profit comparison in Table 3:

[0167] Table 3

[0168] method Daily income relative improvement LP 1810 benchmark MPC 1956 +8.0% DQN 2093 +15.6% AC 2136 +18.0% SAC 2201 +21.6% This invention, Hybrid DRL 2297 +26.9%

[0169] Safety and constraint satisfaction status.

[0170] The present invention satisfies the following conditions: SOC limit: no over-limit (SOC ∈ [15%, 90%]); battery temperature: no over-temperature condition; power limit: no overload operation; market bidding: in compliance with market rules.

[0171] Robustness testing:

[0172] Under the conditions of random electricity price + random PV disturbance: the volatility of the strategy of this invention is < 10%, the volatility of DQN is > 30%, and the volatility of MPC is > 20%;

[0173] This demonstrates that the present invention can maintain high performance even under conditions of strong uncertainty.

[0174] Autonomous Experiment on Communication Interruption:

[0175] Internet outage time: 0–120 minutes;

[0176] Test results: Internet outage <10 minutes: almost no impact; Internet outage 10–30 minutes: revenue decrease <4%; Internet outage 30–120 minutes: revenue decrease <10%, still significantly better than MPC;

[0177] Reason: The local model can reason autonomously, and the local experience pool can support fine-tuning with small steps.

[0178] Multi-policy consistency test: The policy differences of multiple edge terminals under two scenarios, with and without federated learning, are shown in Table 4.

[0179] Table 4

[0180] method Multi-terminal strategy differences Collaborative behavior Independent RL for each terminal Large difference (>35%) Action conflicts and unstable returns Cloud-based centralized training smaller Slow training, requires a lot of communication This invention, Federal RL <8% Unified behavior and increased benefits

[0181] Extended testing of offline self-governance capabilities:

[0182] Simulated 5G interruption: 5 minutes: no significant impact; 30 minutes: revenue decrease of less than 4.2%; 60 minutes: decrease of 8.9%; 120 minutes: decrease of 12.5%; consistently outperforms LP / MPC.

[0183] Model inference performance testing:

[0184] Quantization model (INT8) inference speed: single inference: 28 ms; single loop: 100 ms (including read and write); CPU usage: 36–48%; meets real-time control requirements.

[0185] Based on multiple experiments, this embodiment has the following advantages: significantly improved returns, strong strategy stability, can operate in various extreme scenarios, maintains availability even under communication interruption, and has excellent multi-terminal collaborative performance.

[0186] This embodiment also provides a virtual power plant energy trading self-optimization system based on reinforcement learning, such as... Figure 2 As shown, it includes: a data acquisition module, a reinforcement learning decision-making module, and an optimization module;

[0187] The data acquisition module is used to acquire distributed energy operation status data, market price information and environmental parameters, and construct a state vector based on the acquired data;

[0188] The reinforcement learning decision module is used to input the state vector into a shared feature network for feature extraction to obtain a feature vector; obtain discrete actions and continuous actions based on the feature vector; combine the discrete actions and the continuous actions to form a joint action; receive market clearing results and equipment operation feedback information, and calculate a reward value based on the equipment operation feedback information; store the state vector, the joint action, the reward value, and the state vector at the next time step as experience samples in a distributed experience replay pool; and obtain global model parameters based on the sample data in the experience replay pool.

[0189] The optimization module is used to distribute the global model parameters to each edge terminal and synchronously update the local policy network to achieve policy optimization.

[0190] Specifically, the data acquisition unit is used to collect real-time operating parameters of distributed energy resources within the park, including voltage, current, power, energy storage state of charge (SOC), market electricity price, carbon emission coefficient, etc.

[0191] Signal preprocessing unit: Normalizes, aligns, and filters noise in the acquired signals;

[0192] Reinforcement learning decision unit (AI core): Embedded DQN + Actor-Critic hybrid neural network, used to calculate the optimal trading action in real time based on state information;

[0193] 5G-RedCap communication module: Enables data exchange with the cloud-based virtual power plant control center, including status uploading, parameter downloading, and policy synchronization;

[0194] Execution control unit: responsible for translating strategic decisions into actual energy trading execution signals, such as bidding, power scheduling, or energy storage discharge commands;

[0195] Local storage and learning unit: Saves historical state-action-reward data for the experience replay module to use for model self-learning and updates;

[0196] Power and interface module: Provides stable power supply and multiple industrial communication interfaces (CAN, RS485, Ethernet).

[0197] The self-optimizing virtual power plant energy trading system proposed in this invention has the following technical advantages: it can simultaneously handle discrete bidding behavior and continuous energy storage power, supports end-edge-cloud collaboration, supports federated training, multi-terminal consistency, supports autonomous operation during network outages, significantly improves returns (18%~27%), has high security, and strong robustness.

[0198] This invention can be applied to: independent virtual power plants based on photovoltaic and energy storage, "park-level" zero-carbon energy management, integrated energy stations with "multi-station integration", large-scale virtual power plant operators, flexible dispatching platforms for power distribution networks, and has broad engineering application value.

[0199] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A self-optimization method for virtual power plant energy trading based on reinforcement learning, characterized in that, include: Acquire distributed energy operation status data, market price information, and environmental parameters, and construct a state vector based on the acquired data; The state vector is input into a shared feature network for feature extraction to obtain a feature vector; Based on the feature vector, discrete actions and continuous actions are obtained; The discrete action is combined with the continuous action to form a joint action; Receive market clearing results and equipment operation feedback information, and calculate the reward value based on the equipment operation feedback information; The state vector, the joint action, the reward value, and the state vector at the next moment are stored as experience samples in a distributed experience replay pool. The global model parameters are obtained based on the sample data in the experience replay pool. The global model parameters are distributed to each edge terminal, and the local policy network is updated synchronously to achieve policy optimization.

2. The self-optimization method for virtual power plant energy trading based on reinforcement learning according to claim 1, characterized in that, The state vector includes energy storage state of charge, photovoltaic output, load power, market electricity price sequence, carbon emission coefficient, weather characteristics, historical action sequence, and time characteristics.

3. The self-optimization method for virtual power plant energy trading based on reinforcement learning according to claim 1, characterized in that, Based on the feature vector, obtaining discrete and continuous actions includes: Based on the feature vector, discrete actions and their value functions are calculated using a deep Q-network, and continuous actions are generated using an Actor network. The discrete actions include market electricity price bidding tiers, which include conservative, neutral, and aggressive categories. The continuous actions include energy storage charging and discharging power, which is output through an activation function and mapped and scaled to the rated power range of the energy storage system.

4. The self-optimization method for virtual power plant energy trading based on reinforcement learning according to claim 1, characterized in that, The reward value is calculated based on the equipment operation feedback information, including: ； in, As a reward value, For energy trading revenue, For energy storage safety constraints, This is the fitted value for battery degradation. Weighting for reward penalties For security penalty weighting, This is to reduce the penalty weight.

5. The self-optimization method for virtual power plant energy trading based on reinforcement learning according to claim 3, characterized in that, The global model parameters obtained based on the sample data in the empirical replay pool include: Based on the sample data in the experience replay pool, the temporal difference error is calculated using the Critic network, and the parameters of the deep Q-network, Actor network, and Critic network are updated using the temporal difference error; the network parameters of multiple edge terminals are then uploaded to the cloud. The global model parameters are obtained by weighted aggregation based on the parameters of each terminal and the corresponding number of empirical samples.

6. The self-optimization method for virtual power plant energy trading based on reinforcement learning according to claim 5, characterized in that, Weighted aggregation is performed based on the parameters of each terminal and the corresponding number of empirical samples, including: ； in, Let be the parameters of the i-th edge terminal. Its empirical sample size represents the global model parameters after aggregation, and N represents the number of edge terminals participating in this round of aggregation.

7. A reinforcement learning-based virtual power plant energy trading self-optimization system, used to implement the method as described in any one of claims 1-6, characterized in that, include: Data acquisition module, reinforcement learning decision-making module, and optimization module; The data acquisition module is used to acquire distributed energy operation status data, market price information and environmental parameters, and construct a state vector based on the acquired data; The reinforcement learning decision module is used to input the state vector into a shared feature network for feature extraction to obtain a feature vector; obtain discrete actions and continuous actions based on the feature vector; combine the discrete actions and the continuous actions to form a joint action; receive market clearing results and equipment operation feedback information, calculate the reward value based on the equipment operation feedback information; and store the state vector, the joint action, the reward value, and the state vector at the next time step as experience samples in a distributed experience replay pool. The global model parameters are obtained based on the sample data in the experience replay pool. The optimization module is used to distribute the global model parameters to each edge terminal and synchronously update the local policy network to achieve policy optimization.

Images