Virtual power plant energy storage dispatch optimization method and system based on deep reinforcement learning

Abstract

The application discloses a virtual power plant energy storage scheduling optimization method and system based on deep reinforcement learning, relates to the technical field of virtual power plant energy storage scheduling, and first collects historical and real-time operation data of distributed photovoltaic power generation and distributed wind power generation in a virtual power plant, historical and real-time load data of various users, operation parameters of energy storage batteries and scheduling requirements issued by a power grid scheduling platform; a multimodal source and load probability prediction model is constructed to output distributed power output fluctuation intervals and user load fluctuation intervals in a preset scheduling period; and a deep reinforcement learning scheduling model with a double-delay deep deterministic policy gradient architecture adapted to a continuous action space is constructed. The application can adaptively match the random fluctuation characteristics of distributed power and user load, does not need to set fixed scheduling rules, improves scheduling operation income on the premise that the power of a grid connection point meets the scheduling requirements of the power grid, and effectively reduces the risk of power grid assessment.

Description

Technical Field

[0001] This invention relates to the field of virtual power plant energy storage scheduling technology, and in particular to a virtual power plant energy storage scheduling optimization method and system based on deep reinforcement learning. Background Technology

[0002] With the continuous increase in the scale of distributed photovoltaic and wind power integration, virtual power plants, as the core carrier for aggregating distributed energy, energy storage, and controllable loads to participate in grid dispatch, have become an important support for the coordinated interaction of power system sources, grid, load, and storage. Among them, the charging and discharging dispatch strategy of energy storage system directly determines the operating economy of virtual power plant, grid access compliance, and the full life cycle revenue of energy storage assets, and is the core technical link of virtual power plant operation.

[0003] Currently, mainstream energy storage dispatch technologies are mainly divided into two categories. The first category is the fixed-rule dispatch method, which involves pre-setting charging and discharging trigger thresholds and SOC operating ranges based on historical peak and valley electricity price periods and rated energy storage parameters. During dispatch, charging and discharging commands are strictly executed according to the preset rules. This solution has low implementation cost, simple logic, and high operational stability, and is currently widely used in virtual power plant scenarios with small scale and low proportion of distributed energy. However, this type of solution cannot adapt to the random fluctuation characteristics of distributed power output and user load. Under extreme weather conditions and sudden load changes, the power fluctuation at the grid connection point is prone to exceed the grid assessment threshold. At the same time, it cannot dynamically match the temporary dispatch needs of the grid, and the potential for improving dispatch benefits is limited.

[0004] The second category is traditional mathematical optimization scheduling methods, including mixed-integer linear programming, particle swarm optimization, and genetic algorithms. Their implementation logic involves establishing an optimization function with the goal of maximizing scheduling revenue, and then solving for the optimal charging and discharging strategy by combining grid constraints and energy storage constraints. This type of scheme can obtain the theoretically optimal solution under deterministic scenarios, and its scheduling economy is superior to fixed-rule schemes. It is often used in large-scale centralized energy storage scheduling scenarios. However, the solution speed of this type of scheme increases exponentially with the number of energy storage nodes and constraints, making it unable to meet the real-time scheduling requirements of 15 minutes or even minutes. Furthermore, it can only model based on deterministic source-load predictions, failing to consider the uncertainty of source-load, leading to significant deviations in actual operation. At the same time, most schemes do not include the energy storage's life-cycle losses in the optimization objective, resulting in excessively rapid energy storage degradation rates over long-term operation, significantly increasing the asset investment cost of the virtual power plant.

[0005] As the proportion of distributed energy covered by virtual power plants continues to increase and the requirements for refined grid dispatching continue to rise, existing dispatching schemes can no longer simultaneously adapt to the needs of source-load uncertainty, real-time dispatching response, and multi-objective collaborative optimization. There is an urgent need to develop new energy storage dispatching technologies to solve the above-mentioned common problems in the industry. Summary of the Invention

[0006] The present invention proposes a virtual power plant energy storage scheduling optimization method and system based on deep reinforcement learning to solve the problems mentioned in the prior art, such as insufficient flexibility, poor real-time performance, and failure to take into account energy storage lifetime and multi-objective collaborative optimization.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a virtual power plant energy storage scheduling optimization method based on deep reinforcement learning, comprising the following steps: S1. Through a multi-protocol compatible data acquisition interface, connect to the distributed energy monitoring system, user load acquisition terminal, energy storage battery management system, and power grid dispatching platform within the coverage area of ​​the virtual power plant to collect multi-dimensional source and load operation data, power grid dispatching instructions, and rated parameters and real-time status data of each energy storage node. The source-load operation data includes real-time output data of photovoltaic / wind power, inverter operating parameters, time-of-use load data of industrial and commercial / residential users, and demand-side response reporting data. The grid dispatch instructions include peak-shaving assessment requirements, grid connection point power fluctuation thresholds, peak-valley electricity price time period division rules, and emergency response instructions. The energy storage node data includes rated capacity, rated charge and discharge power, SOC operating limits, charge and discharge conversion efficiency, cycle life decay coefficient, current SOC value, real-time charge and discharge power, and battery health status (SOH). All the collected raw data are preprocessed to obtain a standardized time-series dataset. S2. Input the standardized time series dataset into the multimodal source load probability prediction model that has been trained and converged by the pre-labeled historical operation data, and output the distributed power output probability range and user load fluctuation probability range under the P90 confidence level within the preset scheduling period that can be customized from 1 hour to 24 hours in the future. S3. Construct a deep reinforcement learning scheduling model with a dual-delay deep deterministic strategy gradient TD3 architecture adapted to the continuous action space scenario of virtual power plant energy storage scheduling. The state space input of the deep reinforcement learning scheduling model includes the upper and lower limits of the output probability interval of distributed power sources, the upper and lower limits of the user load fluctuation probability interval, the real-time SOC and SOH values ​​of each energy storage node, and the grid scheduling command parameters. The action space is the continuous adjustment of the charging and discharging power of each energy storage node, with the value range from the negative value of the rated discharge power of the corresponding energy storage node to the positive value of the rated charging power. The reward function is a weighted comprehensive reward function that couples the three dimensions of scheduling economy, grid stability, and energy storage life cycle loss. S4. The scheduling process is triggered every 15 minutes or when an emergency dispatch instruction from the power grid is received. The real-time collected scheduling trigger data is input into the trained deep reinforcement learning scheduling model, and the optimal charging and discharging scheduling instruction with the unique identifier of the corresponding energy storage node, the execution period, and the charging and discharging power value is output and sent to the corresponding energy storage node for execution.

[0008] Preferably, the preprocessing in step S1 specifically includes: setting corresponding 3σ criterion outlier judgment thresholds for different data sources, removing outliers from the collected raw data, performing time-domain alignment on heterogeneous data with sampling frequencies of 1 minute, 5 minutes, and 30 minutes according to a preset 15-minute scheduling step, uniformly mapping all numerical data to the [0,1] interval using the minimum-maximum normalization method, and completing missing data with a missing rate of less than 5% using weighted interpolation of three adjacent sampling points before and after, with the weight inversely proportional to the time distance from the sampling point to the missing position.

[0009] Preferably, the multimodal source-load probability prediction model in step S2 adopts a Transformer-quantile regression joint model that integrates a multi-head time attention mechanism. The input side of the model simultaneously receives standardized numerical time-series data and weather and holiday category features processed by one-hot encoding. The category features include real-time light intensity, wind speed, ambient temperature, rainfall, and weekday / holiday / major event day identifiers. The output side of the model is set with three quantile regression branches, which output the distributed power output value and user load value of the three quantiles P10, P50, and P90, respectively. The corresponding probability interval is formed by the P10 and P90 quantile values.

[0010] Preferably, the calculation logic of the weighted comprehensive reward function in step S3 is as follows: First, calculate the difference between the revenue from purchasing and selling electricity from the virtual power plant and the cost of energy storage operation and maintenance within the scheduling period, and obtain the economic reward component after normalization; calculate the deviation between the actual power fluctuation value at the grid connection point and the preset threshold. The smaller the deviation, the higher the reward value. Obtain the stability reward component after normalization. The mapping loss value between the current charge / discharge depth of energy storage and the rated cycle life is calculated. The lower the loss, the higher the reward value. After normalization, the loss suppression reward component is obtained. Dynamic weights are assigned to the three components according to the current grid dispatch demand and the overall health status of energy storage. The sum of the weights of the three components is 1. The final comprehensive reward value is obtained by summing the weighted three components.

[0011] Preferably, the deep reinforcement learning scheduling model adopts a training mode of offline pre-training plus online fine-tuning. In the offline pre-training stage, a simulation scheduling environment is constructed using the historical operating data of a virtual power plant over the past three years. After at least 1 million iterations of training, the comprehensive reward value fluctuates below a preset threshold for 1,000 consecutive rounds, and the model converges. In the online fine-tuning stage, every 7-day update cycle, the model parameters are updated with small-step gradients using the actual operating data of the most recent 28 scheduling cycles. The learning rate in the fine-tuning stage is set to one-tenth of the learning rate in the offline pre-training stage to avoid large fluctuations in model parameters that could lead to scheduling anomalies.

[0012] Preferably, after outputting the optimal charge / discharge scheduling command in step S4, a command safety verification step is also included. First, it is verified whether a single charge / discharge scheduling command exceeds the actual allowable charge / discharge power range corresponding to the current SOH of the corresponding energy storage node. Second, it is verified whether the power of the grid connection point after the total charge / discharge power of all energy storage nodes is superimposed with the predicted output of the source load is within the fluctuation range required by the power grid. Finally, it is verified whether the scheduling command has the risk of triggering islanding operation in the event of a grid-side fault. The command is sent to the corresponding energy storage node for execution only after all verification items pass. If the verification fails, the scheduling model is triggered to make a secondary decision and output an adjusted scheduling command for re-verification.

[0013] Preferred, a virtual power plant energy storage scheduling optimization system based on deep reinforcement learning includes: The data acquisition and preprocessing module is equipped with interfaces that support multiple communication protocols such as Modbus, MQTT, and IEC104. It is used to connect to distributed energy monitoring systems, user load acquisition terminals, energy storage battery management systems, and power grid dispatching platforms. It collects multi-dimensional source and load operation data, power grid dispatching instructions, and rated parameters and real-time status data of each energy storage node within the coverage area of ​​the virtual power plant. It preprocesses all the collected data to obtain a standardized time-series dataset. The source-load probability prediction module has a built-in multimodal source-load probability prediction model that has been trained and converged using pre-labeled historical data. It is used to output the probability range of distributed power output and the probability range of user load fluctuation within a future preset scheduling period after inputting a standardized time series dataset. The scheduling optimization decision module has a built-in deep reinforcement learning scheduling model with a dual-delay deep deterministic strategy gradient TD3 architecture adapted to virtual power plant energy storage scheduling scenarios. The state space input of the deep reinforcement learning scheduling model includes the output probability range of distributed power sources, the user load fluctuation probability range, the real-time status data of each energy storage node, and the grid scheduling instructions. The action space is the charging and discharging power adjustment of each energy storage node. The reward function is a weighted comprehensive reward function that couples scheduling economy, grid stability, and energy storage life cycle loss, and is used to output the optimal charging and discharging scheduling instructions for each energy storage node. The instruction issuance and execution module is equipped with an encrypted communication link, which is used to issue the optimal charge and discharge scheduling instruction with timestamp and check code to the corresponding energy storage node for execution, and supports real-time feedback of instruction execution status.

[0014] Preferably, the system is also equipped with a time-series database storage module, which uses InfluxDB time-series database to store all historical operation data and scheduling logs. The data storage period is no less than 5 years, and it supports fast querying of historical data by time dimension and node ID dimension. The deep reinforcement learning scheduling model is deployed on the edge computing node of the virtual power plant, without the need to upload the full amount of operation data to the cloud. The end-to-end response latency of scheduling instructions is less than 100 milliseconds, which meets the real-time scheduling requirements.

[0015] Preferably, the source-load probability prediction module is also equipped with a third-party data interface for real-time connection to the public data interface of the meteorological department, the real-time peak-valley electricity price interface of the power trading platform, and the subsidy signal interface of the demand-side response management platform. The update frequency of the third-party data is matched with the scheduling step size, serving as a supplementary input feature for the multimodal source-load probability prediction model and improving the accuracy of the prediction results.

[0016] Preferably, the system is also equipped with a dispatch command security verification module, which has a built-in multi-verification rule library that can be updated online. This module is used to determine whether the charging and discharging dispatch command exceeds the rated charging and discharging power range of the corresponding energy storage node and whether it meets the grid anti-islanding protection requirements. The verification process adopts a parallel computing mode, and the verification time is less than 10 milliseconds, which does not affect the overall dispatch response speed. If the verification fails, in addition to triggering secondary dispatch, an alarm message will be generated and pushed to the virtual power plant operation and management personnel.

[0017] Compared with existing technologies, the beneficial effects of this invention are: This invention outputs the probability range of source-load fluctuations within a preset scheduling period through a multimodal source-load probability prediction model. It combines a deep reinforcement learning model with a dual-delay deep deterministic strategy gradient architecture adapted to continuous action space to make scheduling decisions. It does not require pre-setting fixed scheduling rules and can adaptively match the random fluctuation characteristics of distributed power output and user load. Under the premise of ensuring that the power fluctuation at the grid connection point always meets the grid scheduling requirements, it maximizes the economy of the scheduling process and effectively solves the problems of insufficient flexibility of existing fixed rule scheduling schemes, inability to adapt to source-load uncertainty, high grid assessment risk and low operating benefits.

[0018] This invention employs a weighted dynamic comprehensive reward function that couples scheduling economy, grid stability, and energy storage lifecycle losses to construct the optimization objective of a deep reinforcement learning model. It also utilizes an offline pre-training plus online fine-tuning model training mode, resulting in fast model inference speed. This meets the real-time scheduling needs of virtual power plants and automatically adjusts the weights of multiple objectives under different operating scenarios to balance multi-dimensional scheduling requirements. This effectively mitigates the degradation rate of energy storage over long-term operation, reduces the total lifecycle asset investment cost of virtual power plants, and solves the problems of long solution time, inability to adapt to real-time scheduling requirements, and high asset operation and maintenance costs caused by the lack of energy storage loss optimization in traditional mathematical optimization scheduling methods.

[0019] This invention features a multi-protocol compatible data acquisition interface and a multi-level security verification mechanism, supporting local deployment on the edge side without uploading all operational data to the cloud. This reduces the risk of data leakage and ensures low latency in dispatch response. It is also adaptable to various operational scenarios such as conventional peak shaving and grid emergency response, and compatible with virtual power plant scenarios of different capacities and distributed energy ratios. It can be directly connected to existing mainstream energy storage battery management systems and grid dispatch platforms, with low transformation costs and high industry promotion value. Attached Figure Description

[0020] Figure 1 This is a full-process diagram of the overall operation of the virtual power plant energy storage dispatch optimization system proposed in this invention; Figure 2 This is a flowchart of the multi-source data multi-protocol acquisition and time-domain alignment preprocessing proposed in this invention; Figure 3 This is a flowchart of the hybrid feature input and multi-quantile source load probability prediction proposed in this invention; Figure 4 This is a logic diagram of the state input and reward function calculation of the TD3 deep reinforcement learning model proposed in this invention; Figure 5 This is a control diagram for the parallel security verification gateway and multi-level collaborative instruction issuance proposed in this invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Reference Figures 1 to 5 This invention discloses a virtual power plant energy storage scheduling optimization method based on deep reinforcement learning, comprising the following steps: S1. Through a multi-protocol compatible industrial-grade data acquisition gateway, it connects to the distributed energy monitoring system, user load acquisition terminal, energy storage battery management system (BMS), and power grid dispatching platform within the coverage area of ​​the virtual power plant. The distributed photovoltaic / wind power monitoring system adopts the Modbus RTU protocol with a sampling frequency of 1 minute. The user load acquisition terminal uses the MQTT protocol, with a sampling frequency of 5 minutes for industrial and commercial users and 30 minutes for residential users; The energy storage BMS adopts the IEC104 protocol with a sampling frequency of 1 minute; the power grid dispatching platform adopts a power-specific encrypted communication protocol with a sampling frequency of 15 minutes.

[0023] The collected multi-dimensional source and load operation data specifically include the DC side voltage and current of photovoltaic strings, the AC side output of inverters, the wind turbine speed and actual power generation, the production line load of industrial and commercial users, the load of public facilities, the total electricity load of residential users, and the reported demand-side response load reduction amount. The collected grid dispatch instructions specifically include monthly peak-shaving assessment indicators, 15-minute power fluctuation thresholds at grid connection points, peak-valley electricity price time period division rules, and emergency peak shaving / valley filling instructions. The collected energy storage node data specifically includes rated capacity, maximum continuous charge / discharge power, allowable SOC operating range (default 10%~90%), charge / discharge conversion efficiency, unit charge / discharge cycle life attenuation coefficient, current SOC value, real-time charge / discharge power, and battery health status (SOH) value. All collected raw data undergoes outlier removal, time-domain alignment, normalization, and missing value completion preprocessing to obtain a standardized time-series dataset with a 15-minute time step, where each sample contains 24 hours of historical time-series features.

[0024] S2. Input the standardized time-series dataset into the multimodal source-load probability prediction model that has been trained and converged using 3 years of labeled historical operating data. Output the probability range of distributed power output and the probability range of user load fluctuation under the P90 confidence level within a customizable preset scheduling period. The daily scheduling uses a 24-hour scheduling period by default, while the grid emergency response scenario uses a 1-hour scheduling period. The upper and lower limits of the intervals correspond to the maximum / minimum possible output and the maximum / minimum possible load under a 90% probability, respectively.

[0025] S3. Construct a deep reinforcement learning scheduling model with a dual-delay deep deterministic policy gradient (TD3) architecture adapted to the continuous action space scenario of virtual power plant energy storage scheduling. The deep reinforcement learning scheduling model includes 2 independent commentator networks, 1 actor network, 1 target actor network, and 2 target commentator networks. The state space input dimension is: scheduling cycle step size × number of source load features + number of energy storage nodes × number of state features + number of scheduling command features. Specific input features include the upper and lower limits of distributed power output range, the upper and lower limits of user load fluctuation range, the real-time SOC and SOH values ​​of each energy storage node, grid peak-shaving requirements, and real-time electricity price signals. The action space dimension is consistent with the number of energy storage nodes. The value range of each action is from the negative value of the rated discharge power of the corresponding energy storage node to the positive value of the rated charging power, where negative values ​​represent discharge and positive values ​​represent charging. The reward function is a weighted comprehensive reward function coupling scheduling economy, grid stability, and energy storage life cycle loss.

[0026] S4. Scheduling triggers are divided into two categories: timed triggers, which execute regular scheduling every 15 minutes, and event triggers, which are triggered immediately upon receiving an emergency grid scheduling command. Real-time data collected at the trigger time is input into the trained deep reinforcement learning scheduling model, which outputs an optimal charging / discharging scheduling command containing a unique energy storage node ID, execution start and end times, charging / discharging power values, and an MD5 checksum. This command is then sent to the corresponding energy storage BMS for execution via a communication link encrypted with the national cryptographic standard SM2. After execution, the BMS transmits the execution status back to the scheduling system in real time.

[0027] This invention also discloses the specific implementation logic of the preprocessing stage: For different data sources, corresponding 3σ criterion outlier judgment thresholds are set to remove outliers from the collected raw data. The judgment formula is as follows: Where μ is the average value of the data corresponding to the feature in the data source during the same period over the past 7 days. The currently collected feature values, The sample size is the data for the same period in the last 7 days. Exceeding If the range is considered an outlier, it will be removed directly.

[0028] Time-domain alignment is performed on heterogeneous data with sampling frequencies of 1 minute, 5 minutes, and 30 minutes according to a preset 15-minute scheduling step size. For 1-minute samples, the average value over the past 15 minutes is used as the feature value for that step size; for 5-minute samples, the average value of three sampling points is used; and for 30-minute samples, the same feature value is split into two adjacent step sizes. A min-max normalization method is used to map all numerical data to the [0,1] interval. The normalization formula is: Where x is the original feature value, This is the minimum value in the training set for this feature. This represents the maximum value in the training set for this feature. For missing data with a missing rate below 5%, weighted interpolation based on three adjacent sampling points is used for imputation. The interpolation formula is: ;in The timestamps of adjacent sampling points For timestamps of missing data, These are the feature values ​​of adjacent sampling points, and their weights are inversely proportional to the time distance from the sampling point to the missing location. The core implementation code snippet is as follows: def weighted_interpolate(miss_ts, neighbor_ts, neighbor_vals): # Calculate the weight of each adjacent point weights = [1 / abs(ts - miss_ts) for ts in neighbor_ts] total_weight = sum(weights) # Weighted interpolation to obtain missing values interpolate_val = sum([w*v for w,v in zip(weights, neighbor_vals)]) / total_weight return round(interpolate_val, 2) This invention also discloses the specific structure of a multimodal source-load probability prediction model: It employs a Transformer-quantile regression joint model that integrates a multi-head temporal attention mechanism. Numerical time-series data from the input side is directly input to the Transformer encoder. Categorical features (weather, holiday identifiers) are one-hot encoded and then concatenated with the encoder output, jointly inputting into the quantile regression branch. The computational logic of the multi-head temporal attention mechanism is as follows: Where Q is the query matrix, K is the key matrix, and V is the value matrix. For single-head attention, multi-head attention concatenates the outputs of eight independent attention heads to obtain the final attention feature. The model output side has three quantile regression branches, corresponding to quantiles τ=0.1, τ=0.5, and τ=0.9 respectively. The quantile loss function is: Where y is the actual output or load value, Let τ be the predicted value for the quantile τ, where τ is the quantile value. The predicted values ​​for τ=0.1 and τ=0.9 are respectively the lower and upper limits of the probability interval P90.

[0029] This invention also discloses the specific calculation logic of the weighted comprehensive reward function, and the calculation formula of the weighted comprehensive reward function R is as follows: ; in , , The dynamic weights of the three reward components are set to a default value in daily scheduling scenarios. =0.5、 =0.3、 =0.2, automatically adjusted to [value] in emergency response scenarios. =0.2、 =0.6、 =0.2, when the overall average SOH of energy storage is below 80%. Automatically increased to 0.4. The economic reward component is calculated using the following formula: ,in Revenue from electricity sales during the dispatch cycle, For electricity purchase costs, For energy storage operation and maintenance costs, This represents the theoretical maximum return within that period. The value range is [-1, 1]. The stability reward component is calculated using the following formula: ,in The power fluctuation value at the grid connection point over 15 minutes. If the fluctuation threshold required by the power grid is... but The value is 1. To suppress the reward component due to loss, the calculation formula is as follows: Where DOD is the depth of charge / discharge of the energy storage corresponding to the current scheduling command. The maximum depth of charge and discharge allowed for energy storage, if but The value is 1.

[0030] This invention also discloses the training logic of a deep reinforcement learning scheduling model: an offline pre-training plus online fine-tuning training mode is adopted. In the offline pre-training stage, a simulation scheduling environment is constructed using the historical operation data of a virtual power plant over the past 3 years. The simulation environment can simulate different source-load fluctuation scenarios, grid scheduling requirements, and energy storage attenuation status. During model training, the experience replay buffer size is set to 1 million records, the batch size is 256, the actor network learning rate is 1e-4, the critic network learning rate is 1e-3, the discount factor γ is 0.99, and the soft update coefficient τ is 0.005. After at least 1 million iterations of training, the comprehensive reward value fluctuates below 0.01 for 1000 consecutive rounds, and the model converges.

[0031] During the online fine-tuning phase, an update cycle of 7 days is used, employing actual running data from the most recent 28 scheduling cycles to perform small-step gradient updates on the model parameters. The learning rate during the fine-tuning phase is set to one-tenth of the learning rate during the offline pre-training phase, and only the parameters of the last two layers of the actor network are updated to avoid significant fluctuations in model parameters that could lead to scheduling anomalies. If the reward value is lower than the historical average for 10 consecutive scheduling cycles during the fine-tuning process, the fine-tuning is stopped, and the model is automatically rolled back to the stable parameters of the previous version.

[0032] This invention also discloses a security verification logic for the scheduling command: after step S4 outputs the optimal charging and discharging scheduling command, it further includes a three-level security verification step: The first level is single-node power verification, which determines whether the power value of a single charge and discharge scheduling command exceeds the actual allowable charge and discharge power range corresponding to the current SOH of the corresponding energy storage node. The actual allowable charge and discharge power is the rated power multiplied by the SOH coefficient. If it exceeds the limit, the verification is deemed to have failed. The second level is the grid connection point fluctuation verification. After calculating the total charging and discharging power of all energy storage nodes and superimposing the source and load prediction output of the P50 quantile, the 15-minute power fluctuation value of the grid connection point is checked to see if it is within the threshold range required by the grid. If it is exceeded, the verification is deemed to have failed. The third level is the anti-islanding verification. When a fault occurs on the grid side, it is determined whether the current dispatch command will cause the virtual power plant to disconnect from the main grid and still have power supply to form an island operation. If there is a risk, the verification is deemed to have failed.

[0033] All verification items must pass before being sent to the corresponding energy storage node for execution. If a verification fails, a secondary decision is triggered in the scheduling model. During the secondary decision, the corresponding constraint condition for the verification failure is added to the state space input, and the adjusted scheduling instruction is output for re-verification. If three consecutive verifications fail, a manual intervention alarm is triggered.

[0034] This invention discloses a virtual power plant energy storage scheduling optimization system based on deep reinforcement learning, comprising four core modules: The data acquisition and preprocessing module is equipped with an industrial-grade acquisition gateway that supports Modbus, MQTT, IEC104, and power-specific encryption protocols. The gateway hardware operates in a temperature range of -40℃ to 85℃ and supports power outage resume function. The acquired raw data is first cached in a local 16G memory cache, and then preprocessed. The preprocessed data is then synchronously pushed to the source-load probability prediction module and the time series database.

[0035] The source load probability prediction module has a built-in multimodal source load probability prediction model that has been trained and converged using labeled historical data. It is deployed in the GPU inference container of the edge computing node. The container is allocated 2 CPU cores, 8G of video memory, and 16G of RAM. The time taken for a single 24-hour prediction request is less than 50 milliseconds. The output probability interval data is pushed to the scheduling optimization decision module.

[0036] The scheduling optimization decision module has a built-in TD3 architecture deep reinforcement learning scheduling model adapted to virtual power plant energy storage scheduling scenarios. It is deployed in an independent container on the same edge node, with 4 CPU cores, 16G video memory, and 32G RAM. The time taken to infer a single scheduling request is less than 30 milliseconds, and the output scheduling instructions are pushed to the security verification module.

[0037] The instruction issuance and execution module is equipped with a dedicated encrypted VPN link for power. The instruction transmission adopts the national cryptographic SM2 encryption algorithm. After the instruction is issued, the module receives the execution status returned by the energy storage BMS in real time, and the execution status data is synchronously stored in the time series database.

[0038] This invention also discloses the system's storage and deployment configuration: the system is also equipped with a time-series database storage module, using InfluxDB Enterprise Edition time-series database, with 3 redundant nodes in the cluster, a storage capacity of 10T, and supports a write speed of 100,000 data entries per second. It is used to store all historical running data and scheduling logs, with a data storage period of no less than 5 years. It supports fast querying of historical data by time dimension, node ID dimension, and feature type dimension, and the query time for a single node's historical data spanning 1 year is less than 1 second.

[0039] The deep reinforcement learning scheduling model is deployed on the local edge computing node of the virtual power plant, eliminating the need to upload all operational data to the cloud, thus avoiding data transmission delays and data leakage risks. The end-to-end response latency of scheduling commands is less than 100 milliseconds, meeting the requirements for real-time scheduling. The edge node is configured with redundant power supplies and redundant 5G / fiber dual networks, achieving an availability of 99.99%.

[0040] In this invention, the third-party data access logic of the source-load probability prediction module is also disclosed: the source-load probability prediction module is also configured with a third-party data interface with a RESTful architecture, which supports HTTPS encrypted transmission and is used to connect in real time to the public data interface of the meteorological department, the real-time peak-valley electricity price interface of the power trading platform, and the subsidy signal interface of the demand-side response management platform. The update frequency of the third-party data is matched with the scheduling step size of 15 minutes. Meteorological data includes 15-minute forecasts of sunshine, wind speed, and temperature for the next 24 hours; electricity price data includes 15-minute peak and off-peak electricity prices for the next 24 hours; and subsidy signals include subsidy standards and response periods for demand-side response. After standardization, the above third-party data serves as supplementary input features for the multimodal source-load probability prediction model, which can improve the prediction accuracy by more than 10%.

[0041] This invention also discloses the specific configuration of the scheduling instruction security verification module: the system is also equipped with a scheduling instruction security verification module, which has a built-in multi-verification rule library that can be updated online. The rule library supports operation and management personnel to manually add or modify verification rules. After the rules are updated, they can take effect without restarting the system. The verification process adopts a parallel computing mode, with three verification steps executed simultaneously. The verification time is less than 10 milliseconds, which does not affect the overall scheduling response speed.

[0042] When the verification fails, in addition to triggering secondary scheduling, an alarm message containing the reason for the failure, the ID of the energy storage node involved, and the content of the instruction will be generated. The alarm message will be pushed to the virtual power plant operation and management personnel via SMS and system pop-up. The alarm message will also be stored in the log database for subsequent traceability and analysis.

[0043] Example Example 1: Daily Peak Shaving Scenario of a Virtual Power Plant in an Industrial Park A virtual power plant in a provincial industrial park covers 12MW of distributed photovoltaic power, 8MW / 16MWh of lithium iron phosphate energy storage, and 30MW of industrial and commercial load. The grid requires that the power fluctuation at the grid connection point not exceed 2MW within 15 minutes. The daily dispatching goal is to maximize the benefits of peak shaving and valley filling while reducing energy storage losses.

[0044] The system triggers scheduling every 15 minutes, collecting real-time photovoltaic output, user load, energy storage SOC / SOH, and grid peak-valley electricity price signals. These signals are then input into a multimodal source-load probability prediction model to obtain the P90 photovoltaic output range and load range for the next 24 hours. This data is then input into the TD3 scheduling model, using the weight settings from daily scheduling. It outputs charging and discharging commands for each energy storage node, which are then sent out for execution after passing through three levels of safety verification.

[0045] The core operation flow is as follows: data acquisition → preprocessing → source load prediction → scheduling decision → verification → distribution → execution feedback. The core scheduling trigger code snippet is as follows: import schedule import time def regular_dispatch_task(): # Collect all real-time data real_time_data = data_collect_gateway.get_all_real_data() # Data preprocessing yields a standardized dataset norm_data = preprocess_module.process(real_time_data) # Source load probability prediction yields the interval pred_interval = predict_module.predict_24h(norm_data) # Deep reinforcement learning generates scheduling instructions dispatch_cmd = td3_model.get_optimal_cmd(pred_interval, real_time_data['storage_state'], real_time_data['grid_order']) # Level 3 Security Verification if verify_module.check_all(dispatch_cmd): # Issue instructions exec_module.send_encrypted_cmd(dispatch_cmd) # Perform a routine scheduling every 15 minutes schedule.every(15).minutes.do(regular_dispatch_task) while True: schedule.run_pending() time.sleep(1) Compared to traditional scheduling methods based on fixed rules, this scenario eliminates the need for manual adjustments to scheduling rules for different seasons and weather conditions. It can automatically adapt to random fluctuations in source loads, increasing peak shaving and valley filling benefits by more than 15% within the scheduling cycle. The energy storage charging and discharging depth is controlled within a reasonable range, extending cycle life by 20%. Power fluctuations at the grid connection point meet grid requirements throughout the process, without incurring any performance penalties.

[0046] Example 2: Power Grid Emergency Demand Response Scenario Application scenario description: During the summer peak electricity consumption period, a power grid in a certain region experienced a power supply shortage of 100MW. The power grid issued an emergency peak shaving command to the virtual power plant in the industrial park, requiring a reduction of 5MW of load within 10 minutes. The dispatch priority was to first meet the grid stability requirements, followed by consideration of economic efficiency and energy storage losses.

[0047] refer to Figure 1 This diagram illustrates the complete closed-loop business flow of a virtual power plant energy storage scheduling optimization system based on deep reinforcement learning from a macro-control perspective. The entire operational lifecycle is connected by five core modules or stages: First, the data acquisition and preprocessing module synchronously accesses multi-dimensional heterogeneous time-series parameters from sources, loads, grids, and storage through multi-protocol compatible interfaces at high frequency, and outputs standardized time-series datasets after cleaning and alignment; Secondly, the source-load probability prediction module inputs the dataset into the time attention mechanism model to predict the probability range of distributed power output and user load fluctuations in the future scheduling cycle. Subsequently, the scheduling optimization decision module takes the prediction range, energy storage status and grid instructions as input to the state space, and performs reinforcement learning through a dual-delay deterministic policy gradient architecture to solve the charging and discharging scheduling instructions of each energy storage node. Next, the security verification module performs multi-rule interception and calibration at the initial instruction execution node level, grid connection point level, and anti-islanding level. If the command fails to meet the requirements, the model will trigger a secondary decision. Finally, the optimal scheduling instruction that has been verified to be compliant is accurately distributed to each terminal by the execution module through an encrypted link, and is synchronously fed back to the local time series database and the cloud management platform to achieve long-term closed-loop control.

[0048] refer to Figure 2 This diagram details the control logic within the front-end data acquisition and preprocessing module, including heterogeneous data extraction, noise filtering, and time-domain normalization and alignment. The data acquisition and preprocessing module simultaneously interfaces with four heterogeneous data sources: distributed energy monitoring systems, load terminals, energy storage management systems, and power grid dispatching platforms, encompassing IoT protocols, industrial protocols, and power industry-specific protocols.

[0049] Due to the significant time overlap in the sampling frequencies of various hardware terminals, the system first dynamically adjusts the three-σ criterion threshold based on the characteristics of different channels to eliminate isolated noise points and abrupt interference. For missing data with a proportion less than a very small percentage, the system employs an inverse distance-weighted interpolation algorithm that is inversely proportional to the time distance for lossless completion.

[0050] Subsequently, the system uses a fixed scheduling step size as the clock reference to perform spatial and temporal forced alignment on time-series streams with different frequencies such as one minute, five minutes, and thirty minutes. Finally, it normalizes the numerical parameters to the same normalization interval through the extreme value mapping operator and outputs a standardized multidimensional time-series feature matrix.

[0051] refer to Figure 3 This figure specifically analyzes the internal software calculation path of the source-load probability prediction module, which deeply integrates numerical time-series features with categorical meteorological features after unique thermal encoding, and outputs probability intervals through quantile regression structures.

[0052] To improve the accuracy of predicting new energy output, the prediction module adopts a dual-channel feature input architecture: the first channel inputs the aforementioned standardized numerical time-series data; the second channel connects in real time to a third-party meteorological and power platform to perform one-heat encoding conversion on the captured nonlinear features such as light intensity, wind speed, ambient temperature, and holidays and major event days.

[0053] After the two features are concatenated, they are fed into the model that integrates a multi-head time attention mechanism for forward inference. The model captures the intrinsic relationship between long-term historical patterns and sudden weather variations through the attention mechanism. Finally, through the three independent quantile regression branches at the bottom layer, the multi-quantile output and load prediction values ​​representing different probability boundaries in the future scheduling cycle are calculated in parallel. The upper and lower limits of these values ​​are aligned and combined to form a highly reliable source-load fluctuation probability interval.

[0054] refer to Figure 4 This figure illustrates how a deep reinforcement learning scheduling model deployed on edge computing nodes achieves adaptive iterative control logic through multi-dimensional state space perception, continuous action space mapping, and a three-factor weighted comprehensive reward function.

[0055] The model's state space is highly integrated, receiving real-time inputs from the front-end, including source and load output, upper and lower limits of load probability ranges, remaining lifetime and current state of charge of each energy storage node, and macro-level peak-shaving assessment commands from the power grid. The model's action space is mapped to continuously adjustable physical quantities, allowing for stepless adjustment of the charging and discharging power of each energy storage node.

[0056] To guide the model toward the optimal scheduling path, the reward solver calculates three core reward components throughout the entire time period: the economic component generated by the difference between the purchase and sale price of electricity, the grid stability component that suppresses power fluctuations at the grid connection point, and the loss suppression component calculated based on the lifetime decay coefficient.

[0057] The system dynamically adjusts the weights of the three factors based on the current urgency of the power grid, driving the overall control strategy to achieve long-term self-evolution through pre-training in the simulation environment and periodic small-step online fine-tuning.

[0058] refer to Figure 5 This diagram details the parallel security verification gateway quality control process that the optimal control command undergoes before its formal issuance, as well as the closed-loop reconstruction response mechanism for abnormal situations. When the reinforcement learning model outputs the initial scheduling command in 15-minute increments or under emergency triggering conditions, the command is intercepted and pushed into the security verification module.

[0059] The verification module employs a high-performance parallel computing mode, simultaneously executing triple protection network rules within an extremely short latency: The first verification step is to check whether the single-node command exceeds the actual power safety limit allowed by the current health status of the energy storage unit. The second verification is to check whether the overall fluctuation of the grid connection point after the sum of the total energy storage power of the entire network and the source load output is compliant. The third layer of verification checks for the risk of triggering malicious isolated operation. If all criteria are met, the security lock is released, and the optimal command is sent and executed via an encrypted link carrying a clock stamp and checksum, and recorded in the time-series database. If any item triggers an out-of-bounds interception, the system immediately locks the sending channel, pushes a visual alarm to the management terminal, and forces the scheduling model to enter a secondary strategy decision-making process, reconstructing the control command until it is completely secure.

[0060] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A virtual power plant energy storage dispatch optimization method based on deep reinforcement learning, characterized in that, Includes the following steps: S1. Collect multi-dimensional source and load operation data, power grid dispatch instructions, and rated parameters and real-time status data of each energy storage node within the coverage area of ​​the virtual power plant. Preprocess all collected data to obtain a standardized time-series dataset. S2. Input the standardized time-series dataset into the pre-trained multimodal source-load probability prediction model and output the probability range of distributed power output and the probability range of user load fluctuation within the future preset scheduling period. S3. Construct a deep reinforcement learning scheduling model adapted to the virtual power plant energy storage scheduling scenario. The state space input of the deep reinforcement learning scheduling model includes the output probability range of distributed power sources, the user load fluctuation probability range, the real-time status data of each energy storage node, and the grid scheduling instructions. The action space is the charging and discharging power adjustment of each energy storage node. The reward function is a weighted comprehensive reward function that couples scheduling economy, grid stability, and energy storage life cycle loss. S4. Input the real-time collected scheduling trigger data into the trained deep reinforcement learning scheduling model, output the optimal charging and discharging scheduling instructions for each energy storage node, and send them to the corresponding energy storage node for execution.

2. The virtual power plant energy storage scheduling optimization method based on deep reinforcement learning according to claim 1, characterized in that, The preprocessing in step S1 specifically includes: removing outliers from the collected raw data based on the 3σ criterion; aligning source load data and energy storage data with the time domain at different sampling frequencies according to a preset scheduling step size; scaling all data using the minimum-maximum normalization method; and completing missing data using adjacent sliding window weighted interpolation.

3. The virtual power plant energy storage scheduling optimization method based on deep reinforcement learning according to claim 1, characterized in that, The multimodal source-load probability prediction model in step S2 adopts a Transformer-quantile regression model with an attention fusion mechanism. The input side of the model simultaneously receives numerical time-series data and weather and holiday category features, and outputs the output and load probability intervals at different confidence levels.

4. The virtual power plant energy storage scheduling optimization method based on deep reinforcement learning according to claim 1, characterized in that, The calculation logic of the weighted comprehensive reward function in step S3 is as follows: First, the difference between the electricity purchase and sale revenue and the energy storage operation and maintenance cost within the scheduling cycle is calculated to obtain the economic reward component. The deviation between the power fluctuation at the grid connection point and the preset threshold is calculated to obtain the stability reward component. The mapping loss value between the energy storage charging and discharging depth and the rated cycle life is calculated to obtain the loss suppression reward component. Dynamic weights are assigned to the three components respectively, and then the sum is obtained to obtain the comprehensive reward value.

5. The virtual power plant energy storage scheduling optimization method based on deep reinforcement learning according to claim 1, characterized in that, The deep reinforcement learning scheduling model adopts an offline pre-training plus online fine-tuning training mode. In the offline pre-training stage, the model convergence is completed using the historical operation data of the virtual power plant over the past three years. In the online fine-tuning stage, the model parameters are updated with small-step gradients using the actual operation data of the most recent N scheduling cycles at preset update intervals.

6. The virtual power plant energy storage scheduling optimization method based on deep reinforcement learning according to claim 1, characterized in that, After outputting the optimal charging and discharging scheduling command in step S4, the command security verification step is also included to determine whether the charging and discharging scheduling command exceeds the rated charging and discharging power range of the corresponding energy storage node and whether it meets the grid anti-islanding protection requirements. After the verification is passed, the command is sent to the corresponding energy storage node for execution.

7. A virtual power plant energy storage scheduling optimization system based on deep reinforcement learning, used to implement the virtual power plant energy storage scheduling optimization method based on deep reinforcement learning as described in any one of claims 1-6, characterized in that, include: The data acquisition and preprocessing module is used to collect multi-dimensional source and load operation data, power grid dispatch instructions, and rated parameters and real-time status data of each energy storage node within the coverage area of ​​the virtual power plant. All the collected data are preprocessed to obtain a standardized time-series dataset. The source load probability prediction module has a built-in pre-trained multimodal source load probability prediction model, which is used to output the probability range of distributed power output and the probability range of user load fluctuation within the future preset scheduling period after inputting a standardized time series dataset. The scheduling optimization decision module has a built-in deep reinforcement learning scheduling model adapted to the virtual power plant energy storage scheduling scenario. The state space input of the deep reinforcement learning scheduling model includes the output probability range of distributed power sources, the user load fluctuation probability range, the real-time status data of each energy storage node, and the grid scheduling instructions. The action space is the charging and discharging power adjustment of each energy storage node. The reward function is a weighted comprehensive reward function that couples scheduling economy, grid stability, and energy storage life cycle loss, and is used to output the optimal charging and discharging scheduling instructions for each energy storage node. The instruction issuance and execution module is used to issue the optimal charge and discharge scheduling instruction to the corresponding energy storage node for execution.

8. The virtual power plant energy storage scheduling optimization system based on deep reinforcement learning according to claim 7, characterized in that, The system is also equipped with a time-series database storage module for storing all historical operating data and scheduling logs. The deep reinforcement learning scheduling model is deployed on the edge computing node of the virtual power plant, and the end-to-end response latency of the scheduling command is lower than a preset threshold.

9. The virtual power plant energy storage scheduling optimization system based on deep reinforcement learning according to claim 7, characterized in that, The source-load probability prediction module is also equipped with a third-party data interface for real-time acquisition of weather forecast data, power grid peak-valley electricity price signals, and demand-side response subsidy signals, which serve as supplementary input features for the prediction model.

10. The virtual power plant energy storage scheduling optimization system based on deep reinforcement learning according to claim 7, characterized in that, The system is also equipped with a scheduling instruction security verification module, which is used to determine whether the charging and discharging scheduling instruction exceeds the rated charging and discharging power range of the corresponding energy storage node and whether it meets the grid anti-islanding protection requirements. After the verification is passed, it is pushed to the instruction issuance and execution module.

Images