Industrial and commercial energy storage demand real-time scheduling method and system based on reinforcement learning
By constructing a reinforcement learning scheduling method with a hierarchical constraint space and a dual-timescale action space, the problem of bus voltage drop caused by millisecond-level power surges in GPU load is solved, achieving a balance between highly reliable power supply and optimal cost-effectiveness for AI computing centers.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Filing Date
- 2026-04-20
- Publication Date
- 2026-07-14
AI Technical Summary
Existing energy storage scheduling methods cannot cope with bus voltage drops caused by millisecond-level power surges from GPU loads, and it is difficult to balance high-reliability power supply with optimal economic efficiency.
By constructing a hierarchical constraint space that includes power change rate constraints, and using a dual-timescale action space to generate scheduling instructions, combined with high-frequency sampling and feature extraction, state space construction, hierarchical constraint management, and reinforcement learning decision-making, real-time tracking and fast power compensation of GPU load are achieved.
It effectively solves the risk of power outages caused by power surges in AI computing centers, achieves adaptive optimization of economy and reliability under different operating conditions, and improves the overall utilization efficiency of energy storage systems.
Smart Images
Figure CN122394026A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent scheduling system technology, and more specifically, to a method and system for real-time scheduling of industrial and commercial energy storage demand based on reinforcement learning. Background Technology
[0002] With the rapid development of artificial intelligence technology, AI computing centers centered on GPU servers have become a significant growth driver for industrial and commercial power loads. Large-scale AI training tasks typically employ synchronous batch computing, with GPU clusters periodically switching between computing and communication phases. This results in dramatic fluctuations in system-level power over millisecond timescales—actual data shows that the difference between power lows and highs can exceed 100%, and power ramp-up rates can reach over 100 kW / s. These millisecond-level, high-amplitude power surges are directly transmitted to the power supply and distribution system, causing instantaneous voltage drops on the bus. When the voltage drop exceeds the undervoltage protection threshold of the server power module (typically 10% to 15% of the rated voltage), it triggers a power outage and restart of the GPU server, leading to training task interruption, loss of model parameters, and even hardware damage. For Tier III / Tier IV data centers, the economic losses from a single unexpected power outage can reach hundreds of thousands to millions of yuan.
[0003] Existing energy storage scheduling methods struggle to address the aforementioned issues. Traditional static threshold scheduling methods rely solely on State of Charge (SOC) as the scheduling basis, failing to detect power change rates and lacking any response capability to millisecond-level shocks. While existing reinforcement learning-based energy storage scheduling schemes introduce SOC constraints and penalties, their state space only includes slow variables such as SOC and electricity price, and their action space is on a single time scale (seconds or minutes), unable to output millisecond-level power corrections. Their constraints are primarily energy-based, lacking hard constraints on power change rates; their reward functions focus on economic optimization, with power supply reliability only serving as a soft penalty, potentially violating constraints under extreme shocks. Therefore, achieving real-time tracking and rapid power compensation for millisecond-level power shocks to GPU loads in energy storage systems, while ensuring optimal economic efficiency under high-reliability power supply requirements, is a pressing technical problem in this field. To address this, we propose a reinforcement learning-based real-time scheduling method and system for industrial and commercial energy storage demand. Summary of the Invention
[0004] The purpose of this invention is to provide a real-time scheduling method and system for industrial and commercial energy storage demand based on reinforcement learning, so as to solve the technical problems that existing energy storage scheduling methods cannot cope with the bus voltage drop caused by the millisecond-level power surge of GPU load, and it is difficult to balance high reliable power supply and optimal economic efficiency.
[0005] To address the aforementioned technical problems, this invention provides the following technical solution: a real-time scheduling method for industrial and commercial energy storage demand based on reinforcement learning, comprising the following steps: S1: Collect the instantaneous power signal of the GPU server, and calculate the load power, power change rate, and time-series waveform characteristics of the load power based on the instantaneous power signal; S2: The time-series waveform characteristics of the load power, the state of charge of the energy storage system, and the electricity price signal are jointly constructed into a state space for reinforcement learning; S3: Construct a hierarchical constraint space containing at least two priority levels, where the highest priority level corresponds to the GPU core computing cluster. The constraint conditions of this level include a power change rate constraint, which requires that the instantaneous discharge power change rate of the energy storage system is not less than the predicted value of the GPU load power change rate. S4: The scheduling instructions are generated using a dual-timescale action space. The dual-timescale action space includes: the energy storage reference charging and discharging power output by the main strategy network at the first time scale, and the power correction amount output by the auxiliary controller at the second time scale. The total output power is obtained by superimposing the energy storage reference charging and discharging power and the power correction amount. S5: Train the reinforcement learning agent based on a preset reward function, the reward function including a power supply reliability reward item and a power tracking accuracy reward item; S6: Control the charging and discharging power of the energy storage converter based on the total output power of the trained reinforcement learning agent.
[0006] This invention first collects the instantaneous power signal of the GPU server at a microsecond-level sampling frequency, calculates the load power and its power change rate, and incorporates the temporal waveform characteristics of the power change rate into the reinforcement learning state space, enabling the agent to perceive millisecond-level power fluctuation trends. Second, it sets a dedicated power change rate constraint for the GPU core computing cluster in a hierarchical constraint space, requiring that the instantaneous discharge power change rate of the energy storage system is not less than the predicted value of the GPU load power change rate, thus forcing the energy storage to have sufficient power ramp-up capability from a constraint perspective. Finally, it adopts a dual-timescale architecture that uses the main strategy network to output a second-level benchmark charge and discharge power and the auxiliary controller to output a millisecond-level power correction. The total output power can follow the millisecond-level fluctuations of the GPU load in real time, solving the problem of power outage risk caused by power surges in AI computing centers.
[0007] Preferably, the timing waveform characteristics of the load power in S1 are extracted by a sliding window, the sliding window contains a preset number of continuous sampling points, the sampling frequency of the sampling points is on the order of microseconds or milliseconds, and the power change rate is calculated by dividing the power difference between adjacent sampling points by the sampling interval. The power change rate is calculated using the following formula: ; In the formula, For a moment The rate of change of load power; For a moment The instantaneous load power of the GPU server was collected; This represents the load power at the previous sampling time. The sampling interval is denoted as .
[0008] Preferably, the hierarchical constraint space in S3 includes three priority levels: the first priority level corresponds to the GPU core computing cluster, and its constraints include power change rate constraints and state of charge threshold constraints; the second priority level corresponds to conventional IT equipment, and its constraints include state of charge threshold constraints; the third priority level corresponds to auxiliary equipment, and its constraints include energy storage constraints.
[0009] Preferably, the specific form of the power change rate constraint in S3 is: the ratio of the absolute value of the power change of the energy storage system within a unit control cycle to the control cycle is not less than the product of the predicted value of the GPU load power change rate and the tracking coefficient, wherein the tracking coefficient is dynamically adjusted according to the current grid stability state and the degree of GPU load fluctuation. The power change rate constraint is expressed by the following inequality: ; In the formula, For a moment Total output power of the energy storage system; This indicates that a control cycle has elapsed. The total output power of the post-energy storage system; To control the cycle; For a moment The tracking coefficient; For a moment The predicted rate of change of GPU load power.
[0010] Preferably, in S4, the main strategy network adopts a deep deterministic strategy gradient algorithm or a variant thereof, and outputs the energy storage benchmark charging and discharging power with a decision cycle of seconds; the auxiliary controller is a fast controller based on model predictive control or a proportional-integral-derivative plus feedforward compensation controller, and outputs the power correction amount with a control cycle of milliseconds. The response delay of the auxiliary controller is less than the control cycle of the energy storage converter. The auxiliary controller uses the instantaneous power of the GPU load as the tracking target and the power change rate capability of the energy storage converter as a constraint to output a power correction amount.
[0011] Preferably, after generating scheduling instructions using a dual-timescale action space in step S4, a power buffer reservation step is further included: dynamically calculating the power buffer reservation amount based on the predicted value of the GPU load power change rate and the power response time of the energy storage system; and pre-locking the power buffer reservation amount from the available power of the energy storage system, specifically for dealing with instantaneous power surges in the GPU load. The power buffer reserve is calculated using the following formula: ; In the formula, For a moment The power buffer reserve needs to be locked in advance from the energy storage system; For safety factor; For a moment The predicted rate of change of GPU load power; This refers to the power response time of the energy storage system.
[0012] Preferably, the power buffer reservation step further includes: obtaining GPU training task scheduling information through an interface with an AI scheduling platform, the scheduling information including task start time, expected load curve and task priority; and, based on the scheduling information, increasing the power buffer reservation amount in advance before the GPU training task starts, and automatically releasing the power buffer reservation amount after the task ends. The pre-boosted power buffer reserve is calculated using the following formula: ; In the formula, This indicates the power buffer reserve after pre-reservation enhancement; This is the pre-reservation coefficient; This represents the predicted rate of change in GPU load power at the time of task startup; This allows for advance planning and preparation.
[0013] Preferably, in step S4, a hierarchical controller architecture is used to execute scheduling instructions. The hierarchical controller architecture includes: a millisecond-level power tracking layer, used to track the instantaneous power of the GPU load with a millisecond-level response delay and output the power correction amount; a second-level strategy optimization layer, used to update the energy storage benchmark charging and discharging power with a second-level cycle to optimize economy; and a minute-level constraint update layer, used to update the constraint parameters of the hierarchical constraint space with a minute-level cycle. The hierarchical controller architecture also includes a priority arbitration mechanism: when the energy storage reference charging and discharging power output by the second-level strategy optimization layer conflicts with the power correction amount required by the millisecond-level power tracking layer, the millisecond-level power tracking layer takes over the control right according to the principle that the power change rate constraint corresponding to the first priority level takes precedence over economic scheduling.
[0014] Preferably, the power supply reliability reward item in S5 is negatively correlated with the measured value of the bus voltage drop. When the bus voltage drop exceeds the preset protection threshold, a large negative reward is given or the current training round is terminated. The power tracking accuracy reward item is negatively correlated with the tracking error between the actual output power of the energy storage and the GPU load power. The power supply reliability bonus is calculated using the following formula: ; In the formula, This is an award for power supply reliability. This is the penalty coefficient; The measured value is the bus voltage drop. The preset protection threshold; when hour, Set to negative infinity or terminate the current training round; The power tracking accuracy bonus is calculated using the following formula: ; In the formula, Awarded for power tracking accuracy; This is the tracking error penalty coefficient; For a moment Total output power of the energy storage system; For a moment The instantaneous load power of the GPU server was collected.
[0015] A real-time scheduling system for industrial and commercial energy storage based on reinforcement learning, comprising: The high-frequency sampling and feature extraction module is used to acquire the current and voltage signals of the GPU server power supply circuit at a sampling frequency of microseconds or milliseconds, and calculate the instantaneous power and power change rate based on the current and voltage signals; The state space construction module is used to jointly construct the state space for reinforcement learning by combining the time-series waveform characteristics of the instantaneous power, the power change rate, the state of charge of the energy storage system, and the electricity price signal. The hierarchical constraint management module is used to construct a hierarchical constraint space containing at least two priority levels, where the highest priority level corresponds to the GPU core computing cluster, and the constraints at this level include power change rate constraints. The reinforcement learning decision module includes a main policy network and an auxiliary controller. The main policy network outputs the energy storage reference charge and discharge power at a first time scale, and the auxiliary controller outputs the power correction amount at a second time scale. The total output power is obtained by superimposing the energy storage reference charge and discharge power and the power correction amount. The power buffer reservation module is used to dynamically calculate the power buffer reservation amount based on the predicted value of the GPU load power change rate, and to lock the power buffer reservation amount in advance from the available power of the energy storage system. The execution module is used to control the charging and discharging power of the energy storage converter according to the total output power.
[0016] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention effectively solves the technical problem that existing scheduling methods cannot cope with the voltage drop of the bus caused by millisecond-level power surges in GPU load by constructing a hierarchical constraint space containing power change rate constraints and generating scheduling instructions using a dual-timescale action space. Specifically, this invention first collects the instantaneous power signal of the GPU server at a microsecond-level sampling frequency, calculates the load power and its power change rate, and incorporates the temporal waveform characteristics of the power change rate into the reinforcement learning state space, enabling the agent to perceive the millisecond-level power fluctuation trend. Second, a dedicated power change rate constraint is set for the GPU core computing cluster in the hierarchical constraint space, requiring that the instantaneous discharge power change rate of the energy storage system is not less than the predicted value of the GPU load power change rate, thus forcing the energy storage to have sufficient power ramp-up capability from the constraint level. Finally, a dual-timescale architecture is adopted, in which the main policy network outputs a second-level benchmark charging and discharging power and the auxiliary controller outputs a millisecond-level power correction. The total output power can follow the millisecond-level fluctuations of the GPU load in real time, solving the problem of power outage risk caused by power surges in AI computing centers.
[0017] 2. This invention also addresses the technical challenge of balancing reliability and economy by designing a dynamically weighted multi-objective reward function. Specifically, the reward function includes a power supply reliability reward, a power point tracking accuracy reward, and an economic reward. The reliability reward uses an exponential function, with a sharp increase in penalty when the bus voltage drops close to the protection threshold, driving the agent to actively move away from dangerous areas. The economic reward calculates peak-valley arbitrage profits based on time-of-use pricing and subtracts battery aging costs. The three reward items are summed using dynamic weighting coefficients, which are adjusted in real-time based on grid stability and GPU load power change rate predictions. When load fluctuations are severe, the reliability weight is automatically increased to prioritize power supply safety; when the load is stable, the economic weight is automatically increased to fully utilize price differences for arbitrage. This dynamic trade-off mechanism enables the energy storage system to automatically select the optimal scheduling strategy under different operating conditions. Compared to existing fixed-weight schemes, it improves annualized arbitrage profits while reducing battery aging costs at the same level of reliability, achieving a balance between high reliability and high economy.
[0018] 3. This invention further addresses the issues of differentiated protection for multi-priority loads and proactive response to impact loads by constructing a three-tiered constraint space comprising GPU core computing clusters, conventional IT equipment, and auxiliary equipment, and combining this with the task-aware pre-reserve mechanism of the AI scheduling platform. Specifically, this invention sets the highest priority for GPU core computing clusters, with constraints including both power change rate constraints and state of charge threshold constraints, ensuring priority response to millisecond-level power surges; sets a second-highest priority for conventional IT equipment, with constraints primarily based on state of charge thresholds; and sets the lowest priority for auxiliary equipment, requiring only that energy reserve constraints be met. This hierarchical design avoids resource waste or insufficient protection caused by high-level and low-level loads sharing the same constraint threshold. Simultaneously, this invention obtains information such as the start time and expected load curve of GPU training tasks through an interface with the AI scheduling platform, pre-increasing the power buffer reserve before task start, enabling the energy storage system to provide sufficient power support the instant an impact occurs. This mechanism upgrades passive response scheduling to active perception scheduling, significantly reducing the voltage drop during the initial impact and avoiding economic losses caused by high reserves throughout the day. Under the same power supply reliability, this invention can release additional energy storage capacity for peak-valley arbitrage compared to a single threshold scheme, significantly improving the overall utilization efficiency of the energy storage system. Attached Figure Description
[0019] Figure 1 This is a flowchart illustrating the overall process of real-time scheduling of industrial and commercial energy storage demand based on reinforcement learning, as described in this invention. Figure 2 This is a diagram of the dual-time-scale hierarchical controller architecture of the present invention; Figure 3 This is a flowchart of the power buffer reservation and task pre-reserve mechanism of the present invention; Figure 4 This is a graph of the reinforcement learning training closed loop and multi-objective reward function of the present invention; Figure 5 This is a schematic diagram of the overall framework of the industrial and commercial energy storage real-time dispatching system of the present invention. Detailed Implementation
[0020] Example 1: As Figures 1 to 4 As shown, the present invention relates to a real-time scheduling method for industrial and commercial energy storage demand based on reinforcement learning, comprising the following steps: S1: Collect the instantaneous power signal of the GPU server, and calculate the load power, power change rate, and time-series waveform characteristics of the load power based on the instantaneous power signal; In an embodiment of the present invention, the instantaneous power signal acquisition of the GPU server in step S1 specifically involves: connecting a high-precision Hall effect current sensor in series at the power input terminal of the GPU server, and simultaneously connecting a voltage divider sampling circuit in parallel to synchronously acquire current and voltage signals at a sampling frequency of not less than 100kHz, and using the instantaneous product of current and voltage as the instantaneous power signal. Alternatively, the real-time power consumption register value of the GPU chip can be read through the GPU server baseboard management controller (BMC) to obtain power sampling data in milliseconds. When using a sliding window to extract timing waveform features, the window length is set to 10 to 50 sampling points, and the window sliding step is 1 sampling point.
[0021] In an embodiment of the present invention, the timing waveform characteristics of the load power in S1 are extracted by a sliding window. The sliding window contains a preset number of continuous sampling points. The sampling frequency of the sampling points is on the order of microseconds or milliseconds. The power change rate is calculated by dividing the power difference between adjacent sampling points by the sampling interval.
[0022] The power change rate is calculated using the following formula: ; In the formula: For a moment The load power change rate, expressed in power / time (e.g., kW / s), reflects the rate at which the GPU load power changes per unit time. These are measured values, as opposed to predicted values. ; For a moment The instantaneous load power of the GPU server was collected; This represents the load power at the previous sampling time. The sampling interval is in the microsecond or millisecond range.
[0023] Explanation of the calculation logic: This formula calculates the rate of change of power at the current moment by dividing the power difference between two adjacent sampling points by the sampling time interval. When the sampling frequency is high enough, this rate of change can be approximated as the derivative of load power with respect to time, capturing the transient fluctuation characteristics of GPU load on a millisecond-level time scale. If the difference is positive, it indicates that the power is increasing; if it is negative, it indicates that the power is decreasing; the larger the absolute value, the more drastic the power fluctuation.
[0024] This formula enables a quantitative characterization of millisecond-level power fluctuations in GPU load, allowing reinforcement learning agents to perceive transient impact characteristics that traditional average power metrics cannot reflect. By incorporating this dynamic metric into the state space, the energy storage system can predict load abrupt changes in advance, providing precise input for subsequent power buffer reservations and dual-timescale scheduling, thereby significantly improving its responsiveness to the typical "square wave" load of AI computing centers.
[0025] S2: The time-series waveform characteristics of the load power, the state of charge of the energy storage system, and the electricity price signal are jointly constructed into a state space for reinforcement learning; S3: Construct a hierarchical constraint space containing at least two priority levels, where the highest priority level corresponds to the GPU core computing cluster. The constraint conditions of this level include a power change rate constraint, which requires that the instantaneous discharge power change rate of the energy storage system is not less than the predicted value of the GPU load power change rate. In an embodiment of the present invention, the hierarchical constraint space in S3 includes three priority levels: the first priority level corresponds to the GPU core computing cluster, and its constraints include power change rate constraints and state of charge threshold constraints; the second priority level corresponds to conventional IT equipment, and its constraints include state of charge threshold constraints; the third priority level corresponds to auxiliary equipment, and its constraints include energy storage constraints.
[0026] In an embodiment of the present invention, the specific form of the power change rate constraint in S3 is: the ratio of the absolute value of the power change of the energy storage system within a unit control cycle to the control cycle is not less than the product of the predicted value of the GPU load power change rate and the tracking coefficient, wherein the tracking coefficient is dynamically adjusted according to the current grid stability state and the degree of GPU load fluctuation.
[0027] The power change rate constraint is expressed by the following inequality: ; In the formula: For a moment The total output power of the energy storage system (positive for discharging, negative for charging); This indicates that a control cycle has elapsed. The total output power of the post-energy storage system; The control cycle is measured in seconds or milliseconds, corresponding to the decision cycle of the main strategy network or auxiliary controller. For a moment The tracking coefficient is dimensionless and typically ranges from 0.8 to 1.2, dynamically adjusted based on the stability of the power grid and the degree of GPU load fluctuation. For a moment The predicted rate of change of GPU load power can be obtained by extrapolating historical rate of change sequences or by predicting based on task scheduling information.
[0028] In an embodiment of the present invention, the predicted value of the GPU load power change rate Obtain it through the following methods: As a first implementation method, an autoregressive moving average (ARMA) model is used to extrapolate and predict the historical power change rate sequence. The model order is determined by the Akaike Information Criterion, and the prediction time domain is set to 1 to 5 control periods.
[0029] In the second implementation, when communicating with the AI scheduling platform, the configuration file of the GPU training task is obtained from the platform. Parameters such as the task's computational density, batch size, and communication frequency are parsed. Based on the power change rate feature library collected during historical task execution, similarity matching is performed. The set of historical change rate curves with the highest matching degree is used as the predicted waveform of the current task, and the predicted power change rate at the task start time is extracted from it. .
[0030] As a third implementation method, a Long Short-Term Memory (LSTM) network is used to perform end-to-end prediction of the power change rate time series. The LSTM network contains two hidden layers, each with 64 neurons, and the input sequence length is 20 time steps.
[0031] Explanation of the operational logic: This inequality defines a power rate of change constraint that the energy storage system must satisfy. The left side represents the absolute value of the power rate of change of the energy storage system within a unit control cycle, characterizing the power ramp-up capability of the energy storage; the right side represents the GPU load prediction power rate of change multiplied by the tracking coefficient. The constraint requires that the power regulation rate of the energy storage be no less than a set multiple of the load change rate. When the load rises rapidly, the energy storage must be able to increase the discharge power at a sufficiently fast rate; when the load falls rapidly, the energy storage must be able to quickly reduce the discharge power or switch to charging to prevent bus voltage overshoot.
[0032] This constraint extends the traditional single-dimensional guarantee of State of Charge (SOC) to a dual-dimensional guarantee of "energy + rate of change of power," fundamentally solving the problem that existing scheduling strategies cannot cope with millisecond-level power surges. This is achieved by dynamically adjusting the tracking coefficient. The system can automatically tighten constraints when the power grid is unstable or the load fluctuates drastically, and appropriately relax constraints during stable periods to release economic space, thus achieving an adaptive balance between reliability and economy.
[0033] S4: The scheduling instructions are generated using a dual-timescale action space. The dual-timescale action space includes: the energy storage reference charging and discharging power output by the main strategy network at the first time scale, and the power correction amount output by the auxiliary controller at the second time scale. The total output power is obtained by superimposing the energy storage reference charging and discharging power and the power correction amount. In this embodiment, the total output power is defined as ,in The energy storage reference charge / discharge power, This refers to the power correction amount; In an embodiment of the present invention, the main strategy network in S4 adopts a deep deterministic strategy gradient algorithm or a variant thereof, and outputs the energy storage benchmark charging and discharging power with a decision cycle of seconds; the auxiliary controller is a fast controller based on model predictive control or a proportional-integral-derivative plus feedforward compensation controller, and outputs the power correction amount with a control cycle of milliseconds.
[0034] The auxiliary controller has a response delay that is less than the control cycle of the energy storage converter. The auxiliary controller tracks the instantaneous power of the GPU load and is constrained by the power change rate capability of the energy storage converter, and outputs a power correction amount.
[0035] In embodiments of the present invention, the main policy network employs the Deep Deterministic Policy Gradient (DDPG) algorithm, comprising an Actor network and a Critic network. The Actor network consists of an input layer, three fully connected hidden layers, and an output layer. The input layer dimension is the same as the state space dimension (including load power time-series waveform features, state of charge, electricity price signal, etc.). The number of neurons in the hidden layers are 256, 256, and 128, respectively, using the ReLU activation function. The output layer uses the Tanh activation function to output normalized energy storage baseline charge / discharge power. The Critic network structure is similar to the Actor network, with the output layer being a scalar value used to evaluate the value of state-action pairs. The experience replay pool capacity is set to 10^5 samples, and 64 samples are randomly sampled for mini-batch gradient descent during each training iteration. The soft update coefficient of the target network is set to 0.001, the discount factor is set to 0.95, the learning rate of the Actor network is set to 0.0001, and the learning rate of the Critic network is set to 0.001. During training, the target network is updated every 100 time steps.
[0036] In an embodiment of the present invention, the auxiliary controller employs a fast controller based on model predictive control (MPC), and its predictive model is a first-order inertial plus delay model of the energy storage converter: ,in This is the gain coefficient. It is a time constant. The delay time is defined as . The objective function of the MPC is to minimize the sum of the squared errors between the energy storage output power and the predicted GPU load power over the next N control cycles. The constraints are the maximum power change rate and the maximum output power of the energy storage converter. The control cycle is set to 1 millisecond, and the prediction time N is set to 10. Alternatively, the auxiliary controller can employ a proportional-integral-derivative (PID) controller with feedforward compensation. The PID parameters are tuned using the Ziegler-Nichols method, and the feedforward compensation is dynamically calculated based on the measured value of the GPU load power change rate.
[0037] In an embodiment of the present invention, after generating scheduling instructions using a dual-timescale action space in step S4, a power buffer reservation step is further included: dynamically calculating the power buffer reservation amount based on the predicted value of the GPU load power change rate and the power response time of the energy storage system; and pre-locking the power buffer reservation amount from the available power of the energy storage system, specifically for dealing with instantaneous power surges in GPU load.
[0038] The power buffer reserve is calculated using the following formula: ; In the formula: For a moment The power buffer reserve that needs to be locked in advance from the energy storage system, in units of power (e.g., kW); This is a safety factor, dimensionless, greater than 1 (typically 1.2 to 1.5), used to provide redundancy protection; The same predicted GPU load power change rate as mentioned above; The power response time of the energy storage system is the time required from receiving the command to the actual output power reaching the target value. It is determined by the control delay of the energy storage converter and the dynamic characteristics of the battery itself.
[0039] Explanation of the calculation logic: This formula calculates a power buffer reserve based on the predicted load change rate and the energy storage's own response time. Its physical meaning is: during the energy storage response delay, the load power may have changed significantly, therefore sufficient power capacity needs to be reserved in advance to cover this change. Safety factor The reserve was further increased to address prediction errors and system uncertainties. The calculated... It will be locked from the available energy storage power and will not participate in economic arbitrage scheduling, only being released when a load surge occurs.
[0040] This formula further quantifies the power change rate constraint into an executable reserved capacity instruction, realizing a closed loop from "constraint definition" to "resource reservation". By explicitly incorporating the energy storage response time into the calculation, the formula ensures that the reserved amount matches the dynamic characteristics of the actual physical system, avoiding resource waste caused by excessive reservation and reliability risks caused by insufficient reservation, thereby maximizing the capacity available for arbitrage while ensuring power supply security.
[0041] The power buffer reservation step further includes: obtaining GPU training task scheduling information through an interface with the AI scheduling platform, the scheduling information including task start time, expected load curve and task priority; and, based on the scheduling information, increasing the power buffer reservation amount before the GPU training task starts, and automatically releasing the power buffer reservation amount after the task ends.
[0042] The pre-boosted power buffer reserve is calculated using the following formula: ; In the formula: This indicates the power buffer reserve after pre-reservation enhancement; This is a buffer reserve for the basic power calculated above; This is the pre-reserve coefficient, dimensionless, greater than 0 (typically 0.5 to 1.0), used to adjust the aggressiveness of advance reserves; This represents the predicted rate of change in GPU load power at the time of task startup, estimated based on historical task load characteristics or task configuration files. The pre-reserve lead time is the time window between issuing the pre-reserve instruction and the actual start of the task.
[0043] Explanation of the operational logic: This formula adds an additional reserve proportional to the predicted task's impact intensity on top of the aforementioned basic reserve. When the AI scheduling platform announces the imminent launch of a high-load training task, the system determines the reserve based on the task's expected power impact intensity (determined by...). Characterization) and lead time Calculate the additional capacity that needs to be locked in advance. Pre-reserve coefficient. Control the amount of extra reserves to avoid being overly aggressive.
[0044] This formula upgrades the passively responsive power buffer to an active, sensing pre-reserve mechanism, enabling the energy storage system to prepare capacity before GPU tasks begin. By utilizing prior information from the AI scheduling platform, the system eliminates response delays caused by predictive uncertainty, providing sufficient power support at the moment of task initiation. This significantly reduces the risk of voltage drops in the initial stages of an impact event, while avoiding economic losses caused by high reserves throughout the entire time.
[0045] In an embodiment of the present invention, the scheduling instructions are executed in step S4 using a hierarchical controller architecture, which includes: a millisecond-level power tracking layer for tracking the instantaneous power of the GPU load with a millisecond-level response delay and outputting a power correction amount; a second-level strategy optimization layer for updating the energy storage benchmark charge and discharge power at a second-level cycle to optimize economy; and a minute-level constraint update layer for updating the constraint parameters of the hierarchical constraint space at a minute-level cycle.
[0046] The hierarchical controller architecture also includes a priority arbitration mechanism: when the energy storage reference charging and discharging power output by the second-level strategy optimization layer conflicts with the power correction amount required by the millisecond-level power tracking layer, the millisecond-level power tracking layer takes over the control right according to the principle that the power change rate constraint corresponding to the first priority level takes precedence over economic scheduling.
[0047] S5: Train the reinforcement learning agent based on a preset reward function, the reward function including a power supply reliability reward item and a power tracking accuracy reward item; In an embodiment of the present invention, the power supply reliability reward item in S5 is negatively correlated with the measured value of bus voltage drop. When the bus voltage drop exceeds the preset protection threshold, a large negative reward is given or the current training round is terminated. The power tracking accuracy reward item is negatively correlated with the tracking error between the actual output power of energy storage and the GPU load power.
[0048] The power supply reliability bonus is calculated using the following formula: ; when hour, Set to negative infinity or terminate the current training round.
[0049] In the formula: This is a power supply reliability reward, which can be negative or negative infinity, and is used to punish behaviors that violate reliability constraints during reinforcement learning training. This is the penalty coefficient, a positive number, which controls the overall magnitude of the penalty. The measured value of the bus voltage dip is in volts (V) and is collected in real time by a voltage sensor. This is a preset protection threshold, measured in volts, and is typically taken as the trigger voltage value for undervoltage protection of the server power module.
[0050] Explanation of the operational logic: This formula uses an exponential function to map the voltage drop magnitude to a negative reward value. When the voltage drops... Much smaller than the threshold When the index approaches 1, the reward is... A slight penalty is imposed; when the voltage drops close to the threshold, the exponential growth is rapid; when... At this point, the reward is set to negative infinity or the current training episode is terminated directly, which is equivalent to imposing an absolute prohibition on fatal errors.
[0051] The power tracking accuracy bonus is calculated using the following formula: ; In the formula: This is a power tracking accuracy bonus, and can be a negative value or zero. The tracking error penalty coefficient is a positive number, controlling the penalty intensity corresponding to the unit tracking error. Its dimension is 1 / power, and it is used to convert the power error into a dimensionless penalty value. The dimension of is the reciprocal of power, which makes Dimensionless; The total output power is the sum of the energy storage reference charge / discharge power and the power correction amount, as defined above. The load power is as defined above.
[0052] Explanation of the operational logic: This formula employs a linear penalty for absolute value error. The greater the instantaneous deviation between the actual output power of the energy storage and the load power, the heavier the penalty; when perfect tracking is achieved, the reward is 0. This reward encourages the reinforcement learning agent to make the energy storage output power follow the changes in load power in real time as much as possible, thereby maintaining the stability of the bus voltage.
[0053] The two reward formulas together construct an incentive structure that prioritizes reliability over accuracy. The exponential reliability reward causes the penalty to rise sharply when the voltage drops close to the protection threshold, driving the agent to actively move away from dangerous areas; the linear tracking accuracy reward guides the agent to pursue higher dynamic tracking performance within safe areas. The combination of the two ensures that the trained strategy can guarantee absolute safety in extreme situations while achieving high-quality power following in normal operation.
[0054] The reward function also includes an economic reward item, which calculates peak-valley arbitrage revenue based on time-of-use electricity pricing and subtracts battery aging costs. The power supply reliability reward item, power tracking accuracy reward item, and economic reward item are weighted and summed using dynamic weighting coefficients, which are adjusted in real time based on the grid stability and GPU load fluctuations.
[0055] The economic incentive item is calculated using the following formula: ; In the formula: This is an economic reward item, with the unit being monetary, and can be a positive value (gain) or a negative value (loss). This is a preset benchmark value for economic returns, expressed in monetary units, used to normalize economic rewards into a dimensionless scalar. It can be set as an estimate of the theoretical maximum arbitrage profit within a typical operating cycle (such as one day), or it can be dynamically updated through an online adaptive method.
[0056] For a moment The time-of-use electricity price is expressed in currency units per power per time (e.g., yuan / kWh), during discharge. It is positive when purchasing electricity. Negative; The total output power as defined above; This is a battery aging cost function, expressed in monetary units per time, and is related to factors such as depth of discharge, discharge rate, and temperature. In an embodiment of the present invention, the battery aging cost function C_aging(t) is calculated using the following model: ; in, This represents the total replacement cost of the battery pack. This represents the change in depth of discharge during the current charge-discharge cycle. To achieve the average depth of discharge The battery cycle life is obtained by interpolation using the cycle life curve provided by the battery manufacturer. This is the current current multiplier. For reference current ratio, The aging index is the ratio of the number of years of aging (typically ranging from 1.1 to 1.5). The current battery temperature. This is a reference temperature (usually 25℃). This represents the temperature aging coefficient (typically ranging from 0.05 to 0.1). As a simplified implementation, a linear aging model can be used: ,in, The aging cost coefficient is obtained by fitting battery cycle life test data.
[0057] The integral time window is typically one dispatch cycle or electricity price change cycle.
[0058] The total reward function, calculated by weighting the dynamic weight coefficients, is obtained using the following formula: ; The dynamic weighting coefficient is adaptively adjusted based on the power grid stability state and the predicted value of the GPU load power change rate. In the formula: The total reward function is the objective function for the reinforcement learning agent to optimize. , , They are time points The power supply reliability weight, power point tracking accuracy weight, and economic weight are all non-negative numbers and satisfy the following conditions: , In other words, the more unstable the power grid or the more drastic the load fluctuations, the higher the reliability weight. As an optional implementation, the dynamic weighting coefficients can be calculated using the following linear normalization formula: , ; in, , These are the minimum and maximum values of the tracking coefficient (which can be determined based on the statistical distribution of the tracking coefficient in historical operating data, such as taking the 5th and 95th percentiles of historical values, or setting them according to the maximum and minimum tracking capabilities required by the system design). This is the minimum preset value for the economic weight. It can be set to a fixed small constant or related to the tracking error variance.
[0059] As an alternative implementation method: ,in This is a preset constant (e.g., 0.1); or ,in This is the normalization coefficient.
[0060] The tracking coefficient, as defined above, is used to reflect the current grid stability and load fluctuation. Explanation of the operational logic: The economic reward is obtained by integrating the instantaneous revenue within the time window, where the electricity price multiplied by the power is the arbitrage profit (positive profit during discharge, negative profit or cost during charging), and the net profit is obtained after subtracting the battery aging cost. The total reward function dynamically weights and sums the three objectives of reliability, tracking accuracy, and economy. The sum of the weights being 1 ensures the consistency of the reward dimensions; the reliability weight is related to the tracking coefficient. The positive correlation enables the agent to automatically shift its learning focus to reliability assurance when the risk is high, and to favor economic optimization when the risk is low.
[0061] This overall reward function enables dynamic online trade-offs among multiple objectives. Compared to a fixed-weight reward function, the dynamic weight mechanism allows the reinforcement learning agent to adaptively adjust its behavioral strategy based on real-time operating conditions—prioritizing power supply security during periods of severe grid fluctuations and fully utilizing price differences for arbitrage during stable periods. The introduction of battery aging costs further guides the agent to avoid overly aggressive charging and discharging strategies, extending the overall lifespan of the energy storage system. The three reward items are optimized through a normalized weighted average, avoiding the tediousness and rigidity of manually setting fixed priorities in traditional multi-objective optimization, and significantly improving the generalization ability of the strategy under different operating conditions.
[0062] S6: Control the charging and discharging power of the energy storage converter based on the total output power of the trained reinforcement learning agent.
[0063] Example 2: Figure 5 As shown, the present invention relates to a real-time scheduling system for industrial and commercial energy storage based on reinforcement learning, comprising: The high-frequency sampling and feature extraction module is used to acquire the current and voltage signals of the GPU server power supply circuit at a sampling frequency of microseconds or milliseconds, and calculate the instantaneous power and power change rate based on the current and voltage signals; The state space construction module is used to jointly construct the state space for reinforcement learning by combining the time-series waveform characteristics of the instantaneous power, the power change rate, the state of charge of the energy storage system, and the electricity price signal. The hierarchical constraint management module is used to construct a hierarchical constraint space containing at least two priority levels, where the highest priority level corresponds to the GPU core computing cluster, and the constraints at this level include power change rate constraints. In another embodiment of the present invention, the hierarchical constraint management module is further configured to dynamically adjust the constraint parameters of each priority level in the hierarchical constraint space according to the power grid precursor signal or the degree of GPU load fluctuation, wherein the power grid precursor signal includes at least one of voltage sag, frequency fluctuation and harmonic distortion rate.
[0064] The reinforcement learning decision module includes a main policy network and an auxiliary controller. The main policy network outputs the energy storage reference charge and discharge power at a first time scale, and the auxiliary controller outputs the power correction amount at a second time scale. The total output power is obtained by superimposing the energy storage reference charge and discharge power and the power correction amount. In another embodiment of the present invention, the reinforcement learning decision module further includes a task perception unit, which is communicatively connected to the AI scheduling platform to obtain scheduling information of GPU training tasks and send a pre-reservation instruction to the power buffer reservation module before the task starts according to the scheduling information.
[0065] In another embodiment of the present invention, the reinforcement learning decision module adopts a hierarchical controller architecture, which includes: a millisecond-level power tracking controller for tracking the instantaneous power of GPU load with millisecond-level response latency; a second-level policy optimization controller for optimizing economic scheduling policies with a second-level cycle; a minute-level constraint update controller for updating constraint parameters with a minute-level cycle; and a priority arbitrator for selecting control instructions according to preset priority rules when multiple controller outputs conflict.
[0066] The power buffer reservation module is used to dynamically calculate the power buffer reservation amount based on the predicted value of the GPU load power change rate, and to lock the power buffer reservation amount in advance from the available power of the energy storage system. The execution module is used to control the charging and discharging power of the energy storage converter according to the total output power.
[0067] The embodiments disclosed in this invention are preferred embodiments, but are not limited thereto. Those skilled in the art can easily understand the spirit of this invention based on the above embodiments and make different extensions and variations, but as long as they do not depart from the spirit of this invention, they are all within the protection scope of this invention.
Claims
1. A real-time scheduling method for industrial and commercial energy storage demand based on reinforcement learning, characterized in that, Includes the following steps: S1: Collect the instantaneous power signal of the GPU server, and calculate the load power, power change rate, and time-series waveform characteristics of the load power based on the instantaneous power signal; S2: The time-series waveform characteristics of the load power, the state of charge of the energy storage system, and the electricity price signal are jointly constructed into a state space for reinforcement learning; S3: Construct a hierarchical constraint space containing at least two priority levels, where the highest priority level corresponds to the GPU core computing cluster. The constraint conditions of the highest priority level include a power change rate constraint, which requires that the instantaneous discharge power change rate of the energy storage system is not less than the predicted value of the GPU load power change rate. S4: The scheduling instructions are generated using a dual-timescale action space. The dual-timescale action space includes: the energy storage reference charging and discharging power output by the main strategy network at the first time scale, and the power correction amount output by the auxiliary controller at the second time scale. The total output power is obtained by superimposing the energy storage reference charging and discharging power and the power correction amount. S5: Train the reinforcement learning agent based on a preset reward function, the reward function including a power supply reliability reward item and a power tracking accuracy reward item; S6: Control the charging and discharging power of the energy storage converter based on the total output power of the trained reinforcement learning agent.
2. The method for real-time scheduling of industrial and commercial energy storage demand based on reinforcement learning according to claim 1, characterized in that, The timing waveform characteristics of the load power in S1 are extracted through a sliding window. The sliding window contains a preset number of continuous sampling points. The sampling frequency of the sampling points is on the order of microseconds or milliseconds. The power change rate is calculated by dividing the power difference between adjacent sampling points by the sampling interval. The power change rate is calculated using the following formula: ； In the formula, For a moment The rate of change of load power; For a moment The instantaneous load power of the GPU server was collected; This represents the load power at the previous sampling time. The sampling interval is denoted as .
3. The method for real-time scheduling of industrial and commercial energy storage demand based on reinforcement learning according to claim 1, characterized in that, The hierarchical constraint space in S3 includes three priority levels: the first priority level corresponds to the GPU core computing cluster, and its constraints include power change rate constraints and state of charge threshold constraints; the second priority level corresponds to conventional IT equipment, and its constraints include state of charge threshold constraints; the third priority level corresponds to auxiliary equipment, and its constraints include energy storage constraints.
4. The method for real-time scheduling of industrial and commercial energy storage demand based on reinforcement learning according to claim 1, characterized in that, The specific form of the power change rate constraint in S3 is: the ratio of the absolute value of the power change of the energy storage system within a unit control cycle to the control cycle is not less than the product of the predicted value of the GPU load power change rate and the tracking coefficient, wherein the tracking coefficient is dynamically adjusted according to the current grid stability state and the severity of GPU load fluctuations. The power change rate constraint is expressed by the following inequality: ； In the formula, For a moment Total output power of the energy storage system; This indicates that a control cycle has elapsed. The total output power of the post-energy storage system; To control the cycle; For a moment The tracking coefficient; For a moment The predicted rate of change of GPU load power.
5. The method for real-time scheduling of industrial and commercial energy storage demand based on reinforcement learning according to claim 1, characterized in that, The main strategy network in S4 adopts a deep deterministic strategy gradient algorithm or its variant, and outputs the energy storage benchmark charging and discharging power with a decision cycle of seconds; the auxiliary controller is a fast controller based on model predictive control or a proportional-integral-derivative plus feedforward compensation controller, and outputs the power correction amount with a control cycle of milliseconds. The response delay of the auxiliary controller is less than the control cycle of the energy storage converter. The auxiliary controller uses the instantaneous power of the GPU load as the tracking target and the power change rate capability of the energy storage converter as a constraint to output a power correction amount.
6. The method for real-time scheduling of industrial and commercial energy storage demand based on reinforcement learning according to claim 1, characterized in that, After generating scheduling instructions using a dual-timescale action space in S4, a power buffer reservation step is also included: dynamically calculating the power buffer reservation amount based on the predicted value of the GPU load power change rate and the power response time of the energy storage system; and pre-locking the power buffer reservation amount from the available power of the energy storage system to cope with the instantaneous power surge of the GPU load. The power buffer reserve is calculated using the following formula: ； In the formula, For a moment The power buffer reserve needs to be locked in advance from the energy storage system; For safety factor; For a moment Predicted GPU load power change rate; This refers to the power response time of the energy storage system.
7. The method for real-time scheduling of industrial and commercial energy storage demand based on reinforcement learning according to claim 6, characterized in that, The power buffer reservation step further includes: obtaining GPU training task scheduling information through an interface with the AI scheduling platform, the scheduling information including task start time, expected load curve and task priority; and, based on the scheduling information, increasing the power buffer reservation amount in advance before the GPU training task starts, and automatically releasing the power buffer reservation amount after the task ends. The pre-boosted power buffer reserve is calculated using the following formula: ； In the formula, This indicates the power buffer reserve after pre-reservation enhancement; This is the pre-reservation coefficient; This represents the predicted rate of change in GPU load power at the time of task startup; This allows for advance planning and preparation.
8. The method for real-time scheduling of industrial and commercial energy storage demand based on reinforcement learning according to claim 1, characterized in that, The S4 uses a hierarchical controller architecture to execute scheduling instructions. The hierarchical controller architecture includes: a millisecond-level power tracking layer, which tracks the instantaneous power of the GPU load with a millisecond-level response delay and outputs a power correction amount; a second-level strategy optimization layer, which updates the energy storage benchmark charging and discharging power with a second-level cycle to optimize economy; and a minute-level constraint update layer, which updates the constraint parameters of the hierarchical constraint space with a minute-level cycle. The hierarchical controller architecture also includes a priority arbitration mechanism: when the energy storage reference charging and discharging power output by the second-level strategy optimization layer conflicts with the power correction amount required by the millisecond-level power tracking layer, the millisecond-level power tracking layer takes over the control right according to the principle that the power change rate constraint corresponding to the first priority level takes precedence over economic scheduling.
9. The method for real-time scheduling of industrial and commercial energy storage demand based on reinforcement learning according to claim 1, characterized in that, The power supply reliability reward in S5 is negatively correlated with the measured value of bus voltage drop. When the bus voltage drop exceeds the preset protection threshold, a large negative reward is given or the current training round is terminated. The power tracking accuracy reward is negatively correlated with the tracking error between the actual output power of the energy storage and the GPU load power. The power supply reliability bonus is calculated using the following formula: ； In the formula, This is an award for power supply reliability. This is the penalty coefficient; The measured value is the bus voltage drop. The preset protection threshold; when hour, Set to negative infinity or terminate the current training round; The power tracking accuracy bonus is calculated using the following formula: ； In the formula, Awarded for power tracking accuracy; This is the tracking error penalty coefficient; For a moment Total output power of the energy storage system; For a moment The instantaneous load power of the GPU server was collected.
10. A real-time scheduling system for industrial and commercial energy storage based on reinforcement learning, used to execute the real-time scheduling method for industrial and commercial energy storage demand based on reinforcement learning as described in any one of claims 1 to 9, characterized in that, include: The high-frequency sampling and feature extraction module is used to acquire the current and voltage signals of the GPU server power supply circuit at a sampling frequency of microseconds or milliseconds, and calculate the instantaneous power and power change rate based on the current and voltage signals; The state space construction module is used to jointly construct the state space for reinforcement learning by combining the time-series waveform characteristics of the instantaneous power, the power change rate, the state of charge of the energy storage system, and the electricity price signal. The hierarchical constraint management module is used to construct a hierarchical constraint space containing at least two priority levels, where the highest priority level corresponds to the GPU core computing cluster, and the constraints at this level include power change rate constraints. The reinforcement learning decision module includes a main policy network and an auxiliary controller. The main policy network outputs the energy storage reference charge and discharge power at a first time scale, and the auxiliary controller outputs the power correction amount at a second time scale. The total output power is obtained by superimposing the energy storage reference charge and discharge power and the power correction amount. The power buffer reservation module is used to dynamically calculate the power buffer reservation amount based on the predicted value of the GPU load power change rate, and to lock the power buffer reservation amount in advance from the available power of the energy storage system. The execution module is used to control the charging and discharging power of the energy storage converter according to the total output power.