A data center-oriented energy storage system charging and discharging control method and system

Abstract

The application provides a data center-oriented energy storage system charge-discharge control method and system, the method comprising: identifying thermal inertia coefficient and computer room equivalent thermal resistance according to temperature and refrigeration power historical data, combining charge boundary, transformer margin and temperature control standby power to construct a normalized safety domain, normalizing discrete action set and projecting it to the safety domain to obtain candidate action set, projection displacement and constraint multiplier, fusing projection displacement and constraint multiplier to obtain constraint pressure factor, incorporating state vector to adjust exploration strategy, completing action selection and value evaluation through double networks respectively, cutting off the reverse propagation gradient between the two networks, calculating the instant return containing electricity charges, battery attenuation and out-of-boundary penalty, combining the health reduction factor to generate the Bellman target, updating the double networks respectively by using the truncated Huber loss and the residual gate sliding average, comprehensively scoring to select the optimal power, and issuing the energy storage converter for execution.

Description

Technical Field

[0001] This application belongs to the field of control, and in particular relates to a charging and discharging control method and system for energy storage systems in data centers. Background Technology

[0002] The operating environment of data center energy storage systems is complex, involving fluctuating electricity prices over time and closely related to the thermodynamic characteristics of the computer room, air conditioning cooling load, and real-time transformer capacity. Traditional energy storage charging and discharging control methods are geared towards single-price arbitrage, which can easily lead to fluctuations in cooling power that threaten the thermal safety of servers. Because they cannot combine transformer capacity bottlenecks with the boundaries of battery state of charge, conventional scheduling can easily trigger system overload or overcharging and over-discharging of energy storage under extreme conditions. Therefore, the safe, economical, and coordinated control of data center energy storage systems under multi-dimensional and stringent constraints has become a problem that the industry needs to solve.

[0003] Conventional algorithms often rely on post-hoc soft penalty mechanisms when handling safety constraints, failing to define safety boundaries for actions during the action exploration phase. This makes them prone to outputting dangerous commands that violate transformer constraints or temperature control backup limits during the exploration process. Traditional Bellman objective calculations lack feedback on system constraint pressures and energy storage health degradation, easily leading to "overestimation" of action values ​​and causing unstable model training. Existing reward evaluation systems typically overlook the implicit costs of frequent charge-discharge switching and the long-term degradation penalty of batteries, resulting in oscillating output control commands and accelerated reduction in energy storage battery life, failing to achieve safe and long-term control of data center energy storage under real-world operating conditions. Summary of the Invention

[0004] To address the problems of lack of safety constraints, model training instability, and increased battery wear in traditional energy storage scheduling, this invention proposes a charging and discharging control method and system for energy storage systems in data centers.

[0005] According to one aspect of the present invention, a charging and discharging control method for an energy storage system for data centers is provided, comprising the following steps: The system collects electricity price, temperature, state of charge, and transformer margin to generate a state vector. It identifies the thermal inertia coefficient and the equivalent thermal resistance of the computer room based on historical data of temperature and cooling power. It constructs a normalized safety domain based on the charge boundary, transformer margin, and temperature control backup power determined by the thermal inertia coefficient and the equivalent thermal resistance of the computer room. It projects the normalized discrete action set onto the normalized safety domain to obtain the candidate action set, projected displacement, and constraint multipliers. The projected displacement and constraint multiplier are fused into a constraint pressure factor and attached to the state vector. The exploration strategy for the next cycle is adjusted according to the constraint pressure factor. The action selection network determines the next action from the candidate action set, and the value evaluation network calculates the target value. The backpropagation gradient between the action selection network and the value evaluation network is cut off during the calculation. The system calculates the immediate return, including electricity costs, degradation, and out-of-bounds penalties. It then generates a Bellman target by combining the energy storage health reduction factor and the constraint pressure factor. The action selection network is updated with the current action value and the truncated Huber loss of the Bellman target. The value evaluation network parameters are updated using the residual gated moving average. A comprehensive score is calculated based on the action value, constraint multiplier, and switching penalty cost. The energy storage converter with the highest power output is selected and sent to the data center for execution.

[0006] Optionally, the process of collecting electricity price, temperature, state of charge, and transformer margin to generate a state vector, and identifying the thermal inertia coefficient and equivalent thermal resistance of the computer room from historical temperature and cooling power data, includes: The timestamps corresponding to the control cycle are converted into time slice feature vectors using one-hot encoding. Collect cooling power and computer room temperature data within a preset time window prior to the current moment to construct a time series matrix; Using the step change in cooling power as input and the response data of the computer room temperature as output, the first-order thermal response transfer function of the system is fitted by the least squares method. The time constant in the transfer function is extracted as the thermal inertia coefficient, and the absolute value of the steady-state gain is extracted as the equivalent thermal resistance of the computer room. The state vector is formed by horizontally splicing the time slice feature vector, the various parameters collected in real time, the thermal inertia coefficient, and the equivalent thermal resistance of the computer room.

[0007] Optionally, the step of constructing a normalized safety region based on the charge boundary, transformer margin, and temperature-controlled backup power determined by the thermal inertia coefficient and the equivalent thermal resistance of the computer room, and projecting the normalized discrete action set onto the normalized safety region to obtain the candidate action set, projected displacement, and constraint multipliers, includes: Calculate the transformer margin by subtracting the sum of the real-time information technology equipment load and the basic cooling power from the transformer's rated capacity; Based on the thermal inertia coefficient, the equivalent thermal resistance of the computer room, the heat generation power of real-time information technology equipment, and the safe operating threshold of the computer room temperature, the cooling redundancy power required by the system to resist environmental thermal disturbances is set as the temperature control backup power. Combining the rated capacity of energy storage with the duration of the control cycle, the transformer margin, temperature control backup power, energy storage charge and discharge rate limit, and upper and lower limits of state of charge are converted into power boundaries and normalized to form a normalized safety domain with multi-dimensional boundaries. After normalizing each action in the discrete action set, project it onto the normalized safety region using the minimum Euclidean distance. Map the out-of-bounds action to the nearest safety boundary point to obtain the candidate action set. The absolute difference between the corresponding elements of the motion coordinates before and after projection is calculated as the projection displacement. The values ​​of the dual variables corresponding to the nearest plane output by the optimization solver are extracted to generate the initial constraint multipliers. The projection displacement and constraint multipliers are then dimensionless.

[0008] Optionally, the step of fusing the projected displacement and constraint multipliers into a constraint pressure factor and appending it to the state vector, and adjusting the exploration strategy for the next cycle based on the constraint pressure factor, includes: Weight parameters are assigned to the dimensionless projected displacement and constraint multipliers respectively, and linear weighted summation is performed to obtain the initial pressure value; The initial pressure value is input into a Sigmoid activation function with saturated compression characteristics for nonlinear mapping and normalized to a preset interval to generate a constraint pressure factor. The constraint pressure factor is appended as an additional feature to the end of the current state vector; By comparing the constraint pressure factor with the preset pressure threshold, if the constraint pressure factor exceeds the preset pressure threshold, the random exploration probability of the exploration strategy will be reduced in the next cycle; if it does not exceed the preset pressure threshold, the basic random exploration probability will remain unchanged.

[0009] Optionally, the step of determining the next action from the candidate action set by the action selection network, calculating the target value by the value evaluation network, and cutting off the backpropagation gradient between the action selection network and the value evaluation network during the calculation includes: The state vector with the constraint pressure factor is input into the action selection network. The forward propagation outputs the predicted value of each candidate action in the candidate action set. A greedy strategy is used to select the action with the highest predicted value as the next action to calculate the target value. The state vector is fed into the value evaluation network, and the predicted target value of all candidate actions is output through forward calculation. The predicted target value corresponding to the next action is extracted as the target value. During the backpropagation calculation of network gradients, a stopping gradient operator is inserted at the nodes of the cascaded computation graph of the action selection network and the value evaluation network. The gradient operator is stopped to block the backpropagation chain of the target value calculation on the action selection network parameters, so that the target value is treated as a static constant in the current iteration step, suppressing the overestimation bias caused by the co-current updates of the action selection network and the value evaluation network.

[0010] Optionally, the calculation includes immediate returns on electricity costs, degradation, and out-of-bounds penalties, and combines energy storage health reduction factors and constraint pressure factors to generate Bellman objectives, including: Real-time electricity revenue is calculated based on time-of-use pricing and actual power flow, and peak excess penalty for transformers is calculated based on demand-based billing models. Substituting into the empirical formula for energy storage cycle life, the energy storage capacity decay caused by a single action is assessed by the current depth of discharge and ambient temperature and converted into an economic penalty term. When the predicted temperature of the computer room exceeds the safety limit, an exponentially increasing temperature over-limit penalty is imposed. Add the smoothing factor to the real-time electricity revenue, and then subtract the transformer peak excess penalty, economic penalty, and temperature over-limit penalty to obtain the real-time return. The discount rate is calculated by multiplying the preset base discount rate by the energy storage health reduction factor of the energy storage available life and the safety adjustment coefficient obtained by subtracting the constraint pressure factor from the constant 1. The Bellman target is formed by multiplying the immediate return by the discount rate and the target value at the next moment.

[0011] Optionally, the step of updating the action selection network with the current action value and the truncated Huber loss of the Bellman objective, and updating the value evaluation network parameters by residual gated moving average, includes: Calculate the temporal difference error between the current action value output by the action selection network and the Bellman objective; When the absolute value of the timing difference error is less than the truncation threshold, the mean square error is used to calculate the loss value. When the absolute value of the timing difference error is greater than or equal to the truncation threshold, the error with a linear compensation term is used to calculate the truncated Huber loss. The action selection network is optimized using gradient descent with truncated Huber loss. Extract the parameter residual vector before and after the action selection network update. If the norm of the parameter residual vector is less than the preset gating threshold, the first smoothing coefficient is used to update the parameter exponential moving average of the action selection network to the value evaluation network. If the norm of the parameter residual vector is greater than or equal to the preset gating threshold, a second smoothing coefficient less than the first smoothing coefficient is used to perform a slow update.

[0012] Optionally, the step of calculating a comprehensive score based on action value, constraint multipliers, and switching penalty costs, and selecting the energy storage converter with the highest power output to the data center for execution, includes: For each action in the candidate action set, extract the expected action value score output by the action selection network. Compare the differences between each candidate action and the historical actions executed in the previous cycle, and multiply the difference by the equipment loss coefficient to obtain the switching penalty cost; The overall score for each candidate action is obtained by subtracting the switching penalty cost from the action value expectation score and subtracting the product of the constraint multiplier and the safety weight coefficient. Sort the comprehensive scores of all candidate actions in the candidate action set in descending order and obtain the action number of the first ranked action; The action number ranked first is mapped and parsed into an actual power command, which is then sent to the energy storage converter in the data center for execution via the communication interface.

[0013] According to another aspect of the present invention, a charging and discharging control system for an energy storage system for data centers is provided, comprising the following modules: The data acquisition module is used to collect electricity price, temperature, state of charge, and transformer margin to generate state vectors. It identifies thermal inertia coefficient and equivalent thermal resistance of the computer room from historical data of temperature and cooling power. Based on the charge boundary, transformer margin, and temperature control backup power determined by thermal inertia coefficient and equivalent thermal resistance of the computer room, it constructs a normalized safety domain. After normalizing the discrete action set, it projects it onto the normalized safety domain to obtain candidate action set, projected displacement, and constraint multipliers. The determination module is used to fuse the projected displacement and constraint multipliers into a constraint pressure factor and attach it to the state vector. The exploration strategy for the next cycle is adjusted according to the constraint pressure factor. The action selection network determines the next action from the candidate action set, and the value evaluation network calculates the target value. The backpropagation gradient between the action selection network and the value evaluation network is cut off during the calculation. The execution module is used to calculate the immediate return, including electricity costs, attenuation, and out-of-bounds penalties. It generates a Bellman target by combining the energy storage health reduction factor and the constraint pressure factor. It updates the action selection network with the current action value and the truncated Huber loss of the Bellman target. It updates the value evaluation network parameters by residual gating moving average. It calculates a comprehensive score based on action value, constraint multiplier, and switching penalty cost. It selects the energy storage converter with the highest power to be sent to the data center for execution.

[0014] Preferably, the step of collecting electricity price, temperature, state of charge, and transformer margin to generate a state vector, and identifying the thermal inertia coefficient and equivalent thermal resistance of the computer room from historical data of temperature and cooling power, includes: The timestamps corresponding to the control cycle are converted into time slice feature vectors using one-hot encoding. Collect cooling power and computer room temperature data within a preset time window prior to the current moment to construct a time series matrix; Using the step change in cooling power as input and the response data of the computer room temperature as output, the first-order thermal response transfer function of the system is fitted by the least squares method. The time constant in the transfer function is extracted as the thermal inertia coefficient, and the absolute value of the steady-state gain is extracted as the equivalent thermal resistance of the computer room. The state vector is formed by horizontally splicing the time slice feature vector, the various parameters collected in real time, the thermal inertia coefficient, and the equivalent thermal resistance of the computer room.

[0015] Preferably, the step of constructing a normalized safety region based on the charge boundary, transformer margin, and temperature-controlled backup power determined by the thermal inertia coefficient and the equivalent thermal resistance of the computer room, and then projecting the normalized discrete action set onto the normalized safety region to obtain the candidate action set, projected displacement, and constraint multipliers includes: Calculate the transformer margin by subtracting the sum of the real-time information technology equipment load and the basic cooling power from the transformer's rated capacity; Based on the thermal inertia coefficient, the equivalent thermal resistance of the computer room, the heat generation power of real-time information technology equipment, and the safe operating threshold of the computer room temperature, the cooling redundancy power required by the system to resist environmental thermal disturbances is set as the temperature control backup power. Combining the rated capacity of energy storage with the duration of the control cycle, the transformer margin, temperature control backup power, energy storage charge and discharge rate limit, and upper and lower limits of state of charge are converted into power boundaries and normalized to form a normalized safety domain with multi-dimensional boundaries. After normalizing each action in the discrete action set, project it onto the normalized safety region using the minimum Euclidean distance. Map the out-of-bounds action to the nearest safety boundary point to obtain the candidate action set. The absolute difference between the corresponding elements of the motion coordinates before and after projection is calculated as the projection displacement. The values ​​of the dual variables corresponding to the nearest plane output by the optimization solver are extracted to generate the initial constraint multipliers. The projection displacement and constraint multipliers are then dimensionless.

[0016] Preferably, the step of fusing the projected displacement and constraint multipliers into a constraint pressure factor and appending it to the state vector, and adjusting the exploration strategy for the next cycle based on the constraint pressure factor, includes: Weight parameters are assigned to the dimensionless projected displacement and constraint multipliers respectively, and linear weighted summation is performed to obtain the initial pressure value; The initial pressure value is input into a Sigmoid activation function with saturated compression characteristics for nonlinear mapping and normalized to a preset interval to generate a constraint pressure factor. The constraint pressure factor is appended as an additional feature to the end of the current state vector; By comparing the constraint pressure factor with the preset pressure threshold, if the constraint pressure factor exceeds the preset pressure threshold, the random exploration probability of the exploration strategy will be reduced in the next cycle; if it does not exceed the preset pressure threshold, the basic random exploration probability will remain unchanged.

[0017] Preferably, the step of determining the next action from the candidate action set by the action selection network, calculating the target value by the value evaluation network, and cutting off the backpropagation gradient between the action selection network and the value evaluation network during the calculation includes: The state vector with the constraint pressure factor is input into the action selection network. The forward propagation outputs the predicted value of each candidate action in the candidate action set. A greedy strategy is used to select the action with the highest predicted value as the next action to calculate the target value. The state vector is fed into the value evaluation network, and the predicted target value of all candidate actions is output through forward calculation. The predicted target value corresponding to the next action is extracted as the target value. During the backpropagation calculation of network gradients, a stopping gradient operator is inserted at the nodes of the cascaded computation graph of the action selection network and the value evaluation network. The gradient operator is stopped to block the backpropagation chain of the target value calculation on the action selection network parameters, so that the target value is treated as a static constant in the current iteration step, suppressing the overestimation bias caused by the co-current updates of the action selection network and the value evaluation network.

[0018] Preferably, the calculation includes immediate returns on electricity costs, degradation, and over-limit penalties, and combines energy storage health reduction factors and constraint pressure factors to generate Bellman objectives, including: Real-time electricity revenue is calculated based on time-of-use pricing and actual power flow, and peak excess penalty for transformers is calculated based on demand-based billing models. Substituting into the empirical formula for energy storage cycle life, the energy storage capacity decay caused by a single action is assessed by the current depth of discharge and ambient temperature and converted into an economic penalty term. When the predicted temperature of the computer room exceeds the safety limit, an exponentially increasing temperature over-limit penalty is imposed. Add the smoothing factor to the real-time electricity revenue, and then subtract the transformer peak excess penalty, economic penalty, and temperature over-limit penalty to obtain the real-time return. The discount rate is calculated by multiplying the preset base discount rate by the energy storage health reduction factor of the energy storage available life and the safety adjustment coefficient obtained by subtracting the constraint pressure factor from the constant 1. The Bellman target is formed by multiplying the immediate return by the discount rate and the target value at the next moment.

[0019] Preferably, the step of updating the action selection network with the current action value and the truncated Huber loss of the Bellman objective, and updating the value evaluation network parameters by residual gated moving average, includes: Calculate the temporal difference error between the current action value output by the action selection network and the Bellman objective; When the absolute value of the timing difference error is less than the truncation threshold, the mean square error is used to calculate the loss value. When the absolute value of the timing difference error is greater than or equal to the truncation threshold, the error with a linear compensation term is used to calculate the truncated Huber loss. The action selection network is optimized using gradient descent with truncated Huber loss. Extract the parameter residual vector before and after the action selection network update. If the norm of the parameter residual vector is less than the preset gating threshold, the first smoothing coefficient is used to update the parameter exponential moving average of the action selection network to the value evaluation network. If the norm of the parameter residual vector is greater than or equal to the preset gating threshold, a second smoothing coefficient less than the first smoothing coefficient is used to perform a slow update.

[0020] Preferably, the step of calculating a comprehensive score based on action value, constraint multipliers, and switching penalty costs, and selecting the energy storage converter with the highest power output to the data center for execution, includes: For each action in the candidate action set, extract the expected action value score output by the action selection network. Compare the differences between each candidate action and the historical actions executed in the previous cycle, and multiply the difference by the equipment loss coefficient to obtain the switching penalty cost; The overall score for each candidate action is obtained by subtracting the switching penalty cost from the action value expectation score and subtracting the product of the constraint multiplier and the safety weight coefficient. Sort the comprehensive scores of all candidate actions in the candidate action set in descending order and obtain the action number of the first ranked action; The action number ranked first is mapped and parsed into an actual power command, which is then sent to the energy storage converter in the data center for execution via the communication interface.

[0021] This invention integrates the thermal characteristics of data center server rooms with the operational boundaries of energy storage. By identifying the thermal inertia coefficient and the equivalent thermal resistance of the server room, a normalized safety domain is constructed. Discrete actions are projected onto this safety domain, ensuring the transformer margin and the operational baseline of the temperature control system. The projected displacement and constraint multipliers are transformed into constraint pressure factors to guide the exploration strategy and improve sample utilization efficiency. Combined with the backpropagation gradient truncation mechanism between the two networks, the stability and convergence quality of model training are improved. Based on the consideration of operating electricity costs, battery degradation, out-of-bounds penalties, and equipment switching costs, an improved Bellman objective is constructed using energy storage health reduction factors and constraint pressure factors. Network parameters are updated using a truncated Huber loss and a residual gated moving average mechanism. This mechanism enhances the model's tolerance to extreme outliers, reduces hardware wear, and extends the lifespan of energy storage equipment, achieving a high balance between economic benefits and safety. This ensures that the power commands issued to the energy storage converter are both profitable and reliable. Attached Figure Description

[0022] Figure 1 A flowchart of the first embodiment; Figure 2 A schematic diagram for identifying thermal parameters in a data center computer room; Figure 3 A schematic diagram of the construction of the normalized safety domain and the discrete action quadratic programming projection; Figure 4 This is a diagram showing the comparison of metrics for four sets of algorithm schemes. Detailed Implementation

[0023] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0024] In the first embodiment, the present invention proposes a charging and discharging control method for an energy storage system for data centers, such as... Figure 1 It includes the following steps: S1 collects electricity price, temperature, state of charge, and transformer margin to generate a state vector. It identifies the thermal inertia coefficient and the equivalent thermal resistance of the computer room based on historical data of temperature and cooling power. It constructs a normalized safety domain based on the charge boundary, transformer margin, and temperature control backup power determined by the thermal inertia coefficient and the equivalent thermal resistance of the computer room. It projects the normalized discrete action set onto the normalized safety domain to obtain the candidate action set, projected displacement, and constraint multipliers.

[0025] The system reads the current state of charge (SOC) from the energy storage battery management system and the transformer capacity from the distribution cabinet monitoring terminal using the Modbus TCP protocol of the pymodbus communication library. It uses the requests library to call the local power grid's open interface to obtain time-of-use electricity price data and utilizes data center temperature and humidity sensors to detect the current server room temperature. The electricity price, temperature, SOC, and transformer capacity are concatenated into a one-dimensional floating-point array as the initial state vector. Historical time-series data of server room temperature and cooling power from the past month are extracted from the database. The Recursive Least Squares algorithm from the statsmodels time-series library is used for system identification, fitting a first-order thermodynamic discrete differential equation to solve for the thermal inertia coefficient and equivalent thermal resistance of the data center server room.

[0026] By substituting the thermal inertia coefficient and the equivalent thermal resistance of the data center into the temperature control model, the required temperature control backup power to maintain the data center temperature below the maximum threshold is calculated. The charge boundary of the upper and lower limit operation of the energy storage system, the transformer margin, and the temperature control backup power are combined to generate multiple inequality constraints. The HalfspaceIntersection algorithm in the SciPy scientific computing library is used to construct a polyhedral safe operating space surrounded by multiple linear inequalities. The boundary of this safe operating space is mapped to a continuous interval from -1 to 1 to form a normalized safety domain. The linspace function in the NumPy library is used to divide the rated charge and discharge power of the energy storage converter into 30 discrete action sets in equal proportion. The cvxpy convex optimization library is called to construct a quadratic programming model. The objective function is to minimize the Euclidean distance between the original discrete action and the projected coordinates. Each discrete action is projected as a decision variable into the normalized safety domain. Solving the quadratic programming model yields the projected compliant coordinates as the candidate action set. The absolute difference between the corresponding elements of the original discrete action set and the candidate action set is calculated using the subtract function of the NumPy library as the projected displacement. The values ​​of the dual variables corresponding to the optimal solution output by the cvxpy solver are extracted as constraint multipliers.

[0027] S2 integrates the projected displacement and constraint multipliers into a constraint pressure factor and attaches it to the state vector. The exploration strategy for the next cycle is adjusted according to the constraint pressure factor. The action selection network determines the next action from the candidate action set, and the value evaluation network calculates the target value. The backpropagation gradient between the action selection network and the value evaluation network is cut off during the calculation.

[0028] The `dot` function from the NumPy library is used to perform an inner product operation between the one-dimensional vector of projected displacement and the one-dimensional vector of constraint multipliers. The result is then mapped to a numerical range of 0 to 1 using the sigmoid activation function to generate a constraint pressure factor. This constraint pressure factor is appended to the end of the original initial state vector to form an enhanced state vector. An exploration decay mechanism is implemented by calculating the product of the base exploration rate and the natural exponential function. The exponential part of the natural exponential function is the negative constraint pressure factor multiplied by an adjustment coefficient. This reduces the probability of random exploration as the constraint pressure factor increases to ensure operational safety. The resulting epsilon value is used as the greedy strategy to calculate the actual exploration probability for the next cycle. A multilayer perceptron based on the PyTorch deep learning framework is constructed as the action selection network and the value evaluation network, respectively. The action selection network contains four fully connected linear layers and a ReLU activation function. The augmented state vector is converted into a PyTorch tensor and input into the action selection network for forward propagation to obtain the Q-value of each action in the candidate action set. Based on the previously calculated exploration probability, a multinomial sampling function is used to randomly sample or select the term with the largest Q-value to determine the next action. The augmented state vector and the next action are input into the value evaluation network, which contains a network structure with similar hidden layers, to calculate the expected target value. In the PyTorch backpropagation code logic, the expected target value is separated from the current computation graph by calling the detach function of the target value tensor output by the value evaluation network. This breaks the backpropagation gradient connection between the computation graphs of the action selection network and the value evaluation network, preventing gradient interaction and divergence oscillations between their parameters during optimization and update.

[0029] S3 calculates the immediate return including electricity costs, attenuation, and out-of-bounds penalties. It generates a Bellman target by combining the energy storage health reduction factor and constraint pressure factor. The action selection network is updated with the current action value and the truncated Huber loss of the Bellman target. The value evaluation network parameters are updated by residual gating moving average. A comprehensive score is calculated based on the action value, constraint multiplier, and switching penalty cost. The energy storage converter with the highest power output is selected and sent to the data center for execution.

[0030] Discharging is defined as positive and charging as negative. The next action is converted into the actual charging / discharging power multiplied by the real-time electricity price to obtain the electricity revenue item for the current cycle. The rainflow counting algorithm is used to count the number of cycles of charge / discharge depth change of the energy storage battery within a historical period. This data is then substituted into the lithium battery life empirical decay model to calculate the battery single-action decay penalty item. When the execution of an action causes the temperature or state of charge to exceed a preset hard constraint threshold, a fixed scalar value over-limit penalty item is triggered. The electricity revenue item, the normalized negative decay penalty item, and the negative over-limit penalty item are added together to obtain the single-step instant reward. The current health status characteristic value SOH detected from the battery management system is used as the energy storage health reduction factor. A discount coefficient is generated by multiplying the preset benchmark discount factor by the energy storage health reduction factor and the safety adjustment coefficient obtained by subtracting the constraint pressure factor from a constant 1. The instant reward, the discount coefficient, and the expected target value are multiplied and added together to generate the Bellman target. The scalar error between the current action prediction Q value output by the action selection network and the Bellman target is calculated.

[0031] The SmoothL1Loss function is called and the beta truncation threshold hyperparameter is set to calculate the truncated Huber loss. The Adam moment estimation optimizer is instantiated and backpropagation and gradient update are performed on the weight matrix and bias terms of all neurons in the action selection network based on the truncated Huber loss. The L2 norm of the parameter residual vector before and after the current update of the action selection network is calculated as the residual gating variable and substituted into the nonlinear smoothing function to adjust the smoothing coefficient of the exponential moving average algorithm. Based on the adjusted smoothing coefficient, the updated weights of the action selection network are slowly covered and fused into the corresponding parameter matrix of the value evaluation network according to the residual gating moving average calculation rule.

[0032] The action value is calculated by extracting all node values ​​from the action selection network output layer, i.e., the action value corresponding to the candidate action. The absolute value of the difference between the energy storage power commands of two adjacent control time steps is calculated and multiplied by the wear coefficient as the switching penalty cost. The action value is then calculated using scalar multiplication and subtraction from the NumPy library according to preset weight hyperparameters. The penalty multiplier multiplied by the constraint multiplier is subtracted, and the switching penalty cost is further subtracted to obtain a one-dimensional score vector for each candidate action. The score vectors are sorted in descending order of value. The candidate action corresponding to the first element in the ranking index is extracted and denormalized to be restored to the actual continuous power command. The standard Modbus protocol hexadecimal write register instruction is encapsulated using the PyModbus communication library and sent to the underlying power control address space of the data center field energy storage converter for hardware circuit-level execution.

[0033] In an optional embodiment, the step of collecting electricity price, temperature, state of charge, and transformer margin to generate a state vector, and identifying the thermal inertia coefficient and equivalent thermal resistance of the computer room from historical temperature and cooling power data, includes: The timestamps corresponding to the control cycle are converted into time slice feature vectors using one-hot encoding. Collect cooling power and computer room temperature data within a preset time window prior to the current moment to construct a time series matrix; Using the step change in cooling power as input and the response data of the computer room temperature as output, the first-order thermal response transfer function of the system is fitted by the least squares method. The time constant in the transfer function is extracted as the thermal inertia coefficient, and the absolute value of the steady-state gain is extracted as the equivalent thermal resistance of the computer room. The state vector is formed by horizontally splicing the time slice feature vector, the various parameters collected in real time, the thermal inertia coefficient, and the equivalent thermal resistance of the computer room.

[0034] The energy storage scheduling control cycle is set to 15 minutes, with the entire day divided into 96 fixed time slices. The unique sequence number corresponding to the current timestamp is extracted (e.g., 8:00 AM corresponds to the 32nd time slice), and transformed into a one-dimensional sparse time slice feature vector of length 96 using a one-hot encoding mapping algorithm. The central controller, utilizing environmental sensors and a dynamic environmental monitoring protocol, uses a fixed sampling step of 1 minute to retrospectively extract historical monitoring information within a preset time window of the past 4 hours, allocating and constructing a 240×2 double-column time series matrix in memory. In the parameter identification stage, the step change moment where the air conditioning cooling power increases by more than 10kW is extracted and used as the system's input excitation signal. Simultaneously, the time response trajectory of the computer room temperature is extracted as the output reference value. A nonlinear least squares fitting algorithm based on Levenberg-Marquardt is invoked to perform approximation iterations on a typical first-order inertial link model structure. After model convergence, the absolute value of the steady-state gain of the system is extracted from the parameter terms, for example, measured as 0.18℃ / kW, and assigned to the equivalent thermal resistance variable of the computer room. The time constant of the model's decay exponent is extracted, for example, 420 seconds, and assigned to the thermal inertia coefficient variable. The previously generated 96-dimensional time slice vector is combined with the industrial electricity price read in real time via fieldbus, for example, 0.75 yuan per kilowatt-hour during different time periods, the computer room temperature, for example, 26.5℃, the current state of charge (SOC) of the battery pack, for example, 65%, and the transformer electrical margin, for example, 450kW. Combined with the identified thermal resistance and thermal inertia coefficient, a 102-dimensional real-valued state vector is obtained as the observation basis for the state-space model. Figure 2 As shown, the thermal response parameters of the data center are identified, and the temperature change characteristic curve of the data center under the step disturbance of cooling power is given.

[0035] In an optional embodiment, the step of constructing a normalized safety region based on the charge boundary, transformer margin, and temperature-controlled backup power determined by the thermal inertia coefficient and the equivalent thermal resistance of the computer room, and projecting the normalized discrete action set onto the normalized safety region to obtain the candidate action set, projected displacement, and constraint multipliers, includes: Calculate the transformer margin by subtracting the sum of the real-time information technology equipment load and the basic cooling power from the transformer's rated capacity; Based on the thermal inertia coefficient, the equivalent thermal resistance of the computer room, the heat generation power of real-time information technology equipment, and the safe operating threshold of the computer room temperature, the cooling redundancy power required by the system to resist environmental thermal disturbances is set as the temperature control backup power. Combining the rated capacity of energy storage with the duration of the control cycle, the transformer margin, temperature control backup power, energy storage charge and discharge rate limit, and upper and lower limits of state of charge are converted into power boundaries and normalized to form a normalized safety domain with multi-dimensional boundaries. After normalizing each action in the discrete action set, project it onto the normalized safety region using the minimum Euclidean distance. Map the out-of-bounds action to the nearest safety boundary point to obtain the candidate action set. The absolute difference between the corresponding elements of the motion coordinates before and after projection is calculated as the projection displacement. The values ​​of the dual variables corresponding to the nearest plane output by the optimization solver are extracted to generate the initial constraint multipliers. The projection displacement and constraint multipliers are then dimensionless.

[0036] The main control program obtains the rated capacity of the distribution transformer nameplate configuration, for example, set to 2500kVA. The active power limit calculated based on the power factor is 2250kW. This active power limit is then subtracted from the real-time information technology equipment load fed back by the power quality analyzer and the basic cooling power required by the precision air conditioning unit to maintain the current temperature, thus calculating the transformer margin. The program also obtains the upper limit temperature for safe operation of the computer room, the current observed temperature, and the heat generation power converted from the real-time information technology equipment load. Substituting these into the upstream given thermal inertia coefficient and equivalent thermal resistance, the program uses the closed-loop feedback prediction formula: Temperature Control Backup Power = Real-time Heat Generation Power - (Upper Limit Temperature for Safe Operation of Computer Room - Current Observed Temperature) / Equivalent Thermal Resistance of Computer Room. This calculates the cooling redundancy power required to withstand thermal disturbances under specific operating conditions and sets it as the temperature control backup power.

[0037] After acquiring the boundary information, the aforementioned transformer margin, temperature control backup power, and independent dimensional parameters such as the rated charge / discharge rate limit and the upper and lower limits of safe operation of the state of charge fed back by the battery management system are set, for example, between 15% and 85%. These parameters, combined with the rated capacity of the energy storage battery configuration and the current control cycle duration, are converted into charge / discharge power constraint boundaries and then uniformly fed into a scaler for regularization processing before being projected onto a standard closed interval. The regularized multivariate conditional hyperplanes intersect and enclose the area, resulting in a normalized safety domain representing the safety space in a high-dimensional coordinate system. Outside the geometric domain, the original control command interval is uniformly divided into discrete action sets at preset intervals, for example, a total of 21 command units. A quadratic programming projection mechanism is then initiated to determine the internal and external attributes of each discrete action.

[0038] For actions that exceed or do not belong to the normalized safety domain, such as an action command of charging 1.0 but with a capacity boundary upper limit of 0.6, the out-of-bounds action is orthogonally projected along the normal direction to the boundary of the safety domain, mapping it to the nearest legal boundary point. All filtered legal actions are aggregated to obtain a candidate action set. For each corrected action, the scalar length of the line segment from the original position to the edge projection point is calculated to generate the initial projected displacement. The values ​​of the dual variables corresponding to the nearest plane output by the optimization solver are extracted to generate the initial constraint multipliers. To eliminate dimensional differences caused by different parameters, the initial projected displacement and the initial constraint multipliers are smoothed and dimensionless. Figure 3 As shown, this demonstrates the construction of a multidimensional constraint-normalized safety domain and the principle of projecting a discrete action set onto the safety domain using quadratic programming.

[0039] In an optional embodiment, the step of fusing the projected displacement and constraint multipliers into a constraint pressure factor and appending it to the state vector, and adjusting the exploration strategy for the next cycle based on the constraint pressure factor, includes: Weight parameters are assigned to the dimensionless projected displacement and constraint multipliers respectively, and linear weighted summation is performed to obtain the initial pressure value; The initial pressure value is nonlinearly mapped and normalized to a preset range to generate a constraint pressure factor. The constraint pressure factor is appended as an additional feature to the end of the current state vector; By comparing the constraint pressure factor with the preset pressure threshold, if the constraint pressure factor exceeds the preset pressure threshold, the random exploration probability of the exploration strategy will be reduced in the next cycle; if it does not exceed the preset pressure threshold, the basic random exploration probability will remain unchanged.

[0040] During the perceptual feature construction phase of the reinforcement learning runtime, the controller retrieves a pre-set weight configuration file and assigns core weight parameters to the dimensionless projected displacement output from the previous steps, as well as collaborative weight parameters to the dimensionless constraint multipliers. Weighted merging is then performed, fusing the two features into a single real-valued initial pressure value representing the overall urgency. To prevent pressure factor overflow due to outliers and thus avoid gradient divergence, the initial pressure value is input into a sigmoid activation function with saturation compression characteristics for smoothing filtering and nonlinear mapping, constraining the value within the [0,1] interval to generate a constraint pressure factor.

[0041] The generated constraint pressure factor is treated as an additional one-dimensional spatial feature vector and sequentially appended to the end of the previous round's initial state vector (e.g., a data vector consisting of 102 elements), thus topologically expanding the initial state vector into a high-dimensional panoramic perception state vector, such as 103-dimensional. This constraint pressure factor is then guided to the next round of control's stochastic decision-making module and compared with a preset pressure intervention alarm threshold, such as 0.6. If the constraint pressure factor exceeds the preset pressure threshold, the current operating condition is determined to be a dangerous working mode prone to triggering constraint restrictions. The tree-cutting logic is then synchronously activated, and the original default trial-and-error exploration rate allocation process of reinforcement learning is terminated. The stochastic exploration probability constant in the strategy generation algorithm, such as the epsilon greedy strategy, is lowered to reduce the stochastic exploration probability of the exploration strategy and guide the system towards a safer strategy. If subsequent feedback factors do not exceed the preset pressure threshold, the decision-making algorithm unconditionally uses the basic stochastic exploration probability setting.

[0042] In an optional embodiment, the step of determining the next action from a candidate action set by the action selection network, calculating the target value by the value evaluation network, and cutting off the backpropagation gradient between the action selection network and the value evaluation network during the calculation includes: The state vector with the constraint pressure factor is input into the action selection network. The forward propagation outputs the predicted value of each candidate action in the candidate action set. The greedy strategy selects the action with the highest predicted value as the next action to calculate the target value. The state vector is fed into the value evaluation network, and the predicted target value of all candidate actions is output through forward calculation. The predicted target value corresponding to the next action is extracted as the target value. During the backpropagation calculation of network gradients, a stopping gradient operator is inserted at the nodes of the cascaded computation graph of the action selection network and the value evaluation network. By stopping the gradient operator, the backpropagation chain of the derivative of the target value calculation with the action selection network parameters is blocked, so that the target value is treated as a static constant in the current iteration step, suppressing the overestimation bias caused by the co-current updates of the action selection network and the value evaluation network.

[0043] A deep inference engine comprising an action selection network and a value evaluation network is launched. The action selection network is a deep neural network with a multilayer perceptron architecture. The input is an expanded state vector carrying a constraint pressure factor, such as a 103-dimensional one-dimensional tensor. The network structure includes an input layer, multiple fully connected hidden layers, and an output layer. After multilayer forward propagation matrix operations and nonlinear transformations, the output layer outputs a predicted value score array corresponding to each available candidate action. The Argmax operation is used to traverse and compare the score array, and a greedy decision model is used to select the policy behavior index corresponding to the maximum predicted value, such as the power value corresponding to action number 14, as the next action instruction for subsequent calculations.

[0044] In the valuation phase, the value evaluation network adopts the same multilayer perceptron physical architecture and parameter matrix dimensions as the action selection network, i.e., it has the same input layer dimensions, number of hidden layer nodes, and multi-node output layer. The state vector is fed into the value evaluation network as input, and after forward computation, the output generates a predicted target value array corresponding to all candidate actions. Using the next action instruction determined in the previous step as the action index, a slice extraction operation is performed on the predicted target value array to extract the one-dimensional scalar value corresponding to the specific action, representing the target value of the desired objective. This ensures that subsequent direct mapping and moving average updates between network parameters are executed.

[0045] The model employs an optimizer to perform backpropagation updates using partial differentials in the error chain. A stopping gradient operator is pre-inserted after the computational graph nodes for extracting the target value. This stopping gradient operator isolates the backpropagation process of the loss gradient, blocking the gradient differentiation and backpropagation link from the target value calculation to the parameters of the value evaluation network and action selection network. This freezes the extracted target value into a non-differentiable static constant within the current training iteration, reducing overestimation bias and action activation deadlock caused by simultaneous network updates.

[0046] In an optional embodiment, the calculation includes immediate returns on electricity costs, degradation, and out-of-bounds penalties, combined with an energy storage health reduction factor and a constraint stress factor to generate a Bellman objective, including: Real-time electricity revenue is calculated based on time-of-use pricing and actual power flow, and peak excess penalty for transformers is calculated based on demand-based billing models. Substituting into the empirical formula for energy storage cycle life, the energy storage capacity decay caused by a single action is assessed by the current depth of discharge and ambient temperature and converted into an economic penalty term. When the predicted temperature in the computer room exceeds the safety limit, an exponentially increasing temperature violation penalty is imposed. Add the smoothing factor to the real-time electricity revenue, and then subtract the transformer peak excess penalty, economic penalty, and temperature over-limit penalty to obtain the real-time return. The discount rate is calculated by multiplying the preset base discount rate by the energy storage health reduction factor of the energy storage available life and the safety adjustment coefficient obtained by subtracting the constraint pressure factor from the constant 1. The Bellman target is formed by multiplying the immediate return by the discount rate and the target value at the next moment.

[0047] The current time-of-use tariff rate is obtained by calling the grid time-of-use tariff matrix table. Simultaneously, the power execution command of the energy storage converter within this control period is queried. The two are then multiplied by integration to calculate the instantaneous electricity revenue. The apparent power operating extreme value of the grid-side metering is monitored. If the detected data exceeds the application threshold, a penalty for exceeding the transformer peak value is calculated according to the punitive billing rules and weighting. The depth of discharge (DOD) parameter and the heat dissipation monitoring temperature (T) of the current battery's actual operating state are extracted and substituted into the Arrhenius aging curve fitting equation. The estimated irreversible capacity degradation factor is multiplied by the initial construction cost, which is then converted into an economic penalty for equipment health degradation. Let be the irreversible capacity loss coefficient of the energy storage battery, and a, b, and c be Arrhenius aging fitting empirical constants, fixed coefficients obtained from battery factory characteristics and aging test calibration. The predictive model estimates the data center temperature for future periods; if the temperature exceeds the safety limit, an exponentially increasing temperature exceedance penalty is activated. M is the temperature penalty reference amplitude coefficient. This is the growth coefficient of the temperature exceedance index. The preset temperature safety upper limit threshold is used. The preset fixed bias constant smoothing term is added to the real-time electricity revenue, and then the transformer peak excess penalty, economic penalty term and temperature over-limit penalty are subtracted in sequence. The result is linearly summed to generate the real-time return scalar representing the basis for this round.

[0048] When extrapolating the global evolution trend, a preset Markov base discount rate is extracted, for example... =0.98, the actual SOH estimate of the available life of energy storage (SOH) returned by the underlying controller is obtained as the energy storage health reduction factor. The constraint pressure factor is subtracted from the constant 1 to obtain the safety adjustment coefficient. The above base discount rate, energy storage health reduction factor, and safety adjustment coefficient are multiplied together to obtain the discount rate corrected for safety requirements. The immediate return is then added to the product of the discount rate and the target value of the value network at the next time step. A Bellman objective strategy was constructed for the action network to approximate the target, in which... This represents the current environmental state characteristics of the system at the next moment, including multi-dimensional state variables such as electricity price, battery status, temperature, and grid load. To output control actions to the intelligent agent in the next moment, such as energy storage charging and discharging power and inverter scheduling commands; Output for the value network: the target value of the next state-action pair, representing the current state. Execute action The long-term cumulative value is represented by R, which is the scalar value of the immediate return in this round, and the single-step return after taking into account electricity costs and various penalties.

[0049] In an optional embodiment, the step of updating the action selection network with the current action value and the truncated Huber loss of the Bellman objective, and updating the value evaluation network parameters by residual-gated moving average, includes: Calculate the temporal difference error between the current output action value of the action selection network and the Bellman objective; When the absolute value of the time difference error is less than the truncation threshold, the mean square error is used to calculate the loss value. When the absolute value of the time difference error is greater than or equal to the truncation threshold, the mean absolute error with a linear compensation term is used to calculate the truncated Huber loss. The action selection network is optimized using gradient descent with truncated Huber loss. Extract the parameter residual vectors before and after the action selection network update. If the norm of the parameter residual vector is less than a preset gating threshold, use the first smoothing coefficient to update the parameter exponential moving average of the action selection network to the value evaluation network. If the norm of the parameter residual vector is greater than or equal to the preset gating threshold, use the second smoothing coefficient, which is less than the first smoothing coefficient, to perform a slow update.

[0050] During empirical replay training, the first-order difference between the current positive predicted state score of the action selection network and the Bellman objective is calculated to obtain the temporal difference error for the current round. A stabilization and anti-interference loss filtering channel is established, comparing the absolute value of the TD error with a preset empirical threshold: if the absolute value of the TD error is less than the threshold, the mean squared error is used to calculate the loss; if the absolute value of the TD error is greater than or equal to the threshold, it indicates abnormal noise or abrupt changes in the environment, terminating the quadratic multiplication calculation, and calculating the truncated Huber loss using a truncated Huber processing criterion with a linear compensation term. This truncated Huber loss is then used to perform gradient descent optimization and weight parameter updates on the action selection network through backpropagation.

[0051] After the action selection network parameters are updated, the parameter residual vectors before and after the update are extracted. The displacement difference is updated by calculating the norm of the standard Euclidean L2 distance measurement vector. The calculated parameter residual vector norm is compared with a preset gating threshold: if the norm is determined to be less than the preset gating threshold, a first smoothing coefficient of a preset standard size is used to smoothly update the latest parameters to the value evaluation network using an exponential moving average; if the norm is determined to be greater than or equal to the preset gating threshold, indicating oscillation, a delayed smoothing mechanism is activated, replacing the smoothing coefficient with a second smoothing coefficient with a very small value, for example... =0.001, performing slow updates to the value evaluation network with extremely small step sizes to prevent parameter collapse during model evolution.

[0052] In an optional embodiment, the step of calculating a comprehensive score based on action value, constraint multipliers, and switching penalty costs, and selecting the energy storage converter with the highest power output to the data center for execution, includes: For each action in the candidate action set, extract the expected action value score output by the action selection network. Compare the differences between each candidate action and the historical actions executed in the previous cycle, and multiply the difference by the equipment loss coefficient to obtain the switching penalty cost; The overall score for each candidate action is obtained by subtracting the switching penalty cost from the action value expectation score and then subtracting the product of the constraint multiplier and the safety weight coefficient. The comprehensive scores of all candidate actions in the candidate action set are sorted in descending order, and the action number of the first ranked action is retrieved. The action number ranked first is mapped and parsed into an actual power command, which is then sent to the energy storage converter in the data center for execution via the communication interface.

[0053] During the closed-loop operation of the control decision, for each action in the candidate action set, the expected action value score output by the action selection network is extracted. To reduce fatigue damage to hardware caused by frequent charging and discharging state switching, the historical execution action scale of the previous cycle is obtained from the database, and a difference operation is performed with each current candidate action to obtain the deviation step. The deviation step is multiplied by the loss conversion coefficient built into the equipment maintenance and protection rule list to calculate the switching penalty cost representing equipment wear. The constraint multiplier corresponding to the action is multiplied by the preset safety weight coefficient to obtain the boundary violation penalty term for the urgent state of the environmental boundary.

[0054] The switching penalty cost and the boundary violation penalty are subtracted from the expected action value score to synthesize a comprehensive score that considers global benefits, equipment damage prevention, and boundary violation prevention. The formula is as follows: Score = Expected Action Score – Switching Penalty Cost – Constraint Multiplier × Safety Weight. The comprehensive score set including all candidate actions is sorted in descending order, and the action number ranked first in the descending sequence is extracted. According to the built-in query mapping data table, the action number ranked first is mapped to an actual power command with a physical dimension, such as an active power value. The encapsulated real data command is sent to the energy storage converter in the data center through the underlying port via the communication network for execution.

[0055] This experiment constructs a high-fidelity reinforcement learning simulation environment based on 30 days of continuous historical operation monitoring records of a large data center with an installed capacity of 2500 kVA, and sets the scheduling control cycle to 15 minutes. Control group one uses the basic deep Q-network algorithm without any added temperature inertia features or safety polyhedral projection constraints for conventional electricity price arbitrage. Control group two removes the Bellman objective and anti-blowout truncation mechanism of the value network in the reward feedback function, retaining only the basic loss function and fixed step-size parameter updates. Control group three removes the comprehensive scoring and adjudication stage on the control output side that penalizes frequent cross-regional switching of equipment. The experimental group deploys a complete integrated scheduling scheme including thermal inertia feature extraction and stitching, multi-dimensional safety domain projection screening, Bellman discount, and residual gated moving average.

[0056] After 5000 rounds of simulation and data alignment, the core operational performance of each control system was obtained. Control Group 1 achieved a daily saving of 4200 yuan in electricity purchase costs, but triggered 17 transformer power over-limit alarms and the computer room temperature exceeded safety limits for a cumulative total of 94 minutes. Control Group 2 recorded a daily electricity cost saving of 3900 yuan, but monitoring revealed severe oscillations in the network loss surface and overestimation of actions, leading to a surge in monthly battery capacity life loss to 2.1%. Control Group 3 achieved a daily electricity cost saving of 4600 yuan, but the converter experienced 35 polarity reversals and full-load jump switching actions daily. In contrast, the experimental group achieved a daily electricity cost saving of 4850 yuan, achieved zero transformer over-limit alarms and zero computer room temperature alarms throughout operation, reduced the switching frequency to 8 times per day, and controlled the monthly energy storage capacity decay to an extremely low 0.45%. Figure 4 As shown, the overall performance of each control group and the proposed scheduling scheme in terms of economic benefits, safety constraints and energy storage losses is compared.

[0057] In a second embodiment, the present invention also provides a charging and discharging control system for an energy storage system for data centers, comprising the following modules: The data acquisition module is used to collect electricity price, temperature, state of charge, and transformer margin to generate state vectors. It identifies thermal inertia coefficient and equivalent thermal resistance of the computer room from historical data of temperature and cooling power. Based on the charge boundary, transformer margin, and temperature control backup power determined by thermal inertia coefficient and equivalent thermal resistance of the computer room, it constructs a normalized safety domain. After normalizing the discrete action set, it projects it onto the normalized safety domain to obtain candidate action set, projected displacement, and constraint multipliers. The determination module is used to fuse the projected displacement and constraint multipliers into a constraint pressure factor and attach it to the state vector. The exploration strategy for the next cycle is adjusted according to the constraint pressure factor. The action selection network determines the next action from the candidate action set, and the value evaluation network calculates the target value. The backpropagation gradient between the action selection network and the value evaluation network is cut off during the calculation. The execution module is used to calculate the immediate return, including electricity costs, attenuation, and out-of-bounds penalties. It generates a Bellman target by combining the energy storage health reduction factor and the constraint pressure factor. It updates the action selection network with the current action value and the truncated Huber loss of the Bellman target. It updates the value evaluation network parameters by residual gating moving average. It calculates a comprehensive score based on action value, constraint multiplier, and switching penalty cost. It selects the energy storage converter with the highest power to be sent to the data center for execution.

[0058] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0059] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A charging and discharging control method for an energy storage system for data centers, characterized in that, Includes the following steps: The system collects electricity price, temperature, state of charge, and transformer margin to generate a state vector. It identifies the thermal inertia coefficient and the equivalent thermal resistance of the computer room based on historical data of temperature and cooling power. It constructs a normalized safety domain based on the charge boundary, transformer margin, and temperature control backup power determined by the thermal inertia coefficient and the equivalent thermal resistance of the computer room. It projects the normalized discrete action set onto the normalized safety domain to obtain the candidate action set, projected displacement, and constraint multipliers. The projected displacement and constraint multiplier are fused into a constraint pressure factor and attached to the state vector. The exploration strategy for the next cycle is adjusted according to the constraint pressure factor. The action selection network determines the next action from the candidate action set, and the value evaluation network calculates the target value. The backpropagation gradient between the action selection network and the value evaluation network is cut off during the calculation. The system calculates the immediate return, including electricity costs, degradation, and out-of-bounds penalties. It then generates a Bellman target by combining the energy storage health reduction factor and the constraint pressure factor. The action selection network is updated with the current action value and the truncated Huber loss of the Bellman target. The value evaluation network parameters are updated using the residual gated moving average. A comprehensive score is calculated based on the action value, constraint multiplier, and switching penalty cost. The energy storage converter with the highest power output is selected and sent to the data center for execution.

2. The method according to claim 1, characterized in that, The process of collecting electricity price, temperature, state of charge, and transformer margin to generate a state vector, and identifying the thermal inertia coefficient and equivalent thermal resistance of the computer room from historical temperature and cooling power data, includes: The timestamps corresponding to the control cycle are converted into time slice feature vectors using one-hot encoding. Collect cooling power and computer room temperature data within a preset time window prior to the current moment to construct a time series matrix; Using the step change in cooling power as input and the response data of the computer room temperature as output, the first-order thermal response transfer function of the system is fitted by the least squares method. The time constant in the transfer function is extracted as the thermal inertia coefficient, and the absolute value of the steady-state gain is extracted as the equivalent thermal resistance of the computer room. The state vector is formed by horizontally splicing the time slice feature vector, the various parameters collected in real time, the thermal inertia coefficient, and the equivalent thermal resistance of the computer room.

3. The method according to claim 1, characterized in that, The process involves constructing a normalized safety region based on the charge boundary, transformer margin, and temperature-controlled backup power determined by the thermal inertia coefficient and the equivalent thermal resistance of the computer room. The discrete action set is then normalized and projected onto this normalized safety region to obtain candidate action sets, projected displacements, and constraint multipliers, including: Calculate the transformer margin by subtracting the sum of the real-time information technology equipment load and the basic cooling power from the transformer's rated capacity; Based on the thermal inertia coefficient, the equivalent thermal resistance of the computer room, the heat generation power of real-time information technology equipment, and the safe operating threshold of the computer room temperature, the cooling redundancy power required by the system to resist environmental thermal disturbances is set as the temperature control backup power. After normalizing the transformer margin, temperature control backup power, energy storage charge and discharge rate limits, and upper and lower limits of state of charge, they together constitute a normalized security domain representing the multidimensional boundary. After normalizing each action in the discrete action set, project it onto the normalized safety region using the minimum Euclidean distance. Map the out-of-bounds action to the nearest safety boundary point to obtain the candidate action set. The absolute difference between the corresponding elements of the motion coordinates before and after projection is calculated as the projection displacement. The values ​​of the dual variables corresponding to the nearest plane output by the optimization solver are extracted to generate the initial constraint multipliers. The projection displacement and constraint multipliers are then dimensionless.

4. The method according to claim 1, characterized in that, The process of fusing the projected displacement and constraint multipliers into a constraint pressure factor and attaching it to the state vector, and adjusting the exploration strategy for the next cycle based on the constraint pressure factor, includes: Weight parameters are assigned to the dimensionless projected displacement and constraint multipliers respectively, and linear weighted summation is performed to obtain the initial pressure value; The initial pressure value is input into a Sigmoid activation function with saturated compression characteristics for nonlinear mapping and normalized to a preset interval to generate a constraint pressure factor. The constraint pressure factor is appended as an additional feature to the end of the current state vector; By comparing the constraint pressure factor with the preset pressure threshold, if the constraint pressure factor exceeds the preset pressure threshold, the random exploration probability of the exploration strategy will be reduced in the next cycle; if it does not exceed the preset pressure threshold, the basic random exploration probability will remain unchanged.

5. The method according to claim 1, characterized in that, The process of determining the next action from a candidate action set by the action selection network, calculating the target value by the value evaluation network, and cutting off the backpropagation gradient between the action selection network and the value evaluation network during the calculation includes: The state vector with the constraint pressure factor is input into the action selection network. The forward propagation outputs the predicted value of each candidate action in the candidate action set. A greedy strategy is used to select the action with the highest predicted value as the next action to calculate the target value. The state vector is fed into the value evaluation network, and the predicted target value of all candidate actions is output through forward calculation. The predicted target value corresponding to the next action is extracted as the target value. During the backpropagation calculation of network gradients, a stopping gradient operator is inserted at the nodes of the cascaded computation graph of the action selection network and the value evaluation network. The gradient operator is stopped to block the backpropagation chain of the target value calculation to the action selection network parameters, so that the target value is treated as a static constant in the current iteration step, suppressing the overestimation bias caused by the action selection network and the value evaluation network updating in the same direction.

6. The method according to claim 1, characterized in that, The calculation includes immediate returns on electricity costs, degradation, and over-limit penalties, and combines energy storage health reduction factors and constraint pressure factors to generate Bellman objectives, including: Real-time electricity revenue is calculated based on time-of-use pricing and actual power flow, and peak excess penalty for transformers is calculated based on demand-based billing models. Substituting into the empirical formula for energy storage cycle life, the energy storage capacity decay caused by a single action is assessed by the current depth of discharge and ambient temperature and converted into an economic penalty term. When the predicted temperature of the computer room exceeds the safety limit, an exponentially increasing temperature over-limit penalty is imposed. Add the smoothing factor to the real-time electricity revenue, and then subtract the transformer peak excess penalty, economic penalty, and temperature over-limit penalty to obtain the real-time return. The discount rate is calculated by multiplying the preset base discount rate by the energy storage health reduction factor, which represents the usable lifespan of the energy storage, and the safety adjustment coefficient obtained by subtracting the constraint pressure factor from a constant 1. The Bellman target is formed by multiplying the immediate return by the discount rate and the target value at the next moment.

7. The method according to claim 1, characterized in that, The action selection network is updated using the truncated Huber loss of the current action value and the Bellman objective, and the value evaluation network parameters are updated using the residual gated moving average, including: Calculate the temporal difference error between the current action value output by the action selection network and the Bellman objective; When the absolute value of the timing difference error is less than the truncation threshold, the mean square error is used to calculate the loss value. When the absolute value of the timing difference error is greater than or equal to the truncation threshold, the error with a linear compensation term is used to calculate the truncated Huber loss. The action selection network is optimized using gradient descent with truncated Huber loss. Extract the parameter residual vector before and after the action selection network update. If the norm of the parameter residual vector is less than the preset gating threshold, the first smoothing coefficient is used to update the parameter exponential moving average of the action selection network to the value evaluation network. If the norm of the parameter residual vector is greater than or equal to the preset gating threshold, a second smoothing coefficient less than the first smoothing coefficient is used to perform a slow update.

8. The method according to claim 1, characterized in that, The process involves calculating a comprehensive score based on action value, constraint multipliers, and switching penalty costs, and selecting the energy storage converter with the highest power output to be sent to the data center for execution. This includes: For each action in the candidate action set, extract the expected action value score output by the action selection network. Compare the differences between each candidate action and the historical actions executed in the previous cycle, and multiply the difference by the equipment loss coefficient to obtain the switching penalty cost; The overall score for each candidate action is obtained by subtracting the switching penalty cost from the action value expectation score and subtracting the product of the constraint multiplier and the safety weight coefficient. Sort the comprehensive scores of all candidate actions in the candidate action set in descending order and obtain the action number of the first ranked action; The action number ranked first is mapped and parsed into an actual power command, which is then sent to the energy storage converter in the data center for execution via the communication interface.

9. A charging and discharging control system for an energy storage system for data centers, characterized in that, Includes the following modules: The data acquisition module is used to collect electricity price, temperature, state of charge, and transformer margin to generate state vectors. It identifies thermal inertia coefficient and equivalent thermal resistance of the computer room from historical data of temperature and cooling power. Based on the charge boundary, transformer margin, and temperature control backup power determined by thermal inertia coefficient and equivalent thermal resistance of the computer room, it constructs a normalized safety domain. After normalizing the discrete action set, it projects it onto the normalized safety domain to obtain candidate action set, projected displacement, and constraint multipliers. The determination module is used to fuse the projected displacement and constraint multipliers into a constraint pressure factor and attach it to the state vector. The exploration strategy for the next cycle is adjusted according to the constraint pressure factor. The action selection network determines the next action from the candidate action set, and the value evaluation network calculates the target value. The backpropagation gradient between the action selection network and the value evaluation network is cut off during the calculation. The execution module is used to calculate the immediate return, including electricity costs, attenuation, and out-of-bounds penalties. It generates a Bellman target by combining the energy storage health reduction factor and the constraint pressure factor. It updates the action selection network with the current action value and the truncated Huber loss of the Bellman target. It updates the value evaluation network parameters by residual gating moving average. It calculates a comprehensive score based on action value, constraint multiplier, and switching penalty cost. It selects the energy storage converter with the highest power to be sent to the data center for execution.

10. The system according to claim 9, characterized in that, The process of collecting electricity price, temperature, state of charge, and transformer margin to generate a state vector, and identifying the thermal inertia coefficient and equivalent thermal resistance of the computer room from historical temperature and cooling power data, includes: The timestamps corresponding to the control cycle are converted into time slice feature vectors using one-hot encoding. Collect cooling power and computer room temperature data within a preset time window prior to the current moment to construct a time series matrix; Using the step change in cooling power as input and the response data of the computer room temperature as output, the first-order thermal response transfer function of the system is fitted by the least squares method. The time constant in the transfer function is extracted as the thermal inertia coefficient, and the absolute value of the steady-state gain is extracted as the equivalent thermal resistance of the computer room. The state vector is formed by horizontally splicing the time slice feature vector, the various parameters collected in real time, the thermal inertia coefficient, and the equivalent thermal resistance of the computer room.

Images