A power self-adaptive control method for energy storage charging pile based on reinforcement learning
By introducing NLSTM and ADP strategies into energy storage charging piles, and constructing health memory channels and cyclic resource ledger variables, the flexibility problem of power control in energy storage charging piles is solved, enabling more accurate battery status monitoring and cost-effective power regulation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HEFEI BINNENG INFORMATION TECH CO LTD
- Filing Date
- 2025-12-29
- Publication Date
- 2026-05-19
AI Technical Summary
The power control methods of existing energy storage charging piles lack flexibility and cannot respond in real time to fluctuations in user behavior, changes in battery health status, and time-of-day differences in electricity price structure. This results in insufficient flexibility in charging and discharging regulation, affecting the economy and efficiency of energy storage systems.
We employ an NLSTM-based reinforcement learning method to optimize charge and discharge power regulation by constructing a health memory channel and a cyclic resource ledger variable, combined with the ADP strategy. This approach considers battery health, economic benefits, and response performance, achieving dynamic optimization.
It improves the operational safety and reliability of energy storage systems, enhances the stability and flexibility of strategy optimization, strengthens the economy and efficiency of regulation, and optimizes battery health management.
Smart Images
Figure CN121492737B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent control technology, and in particular to a power adaptive control method for energy storage charging piles based on reinforcement learning. Background Technology
[0002] Energy storage charging piles are gradually becoming an important infrastructure for alleviating peak grid pressure and improving the absorption capacity of new energy sources. Existing power control methods for energy storage charging piles mainly rely on fixed strategies, empirical rules, or simple load forecasting methods. Their core decision-making basis is static indicators such as historical statistical characteristics, battery state of charge, and electricity price information. These methods cannot respond in real-time to fluctuations in user behavior, changes in battery health, and time-of-day differences in electricity price structures, resulting in a lack of flexibility in charging and discharging power adjustment and insufficient peak-valley price arbitrage capabilities, thus affecting the economics of energy storage systems. While existing technologies have attempted to use traditional LSTM or other time-series models to predict power demand, they cannot capture the nonlinear changes in the state of energy storage systems under combined conditions.
[0003] At the decision-making level, traditional reinforcement learning methods typically rely on a single reward signal, lacking a comprehensive characterization of electricity price benefits, dynamic response performance, and resource consumption behavior. Existing methods often rely directly on environmental state transitions, failing to construct an internal state structure that sustainably records the energy evolution process. This prevents the formation of a complete battery resource consumption trajectory and the continuous correction of battery usage patterns during iterative training. Scheduling strategies struggle to achieve a balance among multiple regulatory objectives, easily exhibiting a bias towards high returns and low losses.
[0004] Therefore, how to provide a power adaptive control method for energy storage charging piles based on reinforcement learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0005] One objective of this invention is to propose a power adaptive control method for energy storage charging piles based on reinforcement learning. This invention utilizes NLSTM to predict charging behavior and battery state, and achieves dynamic optimization and control of charging and discharging power through cyclic resource ledger variables and ADP strategy. It takes into account battery health, economic benefits and response performance, and has the advantages of precise decision-making, strong adaptability and high resource utilization efficiency.
[0006] According to an embodiment of the present invention, a power adaptive control method for energy storage charging piles based on reinforcement learning includes the following steps:
[0007] S1. Obtain historical data, perform preprocessing, and generate time-series feature sequences;
[0008] S2. Construct NLSTM, and initialize the outer memory cell state, inner healthy memory channel state, and gating parameters;
[0009] S3. Set up a health memory channel in the inner layer of NLSTM, extract battery health-related features from the time-series feature sequence, output the health-related state vector through the inner layer health memory channel, and combine the gating input of the outer layer memory cell with the health-related state vector.
[0010] S4. Construct a state vector based on the predicted state sequence, input the state vector into the ADP algorithm, set the action set, state transition structure, long-term reward structure and cyclic resource ledger variable, and initialize the cyclic resource ledger variable.
[0011] S5. Calculate the immediate reward based on the action set and state vector, combine the immediate reward with the update result of the cyclic resource ledger variable to form the next state, use adaptive dynamic programming to iteratively update the value function and action selection strategy, and generate a scheduling action sequence.
[0012] S6. Apply the scheduling action sequence to the operation of the storage and charging equipment, dynamically adjust the charging and discharging power, record the updated data during the adjustment process, write the updated data into the cyclic resource ledger variable, and form new runtime sequence data.
[0013] S7. Runtime sequence data is used as new historical data. S1 to S6 are executed repeatedly until the scheduling action sequence meets the preset convergence condition.
[0014] Optionally, the construction of NLSTM in S2 specifically includes:
[0015] S21, NLSTM consists of an outer memory unit and an inner memory unit. The outer memory unit is used to receive the temporal feature sequence input and output the predicted state sequence. The inner memory unit sets up the healthy memory channel and initializes the healthy memory channel state.
[0016] S22. Extract the remaining battery power and battery discharge power from the time-series feature sequence, write the remaining battery power and battery discharge power into the health memory channel, calculate the health-related state vector through the inner memory unit and update the health memory channel state.
[0017] S23. In the outer memory unit, the health-related state vector and the temporal feature sequence are used together as the gating inputs of the input gate, forget gate and output gate. At each time step, the state of the outer memory unit is updated according to the gating input and a predicted state sequence is generated.
[0018] Optionally, the characteristic of step S3 is that it specifically includes:
[0019] S31. Set up a healthy memory channel in the inner memory unit, read the state of the healthy memory channel of the previous time step at the current time step, and input it into the healthy memory channel as the initial state of the healthy memory channel at the current time step.
[0020] S32. Perform nonlinear transformation processing on the battery discharge power at the current time step to form a health decay gate, which serves as the decay coefficient in the health memory channel state update process.
[0021] S33. Perform nonlinear transformation processing on the remaining battery power at the current time step to form a health gain gate, which serves as the gain coefficient in the health memory channel state update process.
[0022] S34. At the current time step, perform nonlinear amplification processing on the remaining battery power in the low power range and nonlinear enhancement processing on the battery discharge power in the high load range. Combine the results of the two types of processing and then perform nonlinear compression processing to form a healthy fluctuation quantity.
[0023] S35. At the current time step, with the health decay gate, health gain gate and health fluctuation amount as input, perform health memory channel state update processing, and write the updated health memory channel state into the health memory channel.
[0024] S36. At the current time step, using the health memory channel state as input, perform feature mapping processing to form a health-related state vector;
[0025] S37. At the current time step, concatenate the health-related state vector with the temporal feature sequence, input the input gate, forget gate, and output gate of the outer memory unit, perform the outer memory unit state update processing, and form the predicted state sequence elements.
[0026] Optionally, the construction of the loss function in the NLSTM specifically includes:
[0027] During the model training phase, output power, charging time per kilowatt-hour, cost per kilowatt-hour, battery energy storage utilization rate, and average charge / discharge time are used as supervision labels, and the difference between the predicted output and the supervision labels is used as the prediction error.
[0028] The prediction errors for peak, valley and flat periods are calculated separately under the time-segmented conditions. The prediction errors include mean square error and mean absolute error. The prediction errors for each period are used as time-segmented error terms.
[0029] During the model training phase, the energy storage utilization rate term is calculated, which is composed of the ratio of discharge amount to energy storage capacity; the power matching degree term is calculated, which is composed of the difference between the discharge output power and the customer demand power; and the equivalent cycle number term is calculated, which is composed of the ratio of discharge amount to battery capacity.
[0030] The loss function is formed by using time-period error, energy storage utilization rate, power matching degree and equivalent cycle number as components, and performing a weighted combination of each component with preset weights.
[0031] Optionally, the characteristic of step S4 is that it specifically includes:
[0032] S41. After constructing the state vector, set the action set, which consists of executable charging power adjustment amount and discharging power adjustment amount;
[0033] S42. After constructing the state vector, set the state transition structure. The state transition structure consists of the mapping relationship between the current state vector, the action set elements and the next state vector.
[0034] S43. Set a long-term return structure, which consists of a weighted sum of immediate returns and discount factors.
[0035] S44. Construct a circular resource ledger variable. The circular resource ledger variable adopts a serialized data structure and consists of several ledger record fields. Each ledger record field includes a time period number field, a battery remaining power field, a battery discharge power field, an energy storage utilization field, and a cumulative cycle amount field.
[0036] S45. In the initialization phase, assign values to the fields of each ledger record in the circular resource ledger variable, set the time period number field in chronological order, and set the remaining battery power field, battery discharge power field, energy storage utilization field and cumulative cycle amount field to initial constant values to form the initialized circular resource ledger variable.
[0037] S46. During the initialization phase, input the state vector, action set, state transition structure, long-term reward structure, and initialized circular resource ledger variables into the ADP algorithm to complete the initialization process of the ADP algorithm.
[0038] Optionally, the characteristic of step S5 is that it specifically includes:
[0039] S51. Calculate the model dynamic response delay time reward parameter based on the difference between the predicted state sequence and the historical real state at the current time step. Calculate the electricity price revenue ratio reward parameter based on the discharge amount, peak price and valley price at the current time step.
[0040] S52. At the current time step, perform a weighted combination of the model dynamic response delay time reward parameter and the electricity price revenue ratio reward parameter to form an instant reward, and perform a weighted combination of the instant reward and the next state value function output under the action of the discount factor to form a value update quantity.
[0041] S53. At the current time step, with the value update amount as input, perform iterative updates on the parameters of the value function approximator according to the temporal differential update method of adaptive dynamic programming, and adjust the action selection strategy to form scheduling action elements based on the output of the value function approximator after iterative update.
[0042] Optionally, the characteristic of step S6 is that it specifically includes:
[0043] S61. At the current time step, retrieve the scheduling action element from the scheduling action sequence and execute the corresponding charging or discharging operation at the current time step.
[0044] S62. At the end of the current time step, collect the charging amount, discharging amount, remaining battery amount, battery current, battery temperature and charging / discharging duration, and arrange them as the running data of the current time step.
[0045] S63. Read the battery remaining power field, energy storage utilization field and cumulative cycle quantity field from the previous time step from the cycle resource ledger variable. Perform a difference operation on the battery remaining power field and the battery remaining power in the current time step running data. Perform a ratio operation on the energy storage utilization field and the charging power and discharging power in the current time step running data. Perform a ratio operation on the cumulative cycle quantity field and the discharging power in the current time step running data to form the resource evolution quantity of the current time step.
[0046] S64. Perform a summation operation on the resource evolution quantity with the energy storage utilization quantity field and the cumulative cycle quantity field value of the previous time step to generate the energy storage utilization quantity field value and the cumulative cycle quantity field value of the current time step. Write the energy storage utilization quantity field value, the cumulative cycle quantity field value, and the remaining battery power, charging power, discharging power, discharging power and battery temperature in the current time step running data into the cycle resource ledger variable, replace the corresponding fields of the previous time step, and obtain the updated cycle resource ledger record.
[0047] S65. The remaining battery power field, energy storage utilization field, and cumulative cycle quantity field are combined with the time-of-use electricity price, user load data, and charging order data in the current time step running data to form the state description vector of the current time step. The state description vector is written into the current time step position of the time series feature sequence.
[0048] S66. When constructing the state vector in the next time step, read the data record written in the current time step from the time-series feature sequence and the cyclic resource ledger variable, and record the new runtime sequence data;
[0049] Optionally, the characteristic of step S7 is that it specifically includes:
[0050] S71. After the generation and execution of the scheduling action sequence are completed in the current iteration round, the runtime sequence data formed in the current iteration round is written into the historical data set.
[0051] S72. Calculate the difference between the value function output of the previous iteration and the value function output of the current iteration, and calculate the difference between the instantaneous return of the current iteration and the instantaneous return of the previous iteration.
[0052] S73. When the difference in the output of the value function is lower than the preset value change threshold and the difference in the immediate return is lower than the preset return change threshold, the scheduling action sequence of the current iteration round is determined to meet the convergence condition, and the iteration is terminated; if the convergence condition is not met, the runtime sequence data of the current iteration round is used as the input of the next iteration round, and S1 to S6 are executed repeatedly.
[0053] Optionally, the scheduling action sequence specifically includes:
[0054] The scheduling action element represents the charging power or discharging power in the form of discrete power levels. During the control process, the controller extracts the scheduling action element of the current time step from the scheduling action sequence and maps the scheduling action element to a specific charging power or discharging power value.
[0055] Optionally, the NLSTM outputs a predicted state sequence at the current time step, which is combined with the battery remaining capacity field, energy storage utilization field, and cumulative cycle quantity field recorded in the cyclic resource ledger variable in a preset order to form the state vector of the ADP algorithm at the current time step. The ADP algorithm calculates the value function based on the state vector and selects scheduling action elements. The scheduling action elements are used as the action output at the current time step to perform charging or discharging operations.
[0056] The beneficial effects of this invention are:
[0057] (1) By setting up a health memory channel in NLSTM and constructing a health-related state vector, the present invention enables the independent expression of the impact of the remaining battery power and discharge power on battery resource consumption. It can capture the nonlinear change law of battery health status under different operating ranges, and can more accurately reflect the resource evolution process, thereby improving the safety and reliability of long-term operation of the energy storage system.
[0058] (2) By constructing a circular resource ledger variable and establishing a cross-time resource evolution field update mechanism, the present invention enables the charging and discharging amount, remaining power, energy storage utilization and cumulative circular amount of each time step to form a traceable continuous record and participate in the construction of subsequent state vectors. This allows reinforcement learning to consider both short-term performance and long-term resource consumption trends during policy update, overcomes the resource constraint expression ability, and improves the stability and interpretability of policy optimization.
[0059] (3) This invention iteratively optimizes the scheduling action through adaptive dynamic programming, and adopts a combination of model dynamic response delay time reward parameter and electricity price revenue ratio reward parameter, so that the scheduling strategy can achieve a balance between peak and valley electricity price difference, user load fluctuation and battery health limitation, with higher regulation flexibility and economy, and can improve the revenue level and operating efficiency of energy storage charging piles. Attached Figure Description
[0060] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:
[0061] Figure 1 This is a flowchart of a power adaptive control method for energy storage charging piles based on reinforcement learning proposed in this invention;
[0062] Figure 2 This is a diagram of the NLSTM double-layer memory structure of the power adaptive control method for energy storage charging piles based on reinforcement learning proposed in this invention.
[0063] Figure 3 This is a flowchart illustrating the reward function composition and value update process of a power adaptive control method for energy storage charging piles based on reinforcement learning proposed in this invention. Detailed Implementation
[0064] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.
[0065] refer to Figure 1-3 A power adaptive control method for energy storage charging piles based on reinforcement learning includes the following steps:
[0066] S1. Obtain historical data, perform preprocessing, and generate time-series feature sequences;
[0067] S2. Construct NLSTM, and initialize the outer memory cell state, inner healthy memory channel state, and gating parameters;
[0068] S3. Set up a health memory channel in the inner layer of NLSTM, extract battery health-related features from the time-series feature sequence, output the health-related state vector through the inner layer health memory channel, and combine the gating input of the outer layer memory cell with the health-related state vector.
[0069] S4. Construct a state vector based on the predicted state sequence, input the state vector into the ADP algorithm, set the action set, state transition structure, long-term reward structure and cyclic resource ledger variable, and initialize the cyclic resource ledger variable.
[0070] S5. Calculate the immediate reward based on the action set and state vector, combine the immediate reward with the update result of the cyclic resource ledger variable to form the next state, use adaptive dynamic programming to iteratively update the value function and action selection strategy, and generate a scheduling action sequence.
[0071] S6. Apply the scheduling action sequence to the operation of the storage and charging equipment, dynamically adjust the charging and discharging power, record the updated data during the adjustment process, write the updated data into the cyclic resource ledger variable, and form new runtime sequence data.
[0072] S7. Runtime sequence data is used as new historical data. S1 to S6 are executed repeatedly until the scheduling action sequence meets the preset convergence condition.
[0073] In this embodiment, S1 specifically includes:
[0074] Data sources include charging order data, customer vehicle data, external environment data, electricity price data, and battery data. For charging order data, customer type information, vehicle model information, charging time period information, and charging duration information are obtained. For external environment data, temperature, humidity, and weather information are obtained. For electricity price data, the charging rate for the corresponding time period is obtained. For battery data, the remaining battery capacity and battery discharge power are obtained. For charging infrastructure data, the location of charging stations is obtained. All of the above data together constitute the model's input vector, which consists of, in order: customer type, vehicle model, charging time period, charging duration, temperature, humidity, weather, charging rate, remaining battery capacity, battery discharge power, and charging station location.
[0075] In the above input vector, the data related to battery health include the remaining battery power and the battery discharge power; the other features reflect user behavior, environmental conditions, electricity price level and spatial location information.
[0076] The output vector is constructed as a supervision label, which includes output power, charging time per kWh, cost per kWh, battery energy storage utilization rate, and average charge / discharge time.
[0077] For numerical features such as temperature, humidity, charging time, charging rate, remaining battery capacity, battery discharge power, output power, battery energy storage utilization rate, and average charging / discharging time, their respective average values and variances are calculated. The original values are then linearly scaled based on the average values and variances to map all numerical features to a value range of 0 to 1. For enumerated features such as weather, customer type, vehicle model, charging time period, and charging station location, different categories are encoded into fixed numerical identifiers through an enumeration mapping method and further normalized to the range of 0 to 1, resulting in a time-series feature sequence.
[0078] In this embodiment, the construction of NLSTM in S2 specifically includes:
[0079] S21, NLSTM consists of an outer memory unit and an inner memory unit. The outer memory unit is used to receive the temporal feature sequence input and output the predicted state sequence. The inner memory unit sets up the healthy memory channel and initializes the healthy memory channel state.
[0080] S22. Extract the remaining battery power and battery discharge power from the time-series feature sequence, write the remaining battery power and battery discharge power into the health memory channel, calculate the health-related state vector through the inner memory unit and update the health memory channel state.
[0081] S23. In the outer memory unit, the health-related state vector and the temporal feature sequence are used together as the gating inputs of the input gate, forget gate and output gate. At each time step, the state of the outer memory unit is updated according to the gating input and a predicted state sequence is generated.
[0082] In this embodiment, S3 specifically includes:
[0083] S31. Set up a healthy memory channel in the inner memory unit, read the state of the healthy memory channel of the previous time step at the current time step, and input it into the healthy memory channel as the initial state of the healthy memory channel at the current time step.
[0084] S32. Perform nonlinear transformation processing on the battery discharge power at the current time step to form a health decay gate, which serves as the decay coefficient in the health memory channel state update process.
[0085] S33. Perform nonlinear transformation processing on the remaining battery power at the current time step to form a health gain gate, which serves as the gain coefficient in the health memory channel state update process.
[0086] S34. At the current time step, perform nonlinear amplification processing on the remaining battery power in the low power range and nonlinear enhancement processing on the battery discharge power in the high load range. Combine the results of the two types of processing and then perform nonlinear compression processing to form a healthy fluctuation quantity.
[0087] S35. At the current time step, with the health decay gate, health gain gate and health fluctuation amount as input, perform health memory channel state update processing, and write the updated health memory channel state into the health memory channel.
[0088] S36. At the current time step, using the health memory channel state as input, perform feature mapping processing to form a health-related state vector;
[0089] S37. At the current time step, concatenate the health-related state vector with the temporal feature sequence, input the input gate, forget gate, and output gate of the outer memory unit, perform the outer memory unit state update processing, and form the predicted state sequence elements.
[0090] In this embodiment, the specific update process in S3 includes:
[0091] The health degradation threshold depends on the battery discharge power and is calculated using a sigmoid nonlinear function.
[0092] The health gain gate depends on the remaining battery power and uses a nonlinear function with the same structure.
[0093] The health fluctuation quantity is constructed in three parts: a nonlinear amplification term in the low-charge range, a nonlinear enhancement term in the high-load range, and a combination followed by nonlinear compression. The nonlinear amplification term in the low-charge range is (1 - remaining battery charge)^2; the nonlinear enhancement term in the high-load range is the cube of the battery discharge power; the nonlinear amplification term in the low-charge range and the nonlinear enhancement term in the high-load range are combined with weights of 1.0 and 0.5, respectively; the nonlinear compression after combination is performed using a hyperbolic tangent function.
[0094] The update method is a sum of two terms: one term is the value calculated by the health decay gate multiplied by the health memory channel state of the previous time step, and the other term is the value calculated by the gain gate multiplied by the health fluctuation amount.
[0095] The updated health memory channel state is linearly mapped to generate a health-related state vector; the health-related state vector is concatenated with the temporal feature sequence according to its dimensions; and the outer state is updated.
[0096] In this embodiment, the construction of the loss function in the NLSTM specifically includes:
[0097] During the model training phase, output power, charging time per kilowatt-hour, cost per kilowatt-hour, battery energy storage utilization rate, and average charge / discharge time are used as supervision labels, and the difference between the predicted output and the supervision labels is used as the prediction error.
[0098] The prediction errors for peak, valley and flat periods are calculated separately under the time-segmented conditions. The prediction errors include mean square error and mean absolute error. The prediction errors for each period are used as time-segmented error terms.
[0099] During the model training phase, the energy storage utilization rate term is calculated, which is composed of the ratio of discharge amount to energy storage capacity; the power matching degree term is calculated, which is composed of the difference between the discharge output power and the customer demand power; and the equivalent cycle number term is calculated, which is composed of the ratio of discharge amount to battery capacity.
[0100] The loss function is composed of time-segmentation error, energy storage utilization rate, power matching degree, and equivalent cycle count. Each component is weighted and combined using preset weights to form the NLSTM loss function. In this embodiment, the weights are 0.35, 0.35, and 0.3, respectively.
[0101] In this embodiment, S4 specifically includes:
[0102] S41. After constructing the state vector, set the action set, which consists of executable charging power adjustment amount and discharging power adjustment amount;
[0103] S42. After constructing the state vector, set the state transition structure. The state transition structure consists of the mapping relationship between the current state vector, the action set elements and the next state vector.
[0104] S43. Set a long-term return structure, which consists of a weighted sum of immediate returns and discount factors.
[0105] S44. Construct a circular resource ledger variable. The circular resource ledger variable adopts a serialized data structure and consists of several ledger record fields. Each ledger record field includes a time period number field, a battery remaining power field, a battery discharge power field, an energy storage utilization field, and a cumulative cycle amount field.
[0106] S45. In the initialization phase, assign values to the fields of each ledger record in the circular resource ledger variable, set the time period number field in chronological order, and set the remaining battery power field, battery discharge power field, energy storage utilization field and cumulative cycle amount field to initial constant values to form the initialized circular resource ledger variable.
[0107] S46. During the initialization phase, input the state vector, action set, state transition structure, long-term reward structure, and initialized circular resource ledger variables into the ADP algorithm to complete the initialization process of the ADP algorithm.
[0108] In this embodiment, S5 specifically includes:
[0109] S51. At the current time step, calculate the model dynamic response delay time reward parameter based on the difference between the predicted state sequence and the historical true state. At the current time step, calculate the electricity price revenue ratio reward parameter based on the discharge amount, peak price, and valley price. The model dynamic response delay time reward parameter is composed of the absolute value of the difference between the predicted change and the actual change, and the difference is scaled to the range of 0 to 1 by a normalization coefficient. The electricity price revenue ratio reward parameter is calculated by the ratio of the revenue item composed of the difference between the peak price and the valley price to the valley price charging cost item. The product of the discharge amount and the peak-valley price difference forms the peak price revenue accumulation, and the product of the charging amount and the valley price forms the valley price cost accumulation.
[0110] S52. At the current time step, the dynamic response delay time reward parameter and the electricity price revenue ratio reward parameter are weighted and combined to form the immediate return. The immediate return is then weighted and combined with the output of the next state value function under the effect of the discount factor to form the value update amount. In this embodiment, the weight of the dynamic response delay time reward parameter is 0.4, the weight of the electricity price revenue ratio reward parameter is 0.6, and the discount factor is 0.95. That is, when calculating the value update amount, the immediate return is linearly combined with 0.95 times the output of the next state value function.
[0111] S53. At the current time step, using the value update as input, iteratively update the parameters of the value function approximator according to the temporal difference update method of adaptive dynamic programming. Adjust the action selection strategy to form scheduling action elements based on the updated value function approximator output. The value function approximator adopts a feedforward neural network structure with a single hidden layer; the value function approximator takes the current state vector as input and the scalar prediction value of the state value as output; the hidden layer has 32 neurons with ReLU activation function, the output layer has 1 dimension and the output layer activation function is identity mapping; the parameter set of the value function approximator includes the input layer weight matrix, the hidden layer weight matrix and the bias of each layer; the update adopts the temporal difference gradient descent method with a learning rate of 0.001, and the action selection strategy adopts epsilon-greedy with an epsilon value of 0.1.
[0112] In this embodiment, S6 specifically includes:
[0113] S61. At the current time step, retrieve the scheduling action element from the scheduling action sequence and execute the corresponding charging or discharging operation at the current time step; in this embodiment, the control period is set to 15 minutes.
[0114] S62. At the end of the current time step, collect the charging amount, discharging amount, remaining battery amount, battery current, battery temperature and charging / discharging duration, and arrange them as the running data of the current time step.
[0115] S63. Read the battery remaining capacity field, energy storage utilization field, and cumulative cycle quantity field from the previous time step from the circular resource ledger variables. Perform a difference operation on the battery remaining capacity field and the battery remaining capacity in the current time step's running data. Perform a ratio operation on the energy storage utilization field and the charging and discharging capacity in the current time step's running data. Perform a ratio operation on the cumulative cycle quantity field and the discharging capacity in the current time step's running data to form the resource evolution quantity for the current time step. Read the battery remaining capacity field value, energy storage utilization field value, and cumulative cycle quantity field value recorded in the previous time step from the circular resource ledger variables. Subtract the battery remaining capacity field value from the previous time step and the battery remaining capacity in the current time step's running data to obtain the change in remaining capacity for the current time step. The change in energy quantity is considered part of the resource evolution quantity. The energy storage utilization value from the previous time step is combined with the charging and discharging quantities in the current time step's operating data. The energy storage utilization increment is obtained by dividing the sum of the charging and discharging quantities in the current time step by the sum of the energy storage utilization value from the previous time step and a small constant to prevent division by zero. This energy storage utilization increment is considered part of the resource evolution quantity. The cumulative cycle quantity from the previous time step is combined with the discharging quantity in the current time step's operating data. The ratio of the discharging quantity to the battery's rated capacity is used as the cycle equivalent increment in the current time step. This cycle equivalent increment is considered another part of the resource evolution quantity. The remaining energy quantity change, energy storage utilization increment, and cycle equivalent increment together constitute the resource evolution quantity for the current time step.
[0116] S64. Perform summation operations on the resource evolution quantity with the energy storage utilization quantity field and the cumulative cycle quantity field value of the previous time step, respectively, to generate the energy storage utilization quantity field value and the cumulative cycle quantity field value of the current time step. Write the energy storage utilization quantity field value, the cumulative cycle quantity field value, and the battery remaining power, charging power, discharging power, discharging power, and battery temperature from the current time step's running data into the cycle resource ledger variable, replacing the corresponding fields of the previous time step, to obtain the updated cycle resource ledger record; sum the energy storage utilization quantity field value of the previous time step with the energy storage utilization increment of the current time step, and use the summation result as the energy storage utilization quantity field value of the current time step; sum the cumulative cycle quantity field value of the previous time step with the cycle equivalent increment of the current time step, and use the summation result as the cumulative cycle quantity field value of the current time step.
[0117] S65. The remaining battery power field, energy storage utilization field, and cumulative cycle quantity field are combined with the time-of-use electricity price, user load data, and charging order data in the current time step running data to form the state description vector of the current time step. The state description vector is written into the current time step position of the time series feature sequence.
[0118] S66. When constructing the state vector in the next time step, read the data record written in the current time step from the time-series feature sequence and the cyclic resource ledger variable, and record the new runtime sequence data;
[0119] In this embodiment, S7 specifically includes:
[0120] S71. After the generation and execution of the scheduling action sequence are completed in the current iteration round, the runtime sequence data formed in the current iteration round is written into the historical data set.
[0121] S72. Calculate the difference between the value function output of the previous iteration and the value function output of the current iteration, and calculate the difference between the instantaneous return of the current iteration and the instantaneous return of the previous iteration.
[0122] S73. When the difference in the output of the value function is lower than the preset value change threshold of 0.005 and the difference in the immediate return is lower than the preset return change threshold of 0.01, the scheduling action sequence of the current iteration round is determined to meet the convergence condition, and the iteration is terminated; if the convergence condition is not met, the runtime sequence data of the current iteration round is used as the input of the next iteration round, and S1 to S6 are repeated.
[0123] In this embodiment, the scheduling action sequence specifically includes:
[0124] The scheduling action element represents the charging power or discharging power in the form of discrete power levels. During the control process, the controller extracts the scheduling action element of the current time step from the scheduling action sequence and maps the scheduling action element to a specific charging power or discharging power value.
[0125] In this embodiment, the predicted state sequence output by NLSTM at the current time step is combined with the battery remaining power field, energy storage utilization field, and cumulative cycle quantity field recorded in the cyclic resource ledger variable in a preset order to form the state vector of the ADP algorithm at the current time step. The ADP algorithm calculates the value function based on the state vector and selects the scheduling action element. The scheduling action element is used as the action output of the current time step to perform charging or discharging operations.
[0126] Example 1:
[0127] To verify the feasibility of this invention in practice, it was applied to a typical charging station with multiple energy storage charging piles. The station hosts a large number of electric vehicles with diverse user behaviors, resulting in significant uncertainties in average arrival time, charging duration, and expected charging power. Simultaneously, the energy storage system's batteries are subject to frequent charge-discharge cycles and power fluctuations during long-term operation, leading to issues such as unstable remaining charge, frequent deep discharge triggers, and high battery temperature rise. Traditional scheduling methods, often based on fixed power strategies or simple load forecasting, struggle to coordinate the relationship between electricity price revenue, battery health, and load changes, frequently resulting in excessive battery health degradation, low scheduling efficiency, and high energy costs.
[0128] In this scenario, the method of this invention introduces a health memory channel of NLSTM to model the differential health impacts of battery remaining capacity and discharge power in different operating ranges. Health decay and health gain gates reflect different health fluctuation trends under high load and high remaining capacity conditions, respectively, enabling NLSTM predictions to more realistically characterize the battery state evolution. Cyclic resource ledger variables record charge / discharge capacity, cycle equivalent, battery temperature, and energy storage utilization at each time step, allowing the scheduling strategy to form a continuous resource consumption trajectory over multiple iterations, achieving a quantitative expression of long-term health constraints. The ADP algorithm combines dynamic response delay time reward parameters and electricity price revenue ratio reward parameters, enabling the scheduling strategy to obtain higher economic returns while ensuring rapid response to user charging needs, avoiding additional cycle losses caused by over-discharge.
[0129] The application process of this invention includes preprocessing historical operating data, charging orders, user behavior characteristics, environmental characteristics, and time-of-use electricity price information, and constructing a time-series feature sequence based on the preprocessing results. NLSTM is responsible for generating the predicted state sequence for the next time step, constructing a more complete state vector for ADP. The scheduling actions output by ADP serve as the charging and discharging power settings for the energy storage charging pile during actual operation. After the operation ends, new operating data is collected again and the cyclic resource ledger variables are updated, entering the next round of iterative training until the scheduling output reaches the convergence condition, ensuring the stability and reliability of the strategy.
[0130] To demonstrate the beneficial effects of this invention, actual operational data was collected after dozens of consecutive runs at the site and compared with operational data from traditional experience-based scheduling methods. The comparison results show that the method of this invention has advantages in power regulation smoothness, battery cycle depth control, improved electricity price revenue, and battery temperature suppression. The following are comparison results of some typical operational data to demonstrate the effectiveness of the method of this invention in practical applications.
[0131] Table 1: Comparison of the operational effects of the method of this invention and the traditional method
[0132] Indicator Name Traditional methods Method of the present invention Improvement range Average charging power fluctuation (kW) 3.8 1.5 -60.5% Maximum depth of discharge (SOC%) 11% 6% -45.4% Daily average cyclic equivalent 1.62 1.03 -36.4% Energy storage utilization efficiency (%) 78% 91% +16.7% Average operating temperature (°C) 38.5 33.2 -13.8% Peak-valley electricity price arbitrage profit (unit value) 12.4 18.7 +50.8% Charging order fulfillment rate (%) 92.1% 98.6% +7.1% Dynamic response latency (seconds) 4.3 2.1 -51.2%
[0133] As shown in Table 1, the method of this invention demonstrates advantages in several key operational dimensions. Regarding average charging power fluctuation, the invention, through a state expression composed of a health memory channel and resource ledger variables, enables the scheduling strategy to avoid frequent and large power fluctuations, reducing the average fluctuation amplitude to less than half that of traditional methods, thus improving battery operational stability. Regarding maximum depth of discharge and daily average cycle equivalent, the method of this invention effectively suppresses excessively deep discharge and high-frequency cycling behavior, reducing cycle consumption by more than one-third, contributing to an improvement in the overall lifespan of the energy storage battery. Regarding energy storage utilization efficiency, thanks to the predictive capabilities of NLSTM and the revenue optimization strategy of ADP, this invention can more fully utilize energy storage capacity, reduce ineffective power exchange, and improve utilization efficiency by more than 16%. Regarding operating temperature, the method of this invention reduces battery temperature rise by nearly 14% by limiting rapid discharge in high-load areas, further ensuring operational safety. Regarding economics, the invention, through optimized scheduling strategies, increases peak-valley arbitrage revenue by more than 50%, demonstrating a cost advantage. By responding more accurately to user behavior, the method of this invention increases the order fulfillment rate to near full capacity and reduces dynamic response latency, effectively improving user experience and the real-time performance of the control system.
[0134] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A power adaptive control method for energy storage charging piles based on reinforcement learning, characterized in that, Includes the following steps: S1. Obtain historical data, perform preprocessing, and generate time-series feature sequences; S2. Construct NLSTM, and initialize the outer memory cell state, inner healthy memory channel state, and gating parameters; S3. Set up a health memory channel in the inner layer of NLSTM, extract battery health-related features from the time-series feature sequence, output the health-related state vector through the inner layer health memory channel, and combine the gating input of the outer layer memory cell with the health-related state vector. S3 specifically includes: S31. Set up a healthy memory channel in the inner memory unit, read the state of the healthy memory channel of the previous time step at the current time step, and input it into the healthy memory channel as the initial state of the healthy memory channel at the current time step. S32. Perform nonlinear transformation processing on the battery discharge power at the current time step to form a health decay gate, which serves as the decay coefficient in the health memory channel state update process. S33. Perform nonlinear transformation processing on the remaining battery power at the current time step to form a health gain gate, which serves as the gain coefficient in the health memory channel state update process. S34. At the current time step, perform nonlinear amplification processing on the remaining battery power in the low power range and nonlinear enhancement processing on the battery discharge power in the high load range. Combine the results of the two types of processing and then perform nonlinear compression processing to form a healthy fluctuation quantity. S35. At the current time step, with the health decay gate, health gain gate and health fluctuation amount as input, perform health memory channel state update processing, and write the updated health memory channel state into the health memory channel. S36. At the current time step, using the health memory channel state as input, perform feature mapping processing to form a health-related state vector; S37. At the current time step, concatenate the health-related state vector with the temporal feature sequence, input the input gate, forget gate and output gate of the outer memory unit, perform the outer memory unit state update processing, and form the predicted state sequence elements. S4. Construct a state vector based on the predicted state sequence, input the state vector into the ADP algorithm, set the action set, state transition structure, long-term reward structure and cyclic resource ledger variable, and initialize the cyclic resource ledger variable. S4 specifically includes: S44. Construct a circular resource ledger variable. The circular resource ledger variable adopts a serialized data structure and consists of several ledger record fields. Each ledger record field includes a time period number field, a battery remaining power field, a battery discharge power field, an energy storage utilization field, and a cumulative cycle amount field. S45. In the initialization phase, assign values to the fields of each ledger record in the circular resource ledger variable, set the time period number field in chronological order, and set the remaining battery power field, battery discharge power field, energy storage utilization field and cumulative cycle amount field to initial constant values to form the initialized circular resource ledger variable. S5. Calculate the immediate reward based on the action set and state vector, combine the immediate reward with the update result of the cyclic resource ledger variable to form the next state, use adaptive dynamic programming to iteratively update the value function and action selection strategy, and generate a scheduling action sequence. S6. Apply the scheduling action sequence to the operation of the storage and charging equipment, dynamically adjust the charging and discharging power, record the updated data during the adjustment process, write the updated data into the cyclic resource ledger variable, and form new runtime sequence data. S7. Runtime sequence data is used as new historical data. S1 to S6 are executed repeatedly until the scheduling action sequence meets the preset convergence condition.
2. The power adaptive control method for energy storage charging piles based on reinforcement learning according to claim 1, characterized in that, The construction of NLSTM in S2 specifically includes: S21, NLSTM consists of an outer memory unit and an inner memory unit. The outer memory unit is used to receive the temporal feature sequence input and output the predicted state sequence. The inner memory unit sets up the healthy memory channel and initializes the healthy memory channel state. S22. Extract the remaining battery power and battery discharge power from the time-series feature sequence, write the remaining battery power and battery discharge power into the health memory channel, calculate the health-related state vector through the inner memory unit and update the health memory channel state. S23. In the outer memory unit, the health-related state vector and the temporal feature sequence are used together as the gating inputs of the input gate, forget gate and output gate. At each time step, the state of the outer memory unit is updated according to the gating input and a predicted state sequence is generated.
3. The power adaptive control method for energy storage charging piles based on reinforcement learning according to claim 2, characterized in that, The construction of the loss function in the NLSTM specifically includes: During the model training phase, output power, charging time per kilowatt-hour, cost per kilowatt-hour, battery energy storage utilization rate, and average charge / discharge time are used as supervision labels, and the difference between the predicted output and the supervision labels is used as the prediction error. The prediction errors for peak, valley and flat periods are calculated separately under the time-segmented conditions. The prediction errors include mean square error and mean absolute error. The prediction errors for each period are used as time-segmented error terms. During the model training phase, the energy storage utilization rate term is calculated, which is composed of the ratio of discharge amount to energy storage capacity; the power matching degree term is calculated, which is composed of the difference between the discharge output power and the customer demand power; and the equivalent cycle number term is calculated, which is composed of the ratio of discharge amount to battery capacity. The loss function is formed by using time-period error, energy storage utilization rate, power matching degree and equivalent cycle number as components, and performing a weighted combination of each component with preset weights.
4. The power adaptive control method for energy storage charging piles based on reinforcement learning according to claim 3, characterized in that, S4 specifically also includes: S41. After constructing the state vector, set the action set, which consists of executable charging power adjustment amount and discharging power adjustment amount; S42. After constructing the state vector, set the state transition structure. The state transition structure consists of the mapping relationship between the current state vector, the action set elements and the next state vector. S43. Set a long-term return structure, which consists of a weighted sum of immediate returns and discount factors. S46. During the initialization phase, input the state vector, action set, state transition structure, long-term reward structure, and initialized circular resource ledger variables into the ADP algorithm to complete the initialization process of the ADP algorithm.
5. The power adaptive control method for energy storage charging piles based on reinforcement learning according to claim 4, characterized in that, S5 specifically includes: S51. Calculate the model dynamic response delay time reward parameter based on the difference between the predicted state sequence and the historical real state at the current time step. Calculate the electricity price revenue ratio reward parameter based on the discharge amount, peak price and valley price at the current time step. S52. At the current time step, perform a weighted combination of the model dynamic response delay time reward parameter and the electricity price revenue ratio reward parameter to form an instant reward, and perform a weighted combination of the instant reward and the next state value function output under the action of the discount factor to form a value update quantity. S53. At the current time step, with the value update amount as input, perform iterative updates on the parameters of the value function approximator according to the temporal differential update method of adaptive dynamic programming, and adjust the action selection strategy to form scheduling action elements based on the output of the value function approximator after iterative update.
6. The power adaptive control method for energy storage charging piles based on reinforcement learning according to claim 5, characterized in that, S6 specifically includes: S61. At the current time step, retrieve the scheduling action element from the scheduling action sequence and execute the corresponding charging or discharging operation at the current time step. S62. At the end of the current time step, collect the charging amount, discharging amount, remaining battery amount, battery current, battery temperature and charging / discharging duration, and arrange them as the running data of the current time step. S63. Read the battery remaining power field, energy storage utilization field and cumulative cycle quantity field from the previous time step from the cycle resource ledger variable. Perform a difference operation on the battery remaining power field and the battery remaining power in the current time step running data. Perform a ratio operation on the energy storage utilization field and the charging power and discharging power in the current time step running data. Perform a ratio operation on the cumulative cycle quantity field and the discharging power in the current time step running data to form the resource evolution quantity of the current time step. S64. Perform a summation operation on the resource evolution quantity with the energy storage utilization quantity field and the cumulative cycle quantity field value of the previous time step to generate the energy storage utilization quantity field value and the cumulative cycle quantity field value of the current time step. Write the energy storage utilization quantity field value, the cumulative cycle quantity field value, and the remaining battery power, charging power, discharging power, discharging power and battery temperature in the current time step running data into the cycle resource ledger variable, replace the corresponding fields of the previous time step, and obtain the updated cycle resource ledger record. S65. The remaining battery power field, energy storage utilization field, and cumulative cycle quantity field are combined with the time-of-use electricity price, user load data, and charging order data in the current time step running data to form the state description vector of the current time step. The state description vector is written into the current time step position of the time series feature sequence. S66. When constructing the state vector at the next time step, read the data record written at the current time step from the time-series feature sequence and the cyclic resource ledger variable, and record the new runtime sequence data.
7. The power adaptive control method for energy storage charging piles based on reinforcement learning according to claim 6, characterized in that, Specifically, S7 includes: S71. After the generation and execution of the scheduling action sequence are completed in the current iteration round, the runtime sequence data formed in the current iteration round is written into the historical data set. S72. Calculate the difference between the value function output of the previous iteration and the value function output of the current iteration, and calculate the difference between the instantaneous return of the current iteration and the instantaneous return of the previous iteration. S73. When the difference in the output of the value function is lower than the preset value change threshold and the difference in the immediate return is lower than the preset return change threshold, the scheduling action sequence of the current iteration round is determined to meet the convergence condition, and the iteration is terminated; if the convergence condition is not met, the runtime sequence data of the current iteration round is used as the input of the next iteration round, and S1 to S6 are executed repeatedly.
8. The power adaptive control method for energy storage charging piles based on reinforcement learning according to claim 7, characterized in that, The scheduling action sequence specifically includes: The scheduling action element represents the charging power or discharging power in the form of discrete power levels. During the control process, the controller extracts the scheduling action element of the current time step from the scheduling action sequence and maps the scheduling action element to a specific charging power or discharging power value.
9. The power adaptive control method for energy storage charging piles based on reinforcement learning according to claim 8, characterized in that, The predicted state sequence output by NLSTM at the current time step is combined with the battery remaining capacity field, energy storage utilization field, and cumulative cycle quantity field recorded in the cyclic resource ledger variable in a preset order to form the state vector of the ADP algorithm at the current time step. The ADP algorithm calculates the value function based on the state vector and selects scheduling action elements. The scheduling action elements are used as the action output of the current time step to perform charging or discharging operations.