Reinforcement learning based charging pile load adaptive allocation and energy consumption optimization system
By using digital twin modeling and reinforcement learning to create a closed-loop control structure, and employing the TD3 algorithm and Gaussian perturbation Lagrange constraints to optimize the power allocation of charging piles, the dynamic fluctuation and energy consumption optimization problems in charging pile load control are solved, achieving stable and efficient load management.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HANGZHOU HANGCHI ELECTRONIC TECH CO LTD
- Filing Date
- 2026-03-04
- Publication Date
- 2026-06-05
AI Technical Summary
Existing charging pile load control methods are unable to cope with dynamic load fluctuations, sudden charging requests and grid response limitations, resulting in unbalanced power distribution, low task response efficiency, and the reinforcement learning strategy is susceptible to abnormal data interference, resulting in unstable scheduling results and a lack of energy consumption optimization capabilities.
A closed-loop control structure based on digital twin modeling and reinforcement learning is constructed. The TD3 algorithm is used to generate continuous policy actions. Power optimization is performed by combining Gaussian perturbation and Lagrangian constraints. Charging behavior simulation and reward calculation are carried out through digital twin model to realize synchronous update of policy network and model parameters.
It improves the stability and execution efficiency of power allocation, significantly enhances the convergence speed and scheduling robustness of policy training, reduces energy consumption, and meets the intelligent energy management needs of multi-stack collaboration and high-frequency scheduling scenarios.
Smart Images
Figure CN122159260A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of load management technology, and in particular to a charging pile load adaptive allocation and energy consumption optimization system based on reinforcement learning. Background Technology
[0002] With the continuous growth of new energy vehicle ownership, the deployment density of charging piles in urban areas is constantly increasing, bringing about a demand for large-scale, concurrent, and dynamic charging load management. In order to achieve unified scheduling of regional charging behavior, reduce the impact of peak load on the power grid, and optimize energy utilization efficiency, intelligent scheduling technology for group charging piles has gradually become a research hotspot.
[0003] Existing charging load control methods mostly employ rule-based static scheduling strategies or linear programming optimization methods. These methods essentially rely on pre-defined parameter models and scheduling boundary conditions, which often struggle to cope with complex operating conditions such as dynamic load fluctuations, sudden charging requests, and grid response limitations in practical applications. Especially in scenarios involving multiple charging piles working together, traditional methods lack real-time learning and adjustment capabilities, leading to uneven power distribution, low task response efficiency, and potentially even wasting electrical resources and amplifying peak-valley load fluctuations.
[0004] Meanwhile, some studies have begun to try to introduce reinforcement learning algorithms to achieve adaptive optimization of charging strategies, but the following problems still exist in actual deployment: First, the training process cannot be carried out in real power systems, and there is a lack of a safe and effective policy trial and error environment; second, reinforcement learning strategies are easily affected by abnormal data, and the scheduling results are unstable; third, there is a lack of systematic modeling and optimization capabilities for actual feedback results, resulting in poor policy generalization.
[0005] Furthermore, current reinforcement learning control structures commonly employ update mechanisms based on immediate rewards, failing to incorporate the energy consumption evolution characteristics and task response efficiency throughout the charging process. This makes it difficult to achieve coordinated optimization control of energy consumption and system load stability. Simultaneously, inadequate design of policy perturbation methods and action constraint mechanisms leads to frequent fluctuations in power commands near boundary intervals, impacting the robustness and energy efficiency of policy execution.
[0006] Therefore, how to provide a charging pile load adaptive allocation and energy consumption optimization system based on reinforcement learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0007] One objective of this invention is to propose a charging pile load adaptive allocation and energy consumption optimization system based on reinforcement learning. This invention integrates digital twin modeling and reinforcement learning algorithms to construct a closed-loop control structure encompassing data acquisition, policy perturbation, simulation optimization, and feedback correction. This solves the problems of uneven power distribution, large load fluctuations, and high energy consumption in multi-charging pile scenarios. The system uses the TD3 algorithm to generate continuous policy actions, combines Gaussian perturbation and Lagrange constraints for power optimization, simulates charging behavior and calculates rewards through a digital twin model, and uses an error comparison mechanism to synchronize the update of policy network and model parameters. It possesses advantages such as low energy consumption, strong adaptability, and high scheduling stability.
[0008] The reinforcement learning-based adaptive load allocation and energy consumption optimization system for charging piles according to embodiments of the present invention includes:
[0009] The data acquisition module is used to build a digital twin model corresponding to the charging area, collect the operating data of the charging area in real time at a fixed frequency, and generate a charging state vector.
[0010] The strategy allocation module is used to construct a strategy network, perform continuous action generation operations on the charging state vector, introduce Gaussian noise to perturb the strategy during the generation process, and perform range compression and constraint correction on the perturbated action vector to generate a power allocation vector.
[0011] The digital simulation module is used to simulate charging behavior in a digital twin model using power allocation vector as control command, record simulation data during the simulation process to generate a set of simulation results, and calculate the simulation reward value based on the set of simulation results.
[0012] The strategy update module is used to construct reinforcement learning samples based on the charging state vector, power allocation vector and simulation reward value, perform delay policy optimization operation in TD3 algorithm based on reinforcement learning samples, and update the policy network parameters.
[0013] The strategy execution module is used to regenerate the power allocation vector through the updated strategy network and send it to each charging pile for execution, and build a set of feedback information based on the execution results;
[0014] The feedback correction module is used to compare the feedback information set with the simulation result set to update the parameters of the digital twin model and the policy network based on the comparison results.
[0015] Optionally, modules can be integrated using the following methods:
[0016] S1. Construct a digital twin model corresponding to the charging area, and collect the power status, charging request information and grid load limit parameters of each charging pile in the charging area in real time at a fixed frequency to generate a charging status vector.
[0017] S2. Construct a policy network to perform continuous action generation operations on the charging state vector. In the generation process, Gaussian noise is introduced to perturb the policy, and range compression and constraint correction are performed on the perturbated action vector to generate a power allocation vector.
[0018] S3. In the digital twin model, the charging behavior simulation is executed using the power allocation vector as the control command. The energy consumption, task completion time and load change curve after smoothing with Hampel filter are recorded during the simulation process. A set of simulation results is generated, and the simulation reward value is calculated based on the set of simulation results.
[0019] S4. Construct reinforcement learning samples based on the charging state vector, power allocation vector and simulation reward value, and perform delay policy optimization operation in TD3 algorithm based on reinforcement learning samples to update policy network parameters.
[0020] S5. Regenerate the power allocation vector through the updated strategy network and send it to each charging pile for execution. Collect the power usage data, load change data and vehicle service data generated by the actual operation to build a set of feedback information.
[0021] S6. Compare the feedback information set with the simulation result set to determine the error, correct the state modeling parameters of the digital twin model based on the comparison results, and update the policy network parameters.
[0022] Optionally, the charging area refers to a spatial range where multiple charging piles are deployed and there is a unified power scheduling requirement; the power status refers to the output power value and power adjustment capability of each charging pile at the current time step; the charging request information refers to the charging intention, charging demand, and estimated completion time of the vehicle connected to the charging pile at the current time step; the grid load limit parameter refers to the maximum available total power and power fluctuation range of the charging area as specified by the upper-level grid; the historical load data refers to the power usage and load change records of each charging pile in the past, recorded in a time series; and the power allocation vector represents the power allocation value of each charging pile.
[0023] Optionally, the construction process of the digital twin model specifically includes:
[0024] The physical parameters and operational configuration information of each charging pile in the charging area are obtained to construct a charging unit model. The physical parameters include maximum output power, voltage level and interface type, and the operational configuration information includes working hours, power metering method and control protocol.
[0025] Obtain grid constraint parameters and load response rules related to the charging area, and construct a grid dispatch model. The grid constraint parameters include regional power supply capacity, peak and valley load threshold and real-time electricity price index. The load response rules represent the power flow behavior under different power dispatch strategies.
[0026] A time-series state mapping relationship is constructed based on historical load data, and state response rules are set. These state response rules are used to simulate the behavioral evolution process of each charging pile under different load conditions.
[0027] Define the power command input interface and the status information output interface, and establish an interface mechanism for receiving the policy network allocation results and outputting the simulation status.
[0028] The charging unit model, power grid scheduling model, time-series state mapping relationship and interface mechanism are integrated to form a digital twin model.
[0029] Optionally, S2 specifically includes:
[0030] S21. Construct a policy network, perform forward propagation on the charging state vector, extract high-dimensional feature representations, and generate policy action vectors through a multilayer perceptron.
[0031] S22. Introduce Gaussian noise that follows a zero-mean normal distribution into the policy action vector to construct a perturbation term, and perform element-wise addition on the perturbation term and the policy action vector to generate the perturbation action vector.
[0032] S23. Perform range compression processing on the disturbance action vector, restrict the elements of the corresponding dimension to the preset power range of each charging pile, and construct a Lagrangian objective function with grid load limit parameters as constraints. Perform constraint optimization on the compression result to generate a power allocation vector.
[0033] Optionally, the specific process of constructing the Lagrange objective function and performing constrained optimization includes:
[0034] Using the power values represented by the elements of each dimension in the perturbation action vector as optimization variables, an objective function representing the power allocation error is constructed.
[0035] Set power constraints, and set the maximum available total power and power fluctuation range in the grid load limit parameters as equality constraints and inequality constraints, respectively;
[0036] By introducing Lagrange multipliers, the objective function and constraints are coupled with variables to construct the Lagrange objective function;
[0037] The optimization process is performed using a gradient descent strategy based on the Lagrange objective function, iteratively updating the power values of each dimension in the perturbation action vector to obtain the optimization results that satisfy the constraints.
[0038] The power values of each dimension in the optimization results are sorted according to the actual charging pile numbers to generate a power allocation vector.
[0039] Optionally, S3 specifically includes:
[0040] S31. Based on the charging state vector, set the virtual charging behavior parameters of each charging pile in the digital twin model, and use the power distribution vector as the control command to trigger the simulated charging process.
[0041] S32. During the simulated charging process, record the real-time energy consumption value of each charging pile, calculate the task completion time of each vehicle from connection to completion of charging, and obtain the load change data in the area at a fixed sampling interval to form a load change curve.
[0042] S33. Perform a sliding window extraction operation on the load change curve, use a Hampel filter to detect local outliers in the sliding interval, calculate the median absolute deviation to construct a smoothing threshold, perform replacement processing on outliers, and generate a smoothed load change curve.
[0043] S34. Organize the power consumption, task completion time and smoothed load change curve into a simulation result set according to time steps, and calculate the simulation reward value according to the preset reward evaluation criteria.
[0044] Optionally, the simulation reward value calculation process specifically includes:
[0045] Based on the energy consumption of each charging pile and the charging demand of the corresponding vehicle during the simulation, the average energy consumption of a unit task is calculated and mapped to an energy consumption index at a fixed ratio.
[0046] The response time of each vehicle from connecting to the charging pile to completing the charging task is statistically analyzed. A task completion time threshold is set, the task completion rate and average response time are calculated, and a Min-Max normalization operation is performed to generate task response indicators.
[0047] The standard deviation is calculated using a fixed-length sliding window on the smoothed load change curve. The maximum standard deviation value in all windows is extracted as the load fluctuation amplitude, and Z-Score standardization is performed to generate a load stability index.
[0048] A set of weighting factors is defined, and a weighted addition operation is performed on the energy consumption index, task response index, and load stability index to generate a simulation reward value.
[0049] Optionally, S4 specifically includes:
[0050] S41. The charging state vector, power allocation vector and simulation reward value are organized into reinforcement learning samples according to a unified structure and stored in the experience playback buffer. The experience playback buffer contains the reinforcement learning samples stored at the current time step and multiple preset reinforcement learning samples.
[0051] S42. Randomly select a fixed number of reinforcement learning samples from the experience replay area, calculate the action output corresponding to the current policy in each state, and construct a performance evaluation sequence based on the simulation reward value.
[0052] S43. Based on the delay policy optimization mechanism in the TD3 algorithm, a delay optimization frequency threshold for the policy network is set, and the policy network update operation is triggered at the time step that meets the optimization frequency requirement.
[0053] S44. During the policy network update process, the simulation reward values in the reinforcement learning samples are processed by moving average, and the policy gradient is calculated based on the moving average result. The policy network parameters are optimized and adjusted through backpropagation.
[0054] Optionally, S6 specifically includes:
[0055] S61. Perform structured processing on the feedback information set and build a data index structure;
[0056] S62. Perform feature extraction on the simulation result set to extract the feature state nodes and continuous change trends in the simulation process, and perform time and value alignment with the corresponding data in the feedback information set.
[0057] S63. Calculate the difference based on the alignment results, and combine the difference calculation results into an error comparison sequence according to time steps;
[0058] S64. Based on the error comparison sequence, perform incremental updates on the numerical parameters involved in load response rules, energy consumption mapping and task process modeling in the digital twin model, and write the updated parameters into the state construction structure of the digital twin model.
[0059] S65. Perform moving average processing on the error comparison sequence, and set the policy update trigger condition according to the moving average result. When the trigger condition is met, perform the policy network parameter update operation.
[0060] The beneficial effects of this invention are:
[0061] First, this invention constructs a digital twin model corresponding to the actual charging area to achieve high-fidelity modeling of each charging pile device, grid load boundary, and vehicle behavior characteristics. Using the charging state vector as input, it uses the TD3 algorithm to generate continuous policy actions, combines Gaussian perturbation to enhance policy exploration capabilities, and introduces Lagrange constraints to correct the boundary of the action space. Thus, it balances global optimality and constraint compliance in the power scheduling process, improving the stability and execution efficiency of power allocation.
[0062] Secondly, this invention utilizes a digital twin model to construct a controllable simulation environment, simulating charging behavior and evaluating load response of policy actions without affecting the actual system operation. By introducing a Hampel filter and a sliding standard deviation analysis method, load fluctuations are smoothed and anomalies are eliminated. A multi-objective reward function is constructed, oriented towards energy consumption level, task completion efficiency, and load stability. This significantly improves the convergence speed and scheduling robustness of policy training, avoiding the problems of high trial-and-error costs and unstable training in actual deployment of existing reinforcement learning methods.
[0063] Finally, this invention constructs a feedback correction mechanism centered on error comparison. It performs multi-dimensional difference calculations between actual operating data and simulation results to generate an error sequence for adjusting the structure of the digital twin model and optimizing the policy network parameters. This achieves a unified update process for model adaptive evolution and online policy fine-tuning, constructing a complete closed-loop control architecture. Through this mechanism, the system possesses the ability to continuously adapt to complex operating conditions, significantly improving the efficiency of charging task processing and the utilization rate of energy per unit, meeting the practical needs of intelligent energy management in multi-pile collaborative and high-frequency scheduling scenarios. Attached Figure Description
[0064] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:
[0065] Figure 1 This is a block diagram of the charging pile load adaptive allocation and energy consumption optimization system based on reinforcement learning proposed in this invention.
[0066] Figure 2 This is a diagram of the adaptive closed-loop control structure of the charging pile load adaptive allocation and energy consumption optimization system based on reinforcement learning proposed in this invention.
[0067] Figure 3 This is a flowchart of the strategy generation and power allocation optimization process for the reinforcement learning-based adaptive load allocation and energy consumption optimization system for charging piles proposed in this invention. Detailed Implementation
[0068] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.
[0069] refer to Figure 1 A reinforcement learning-based adaptive load allocation and energy consumption optimization system for charging piles includes:
[0070] The data acquisition module is used to build a digital twin model corresponding to the charging area, collect the operating data of the charging area in real time at a fixed frequency, and generate a charging state vector.
[0071] The strategy allocation module is used to construct a strategy network, perform continuous action generation operations on the charging state vector, introduce Gaussian noise to perturb the strategy during the generation process, and perform range compression and constraint correction on the perturbated action vector to generate a power allocation vector.
[0072] The digital simulation module is used to simulate charging behavior in a digital twin model using power allocation vector as control command, record simulation data during the simulation process to generate a set of simulation results, and calculate the simulation reward value based on the set of simulation results.
[0073] The strategy update module is used to construct reinforcement learning samples based on the charging state vector, power allocation vector and simulation reward value, perform delay policy optimization operation in TD3 algorithm based on reinforcement learning samples, and update the policy network parameters.
[0074] The strategy execution module is used to regenerate the power allocation vector through the updated strategy network and send it to each charging pile for execution, and build a set of feedback information based on the execution results;
[0075] The feedback correction module is used to compare the feedback information set with the simulation result set to update the parameters of the digital twin model and the policy network based on the comparison results.
[0076] refer to Figure 2-3 In this embodiment, the modules are interconnected using the following method:
[0077] S1. Construct a digital twin model corresponding to the charging area, and collect the power status, charging request information and grid load limit parameters of each charging pile in the charging area in real time at a fixed frequency to generate a charging status vector.
[0078] S2. Construct a policy network to perform continuous action generation operations on the charging state vector. In the generation process, Gaussian noise is introduced to perturb the policy, and range compression and constraint correction are performed on the perturbated action vector to generate a power allocation vector.
[0079] S3. In the digital twin model, the charging behavior simulation is executed using the power allocation vector as the control command. The energy consumption, task completion time and load change curve after smoothing with Hampel filter are recorded during the simulation process. A set of simulation results is generated, and the simulation reward value is calculated based on the set of simulation results.
[0080] S4. Construct reinforcement learning samples based on the charging state vector, power allocation vector and simulation reward value, and perform delay policy optimization operation in TD3 algorithm based on reinforcement learning samples to update policy network parameters.
[0081] S5. Regenerate the power allocation vector through the updated strategy network and send it to each charging pile for execution. Collect the power usage data, load change data and vehicle service data generated by the actual operation to build a set of feedback information.
[0082] S6. Compare the feedback information set with the simulation result set to determine the error, correct the state modeling parameters of the digital twin model based on the comparison results, and update the policy network parameters.
[0083] In this embodiment, the charging area refers to a spatial range where multiple charging piles are deployed and there is a unified power scheduling requirement; the power status refers to the output power value and power adjustment capability of each charging pile at the current time step; the charging request information refers to the charging intention, charging demand, and estimated completion time of the vehicle connected to the charging pile at the current time step; the grid load limit parameter refers to the maximum available total power and power fluctuation range of the charging area as specified by the upper-level grid; the historical load data refers to the power usage and load change records of each charging pile in the past, recorded in a time series; and the power allocation vector represents the power allocation value of each charging pile.
[0084] In this embodiment, the construction process of the digital twin model specifically includes:
[0085] The physical parameters and operational configuration information of each charging pile in the charging area are obtained to construct a charging unit model. The physical parameters include maximum output power, voltage level and interface type, and the operational configuration information includes working hours, power metering method and control protocol.
[0086] Obtain grid constraint parameters and load response rules related to the charging area, and construct a grid dispatch model. The grid constraint parameters include regional power supply capacity, peak and valley load threshold and real-time electricity price index. The load response rules represent the power flow behavior under different power dispatch strategies.
[0087] A time-series state mapping relationship is constructed based on historical load data, and state response rules are set. These state response rules are used to simulate the behavioral evolution process of each charging pile under different load conditions.
[0088] Define the power command input interface and the status information output interface, and establish an interface mechanism for receiving the policy network allocation results and outputting the simulation status.
[0089] The charging unit model, power grid scheduling model, time-series state mapping relationship and interface mechanism are integrated to form a digital twin model.
[0090] In this embodiment, S2 specifically includes:
[0091] S21. Construct a policy network, perform forward propagation on the charging state vector, extract high-dimensional feature representations, and generate policy action vectors through a multilayer perceptron.
[0092] S22. Introduce Gaussian noise that follows a zero-mean normal distribution into the policy action vector to construct a perturbation term, and perform element-wise addition on the perturbation term and the policy action vector to generate the perturbation action vector.
[0093] S23. Perform range compression processing on the disturbance action vector, restrict the elements of the corresponding dimension to the preset power range of each charging pile, and construct a Lagrangian objective function with grid load limit parameters as constraints. Perform constraint optimization on the compression result to generate a power allocation vector.
[0094] In this embodiment, the specific process of constructing the Lagrange objective function and performing constrained optimization includes:
[0095] Using the power values represented by the elements of each dimension in the perturbation action vector as optimization variables, an objective function representing the power allocation error is constructed.
[0096] Set power constraints, and set the maximum available total power and power fluctuation range in the grid load limit parameters as equality constraints and inequality constraints, respectively;
[0097] By introducing Lagrange multipliers, the objective function and constraints are coupled with variables to construct the Lagrange objective function;
[0098] The optimization process is performed using a gradient descent strategy based on the Lagrange objective function, iteratively updating the power values of each dimension in the perturbation action vector to obtain the optimization results that satisfy the constraints.
[0099] The power values of each dimension in the optimization results are sorted according to the actual charging pile numbers to generate a power allocation vector.
[0100] In this embodiment, S3 specifically includes:
[0101] S31. Based on the charging state vector, set the virtual charging behavior parameters of each charging pile in the digital twin model, and use the power distribution vector as the control command to trigger the simulated charging process.
[0102] S32. During the simulated charging process, record the real-time energy consumption value of each charging pile, calculate the task completion time of each vehicle from connection to completion of charging, and obtain the load change data in the area at a fixed sampling interval to form a load change curve.
[0103] S33. Perform a sliding window extraction operation on the load change curve, use a Hampel filter to detect local outliers in the sliding interval, calculate the median absolute deviation to construct a smoothing threshold, perform replacement processing on outliers, and generate a smoothed load change curve.
[0104] S34. Organize the power consumption, task completion time and smoothed load change curve into a simulation result set according to time steps, and calculate the simulation reward value according to the preset reward evaluation criteria.
[0105] In this embodiment, S33 specifically includes:
[0106] S331. Set a fixed-length sliding window and extract the load change curve in segments according to time sequence to construct a load segment set containing multiple sliding intervals;
[0107] S332. Calculate the median and median absolute deviation of all load values in each sliding interval;
[0108] S333. Calculate the difference between the load value at each time step and the median of the corresponding sliding interval. If the difference exceeds the median absolute deviation by a preset multiple, it is determined to be a local anomaly.
[0109] S334. For the time step that is determined to be a local anomaly, select the load values of other time steps within the sliding interval that are not an anomaly, and replace the values by using a weighted average method based on time distance to generate a smoothed load change curve.
[0110] In this embodiment, the simulation reward value calculation process specifically includes:
[0111] Based on the energy consumption of each charging pile and the charging demand of the corresponding vehicle during the simulation, the average energy consumption of a unit task is calculated and mapped to an energy consumption index at a fixed ratio.
[0112] The response time of each vehicle from connecting to the charging pile to completing the charging task is statistically analyzed. A task completion time threshold is set, the task completion rate and average response time are calculated, and a Min-Max normalization operation is performed to generate task response indicators.
[0113] The smoothed load variation curve is subjected to a fixed-length sliding window for standard deviation calculation. The maximum standard deviation value across all windows is extracted as the load fluctuation amplitude. Z-score standardization is then performed to generate a load stability index, which includes:
[0114] A sliding window of fixed length is constructed for the smoothed load change curve, and the standard deviation is calculated for the load data in each sliding window to generate a standard deviation sequence.
[0115] Extract the maximum standard deviation value corresponding to all time steps in the standard deviation sequence. The maximum standard deviation value represents the maximum fluctuation range of load change within the charging cycle.
[0116] Z-Score standardization is performed based on the maximum standard deviation to calculate the average fluctuation amplitude and standard deviation within each charging cycle. The current maximum standard deviation is then processed according to the standardization formula to generate the standardized result.
[0117] The standardized results are mapped to the [0,1] interval, and a load stability index is generated based on the set stability evaluation function. The stability evaluation function is used to compress the impact of extreme fluctuation values on the index results.
[0118] A set of weighting factors is defined, and a weighted addition operation is performed on the energy consumption index, task response index, and load stability index to generate a simulation reward value.
[0119] In this embodiment, S4 specifically includes:
[0120] S41. The charging state vector, power allocation vector and simulation reward value are organized into reinforcement learning samples according to a unified structure and stored in the experience playback buffer. The experience playback buffer contains the reinforcement learning samples stored at the current time step and multiple preset reinforcement learning samples.
[0121] S42. Randomly select a fixed number of reinforcement learning samples from the experience replay area, calculate the action output corresponding to the current policy in each state, and construct a performance evaluation sequence based on the simulation reward value.
[0122] S43. Based on the delay policy optimization mechanism in the TD3 algorithm, a delay optimization frequency threshold for the policy network is set, and the policy network update operation is triggered at the time step that meets the optimization frequency requirement.
[0123] S44. During the policy network update process, the simulation reward values in the reinforcement learning samples are processed by moving average, and the policy gradient is calculated based on the moving average result. The policy network parameters are optimized and adjusted through backpropagation.
[0124] In this embodiment, S5 specifically includes:
[0125] S51. Regenerate the power allocation vector through the updated strategy network and send it to the control interface of each charging pile to perform the corresponding charging power adjustment operation.
[0126] S52. While performing the charging task, collect real-time data on the power usage, load change, and vehicle service generated by each charging pile, and construct data segments containing timestamps.
[0127] S53. Perform missing value imputation, Min-Max normalization, and time alignment operations on the data fragments to construct a unified set of feedback information.
[0128] In this embodiment, S6 specifically includes:
[0129] S61. Perform structured processing on the feedback information set and build a data index structure;
[0130] S62. Perform feature extraction on the simulation result set to extract the feature state nodes and continuous change trends in the simulation process, and perform time and value alignment with the corresponding data in the feedback information set.
[0131] S63. Calculate the difference based on the alignment results, and combine the difference calculation results into an error comparison sequence according to time steps;
[0132] S64. Based on the error comparison sequence, perform incremental updates on the numerical parameters involved in load response rules, energy consumption mapping and task process modeling in the digital twin model, and write the updated parameters into the state construction structure of the digital twin model.
[0133] S65. Perform moving average processing on the error comparison sequence, and set the policy update trigger condition according to the moving average result. When the trigger condition is met, perform the policy network parameter update operation.
[0134] Example 1:
[0135] To verify the feasibility of this invention in practice, it was applied to a charging station scenario. An energy consumption optimization experiment was conducted by scheduling and controlling multiple concurrently connected charging piles. This charging station area has 24 independent charging piles, supporting power output ranging from 7kW to 22kW. The number of vehicles connecting for charging daily is large, and it exhibits typical characteristics of concentrated load during peak periods.
[0136] Traditional scheduling systems use fixed thresholds and polling allocation methods, which result in uneven power distribution, response delays, and high energy consumption. Especially during periods of high load, some charging piles may have power supply over-limit warnings while others are idle, leading to low overall scheduling efficiency and energy utilization.
[0137] In this scenario, the present invention constructs a digital twin model to model the charging piles, power grid boundary constraints, and historical load changes in the charging station. The model can continuously reconstruct the state of each device based on real-time data and compare the differences with the actual execution process.
[0138] At the policy execution level, the TD3 algorithm is used as the core reinforcement learning method to generate continuous power commands for each charging pile at each time step through the policy network. To improve the generalization ability of the policy, a Gaussian perturbation mechanism is added. At the same time, the total power constraint and the adjustable range of each charging pile are introduced into the policy optimization process through the Lagrangian objective function to ensure that the allocation result runs smoothly within the scheduling constraint range.
[0139] In practical applications, the system collects power status, vehicle requests, and load feedback information at a fixed frequency for each time step. After receiving the policy output, the digital twin model first performs simulation to generate load curves, energy consumption, and task response times, and constructs simulation reward values for training.
[0140] After optimizing based on simulation results, the policy network generates a formal power allocation scheme and distributes it to each charging station. Upon completion, the actual results are compared and updated with the simulation results, forming a complete simulation-optimization-execution-feedback closed-loop process.
[0141] To evaluate the scheduling optimization effect of this invention, key indicators such as energy consumption, load fluctuation, and task completion efficiency were collected for both the proposed and existing schemes under three typical operating cycles. The scheduling performance of the two methods under the same access requirements was compared. The data results are shown in the table below.
[0142] Table 1. Comparison of load scheduling effects between the present invention and existing strategies.
[0143]
[0144] As can be seen from the data in Table 1, under the same charging task scale, the load scheduling scheme proposed in this invention reduces the total power consumption by an average of 6.3%, indicating that the reinforcement learning strategy reduces energy redundancy by rationally allocating power commands.
[0145] In terms of task completion time, the strategy of this invention reduces the average time by 10.8%, showing that the system is more efficient in task response and can effectively improve vehicle service capabilities.
[0146] The average load fluctuation amplitude decreased by more than 22%, reflecting that the strategy of using Lagrange optimization and disturbance control resulted in more stable load distribution, smaller load changes, and lower impact on the power grid.
[0147] The utilization rate of charging piles has also been significantly improved, indicating that the system has achieved balanced load distribution among multiple charging piles and avoided resource waste.
[0148] In summary, this invention demonstrates significant advantages in improving system energy efficiency, optimizing load scheduling stability, and shortening task response time, making it valuable for practical deployment and promising for widespread adoption. Applying it to larger-scale charging scenarios or integrating more external constraint models will further enhance its adaptive capabilities and intelligent management level.
[0149] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A charging pile load adaptive allocation and energy consumption optimization system based on reinforcement learning, characterized in that, include: The data acquisition module is used to build a digital twin model corresponding to the charging area, collect the operating data of the charging area in real time at a fixed frequency, and generate a charging state vector. The strategy allocation module is used to construct a strategy network, perform continuous action generation operations on the charging state vector, introduce Gaussian noise to perturb the strategy during the generation process, and perform range compression and constraint correction on the perturbated action vector to generate a power allocation vector. The digital simulation module is used to simulate charging behavior in a digital twin model using power allocation vector as control command, record simulation data during the simulation process to generate a set of simulation results, and calculate the simulation reward value based on the set of simulation results. The strategy update module is used to construct reinforcement learning samples based on the charging state vector, power allocation vector and simulation reward value, perform delay policy optimization operation in TD3 algorithm based on reinforcement learning samples, and update the policy network parameters. The strategy execution module is used to regenerate the power allocation vector through the updated strategy network and send it to each charging pile for execution, and build a set of feedback information based on the execution results; The feedback correction module is used to compare the feedback information set with the simulation result set to update the parameters of the digital twin model and the policy network based on the comparison results.
2. The charging pile load adaptive allocation and energy consumption optimization system based on reinforcement learning according to claim 1, characterized in that, The modules are connected in the following way: S1. Construct a digital twin model corresponding to the charging area, and collect the power status, charging request information and grid load limit parameters of each charging pile in the charging area in real time at a fixed frequency to generate a charging status vector. S2. Construct a policy network to perform continuous action generation operations on the charging state vector. In the generation process, Gaussian noise is introduced to perturb the policy, and range compression and constraint correction are performed on the perturbated action vector to generate a power allocation vector. S3. In the digital twin model, the charging behavior simulation is executed using the power allocation vector as the control command. The energy consumption, task completion time and load change curve after smoothing with Hampel filter are recorded during the simulation process. A set of simulation results is generated, and the simulation reward value is calculated based on the set of simulation results. S4. Construct reinforcement learning samples based on the charging state vector, power allocation vector and simulation reward value, and perform delay policy optimization operation in TD3 algorithm based on reinforcement learning samples to update policy network parameters. S5. Regenerate the power allocation vector through the updated strategy network and send it to each charging pile for execution. Collect the power usage data, load change data and vehicle service data generated by the actual operation to build a set of feedback information. S6. Compare the feedback information set with the simulation result set to determine the error, correct the state modeling parameters of the digital twin model based on the comparison results, and update the policy network parameters.
3. The charging pile load adaptive allocation and energy consumption optimization system based on reinforcement learning according to claim 2, characterized in that, The charging area refers to a spatial range where multiple charging piles are deployed and there is a unified power scheduling requirement. The power status refers to the output power value and power adjustment capability of each charging pile at the current time step. The charging request information refers to the charging intention, charging demand, and estimated completion time of the vehicle connected to the charging pile at the current time step. The grid load limit parameter refers to the maximum available total power and power fluctuation range of the charging area as specified by the upper-level grid. The historical load data refers to the power usage and load change records of each charging pile in the past, recorded in time series. The power allocation vector represents the power allocation value of each charging pile.
4. The charging pile load adaptive allocation and energy consumption optimization system based on reinforcement learning according to claim 2, characterized in that, The construction process of the digital twin model specifically includes: The physical parameters and operational configuration information of each charging pile in the charging area are obtained to construct a charging unit model. The physical parameters include maximum output power, voltage level and interface type, and the operational configuration information includes working hours, power metering method and control protocol. Obtain grid constraint parameters and load response rules related to the charging area, and construct a grid dispatch model. The grid constraint parameters include regional power supply capacity, peak and valley load threshold and real-time electricity price index. The load response rules represent the power flow behavior under different power dispatch strategies. A time-series state mapping relationship is constructed based on historical load data, and state response rules are set. These state response rules are used to simulate the behavioral evolution process of each charging pile under different load conditions. Define the power command input interface and the status information output interface, and establish an interface mechanism for receiving the policy network allocation results and outputting the simulation status. The charging unit model, power grid scheduling model, time-series state mapping relationship and interface mechanism are integrated to form a digital twin model.
5. The charging pile load adaptive allocation and energy consumption optimization system based on reinforcement learning according to claim 2, characterized in that, S2 specifically includes: S21. Construct a policy network, perform forward propagation on the charging state vector, extract high-dimensional feature representations, and generate policy action vectors through a multilayer perceptron. S22. Introduce Gaussian noise that follows a zero-mean normal distribution into the policy action vector to construct a perturbation term, and perform element-wise addition on the perturbation term and the policy action vector to generate the perturbation action vector. S23. Perform range compression processing on the disturbance action vector, restrict the elements of the corresponding dimension to the preset power range of each charging pile, and construct a Lagrangian objective function with grid load limit parameters as constraints. Perform constraint optimization on the compression result to generate a power allocation vector.
6. The charging pile load adaptive allocation and energy consumption optimization system based on reinforcement learning according to claim 5, characterized in that, The specific process of constructing the Lagrange objective function and performing constrained optimization includes: Using the power values represented by the elements of each dimension in the perturbation action vector as optimization variables, an objective function representing the power allocation error is constructed. Set power constraints, and set the maximum available total power and power fluctuation range in the grid load limit parameters as equality constraints and inequality constraints, respectively; By introducing Lagrange multipliers, the objective function and constraints are coupled with variables to construct the Lagrange objective function; The optimization process is performed using a gradient descent strategy based on the Lagrange objective function, iteratively updating the power values of each dimension in the perturbation action vector to obtain the optimization results that satisfy the constraints. The power values of each dimension in the optimization results are sorted according to the actual charging pile numbers to generate a power allocation vector.
7. The charging pile load adaptive allocation and energy consumption optimization system based on reinforcement learning according to claim 2, characterized in that, S3 specifically includes: S31. Based on the charging state vector, set the virtual charging behavior parameters of each charging pile in the digital twin model, and use the power distribution vector as the control command to trigger the simulated charging process. S32. During the simulated charging process, record the real-time energy consumption value of each charging pile, calculate the task completion time of each vehicle from connection to completion of charging, and obtain the load change data in the area at a fixed sampling interval to form a load change curve. S33. Perform a sliding window extraction operation on the load change curve, use a Hampel filter to detect local outliers in the sliding interval, calculate the median absolute deviation to construct a smoothing threshold, perform replacement processing on outliers, and generate a smoothed load change curve. S34. Organize the power consumption, task completion time and smoothed load change curve into a simulation result set according to time steps, and calculate the simulation reward value according to the preset reward evaluation criteria.
8. The charging pile load adaptive allocation and energy consumption optimization system based on reinforcement learning according to claim 7, characterized in that, The simulation reward value calculation process specifically includes: Based on the energy consumption of each charging pile and the charging demand of the corresponding vehicle during the simulation, the average energy consumption of a unit task is calculated and mapped to an energy consumption index at a fixed ratio. The response time of each vehicle from connecting to the charging pile to completing the charging task is statistically analyzed. A task completion time threshold is set, the task completion rate and average response time are calculated, and a Min-Max normalization operation is performed to generate task response indicators. The standard deviation is calculated using a fixed-length sliding window on the smoothed load change curve. The maximum standard deviation value in all windows is extracted as the load fluctuation amplitude, and Z-Score standardization is performed to generate a load stability index. A set of weighting factors is defined, and a weighted addition operation is performed on the energy consumption index, task response index, and load stability index to generate a simulation reward value.
9. The charging pile load adaptive allocation and energy consumption optimization system based on reinforcement learning according to claim 2, characterized in that, S4 specifically includes: S41. The charging state vector, power allocation vector and simulation reward value are organized into reinforcement learning samples according to a unified structure and stored in the experience playback buffer. The experience playback buffer contains the reinforcement learning samples stored in the current time step and multiple preset reinforcement learning samples. S42. Randomly select a fixed number of reinforcement learning samples from the experience replay area, calculate the action output corresponding to the current policy in each state, and construct a performance evaluation sequence based on the simulation reward value. S43. Based on the delay policy optimization mechanism in the TD3 algorithm, a delay optimization frequency threshold for the policy network is set, and the policy network update operation is triggered at the time step that meets the optimization frequency requirement. S44. During the policy network update process, the simulation reward values in the reinforcement learning samples are processed by moving average, and the policy gradient is calculated based on the moving average result. The policy network parameters are optimized and adjusted through backpropagation.
10. The charging pile load adaptive allocation and energy consumption optimization system based on reinforcement learning according to claim 2, characterized in that, S6 specifically includes: S61. Perform structured processing on the feedback information set and build a data index structure; S62. Perform feature extraction on the simulation result set to extract the feature state nodes and continuous change trends in the simulation process, and perform time and value alignment with the corresponding data in the feedback information set. S63. Calculate the difference based on the alignment results, and combine the difference calculation results into an error comparison sequence according to time steps; S64. Based on the error comparison sequence, perform incremental updates on the numerical parameters involved in load response rules, energy consumption mapping and task process modeling in the digital twin model, and write the updated parameters into the state construction structure of the digital twin model. S65. Perform moving average processing on the error comparison sequence, and set the policy update trigger condition according to the moving average result. When the trigger condition is met, perform the policy network parameter update operation.