Charging pile cluster load balancing optimization method and system based on reinforcement learning

Abstract

The invention relates to the technical field of electric vehicle charging facility intelligent control, and discloses a reinforcement learning-based charging pile cluster load balancing optimization method and system. The system comprises a digital twinborn simulation module, a data sensing and processing module, a state characterization module, a double-track decision module, a baseline strategy unit, a shadow learning agent, a safety verification and execution module, a double-path reward calculation module, a shadow experience learning and verification module and an online deployment and continuous learning module. Through the self-evolution system normal form of'evaluation-learning-verification-online 'and the dual security isolation design, the autonomous management system for the charging pile cluster, which can continuously improve the performance on the premise of ensuring absolute security, is successfully constructed, and a replicable solution is provided for engineering application of reinforcement learning in a complex industrial scene.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology for electric vehicle charging facilities, and more specifically to a method and system for load balancing optimization of charging pile clusters based on reinforcement learning. Background Technology

[0002] With the rapid development of the global energy transition and the electric vehicle industry, the scale of charging pile clusters in public and private locations is constantly expanding. As a key node connecting electric vehicles and the power grid, their operational complexity is becoming increasingly prominent. The load balancing optimization problem of charging pile clusters—that is, how to dynamically allocate the limited total power of a limited number of sites in real time while meeting the differentiated charging needs of various electric vehicles and the physical constraints of the charging piles themselves—in order to synergistically improve charging efficiency, user fairness, operational economy, and ensure the safety and stability of the power grid, has become a core bottleneck restricting the improvement of the service capacity and economic benefits of charging stations.

[0003] Currently, power scheduling control of charging pile clusters mainly relies on static strategies based on preset rules, such as first-come, first-served, average allocation, or priority allocation based on simple power thresholds. While these traditional methods are simple to implement and logically intuitive, their core decision-making logic is fixed and cannot perceive or adapt to complex environmental information such as real-time changes in vehicle status, grid conditions, market prices, power generation technology characteristics, user behavior, and energy storage status. Essentially, they are open-loop, reactive control methods, which are difficult to finely balance multi-dimensional and often conflicting optimization objectives.

[0004] Existing technical solutions suffer from a series of inherent limitations: their strategies are rigid, lacking the ability to learn autonomously and iteratively optimize from historical operational data, resulting in system performance stagnating at a suboptimal state for a long time; their optimization dimensions are singular, often neglecting other aspects and failing to systematically coordinate multiple objectives such as charging speed, fair experience, electricity costs, equipment losses, and synergy with power generation technologies and energy storage systems; more importantly, directly deploying advanced algorithms with exploratory capabilities (such as reinforcement learning) in real charging systems carries extremely high safety risks and trial-and-error costs, while traditional architectures lack necessary security isolation and progressive verification mechanisms. Furthermore, facing emerging demands such as vehicle-to-grid (V2G), local renewable energy co-consumption, and interaction with new power generation technologies and on-site / grid-side energy storage systems, the existing rigid system architecture struggles to achieve low-cost, smooth functional expansion and adaptation.

[0005] Therefore, methods that facilitate smooth transitions, multi-objective optimization, continuous evolution, and adaptability to new requirements are needed to address these issues. Summary of the Invention

[0006] To overcome the aforementioned deficiencies in the prior art, this invention provides a load balancing optimization method and system for charging pile clusters based on reinforcement learning, in order to solve the problems existing in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a load balancing optimization method for charging pile clusters based on reinforcement learning, specifically including the following steps: S1. Construct a high-fidelity digital twin of the charging station based on a neural network of physical information, and pre-train the shadow learning agent in the simulation; S2. Deploy a data acquisition system to collect multi-source data in real time, including charging pile data, connected electric vehicle data, power grid market data and context data, and perform preprocessing and feature engineering on the collected multi-source data; S3. Construct the preprocessed feature data into a state vector that can be understood by the baseline policy unit and the shadow learning agent, which is used to describe the complete state of the environment. S4. The state vector is simultaneously input to the baseline rule policy controller and the pre-trained shadow learning agent. The rule policy controller generates the actual action vector based on the state vector, and the shadow learning agent generates the shadow action vector based on the same state vector in the isolated decision environment. S5. Perform a safety check on the actual action vector. If the safety check passes, send it to the charging pile for execution. If it fails, correct the violation action and send it for execution again. The shadow action vector is isolated and not executed. S6. Construct a multi-objective reward function, calculate the actual reward score based on the real new state after the actual action vector is executed, and call the digital twin to simulate the execution of the shadow action vector and predict the simulated new state, and calculate the simulated reward score. S7. The state vector, shadow action vector, simulated reward score and simulated new state obtained from multiple interactions are combined into a shadow experience tuple and stored in the shadow experience playback buffer. Data is periodically sampled from the buffer to perform offline learning and fine-tuning of the agent. After verification, the agent is triggered to go online. S8. Deploy the verified shadow learning agent, and send out the generated action vectors for execution after security verification. Calculate the reward score based on the actual execution results, and store the actual experience tuples in the online experience replay buffer to continuously fine-tune the online strategy.

[0008] The reinforcement learning-based load balancing optimization system for charging pile clusters specifically includes: Digital twin simulation module: used to construct a high-fidelity digital twin of a charging station based on a neural network of physical information, providing a pre-training environment for the shadow learning agent and predicting state transitions in shadow mode; Data perception and processing module: used to deploy the data acquisition system to collect multi-source data in real time, and to preprocess and feature-engineer the data; State representation module: Its input is connected to the data perception and processing module, and it is used to construct the preprocessed feature data into a state vector that can be understood by the baseline policy unit and the shadow learning agent; Dual-track decision module: Its input is connected to the state representation module to receive the state vector; this module includes: Baseline strategy unit: used to generate actual action vectors based on the state vector; Shadow learning agent: operates in parallel with the baseline policy unit to generate shadow action vectors based on the same state vector in an isolated environment; Security verification and execution module: Its input is connected to the baseline strategy unit, which is used to perform security verification on the actual action vector. If it passes the verification, it is sent to the charging pile for execution. If it fails the verification, the violation action is corrected and then sent for execution. At the same time, the shadow action vector is isolated and not executed. Dual-path reward calculation module: Its input terminals are respectively connected to the security verification and execution module and the digital twin simulation module. It is used to construct a multi-objective reward function, calculate the actual reward score based on the real new state after the actual action vector is executed, and call the digital twin simulation module to predict and simulate the new state based on the shadow action vector to calculate the simulated reward score. Shadow Experience Learning and Verification Module: Its input end is connected to the reward calculation and evaluation module. It is used to combine the state vector, shadow action vector, simulated reward score and simulated new state obtained from multiple interactions into a shadow experience tuple, store it in the shadow experience playback buffer, periodically sample data for offline learning and fine-tuning, and trigger online after performance verification is passed. Online Deployment and Continuous Learning Module: After the Shadow Experience Learning and Verification Module is triggered to go online, it deploys the verified shadow learning agent, sends out the generated action vectors after security verification, calculates the reward score based on the actual execution results, and stores the actual experience tuples in the online experience replay buffer to continuously fine-tune the online strategy.

[0009] The technical effects and advantages of this invention are as follows: 1. This invention, by constructing a progressive technical path of "simulation pre-training - shadow parallel evaluation - security mandatory verification", realizes the safe and smooth transition and deployment of reinforcement learning agents from virtual to reality in charging pile cluster control scenarios, fundamentally solving the industry problem of high trial and error costs and huge risks of direct deployment of artificial intelligence in key charging infrastructure; 2. This invention designs a composite reward function that integrates multiple objectives such as efficiency, fairness, economy, power grid safety, and equipment lifespan. By leveraging the autonomous interaction and learning between intelligent agents and the environment, it achieves dynamic and collaborative optimization of power allocation for charging pile clusters, significantly improving the overall operational efficiency of charging stations and the overall user satisfaction. 3. This invention ensures the absolute safety and long-term autonomous evolution capability of the charging control system by deploying a security verification module with real-time auditing and correction capabilities at the front end of the execution layer, and establishing a dual evolution mechanism that combines "offline shadow learning" and "online continuous fine-tuning", enabling it to continuously adapt to dynamically changing environments and business needs. 4. By proactively integrating elements such as V2G services and renewable energy consumption into the state representation and reward function design, this invention constructs a highly scalable technical architecture, enabling the system to not only solve the current load balancing problem, but also smoothly adapt to the new business models and grid interaction needs in the future energy internet context. 5. This invention, through a self-evolving system paradigm of "evaluation-learning-verification-deployment" and a dual security isolation design, successfully constructs an autonomous management system for charging pile clusters that can continuously improve performance while ensuring absolute security, providing a replicable solution for the engineering application of reinforcement learning in complex industrial scenarios. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the system architecture of the present invention.

[0011] Figure 2 This is a flowchart of the method steps of the present invention.

[0012] Figure 3 This is a diagram illustrating the shadow mode operation of the present invention. Detailed Implementation

[0013] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. In addition, the forms of the various structures described in the following embodiments are merely illustrative. The reinforcement learning-based charging pile cluster load balancing optimization method and system involved in the present invention are not limited to the structures described in the following embodiments. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] Reference Figure 1This invention provides a method and system for optimizing load balancing of charging pile clusters based on reinforcement learning, including a digital twin simulation module, a data perception and processing module, a state representation module, a dual-track decision-making module, a baseline strategy unit, a shadow learning agent, a security verification and execution module, a dual-path reward calculation module, a shadow experience learning and verification module, and an online deployment and continuous learning module.

[0015] Reference Figure 2 The specific implementation steps of the present invention include the following steps: S1. Construct a high-fidelity digital twin of the charging station based on a neural network of physical information, and pre-train the shadow learning agent in the simulation.

[0016] It should be noted that the input data for digital twin modeling includes: Physical model data: charging pile circuit topology, transformer thermal characteristic parameters, cable impedance and efficiency curves; Historical operational data: Records of past vehicle charging sessions, including arrival time, initial charging status, end charging status, dwell time, and charging power curve; External data: historical electricity prices, weather data, and holiday information.

[0017] Using PyBullet, MATLAB / Simulink, or a self-developed physics-based simulator, a high-fidelity digital twin model of the charging station is constructed. This model primarily simulates power constraints and vehicle dynamics, such as the total power not exceeding the transformer capacity and the curve of the BMS adjusting the charging power according to the charging status. A shadow learning agent is constructed using the PPO or SAC algorithm, with the input layer dimension matching the state vector and the output layer dimension matching the charging pile power value. Pre-training is performed, and in the simulation, tens of thousands to millions of rounds are simulated, each round representing a day's operations. Through trial and error, the agent learns how to allocate power to maximize cumulative rewards in various simulated scenarios, such as morning rush hour, peak electricity prices, and vehicle congestion.

[0018] S2. Deploy a data acquisition system to collect multi-source data in real time, including charging pile data, connected electric vehicle data, power grid market data, and contextual data, and perform preprocessing and feature engineering on the collected multi-source data.

[0019] It should be specifically noted that the multi-source data includes: Charging pile data: charging pile ID, timestamp, active power, rated power, cumulative power consumption, and operating status; Electric vehicle data: Session ID, Vehicle VIN, Initial charging status, Current charging status, Target charging status, Dwell time, Estimated remaining dwell time, BMS maximum requested power, and Vehicle battery capacity. Grid market data: timestamps, real-time electricity prices, transformer rated capacity, transformer remaining capacity, and total power at the site; Contextual data: timestamp, day of the week, whether it is a holiday, and weather conditions.

[0020] The preprocessing includes data cleaning and normalization. Data cleaning will process abnormal data such as negative power and charging status exceeding 100%, and use front and back interpolation or KNN-based filling methods for missing values. Normalization will scale the values ​​of charging status, power, and electricity price to the range of [0, 1] or [-1, 1].

[0021] The feature engineering specifically includes: Charging urgency = (Target charging state - Current charging state) / Estimated remaining dwell time; Load factor = Total power of the site / Remaining capacity of the transformer; Time feature encoding: Convert hours into sin(hour / 24*2π) and cos(hour / 24*2π) to reflect the periodicity of time.

[0022] S3. Construct the preprocessed feature data into a state vector that can be understood by the baseline policy unit and the shadow learning agent, which is used to describe the complete state of the environment.

[0023] It should be noted that the state vector is a fixed-dimensional vector St, which includes the global state and the local state of each charging station, specifically: Global state: [load rate, normalized real-time electricity price, normalized electricity price trend, predicted short-term load changes, sine time, cosine time, day of the week code, whether it is a holiday], the global state features are fixed to m dimensions; Local state: [Normalized current power, normalized current charging state, normalized target charging state, normalized vehicle maximum power, normalized dwell time, historical power allocation, vehicle connection state (0 / 1)]; For N charging piles, repeat N times. Each charging pile, regardless of whether there is a vehicle connected, is assigned a feature sub-vector of fixed length. The feature sub-vector of each charging pile is defined to contain k dimensions. To ensure that the vector dimensions are fixed, the maximum number of piles is used. For a pile without a vehicle connected, all numerical features in its local state sub-vector are set to 0, except for the vehicle connection state which is 0.

[0024] It is important to note that the specific characteristics and dimensions of the global and local states are not static and should be adjusted in actual deployment based on specific business objectives, data availability, and model performance requirements.

[0025] It should be noted that the global state vector and the local state vectors of N stakes are concatenated in order, with a total dimension z = m + N*k. The state vector St = [global feature 1, ..., global feature m, stake 1 feature 1, ..., stake 1 feature k, ..., stake N feature 1, ..., stake N feature k] is a fixed-dimensional array that describes the complete state of the environment at the current moment.

[0026] It is important to note that if the number of charging piles exceeds a certain number, such as N > 50, a graph neural network (GNN) is used to avoid lengthy state vectors. Each charging pile is treated as a node in the graph, and the features of the node are local state vectors. The GNN can actively learn the relationships between nodes (piles), thereby more intelligently discovering strategies such as allocating power from idle areas to busy areas. The global state is the attribute of the entire graph.

[0027] S4. The state vector is simultaneously input to the baseline rule policy controller and the pre-trained shadow learning agent. The rule policy controller generates the actual action vector based on the state vector, and the shadow learning agent generates the shadow action vector based on the same state vector in the isolated decision environment.

[0028] It should be specifically explained that a dual-track decision-making architecture is adopted: the existing rule strategy generates the actual action vector based on the state vector. At the same time, the pre-trained shadow learning agent receives the same state vector in a completely isolated sandbox environment and generates a shadow action vector. The two decision-making processes are parallel and synchronous, but the performance is compared only at the logical level. The physical control signals are completely separated, ensuring the physical or logical isolation between the optimization decision-making process and the actual control process. This fundamentally eliminates the risk of unverified shadow learning agent decisions interfering with the real system.

[0029] The actual action vector is used to allocate a specific charging power value to each charging pile. Specifically, the state vector St is input to the baseline rule strategy controller, which calculates a power allocation scheme based on St, prioritizing a base power for each charging pile, and then allocating the remaining power proportionally according to the charging capacity or power demand of each vehicle's battery, and outputting the actual action vector Ap=[P1, P2, ..., Pn], where Pi is a real value in the range [0, Pimax], representing the power value directly allocated to the i-th charging pile.

[0030] In parallel, the system inputs the exact same state vector St into the pre-trained shadow learning agent. The agent's policy network performs forward inference on St and outputs a shadow action vector Aq, where Aq = [Q1, Q2, ..., Qn]. This process only performs mathematical calculations and does not generate any actual control signals.

[0031] S5. Perform a safety check on the actual action vector. If the safety check passes, send it to the charging pile for execution. If it fails, correct the violation action and send it for execution again. The shadow action vector is isolated and not executed.

[0032] It should be noted that the security verification includes individual unit power constraints, total system power constraints, and other constraints.

[0033] The specific individual power constraint is as follows: for each power value Pi in the vector, check whether 0≤Pi≤Pimax is satisfied, where Pimax is the rated maximum power of the charging pile or the maximum receiving power currently allowed by the vehicle's BMS.

[0034] The system's total power constraint is specifically defined as follows: check whether the sum of all power values ​​satisfies ΣPi≤Pt, where Pt is the upper limit of the total capacity of the transformer or grid inlet.

[0035] Other constraints can be set according to the actual situation of the charging station, such as three-phase balance constraints, cable current carrying capacity constraints, and temperature limits.

[0036] It should be explained that if Ap satisfies all constraints, the security check passes, and Ap is set as a safe action As. If Ap violates any constraint, the security check fails, triggering an action correction. The correction logic is as follows: This module models the correction problem as a constraint optimization problem: Objective: To find a new vector As that is closest to the original Ap; Constraint: All of the above safety conditions must be met; Solution method: Use computationally efficient projective gradient descent or quadratic programming methods for fast solution; Output: Corrected safety action As.

[0037] For example, if the total power exceeds the limit, i.e., ΣPi>Pt, the correction method is to scale it proportionally: Pis = Pi * (Pt / ΣPi), check again whether each Pis exceeds the individual power constraint. If it does, clamp it to Pimax and recursively repeat this process.

[0038] The final determined As is sent to the corresponding charging pile controller through the communication protocol to perform power adjustment. The shadow action vector Aq is strictly restricted to the data level and completely isolated from the actual control signal, forming a zero-risk shadow evaluation channel. It is only used for subsequent simulation reward calculation and offline learning, and is not directly or indirectly sent to the charging pile equipment.

[0039] S6. Construct a multi-objective reward function, calculate the actual reward score based on the real new state after the actual action vector is executed, and call the digital twin to simulate the execution of the shadow action vector and predict the simulated new state, and calculate the simulated reward score.

[0040] It should be specifically noted that the multi-objective reward function is a weighted sum of multiple sub-rewards, used to balance sometimes conflicting optimization objectives. These rewards include, but are not limited to: Efficiency reward Re is used to encourage more charging in order to maximize the total charging throughput. Re = ∑ (the amount of electricity charged by each charging station in time Δt). The fairness reward Rf prevents any vehicle from failing to charge and promotes even distribution. Rf = -std (the amount of charge each vehicle receives in this cycle). Rf is a large negative value, representing a penalty. A power grid safety reward Rg is given to prevent power over-limit. Rg = -max(0, actual total power - capacity limit). 2 ; The economic incentive Rc reduces charging costs. Rc = -(total charging capacity * real-time electricity price). User satisfaction reward Rs ensures that the user's charging goal is completed in a timely manner and provides positive feedback to vehicles that complete the charging task. Rs = Σ (when vehicle i reaches or exceeds its target charging state in this cycle, a fixed score is awarded). Power smoothing reward Rp protects the charging pile hardware and extends the equipment life. It also punishes drastic fluctuations in power allocation commands to protect the equipment. Rp = -Σ|Pi(t) - Pi(t-1)|, which is the sum of the absolute values ​​of the difference between the current power and the power of all charging piles in the previous cycle. The V2G service reward Rv2g guides electric vehicles to participate in discharge during peak grid load periods, achieving peak shaving and valley filling. When supporting the vehicle-to-grid function, the charging station is used as a virtual power plant resource to participate in grid regulation. Rv2g = +Σ (when Pi < 0 and it is during peak grid period, |Pi| * unit power reward coefficient), where Pi < 0 indicates that the charging pile is in the discharge state. The new energy consumption reward Rn encourages charging stations to prioritize the consumption of local renewable energy sources such as photovoltaic and wind power for charging. Rn = + (the amount of electricity from local renewable energy in the total charging volume during this cycle). The equipment balance loss reward Rw is used to balance the historical workload of each charging pile to extend the overall lifespan. Rw = -std (the historical cumulative running time of each charging pile), which is a penalty for the variance of the cumulative running time of each pile.

[0041] The multi-objective reward function is a combination of at least two of the above sub-rewards, and its general formula is R=Σ(wi*Ri), where wi is the weight coefficient corresponding to each sub-reward.

[0042] The actual reward score is the real result generated after using As, that is, the data obtained from the real state transition from St to St+1, specifically: Ra=R(St, Ap, St+1), which reflects how many points were actually obtained when As was executed in state St.

[0043] Reference Figure 3 The simulated reward score is predicted by the digital twin to accurately predict the simulated new state St+1′ that the environment will enter after executing Aq. It should be noted that since Aq is not actually executed, St+1′ in the tuple is not the next state that is actually observed, but a simulated state predicted by the digital twin. In order to build effective experience data for offline training in shadow mode, while ensuring the absolute security of the production system, the agent can obtain near-realistic policy performance feedback without bearing any execution risk.

[0044] The specific content predicted by the intelligent agent includes: Simulated charging amount at each charging station: calculated based on power Aq and time Δt, combined with the vehicle battery model; Simulated total power: ΣAq[i], this value can be obtained directly without prediction; Simulated changes in vehicle charging state, etc. Substitute the predicted data from St+1′ into the same reward function R, specifically: Rq = R(St, Aq, St+1′); This reflects how many points you can expect to get if Aq is executed under the same state St.

[0045] The dual-path reward calculation mechanism enables the system to continuously evaluate and train the shadow learning agent's strategy without interrupting normal service or incurring any security risks.

[0046] S7. The state vector, shadow action vector, simulated reward score and simulated new state obtained from multiple interactions are combined into a shadow experience tuple and stored in the shadow experience replay buffer. Data is periodically sampled from the buffer to perform offline learning and fine-tuning of the agent. After verification, the agent is triggered to go online.

[0047] It should be specifically noted that the shadow experience tuple is (St, Aq, Rq, St+1′), where St is the state vector, Aq is the shadow action vector, Rq is the simulated reward score, and St+1′ is the simulated new state.

[0048] The shadow experience accumulation is specifically as follows: the system runs continuously, and the shadow experience tuples generated in each interaction are filtered by a confidence level mechanism, retaining only the data whose prediction error is below the threshold. After being timestamped, the data is stored in the shadow experience replay buffer. This buffer is a first-in-first-out queue. When it is full, the oldest data is automatically discarded to maintain the freshness of the data. The capacity is tens of thousands to hundreds of thousands of experiences to ensure the diversity and sufficiency of the data.

[0049] The offline learning and fine-tuning of the intelligent agent specifically involves: At fixed time intervals, a batch of data is randomly sampled from the shadow experience replay buffer. The target network of the agent is trained offline in multiple rounds using sampled shadow experience data.

[0050] It is important to note that the training process is completely decoupled from the online decision-making system. The old strategy is still running online, and the training does not affect the stability of the current production environment. Furthermore, a small learning rate is used to allow the agent to slowly adapt to the real-world data distribution based on the existing pre-training.

[0051] It should be noted that the specific steps for triggering the online launch after verification are as follows: Set clear and quantifiable go-live thresholds: the fine-tuned shadow learning agent must have an average cumulative simulated reward that consistently exceeds the existing rules and strategies by more than 10% for at least 7 consecutive days, and the agent must have zero simulated security violations during the evaluation period; when all conditions are met, the system will automatically generate a report and notify the administrator, triggering the go-live process and preparing to switch the agent from shadow state to online control state.

[0052] S8. Deploy the verified shadow learning agent, and send out the generated action vectors for execution after security verification. Calculate the reward score based on the actual execution results, and store the actual experience tuples in the online experience replay buffer to continuously fine-tune the online strategy.

[0053] It should be specifically noted that deploying the validated shadow learning agent into the production environment, taking over system control, and continuously adapting to environmental changes through continuous learning during operation to achieve long-term performance optimization involves the following steps: A1. Switch systems and deploy intelligent agents online; By configuring the management system or publishing platform, the decision-making module of the online system can be seamlessly switched from rules and policies to intelligent agents to minimize risks, while the security verification module is retained and continues to run.

[0054] A2. Decision-making generates action vectors, verifies safety, and executes them; The agent receives the state vector St, and generates an online action vector through forward computation. The online action vector Ao is sent to the security verification module. If it passes the verification, it is sent directly; if it fails, it is corrected and then sent for execution.

[0055] A3. Collect real-world experience and calculate rewards; Wait for one time step, collect the real new state St+1, and calculate the reward score Rr based on the execution results of St and Ao and the real data in St+1 according to the multi-objective reward function.

[0056] A4. Continuous online fine-tuning based on real interaction data; The real experience tuples (St, Ao, Rr, St+1) obtained from this interaction are stored in the online experience replay buffer. This process uses an incremental update algorithm with a low learning rate to sample from the online experience replay buffer in real time and perform gradient optimization on the policy network. At the same time, the system maintains a dual feedback mechanism of shadow experience and online experience. Even in the online control phase, the digital twin still runs in parallel, providing a safe simulation environment for policy exploration and forming a continuous evolutionary closed loop of simulation-shadow-online three-element linkage.

[0057] It should be explained that a low learning rate is used to ensure the stability of learning and to prevent the strategy from fluctuating drastically due to a single batch of bad data, which could affect the operation of the system.

[0058] It is important to note that learning can lead to unstable policy performance. To mitigate this, a low learning rate, a strict trust zone algorithm, and continuous monitoring of performance metrics should be used, along with a rollback mechanism triggered when performance degrades. Even with security checks, flawed policies can still result in suboptimal performance. Therefore, rule-based policies should be retained as a fallback option, and a comprehensive alarm mechanism should be implemented.

[0059] Through the above description of the embodiments, those skilled in the art can clearly understand that the various embodiments of this application can be implemented by means of software or software combined with necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware functions. Based on this understanding, the technical solution of this application, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions to cause a computer device, such as including but not limited to a personal computer, server, or network device, to execute all or part of the steps of the method described in any embodiment of this application.

[0060] The foregoing has described exemplary embodiments of this application. It should be understood that the above exemplary embodiments are not restrictive but illustrative, and the scope of protection of this application is not limited thereto. It should be understood that those skilled in the art can make modifications and variations to the embodiments of this application without departing from the spirit and scope of this application, and such modifications and variations should be within the scope of protection of this application.

Claims

1. A charging pile cluster load balancing optimization method based on reinforcement learning, characterized in that, Specifically comprising the following steps: S1, a high-fidelity digital twin of the charging station is constructed based on a physical information neural network, and a shadow learning agent is pre-trained in simulation; S2, a data acquisition system is deployed to collect multi-source data in real time, including charging pile data, connected electric vehicle data, power grid market data, and context data, and the collected multi-source data is preprocessed and feature engineered; S3, the preprocessed feature data is constructed into a state vector understandable by the baseline strategy unit and the shadow learning agent, which is used to describe the complete state of the environment; S4, the state vector is input into the baseline rule strategy controller and the pre-trained shadow learning agent at the same time, the rule strategy controller generates an actual action vector according to the state vector, and the shadow learning agent generates a shadow action vector according to the same state vector in an isolated decision environment; S5, the actual action vector is safety checked, and after passing the safety check, it is issued for execution, and if it does not pass, the illegal action is corrected and then issued for execution, and the shadow action vector is isolated and not executed; S6, a multi-objective reward function is constructed, the actual reward points are calculated according to the real new state after the actual action vector is executed, and the digital twin is called to simulate the execution of the shadow action vector and predict the simulated new state, and the simulated reward points are calculated; S7, the state vector, shadow action vector, simulated reward points and simulated new state obtained through multiple interactions are combined into a shadow experience tuple, which is stored in a shadow experience replay buffer, and data is periodically sampled from the buffer to perform offline learning and fine-tuning of the agent, and the agent is verified and verified to trigger online; S8, the verified shadow learning agent is deployed, the generated action vector is safety checked and then issued for execution, the reward points are calculated according to the real execution result, and the real experience tuple is stored in the online experience replay buffer for continuous fine-tuning of the online strategy.

2. The reinforcement learning-based charging pile cluster load balancing optimization method according to claim 1, characterized in that: The multi-source data specifically includes: Charging pile data: charging pile ID, timestamp, active power, rated power, cumulative power, and working state; electric vehicle data: session ID, vehicle VIN, initial charging state, current charging state, target charging state, time spent, estimated remaining time, BMS maximum request power, and vehicle battery capacity; power grid market data: timestamp, real-time electricity price, transformer rated capacity, transformer remaining capacity, and total power of the site; context data: timestamp, day of the week, whether it is a holiday, and weather conditions.

3. The reinforcement learning based charging pile cluster load balancing optimization method according to claim 1, characterized in that: The feature engineering specifically includes: Charging urgency = (target charging state - current charging state) / estimated remaining time; load rate = total power of the site / transformer remaining capacity; time feature encoding: convert hours to sin(hour / 24*2π) and cos(hour / 24*2π).

4. The reinforcement learning based charging pile cluster load balancing optimization method according to claim 1, characterized in that: The state vector is a fixed-dimensional vector St, which specifically includes: Global state: [load rate, normalized real-time electricity price, normalized electricity price trend, predicted short-term load change, sine time, cosine time, day-of-week encoding, whether it is a holiday], the global state feature is fixed to m dimensions; Local state: [normalized current power, normalized current state of charge, normalized target state of charge, normalized vehicle maximum power, normalized dwell time, historical allocated electricity, vehicle connection state (0 / 1)]; Each charging pile is assigned a fixed-length feature sub-vector containing k dimensions; For the pile without vehicle connection, all numerical features in the local state sub-vector are set to 0, the global state vector and the local state vectors of N piles are spliced in order, the total dimension z = m + N * k, the state vector St = [global feature 1,..., global feature m, pile 1 feature 1,..., pile 1 feature k,..., pile N feature 1,..., pile N feature k], which is a fixed-dimensional array, describing the complete state of the environment at the current time.

5. The reinforcement learning based charging pile cluster load balancing optimization method according to claim 1, characterized in that: The action vector is used to assign a specific charging power value to each charging pile, specifically: The state vector St is input into the baseline rule policy controller, which calculates a power allocation scheme according to St, prioritizes a base power for each charging pile, and proportionally allocates the remaining power according to the charging capacity or electricity demand of each vehicle battery, outputs the actual action vector Ap = [P1, P2,..., Pn], where Pi is a real value in the range [0, Pimax], representing the power value directly assigned to the i-th charging pile; In parallel, the system inputs the same state vector St into the pre-trained shadow learning agent, and the agent's policy network performs forward inference on St, outputting a shadow action vector Aq = [Q1, Q2,..., Qn], this process only performs mathematical calculations and does not produce any actual control signals.

6. The reinforcement learning based charging pile cluster load balancing optimization method according to claim 1, characterized in that: The safety check specifically includes: Single power constraint: for each power value Pi in the vector, check whether it satisfies 0 ≤ Pi ≤ Pimax, where Pimax is the rated maximum power of the charging pile or the maximum receiving power currently allowed by the vehicle BMS; System total power constraint: check whether the sum of all power values satisfies ΣPi ≤ Pt, where Pt is the upper limit of the total capacity of the transformer or grid inlet; Other constraints: set according to the actual situation of the charging station; If Ap satisfies all constraints, the safety check is passed, and Ap is set as the safe action As; If Ap violates any constraint, the safety check fails, and action correction is triggered, and the final As is determined and issued to the corresponding charging pile controller through the communication protocol to perform power adjustment, the shadow action vector Aq is completely isolated from the execution channel and does not participate in the check and issuance, where Ap is the actual action vector.

7. The reinforcement learning based charging pile cluster load balancing optimization method according to claim 1, characterized in that: The multi-objective reward function is the weighted sum of multiple sub-rewards, specifically including: efficiency reward Re, Re =∑ (the amount of electricity charged by each charging pile in Δt time); fairness reward Rf, Rf =-std (the amount of electricity obtained by each vehicle in this period), Rf is a large negative value, indicating punishment; grid safety reward Rg, Rg =-max (0, actual total power-capacity upper limit) 2 ; economic reward Rc, Rc =- (total charging amount * real-time electricity price); user satisfaction reward Rs, Rs =∑ (when vehicle i reaches or exceeds its target state of charge in this period, reward a fixed score); power smoothing reward Rp, Rp =-∑|Pi (t)-Pi (t-1)|, that is, punish the sum of absolute values of the difference between the current power of all charging piles and the power of the last period; V2G service reward Rv2g, Rv2g =+∑ (when Pi <0 and in the peak period of the grid, |Pi| * unit power reward coefficient), wherein Pi <0 indicates that the charging pile is in the discharging state; new energy consumption reward Rn, encourage charging stations to preferentially consume local renewable energy such as local photovoltaic and wind power for charging, Rn =+ (the amount of electricity sourced from local renewable energy in the total charging amount in this period); equipment balanced loss reward Rw, Rw =-std (the historical cumulative running time of each charging pile), that is, punish the variance of the cumulative running time of each pile; the multi-objective reward function is a combination of at least two of the above sub-rewards, and its general formula is R =∑ (wi*Ri), wherein wi is the weight coefficient corresponding to each sub-reward. 8.The method of claim 1, wherein: The actual reward score is specifically: the real result generated after using As, that is, data obtained from the real state transition from St to St+1, specifically: Ra=R(St, Ap, St+1), reflecting that in state St, how many points are actually obtained by executing As, wherein Ap is the actual action vector, and As is the safety action; The simulation reward score is specifically: the simulation new state St+1' that the digital twin predicts the environment will enter after executing Aq, the data in St+1' predicted is substituted into the same reward function R, Rq=R(St, Aq, St+1'); reflecting that in the same state St, how many points are expected to be obtained if Aq is executed, wherein Aq is the shadow action vector. 9.The method of claim 1, wherein: The triggering of the online after verification is specifically: Set a clear and quantitative online threshold: the average cumulative simulation reward of the fine-tuned shadow learning agent is more than 10% higher than the existing rule strategy, is continuously maintained for at least 7 days, and the number of simulation safety violations of the agent during the evaluation period is zero; when all conditions are met, the system automatically generates a report and prompts the administrator to trigger the online process, and prepare to switch the agent from the shadow state to the online control state.

10. The charging pile cluster load balancing optimization system based on reinforcement learning, applied to the charging pile cluster load balancing optimization method based on reinforcement learning of any one of claims 1-9, characterized in that, Specifically comprising: A digital twin simulation module: for constructing a high-fidelity digital twin of the charging station based on a physical information neural network, providing a pre-training environment for the shadow learning agent and a state transition prediction in the shadow mode; A data perception and processing module: for deploying a data acquisition system to collect multi-source data in real time, and pre-processing and feature engineering of the data; A state representation module: its input end is connected to the data perception and processing module, for constructing the pre-processed feature data into a state vector that can be understood by the baseline strategy unit and the shadow learning agent; A dual-track decision module: its input end is connected to the state representation module, for receiving the state vector; the module includes: A baseline strategy unit: for generating an actual action vector according to the state vector; A shadow learning agent: runs in parallel with the baseline strategy unit, for generating a shadow action vector according to the same state vector in an isolated environment; A safety verification and execution module: its input end is connected to the baseline strategy unit, for safety verification of the actual action vector, and issuing execution of the charging pile after passing, and modifying and issuing execution of the charging pile again after failing; at the same time, the shadow action vector is isolated and not executed; A dual-path reward calculation module: its input end is connected to the safety verification and execution module and the digital twin simulation module respectively, for constructing a multi-objective reward function, calculating the actual reward score according to the real new state after the actual action vector is executed, and calling the digital twin simulation module to calculate the simulation reward score according to the predicted simulation new state of the shadow action vector; The shadow experience learning and verification module is connected to the reward calculation and evaluation module, and is used for storing the state vector, the shadow action vector, the simulation reward score and the simulation new state obtained through multiple interactions into a shadow experience replay buffer to form a shadow experience tuple, regularly sampling data for offline learning and fine-tuning, and triggering online after performance verification is passed; The online deployment and continuous learning module is used for deploying the verified shadow learning agent after the shadow experience learning and verification module triggers online, issuing and executing the action vector generated by the agent after security verification, calculating the reward score according to the real execution result, storing the real experience tuple into an online experience replay buffer, and continuously fine-tuning the online strategy.

Images