Charging pile load balancing method and system based on reinforcement learning

Abstract

The invention relates to the technical field of electric vehicle charging facility intelligent control, and discloses a charging pile load balancing method and system based on reinforcement learning. The system comprises a digital twinborn simulation module, a data sensing and processing module, a state characterization module, a double-track decision module, a baseline strategy unit, a shadow learning agent, a safety verification and execution module, a double-path reward calculation module, a shadow experience learning and verification module and an online deployment and continuous learning module. Through the technical path of simulation pre-training, shadow evaluation, safety verification and continuous learning, the problem of high deployment risk of reinforcement learning in a real system is solved, safe and smooth transition from simulation to reality is realized, absolute safety and continuous evolution of system operation are ensured, and the method is suitable for popularization and application. A prospective technical architecture is constructed to cover future scenes, and dynamic collaborative optimization of multiple targets such as charging efficiency, fairness, economy and power grid safety is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent control of electric vehicle charging facilities, and more particularly to a charging pile load balancing method and system based on reinforcement learning. BACKGROUND

[0002] With the popularization of electric vehicles, the operation efficiency and power grid stability of charging stations as key infrastructure are facing severe challenges. The load balancing problem of charging piles, i.e., reasonably allocating limited power resources among multiple charging requests, has become a core bottleneck restricting the service capacity and economic benefits of charging stations.

[0003] Currently, the load balancing strategy of charging stations mainly relies on traditional rule-based control methods, which usually include but are not limited to: first-come-first-served strategy: allocating power according to the order of vehicle connection until the total power reaches the upper limit, this method is simple to implement; average allocation strategy: evenly distributing the available total power to all vehicles charging, this method ensures basic fairness; priority strategy based on simple threshold: for example, allocating higher power to vehicles with low battery levels.

[0004] The existing methods also have the following defects: The existing technology lacks safe and efficient self-evolution ability, and once the strategy is deployed, it is fixed and cannot be changed. If optimization or defects are found, it must rely on manual analysis, redesign and manual update, the whole process is long, high cost and high risk, and cannot realize the autonomous learning and online upgrade of the system, which seriously lags behind the dynamically changing environment.

[0005] The decision-making dimension of the existing technology is single and rigid, and it is difficult to balance complex optimization goals. The strategy based on pre-defined rules (such as first-come-first-served, average allocation) cannot simultaneously quantify and weigh multiple dimensions such as efficiency, fairness, economy and safety, often sacrificing other aspects in order to meet a certain goal, resulting in sub-optimal overall system performance and inability to achieve globally optimal resource allocation.

[0006] The existing technology lacks forward-looking safety constraints and continuous optimization capabilities. Rule-based strategies usually only consider safety boundaries at design time and cannot perform real-time, online audits and corrections on all potential violations during execution, which poses a certain safety risk. At the same time, the system does not have the ability to learn from operational data, and its performance will gradually degrade over time and with changes in the environment.

[0007] The prior art architecture is closed and difficult to adapt to new needs in the future development, the traditional control method is designed for specific and known scenarios, and the architecture lacks scalability, when facing new business models or technical needs such as V2G and new energy cooperation, the system often needs to be restructured, which cannot realize smooth evolution and function expansion, protects past investment but is difficult to embrace future changes.

[0008] Therefore, a method that can adapt to new needs is needed to solve the above problems. SUMMARY

[0009] In order to overcome the above defects of the prior art, the application provides a charging pile load balancing method and system based on reinforcement learning to solve the problems in the above background art.

[0010] To achieve the above purpose, the application provides the following technical scheme: a charging pile load balancing method based on reinforcement learning, specifically comprising the following steps: S1, a neural network based on physical information is used to construct a high-fidelity digital twin of a charging station, and a shadow learning agent is pre-trained in simulation; S2, a data acquisition system is deployed to collect multi-source data in real time, including charging pile data, connected electric vehicle data, power grid market data and context data, and the collected multi-source data is preprocessed and feature engineered; S3, the preprocessed feature data is constructed into a state vector that can be understood by the baseline strategy unit and the shadow learning agent, which is used to describe the complete state of the environment; S4, the state vector is input into the baseline rule strategy controller and the pre-trained shadow learning agent at the same time, the rule strategy controller generates an actual action vector according to the state vector, and the shadow learning agent generates a shadow action vector according to the same state vector in an isolated decision environment; S5, the actual action vector is safety checked, and after passing the safety check, it is issued for execution, and if it does not pass, the illegal action is corrected and then issued for execution, and the shadow action vector is isolated and not executed; S6, a multi-objective reward function is constructed, the actual reward points are calculated according to the real new state after the actual action vector is executed, and the digital twin is called to simulate the execution of the shadow action vector and predict the simulated new state, and the simulated reward points are calculated; S7, the state vector, shadow action vector, simulated reward points and simulated new state obtained through multiple interactions are combined into a shadow experience tuple, which is stored in a shadow experience replay buffer, data is periodically sampled from the buffer, the agent is offline learned and fine-tuned, and verified and passed to trigger online; S8. Deploy the verified shadow learning agent, and send out the generated action vectors for execution after security verification. Calculate the reward score based on the actual execution results, and store the actual experience tuples in the online experience replay buffer to continuously fine-tune the online strategy.

[0011] A charging pile load balancing system based on reinforcement learning, specifically including: Digital twin simulation module: used to construct a high-fidelity digital twin of a charging station based on a neural network of physical information, providing a pre-training environment for the shadow learning agent and predicting state transitions in shadow mode; Data perception and processing module: used to deploy the data acquisition system to collect multi-source data in real time, and to preprocess and feature-engineer the data; State representation module: Its input is connected to the data perception and processing module, and it is used to construct the preprocessed feature data into a state vector that can be understood by the baseline policy unit and the shadow learning agent; Dual-track decision module: Its input is connected to the state representation module to receive the state vector; this module includes: Baseline strategy unit: used to generate actual action vectors based on the state vector; Shadow learning agent: operates in parallel with the baseline policy unit to generate shadow action vectors in an isolated environment based on the same state vector; Security verification and execution module: Its input is connected to the baseline strategy unit, which is used to perform security verification on the actual action vector. If it passes the verification, it is sent to the charging pile for execution. If it fails the verification, the violation action is corrected and then sent for execution. At the same time, the shadow action vector is isolated and not executed. Dual-path reward calculation module: Its input terminals are respectively connected to the security verification and execution module and the digital twin simulation module. It is used to construct a multi-objective reward function, calculate the actual reward score based on the real new state after the actual action vector is executed, and call the digital twin simulation module to predict and simulate the new state based on the shadow action vector to calculate the simulated reward score. Shadow Experience Learning and Verification Module: Its input end is connected to the reward calculation and evaluation module. It is used to combine the state vector, shadow action vector, simulated reward score and simulated new state obtained from multiple interactions into a shadow experience tuple, store it in the shadow experience playback buffer, periodically sample data for offline learning and fine-tuning, and trigger online after performance verification is passed. Online Deployment and Continuous Learning Module: After the Shadow Experience Learning and Verification Module is triggered to go online, it deploys the verified shadow learning agent, sends out the generated action vectors after security verification, calculates the reward score based on the actual execution results, and stores the actual experience tuples in the online experience replay buffer to continuously fine-tune the online strategy.

[0012] The technical effects and advantages of this invention are as follows: 1. This invention achieves a safe and smooth transition from simulation to reality. By combining digital twin pre-training with parallel evaluation of shadow mode, a complete engineering implementation path for reinforcement learning is constructed. The intelligent agent first safely learns basic policies in a high-fidelity simulation environment, then learns and verifies risk-free in a real system as an observer, and finally takes over control only after meeting strict performance indicators. This fundamentally solves the industry problem of high trial-and-error costs and high deployment risks of AI in critical infrastructure.

[0013] 2. This invention possesses intelligent decision-making capabilities for multi-objective collaborative optimization. Unlike the single objective of traditional rule-based strategies, the multi-objective reward function designed in this invention can simultaneously quantify performance across multiple dimensions such as efficiency, fairness, safety, economy, and user experience. The shadow learning agent, through interaction with the environment, autonomously learns how to dynamically weigh these often conflicting objectives, thereby achieving globally optimal power allocation and significantly improving the overall operational efficiency of charging stations.

[0014] 3. This invention ensures the absolute safety and continuous evolution of the system operation. A mandatory safety verification module is set up between the decision-making and execution layers to audit and correct all power commands in real time, ensuring that they never exceed the limits. At the same time, the system is designed with a dual learning mechanism of offline fine-tuning and online learning, which enables the agent to continuously evolve using real data, adapt to environmental changes, and become an autonomous system with long-term vitality and increasingly stronger performance.

[0015] 4. This invention constructs a forward-looking technical architecture to cover future scenarios. The design of the state vector and reward function not only meets current needs, but also forward-lookingly incorporates key elements of future smart grids and green energy such as V2G, new energy consumption, and equipment life balance. This enables the system to not only solve the current load balancing problem, but also smoothly adapt to and serve future energy interconnection and advanced business models.

[0016] 5. This invention constructs a self-evolving system paradigm of "evaluation-learning-verification-deployment". Through the dual protection of shadow mode and security verification, it solves the cold start risk and high trial-and-error cost problems of deploying reinforcement learning in real systems. Through the dual learning mechanism of offline and online, it ensures that the system strategy can continuously adapt to the dynamic environment and become an autonomous system with long-term vitality and increasingly stronger performance. Attached Figure Description

[0017] Figure 1 This is a structural block diagram of the present invention.

[0018] Figure 2 This is a flowchart of the present invention.

[0019] Figure 3 This is a diagram illustrating the shadow mode operation of the present invention. Detailed Implementation

[0020] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. In addition, the forms of the various structures described in the following embodiments are merely illustrative. The reinforcement learning-based charging pile load balancing method and system involved in the present invention are not limited to the structures described in the following embodiments. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Reference Figure 1 This invention provides a charging pile load balancing method and system based on reinforcement learning, including a digital twin simulation module, a data perception and processing module, a state representation module, a dual-track decision-making module, a baseline strategy unit, a shadow learning agent, a security verification and execution module, a dual-path reward calculation module, a shadow experience learning and verification module, and an online deployment and continuous learning module.

[0022] Reference Figure 2 The specific implementation steps of the present invention include the following steps: S1. Construct a high-fidelity digital twin of the charging station based on a neural network of physical information, and pre-train the shadow learning agent in the simulation.

[0023] It should be noted that the input data for digital twin modeling includes: Physical model data: charging pile circuit topology, transformer thermal characteristic parameters, cable impedance and efficiency curves; Historical operational data: Records of past vehicle charging sessions, including arrival time, initial charging status, end charging status, dwell time, and charging power curve; External data: historical electricity prices, weather data, and holiday information.

[0024] Using PyBullet, MATLAB / Simulink, or a self-developed physics-based simulator, a high-fidelity digital twin model of the charging station is constructed. This model primarily simulates power constraints and vehicle dynamics, such as the total power not exceeding the transformer capacity and the curve of the BMS adjusting the charging power according to the charging status. A shadow learning agent is constructed using the PPO or SAC algorithm, with the input layer dimension matching the state vector and the output layer dimension matching the charging pile power value. Pre-training is performed, and in the simulation, tens of thousands to millions of rounds are simulated, each round representing a day's operations. Through trial and error, the agent learns how to allocate power to maximize cumulative rewards in various simulated scenarios, such as morning rush hour, peak electricity prices, and vehicle congestion.

[0025] S2. Deploy a data acquisition system to collect multi-source data in real time, including charging pile data, connected electric vehicle data, power grid market data, and contextual data, and perform preprocessing and feature engineering on the collected multi-source data.

[0026] It should be specifically noted that the multi-source data includes: Charging pile data: charging pile ID, timestamp, active power, rated power, cumulative power consumption, and operating status; Electric vehicle data: Session ID, Vehicle VIN, Initial charging status, Current charging status, Target charging status, Dwell time, Estimated remaining dwell time, BMS maximum requested power, and Vehicle battery capacity. Grid market data: timestamps, real-time electricity prices, transformer rated capacity, transformer remaining capacity, and total power at the site; Contextual data: timestamp, day of the week, whether it is a holiday, and weather conditions.

[0027] The preprocessing includes data cleaning and normalization. Data cleaning will process abnormal data such as negative power and charging status exceeding 100%, and use front and back interpolation or KNN-based filling methods for missing values. Normalization will scale the values ​​of charging status, power, and electricity price to the range of [0, 1] or [-1, 1].

[0028] The feature engineering specifically includes: Charging urgency = (Target charging state - Current charging state) / Estimated remaining dwell time; Load factor = Total power of the site / Remaining capacity of the transformer; Time feature encoding: Convert hours into sin(hour / 24*2π) and cos(hour / 24*2π) to reflect the periodicity of time.

[0029] S3. Construct the preprocessed feature data into a state vector that can be understood by the baseline policy unit and the shadow learning agent, which is used to describe the complete state of the environment.

[0030] It should be noted that the state vector is a fixed-dimensional vector St, which includes the global state and the local state of each charging station, specifically: Global state: [load rate, normalized real-time electricity price, normalized electricity price trend, predicted short-term load changes, sine time, cosine time, day of the week code, whether it is a holiday], the global state features are fixed to m dimensions; Local state: [Normalized current power, normalized current charging state, normalized target charging state, normalized vehicle maximum power, normalized dwell time, historical power allocation, vehicle connection state (0 / 1)]; For N charging piles, repeat N times. Each charging pile, regardless of whether there is a vehicle connected, is assigned a feature sub-vector of fixed length. The feature sub-vector of each charging pile is defined to contain k dimensions. To ensure that the vector dimensions are fixed, the maximum number of piles is used. For a pile without a vehicle connected, all numerical features in its local state sub-vector are set to 0, except for the vehicle connection state which is 0.

[0031] It is important to note that the specific characteristics and dimensions of the global and local states are not static and should be adjusted in actual deployment based on specific business objectives, data availability, and model performance requirements.

[0032] It should be noted that the global state vector and the local state vectors of N stakes are concatenated in order, with a total dimension z = m + N*k. The state vector St = [global feature 1, ..., global feature m, stake 1 feature 1, ..., stake 1 feature k, ..., stake N feature 1, ..., stake N feature k] is a fixed-dimensional array that describes the complete state of the environment at the current moment.

[0033] It is important to note that if the number of charging piles exceeds a certain number, such as N > 50, a graph neural network (GNN) is used to avoid lengthy state vectors. Each charging pile is treated as a node in the graph, and the features of the node are local state vectors. The GNN can actively learn the relationships between nodes (piles), thereby more intelligently discovering strategies such as allocating power from idle areas to busy areas. The global state is the attribute of the entire graph.

[0034] S4. The state vector is simultaneously input to the baseline rule policy controller and the pre-trained shadow learning agent. The rule policy controller generates the actual action vector based on the state vector, and the shadow learning agent generates the shadow action vector based on the same state vector in the isolated decision environment.

[0035] It should be specifically explained that a dual-track decision-making architecture is adopted: the existing rule strategy generates the actual action vector based on the state vector. At the same time, the pre-trained shadow learning agent receives the same state vector in a completely isolated sandbox environment and generates a shadow action vector. The two decision-making processes are parallel and synchronous, but the performance is compared only at the logical level. The physical control signals are completely separated, ensuring the physical or logical isolation between the optimization decision-making process and the actual control process. This fundamentally eliminates the risk of unverified shadow learning agent decisions interfering with the real system.

[0036] The actual action vector is used to allocate a specific charging power value to each charging pile. Specifically, the state vector St is input to the baseline rule strategy controller, which calculates a power allocation scheme based on St, prioritizing a base power for each charging pile, and then allocating the remaining power proportionally according to the charging capacity or power demand of each vehicle's battery, and outputting the actual action vector Ap=[P1, P2, ..., Pn], where Pi is a real value in the range [0, Pimax], representing the power value directly allocated to the i-th charging pile.

[0037] In parallel, the system inputs the exact same state vector St into the pre-trained shadow learning agent. The agent's policy network performs forward inference on St and outputs a shadow action vector Aq, where Aq = [Q1, Q2, ..., Qn]. This process only performs mathematical calculations and does not generate any actual control signals.

[0038] S5. Perform a safety check on the actual action vector. If the safety check passes, send it to the charging pile for execution. If it fails, correct the violation action and send it for execution again. The shadow action vector is isolated and not executed.

[0039] It should be noted that the security verification includes individual unit power constraints, total system power constraints, and other constraints.

[0040] The specific individual power constraint is as follows: for each power value Pi in the vector, check whether 0≤Pi≤Pimax is satisfied, where Pimax is the rated maximum power of the charging pile or the maximum receiving power currently allowed by the vehicle's BMS.

[0041] The system's total power constraint is specifically defined as follows: check whether the sum of all power values ​​satisfies ΣPi≤Pt, where Pt is the upper limit of the total capacity of the transformer or grid inlet.

[0042] Other constraints can be set according to the actual situation of the charging station, such as three-phase balance constraints, cable current carrying capacity constraints, and temperature limits.

[0043] It should be explained that if Ap satisfies all constraints, the security check passes, and Ap is set as a safe action As. If Ap violates any constraint, the security check fails, triggering an action correction. The correction logic is as follows: This module models the correction problem as a constraint optimization problem: Objective: Find a new vector As that is closest to the original Ap; Constraint: All of the above safety conditions must be met; Solution method: Use computationally efficient projective gradient descent or quadratic programming methods for fast solution; Output: Corrected safety action As.

[0044] For example, if the total power exceeds the limit, i.e., ΣPi>Pt, the correction method is to scale it proportionally: Pis = Pi * (Pt / ΣPi), check again whether each Pis exceeds the individual power constraint. If it does, clamp it to Pimax and recursively repeat this process.

[0045] The final determined As is sent to the corresponding charging pile controller through the communication protocol to perform power adjustment. The shadow action vector Aq is strictly restricted to the data level and completely isolated from the actual control signal, forming a zero-risk shadow evaluation channel. It is only used for subsequent simulation reward calculation and offline learning, and is not directly or indirectly sent to the charging pile equipment.

[0046] S6. Construct a multi-objective reward function, calculate the actual reward score based on the real new state after the actual action vector is executed, and call the digital twin to simulate the execution of the shadow action vector and predict the simulated new state, and calculate the simulated reward score.

[0047] It should be specifically noted that the multi-objective reward function is a weighted sum of multiple sub-rewards, used to balance sometimes conflicting optimization objectives. These rewards include, but are not limited to: Efficiency reward Re is used to encourage more charging in order to maximize the total charging throughput. Re = ∑ (the amount of electricity charged by each charging station in time Δt). The fairness reward Rf prevents any vehicle from failing to charge and promotes even distribution. Rf = -std (the amount of charge each vehicle receives in this cycle). Rf is a large negative value, representing a penalty. A power grid safety reward Rg is given to prevent power over-limit. Rg = -max(0, actual total power - capacity limit). 2 ; The economic incentive Rc reduces charging costs. Rc = -(total charging capacity * real-time electricity price). User satisfaction reward Rs ensures that the user's charging goal is completed in a timely manner and provides positive feedback to vehicles that complete the charging task. Rs = Σ (when vehicle i reaches or exceeds its target charging state in this cycle, a fixed score is awarded). Power smoothing reward Rp protects the charging pile hardware and extends the equipment life. It also punishes drastic fluctuations in power allocation commands to protect the equipment. Rp = -Σ|Pi(t) - Pi(t-1)|, which is the sum of the absolute values ​​of the difference between the current power and the power of all charging piles in the previous cycle. The V2G service reward Rv2g guides electric vehicles to participate in discharge during peak grid load periods, achieving peak shaving and valley filling. When supporting the vehicle-to-grid function, the charging station is used as a virtual power plant resource to participate in grid regulation. Rv2g = +Σ(when Pi < 0 and it is during peak grid period, |Pi| * unit power reward coefficient), where Pi < 0 indicates that the charging pile is in the discharge state. The new energy consumption reward Rn encourages charging stations to prioritize the consumption of local renewable energy sources such as photovoltaic and wind power for charging. Rn = + (the amount of electricity from local renewable energy in the total charging volume during this cycle). The equipment balance loss reward Rw balances the historical workload of each charging pile to extend the overall lifespan. Rw = -std (the historical cumulative running time of each charging pile), which is a penalty for the variance of the cumulative running time of each pile.

[0048] The multi-objective reward function is a combination of at least two of the above sub-rewards, and its general formula is R=Σ(wi*Ri), where wi is the weight coefficient corresponding to each sub-reward.

[0049] The actual reward score is the real result generated after using As, that is, the data obtained from the real state transition from St to St+1, specifically: Ra=R(St, Ap, St+1), which reflects how many points were actually obtained when As was executed in state St.

[0050] Reference Figure 3 The simulated reward score is predicted by the digital twin to accurately predict the simulated new state St+1′ that the environment will enter after executing Aq. It should be noted that since Aq is not actually executed, St+1′ in the tuple is not the next state that is actually observed, but a simulated state predicted by the digital twin. In order to build effective experience data for offline training in shadow mode, while ensuring the absolute security of the production system, the agent can obtain near-realistic policy performance feedback without bearing any execution risk.

[0051] The specific content predicted by the intelligent agent includes: Simulated charging amount at each charging station: calculated based on power Aq and time Δt, combined with the vehicle battery model; Simulated total power: ΣAq[i], this value can be obtained directly without prediction; Simulated changes in vehicle charging state, etc. Substitute the predicted data from St+1′ into the same reward function R, specifically: Rq = R(St, Aq, St+1′); This reflects how many points you can expect to get if Aq is executed under the same state St.

[0052] The dual-path reward calculation mechanism enables the system to continuously evaluate and train the shadow learning agent's strategy without interrupting normal services or incurring any security risks.

[0053] S7. The state vector, shadow action vector, simulated reward score and simulated new state obtained from multiple interactions are combined into a shadow experience tuple and stored in the shadow experience replay buffer. Data is periodically sampled from the buffer to perform offline learning and fine-tuning of the agent. After verification, the agent is triggered to go online.

[0054] It should be specifically noted that the shadow experience tuple is (St, Aq, Rq, St+1′), where St is the state vector, Aq is the shadow action vector, Rq is the simulated reward score, and St+1′ is the simulated new state.

[0055] The shadow experience accumulation is specifically as follows: the system runs continuously, and the shadow experience tuples generated in each interaction are filtered by a confidence level mechanism, retaining only the data whose prediction error is below the threshold. After being timestamped, the data is stored in the shadow experience replay buffer. This buffer is a first-in-first-out queue. When it is full, the oldest data is automatically discarded to maintain the freshness of the data. The capacity is tens of thousands to hundreds of thousands of experiences to ensure the diversity and sufficiency of the data.

[0056] The offline learning and fine-tuning of the intelligent agent specifically refers to: At fixed time intervals, a batch of data is randomly sampled from the shadow experience replay buffer. The target network of the agent is trained offline in multiple rounds using sampled shadow experience data.

[0057] It is important to note that the training process is completely decoupled from the online decision-making system. The old strategy is still running online, and the training does not affect the stability of the current production environment. Furthermore, a small learning rate is used to allow the agent to slowly adapt to the real-world data distribution based on the existing pre-training.

[0058] It should be noted that the specific steps for triggering the online launch after verification are as follows: Set clear and quantifiable go-live thresholds: the fine-tuned shadow learning agent must have an average cumulative simulated reward that consistently exceeds the existing rules and strategies by more than 10% for at least 7 consecutive days, and the agent must have zero simulated security violations during the evaluation period; when all conditions are met, the system will automatically generate a report and notify the administrator, triggering the go-live process and preparing to switch the agent from shadow state to online control state.

[0059] S8. Deploy the verified shadow learning agent, and send out the generated action vectors for execution after security verification. Calculate the reward score based on the actual execution results, and store the actual experience tuples in the online experience replay buffer to continuously fine-tune the online strategy.

[0060] It should be specifically noted that deploying the validated shadow learning agent into the production environment, taking over system control, and continuously adapting to environmental changes through continuous learning during operation to achieve long-term performance optimization involves the following steps: A1. Switch systems and deploy intelligent agents online; By configuring the management system or publishing platform, the decision-making module of the online system can be seamlessly switched from rules and policies to intelligent agents to minimize risks, while the security verification module is retained and continues to run.

[0061] A2. Decision-making generates action vectors, verifies safety, and executes them; The agent receives the state vector St, and generates an online action vector through forward computation. The online action vector Ao is sent to the security verification module. If it passes the verification, it is sent directly; if it fails, it is corrected and then sent for execution.

[0062] A3. Collect real-world experience and calculate rewards; Wait for one time step, collect the real new state St+1, and calculate the reward score Rr based on the execution results of St and Ao and the real data in St+1 according to the multi-objective reward function.

[0063] A4. Continuous online fine-tuning based on real interaction data; The real experience tuples (St, Ao, Rr, St+1) obtained from this interaction are stored in the online experience replay buffer. This process uses an incremental update algorithm with a low learning rate to sample from the online experience replay buffer in real time and perform gradient optimization on the policy network. At the same time, the system maintains a dual feedback mechanism of shadow experience and online experience. Even in the online control phase, the digital twin still runs in parallel, providing a safe simulation environment for policy exploration and forming a continuous evolutionary closed loop of simulation-shadow-online three-element linkage.

[0064] It should be explained that a low learning rate is used to ensure the stability of learning and to prevent the strategy from fluctuating drastically due to a single batch of bad data, which could affect the operation of the system.

[0065] It is important to note that learning can lead to unstable policy performance. To mitigate this, a low learning rate, a strict trust zone algorithm, and continuous monitoring of performance metrics should be used, along with a rollback mechanism triggered when performance degrades. Even with security checks, flawed policies can still result in suboptimal performance. Therefore, rule-based policies should be retained as a fallback option, and a comprehensive alarm mechanism should be implemented.

[0066] Through the above description of the embodiments, those skilled in the art can clearly understand that the various embodiments of this application can be implemented by means of software or software combined with necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware functions. Based on this understanding, the technical solution of this application, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions to cause a computer device, such as including but not limited to a personal computer, server, or network device, to execute all or part of the steps of the method described in any embodiment of this application.

[0067] The foregoing has described exemplary embodiments of this application. It should be understood that the above exemplary embodiments are not restrictive but illustrative, and the scope of protection of this application is not limited thereto. It should be understood that those skilled in the art can make modifications and variations to the embodiments of this application without departing from the spirit and scope of this application, and such modifications and variations should be within the scope of protection of this application.

Claims

1. A load balancing method for charging piles based on reinforcement learning, characterized in that, Specifically, the following steps are included: S1. Construct a high-fidelity digital twin of the charging station based on a neural network of physical information, and pre-train the shadow learning agent in the simulation; S2. Deploy a data acquisition system to collect multi-source data in real time, including charging pile data, connected electric vehicle data, power grid market data and context data, and perform preprocessing and feature engineering on the collected multi-source data; S3. Construct the preprocessed feature data into a state vector that can be understood by the baseline policy unit and the shadow learning agent, which is used to describe the complete state of the environment. S4. The state vector is simultaneously input to the baseline rule policy controller and the pre-trained shadow learning agent. The rule policy controller generates the actual action vector based on the state vector, and the shadow learning agent generates the shadow action vector based on the same state vector in the isolated decision environment. S5. Perform a safety check on the actual action vector. If the safety check passes, send it to the charging pile for execution. If it fails, correct the violation action and send it for execution again. The shadow action vector is isolated and not executed. S6. Construct a multi-objective reward function, calculate the actual reward score based on the real new state after the actual action vector is executed, and call the digital twin to simulate the execution of the shadow action vector and predict the simulated new state, and calculate the simulated reward score. S7. The state vector, shadow action vector, simulated reward score and simulated new state obtained from multiple interactions are combined into a shadow experience tuple and stored in the shadow experience playback buffer. Data is periodically sampled from the buffer to perform offline learning and fine-tuning of the agent. After verification, the agent is triggered to go online. S8. Deploy the verified shadow learning agent, and send out the generated action vectors for execution after security verification. Calculate the reward score based on the actual execution results, and store the actual experience tuples in the online experience replay buffer to continuously fine-tune the online strategy.

2. The charging pile load balancing method based on reinforcement learning according to claim 1, characterized in that: The multi-source data specifically includes: Charging pile data: charging pile ID, timestamp, active power, rated power, cumulative power consumption, and operating status; Electric vehicle data: session ID, vehicle VIN, initial charging status, current charging status, target charging status, elapsed dwell time, estimated remaining dwell time, maximum power requested by the BMS, and vehicle battery capacity; Grid market data: timestamp, real-time electricity price, transformer rated capacity, remaining transformer capacity, and total power of the site; Contextual data: timestamp, day of the week, whether it is a holiday, and weather conditions.

3. The reinforcement learning-based charging pile load balancing method according to claim 1, characterized in that: The feature engineering specifically includes: Charging urgency = (target charging state - current charging state) / estimated remaining dwell time; Load rate = total power of the site / remaining capacity of the transformer; Time feature encoding: convert hours to sin(hour / 24*2π) and cos(hour / 24*2π).

4. The charging pile load balancing method based on reinforcement learning according to claim 1, characterized in that: The state vector is a fixed-dimensional vector St, specifically including: Global state: [load rate, normalized real-time electricity price, normalized electricity price trend, predicted short-term load change, sine time, cosine time, day of the week code, whether it is a holiday], the global state features are fixed to m dimensions; Local state: [normalized current power, normalized current charging state, normalized target charging state, normalized vehicle maximum power, normalized dwell time, historical allocated electricity, vehicle connection status (0 / 1)]; Each charging station is assigned a fixed-length feature sub-vector containing k dimensions. For charging stations without connected vehicles, all numerical features in their local state sub-vector are set to 0. The global state vector and the local state vectors of N charging stations are concatenated in sequence, with a total dimension z = m + N * k. The state vector St = [global feature 1, ..., global feature m, station 1 feature 1, ..., station 1 feature k, ..., station N feature 1, ..., station N feature k] is a fixed-dimensional array that describes the complete state of the environment at the current moment.

5. The charging pile load balancing method based on reinforcement learning according to claim 1, characterized in that: The action vector is used to assign a specific charging power value to each charging pile, specifically: The state vector St is input to the baseline rule policy controller, which calculates a power allocation scheme based on St. Prioritizing a base power supply for each charging station, the remaining power is then allocated proportionally according to the charging capacity or power demand of each vehicle's battery. The controller outputs an actual action vector Ap = [P1, P2, ..., Pn], where Pi is a real value in the range [0, Pimax], representing the power value directly allocated to the i-th charging station. In parallel, the system inputs the identical state vector St to a pre-trained shadow learning agent. The agent's policy network performs forward inference on St, outputting a shadow action vector Aq, Aq = [Q1, Q2, ..., Qn]. This process involves only mathematical calculations and does not generate any actual control signals.

6. The charging pile load balancing method based on reinforcement learning according to claim 1, characterized in that: The security verification specifically includes: Individual power constraint: For each power value Pi in the vector, check whether 0≤Pi≤Pimax, where Pimax is the rated maximum power of the charging pile or the maximum allowable receiving power of the vehicle's BMS; Total system power constraint: Check whether the sum of all power values ​​satisfies ΣPi≤Pt, where Pt is the upper limit of the total capacity of the transformer or grid inlet; Other constraints: Set according to the actual situation of the charging station; If Ap satisfies all constraints, the safety check passes and Ap is set as a safe action As; If Ap violates any constraint, the safety check fails and action correction is triggered. The final determined As is sent to the corresponding charging pile controller through the communication protocol to perform power adjustment. The shadow action vector Aq is completely isolated from the execution channel and does not participate in the check and sending.

7. The reinforcement learning-based charging pile load balancing method according to claim 1, characterized in that: The multi-objective reward function is a weighted sum of multiple sub-rewards, specifically including: Efficiency reward Re, Re = ∑ (the amount of electricity charged by each charging station in time Δt); Fairness reward Rf, Rf = -std (the amount of electricity charged by each vehicle in this cycle), Rf is a large negative value, indicating a penalty; Grid safety reward Rg, Rg = -max(0, actual total power - capacity limit). 2 Economic reward Rc, Rc = -(total charging capacity * real-time electricity price); User satisfaction reward Rs, Rs = Σ(a fixed score is awarded when vehicle i reaches or exceeds its target charging state in this cycle); Power smoothing reward Rp, Rp = -Σ|Pi(t) - Pi(t-1)|, which is the sum of the absolute values ​​of the differences between the current power and the power of all charging piles in the previous cycle; V2G service reward Rv2g, Rv2g = +Σ(|Pi| * unit power reward coefficient when Pi < 0 and it is during the peak period of the power grid), where Pi <0 indicates that the charging pile is in a discharging state; the new energy consumption reward Rn encourages the charging station to prioritize the consumption of local renewable energy sources such as photovoltaic and wind power for charging, Rn=+(the amount of electricity from local renewable energy in the total charging amount in this cycle); the equipment balancing loss reward Rw, Rw=-std(the historical cumulative running time of each charging pile), that is, to penalize the variance of the cumulative running time of each pile; the multi-objective reward function is a combination of at least two of the above sub-rewards, and its general formula is R=Σ(wi*Ri), where wi is the weight coefficient corresponding to each sub-reward.

8. The charging pile load balancing method based on reinforcement learning according to claim 1, characterized in that: The actual reward score is specifically: the real result generated after using As, that is, the data obtained from the real state transition from St to St+1, specifically: Ra=R(St, Ap, St+1), which reflects how many points were actually obtained when As was executed in state St; The simulated reward score is specifically calculated as follows: the simulated new state St+1′ that the environment will enter after executing Aq is predicted using a digital twin. The data in the predicted St+1′ is then substituted into the same reward function R, Rq=R(St, Aq, St+1′). This reflects how many points can be expected to be obtained if Aq is executed under the same state St.

9. The reinforcement learning-based charging pile load balancing method according to claim 1, characterized in that: The specific steps for triggering the online launch after verification are as follows: Set clear and quantifiable go-live thresholds: the fine-tuned shadow learning agent must have an average cumulative simulated reward that consistently exceeds the existing rules and strategies by more than 10% for at least 7 consecutive days, and the agent must have zero simulated security violations during the evaluation period; when all conditions are met, the system will automatically generate a report and notify the administrator, triggering the go-live process and preparing to switch the agent from shadow state to online control state.

10. The charging pile load balancing system based on reinforcement learning according to claim 1, characterized in that, Specifically, it includes: Digital twin simulation module: used to construct a high-fidelity digital twin of a charging station based on a neural network of physical information, providing a pre-training environment for the shadow learning agent and predicting state transitions in shadow mode; Data perception and processing module: used to deploy the data acquisition system to collect multi-source data in real time, and to preprocess and feature-engineer the data; State representation module: Its input is connected to the data perception and processing module, and it is used to construct the preprocessed feature data into a state vector that can be understood by the baseline policy unit and the shadow learning agent; Dual-track decision module: Its input is connected to the state representation module to receive the state vector; this module includes: Baseline strategy unit: used to generate actual action vectors based on the state vector; Shadow learning agent: operates in parallel with the baseline policy unit to generate shadow action vectors based on the same state vector in an isolated environment; Security verification and execution module: Its input is connected to the baseline strategy unit, which is used to perform security verification on the actual action vector. If it passes the verification, it is sent to the charging pile for execution. If it fails the verification, the violation action is corrected and then sent for execution. At the same time, the shadow action vector is isolated and not executed. Dual-path reward calculation module: Its input terminals are respectively connected to the security verification and execution module and the digital twin simulation module. It is used to construct a multi-objective reward function, calculate the actual reward score based on the real new state after the actual action vector is executed, and call the digital twin simulation module to predict and simulate the new state based on the shadow action vector to calculate the simulated reward score. Shadow Experience Learning and Verification Module: Its input end is connected to the reward calculation and evaluation module. It is used to combine the state vector, shadow action vector, simulated reward score and simulated new state obtained from multiple interactions into a shadow experience tuple, store it in the shadow experience playback buffer, periodically sample data for offline learning and fine-tuning, and trigger online after performance verification is passed. Online Deployment and Continuous Learning Module: After the Shadow Experience Learning and Verification Module is triggered to go online, it deploys the verified shadow learning agent, sends out the generated action vectors after security verification, calculates the reward score based on the actual execution results, and stores the actual experience tuples in the online experience replay buffer to continuously fine-tune the online strategy.

Images