Mobile vehicle charging and storage autonomous charging optimization method and system based on virtual environment emergence data and reinforcement learning

Abstract

The invention discloses a mobile vehicle charging and storage autonomous charging optimization method and system based on virtual environment emergence data and reinforcement learning. According to the method, a high-fidelity dynamic parking lot simulation environment is constructed through AnyLogic, and parking space occupation, charging requirements and obstacle behaviors are simulated; in combination with a Markov chain Monte Carlo method, multi-modal virtual data covering normal scenes, peak periods and equipment faults are generated, and Pareto distribution extreme events are injected to improve model robustness. A multi-target depth deterministic strategy gradient algorithm is adopted, a reward function fusing energy efficiency, time cost, safety, demand coverage, parking duration priority and peak-valley electricity price economy is designed, weight coefficients are dynamically adjusted through Pareto frontier analysis and Bayesian optimization, and end-to-end charging resource allocation and differential pricing are achieved.

Description

Technical Field

[0001] This invention belongs to the field of intelligent charging equipment control technology, specifically relating to a mobile charging vehicle autonomous charging optimization method and system based on virtual environment and reinforcement learning. Background Technology

[0002] Existing charging pile management technologies suffer from the following problems: Static strategies are limited, with fixed charging power and prices, failing to consider factors such as user parking time and dynamic changes in grid load. Supply-demand imbalance: Concentrated charging demand during peak hours puts significant pressure on the grid, while the lack of user guidance strategies exacerbates uneven resource allocation. Insufficient data-driven approaches: Traditional models rely on historical data, making it difficult to cover the generalization of strategies under extreme scenarios and combinations of random variables, such as surges in charging demand and competition among multiple vehicles, resulting in poor generalization of reinforcement learning models. Multi-objective optimization conflicts: The trade-off between charging efficiency, time cost, and safety lacks mathematical theoretical support, easily leading to local optima.

[0003] Public patent CN112987711A proposes a charging scheduling method based on Q-learning, but it fails to address the issues of virtual environment data generation and multimodal scene coverage. Therefore, how to construct a high-fidelity virtual environment using a simulation engine to cover long-tailed distributed random scene data, and to simulate more realistic scenarios for autonomous charging of mobile charging vehicles, has become an urgent problem to be solved. This invention constructs a high-fidelity virtual environment using the AnyLogic simulation engine, combines it with the Markov chain Monte Carlo method to generate random scene data covering a long-tailed distribution, and designs a multi-objective deep deterministic policy gradient algorithm to achieve end-to-end optimization. Summary of the Invention

[0004] This invention proposes a method and system for optimizing autonomous charging of mobile charging and storage vehicles based on emergent data from virtual environments and reinforcement learning, specifically including the following steps: Virtual environment modeling: Based on AnyLogic, a 3D dynamic scene of a parking lot is constructed, and the intelligent agent behavior rules of mobile charging vehicles, charging piles, obstacles, and dynamic vehicles are defined.

[0005] Monte Carlo scene generation: The MCMC method is used to randomly sample parameters such as parking space occupancy, charging demand intensity, and vehicle movement patterns to generate a multimodal dataset containing normal distributions and extreme events.

[0006] Reinforcement learning training: Design a state-action space and reward function, train a decision model using the MO-DDPG algorithm, and output the optimal charging strategy.

[0007] The detailed technical solution of this invention is as follows: S1. Define the scenario elements including: MCSV agents, covering their battery capacity. Smart charging station agents, covering their location Occupancy mark Charging demand intensity Users' urgent charging needs follow a Gaussian distribution. .

[0008] S2. Satisfies the physical interaction rules: Charging station occupancy status update rules: If at coordinates With the center of the circle, If a vehicle exists within a circular area of ​​radius [radius], then the charging / storage vehicle... Constant Occupied State It is set to 1, and conversely, it is set to 0. Among these, The effective charging radius for charging vehicles.

[0009] S3. Monte Carlo Data Generation Define the global state vector Includes the following key parameters: .

[0010] in, , represents the parking space occupancy matrix (N is the total number of parking spaces). Indicates the first One parking space was occupied; , representing the charging demand intensity vector, ; , represents the motion state vector of a dynamic obstacle (vehicle, pedestrian), including position, velocity, and heading angle.

[0011] Built-in periodic electricity price meter , The probability of a user's charging demand follows a Bernoulli distribution. , The total parking time of the vehicles follows a Weibull distribution. When parking space When occupied, activate the user's charging demand. Arrival time (Evenly coverage throughout the day). Initial battery level. .

[0012] S4. Markov Chain State Transition Process: The Metropolis-Hastings algorithm is used to generate the parking space occupancy state sequence. .Every Generate candidate states Randomly flip the occupancy sign of a parking space.

[0013] Define candidate states The probability of acceptance is: ; ; Among the proposal distribution To ensure uniform distribution, Target distribution Defined as: ; This represents the historical average occupancy rate. Control the frequency of sudden scene changes.

[0014] The above formula can be simplified to: ; Generate random numbers ,like Then accept the candidate state. Otherwise, maintain the original state. .

[0015] To simulate rare but high-risk events in real-world scenarios, the following perturbation strategy is designed to inject extreme events. (This relates to the intensity of charging demand.) Introduce heavy-tailed distribution perturbation: ; ; Extreme events include surges in charging demand, equipment malfunctions, and sudden behavior from dynamic obstacles.

[0016] S5. Reinforcement Learning Model (MO-DDPG Layer) Define state space Action space .in, Δθ represents the speed increment, and Δθ represents the steering angle increment.

[0017] Design a multi-objective reward function: ; ; in, The items are, in order: energy efficiency bonus; time penalty; safety penalty; demand coverage bonus; parking time priority bonus; peak-valley electricity price economic penalty. The unit impulse function, The distance to the nearest obstacle. Let be the remaining parking time for the i-th vehicle. Total parking time. For real-time electricity prices, For charging power, This represents the charge level.

[0018] Meet the constraints: power constraint, the final power must meet the following requirements. Temperature rise constraint: battery temperature An empirical model is used.

[0019] The algorithm is designed using the Multi-Objective Deep Deterministic Policy Gradient (MO-DDPG) algorithm and includes the following core components: a dual Actor-Critic structure that separates the policy network and the objective network, and adds a dual Critic network to alleviate the Q-value overestimation problem; and a dynamic weight adjustment module that optimizes the reward function weights in real time through Pareto front analysis.

[0020] Construct the input, hidden, and output layers of the Actor and Critic networks respectively, and select the optimizer and activation function.

[0021] Target network parameters Synchronize via soft update For soft update coefficients: ; A dual-Critic network structure is employed to mitigate the Q-value overestimation problem. The mean squared error (MSE) is defined as the loss function to measure the Q-value estimated by the Critic network. The difference between the target Q value y and the target Q value y.

[0022] The objective Q-value is determined by the reward the agent receives immediately after taking action a in the current state s and the reward from the two Critic networks for the next state. The following is from the policy network Generated actions The minimum value is determined in the Q-value estimation.

[0023] Actor network updates gradients: ; Charging strategy decision tree generation, The system is divided into intervals, each corresponding to one of the following strategies: Fast charging strategy: short-term high-power charging, triggering a premium; Balanced strategy: adjusting power in stages to balance battery life and speed; Slow charging strategy: low-power charging, adapting to off-peak electricity prices and providing discounts.

[0024] Virtual-to-real migration and online adaptation: During virtual training, environmental parameters such as illumination noise and sensor errors are dynamically adjusted to improve model robustness; after deployment to a real MCSV, reward weights are dynamically adjusted through Bayesian optimization, with the objective function being: ; Experience replay and priority sampling. Define the capacity of the replay buffer as D, and adopt priority experience replay. According to the TD error Allocate sampling probabilities .

[0025] Exploration strategy design. For noise injection, the OU process noise is adopted to enhance exploration in the initial stage; the noise standard deviation decays exponentially in the later stage. Train in stages, and gradually increase the environmental complexity. Stage 1: Static obstacles, no dynamic vehicles; Stage 2: Add dynamic vehicles; Stage 3: Inject extreme events, such as faulty charging piles and sudden surges in demand.

[0026] Define the training termination conditions: The average reward volatility in consecutive m rounds < a%; If there is no significant improvement (reward increase < b%) within n rounds, terminate the training.

[0027] The present invention also proposes an autonomous charging optimization system for a mobile charging and storage vehicle to implement the above method, including: A user terminal module, used to collect the estimated parking duration, target power, vehicle battery capacity, and current power input by the user; An environment perception module, used to obtain grid load data, time-of-use electricity price information, and charging pile occupancy status in real time; A Monte Carlo data engine module, which generates a simulation data set based on the probability distribution model of the parking duration. The data set includes random combinations of parking duration, user arrival time, initial power, and charging demand parameters; A policy optimization model module, which trains the simulation data set through a deep learning algorithm and outputs an optimal charging power curve matching the parking duration interval; A dynamic pricing controller module, which generates differentiated charging prices according to the charging strategy, grid load, and user selection. Description of the Drawings

[0028] Figure 1 is the system architecture and operation principle diagram of the present invention; Figure 2 is a schematic diagram of the parking lot model built by AnyLogic; Figure 3 is the architecture diagram of the Actor and Critic networks; Figure 4 is the fluctuation diagram of each reward function; Figure 5 is the schematic diagram of the pseudocode principle of the reinforcement learning process. Detailed Implementation Modes

[0029] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings: In one specific embodiment, the specific implementation process of the present invention is as follows: Figure 1 As shown.

[0031] S1. Scene Modeling and Agent Definition: In one specific embodiment, a parking lot CAD model is imported using AnyLogic 8.7.2. The model is in DWG format, with dimensions of 50m × 30m. Four parking and charging stations are set up, along with fixed obstacles (such as pillars and walls) and dynamic obstacles (pedestrians and vehicles). The dynamic vehicle flow follows a Poisson process. ),like Figure 2 As shown.

[0032] Furthermore, the battery model for the charging and storage vehicle is set with a capacity of Power during charging Discharge power .

[0033] In one specific embodiment, dynamic vehicle and pedestrian behavior rules are defined. Vehicle generation follows a Poisson distribution. .

[0034] In one specific embodiment, the time-of-use electricity price table of the power grid is obtained. and duration coefficient : ; With parking duration Negative correlation is defined as: ; The parameter is set to k=0.5. This ensures a smooth transition of short-term charging premiums.

[0035] In one specific embodiment, the parking duration follows a Weibull distribution. To simulate a long-tailed distribution; to satisfy the initial charge .

[0036] In one specific embodiment, the working radius of the charging and storage vehicle is detected every 0.1 seconds. Is there a vehicle inside? Update the occupancy sign. Charging demand intensity Decays over time: .

[0037] S2. Monte Carlo Data Generation (MCMC Layer) In one specific embodiment, scene sampling is performed based on Metropolis-Hastings. Initial state This is a matrix of all zeros (all parking spaces are vacant). Candidate states are generated every 10 seconds. Randomly flip the occupancy sign of a parking space.

[0038] ; in, Indicates the first Invert the state of each parking space ( .

[0039] Calculate the target distribution and acceptance probability: ; ; in, Historical average occupancy rate (initial value) ), Control the frequency of sudden scene changes.

[0040] If random number Then accept This is the next state.

[0041] In one specific embodiment, a multimodal dataset is generated. 10,000 sets of scene data are generated, with 80% of them being normal scenes, and parameters... Peak-hour charging demand increases 2.6 times, accounting for 15% of the total demand. =0.5, =8; Equipment failure rate 5%, randomly mark 2 charging and storage vehicles as permanent failures. Save each frame of data as a JSON file, including fields such as timestamp, MCSV status, parking space status, and charging demand intensity.

[0042] S3. Reinforcement Learning Training (MO-DDPG Layer) In one specific embodiment, a multi-objective reward function is designed to update the state space S and the action space A: ; The items are, in order: energy efficiency reward; time penalty; safety penalty; demand coverage reward; parking time priority reward; peak-valley electricity price economic penalty. The unit impulse function, This is the distance to the nearest obstacle. This is the charging efficiency coefficient. This is the time penalty coefficient. For safety factor, weighting factor After dynamic adjustment using Pareto front optimization, the initial values ​​were set to 0.79, 0.68, 0.94, and 0.86, respectively.

[0043] In one specific embodiment, the power constraint satisfies , For charging efficiency; temperature rise constraints adopt an empirical model: , For ambient temperature, This represents the battery's thermal characteristic coefficient.

[0044] In one specific embodiment, such as Figure 3 As shown, the Actor network uses a Transformer architecture. The input state is encoded into a sequence and fed into multiple Transformer blocks. Within each block, a multi-head attention mechanism extracts state features in parallel from different subspaces. A feedforward neural network then performs a non-linear transformation, and finally, the output action is passed through a fully connected layer. The Critic network fuses a graph neural network (GNN) with a generative adversarial network to output a Q-value.

[0045] In one specific embodiment, the relevant hyperparameter is set as: discount factor Playback buffer size ; Actor learning rate Actor learning rate Explore noise Soft update parameters .

[0046] In one specific embodiment, the training process is designed as follows: Randomly initialize the Actor and Critic network parameters, and clear the replay buffer; each training episode contains a maximum of Status is collected every 0.1 seconds. Execute actions Calculate rewards ,storage To the buffer.

[0047] 256 data sets were sampled in each batch, and the target Q value was calculated. ; Update the Critic network parameters to minimize the mean squared error: ; Update the Actor network parameters to maximize the Q-value: ; In one specific embodiment, the convergence condition is defined as the average reward fluctuation being less than 1% over 100 consecutive rounds.

[0048] In one specific embodiment, the fluctuation graphs of the various reward functions are as follows: Figure 4 As shown, its average reward fluctuation over 100 consecutive rounds is less than 1%, which satisfies the convergence condition.

[0049] In one specific embodiment, the implementation of the entire algorithm process described above is as follows: Figure 5 As shown.

[0050] In one specific embodiment, a charging strategy decision tree is generated.

[0051] Fast charging strategy : .

[0052] Equilibrium strategy ( Charging in two stages: ; Slow charging strategy ( Charging during off-peak electricity pricing periods: ; In one specific embodiment, the Logit model for user selection strategy is as follows: ; For the calibrated user preference parameters, take .

[0053] S4. Hardware Deployment and Real-Time Control In one specific implementation, the trained model is converted to ONNX format using PyTorch's ONNX export tool. Quantization (FP16 precision) is then performed using TensorRT 8.2 to generate a lightweight engine file.

[0054] In one specific embodiment, the NVIDIA Jetson AGX Xavier (CUDA 11.4, TensorRT8.2) is used as the main control unit; the Velodyne VLP-16 with a 10Hz scanning frequency is used as the LiDAR; the U-blox NEO-M8N with a positioning accuracy of ±0.1 m ±0.1 m is used as the GPS module; and ROS2 Galactic is used as the communication protocol.

[0055] In one specific embodiment, initial weights are set. The NSGA-II algorithm is used to find the Pareto optimal solution set, and non-dominated solutions are filtered out. After every 10 tasks, Bayesian optimization (GPyOpt library) is used to search for the optimal weights. The objective function is: ; The constraints are .

[0056] In one specific embodiment, the allocation sampling probability is calculated: ; In one specific embodiment, OU process noise is used in the initial stage. Enhanced exploration. Later attenuation, with the noise standard deviation decreasing exponentially: ; In one specific embodiment, the experimental results of the present invention are as follows: ; In another specific embodiment, cross-domain generalization testing is performed. An untrained parking lot layout is imported, wherein the parking lot has an L-shaped lane and 12 parking spaces; the dynamic obstacle density is increased to 0.2 obstacles / m². 2 The results showed that its charging success rate was 95.2%.

[0057] In another specific embodiment, during peak parking periods, the remaining parking time of the vehicle is considered. Dynamically adjust charging priorities. For vehicles with short parking times, prioritize charging to ensure they complete their charging task quickly; for vehicles with longer parking times, appropriately postpone charging to balance charging demand and grid load.

Claims

1. A method for optimizing autonomous charging of mobile charging vehicles based on emergent data from virtual environments and reinforcement learning, characterized in that, Includes the following steps: S1. Construct a dynamic parking lot virtual environment using AnyLogic simulation software, and define the intelligent agent models and interaction rules of mobile charging and storage vehicles (MCSV), charging piles, obstacles, and dynamic vehicles; S2. A multimodal scenario dataset is generated using the Markov Chain Monte Carlo (MCMC) method, including parking space occupancy state transition sequences and extreme event injections of charging demand intensity; S3. Based on the Multi-Objective Deep Deterministic Policy Gradient (MO-DDPG) algorithm, design a state space, action space, and a reward function that integrates energy efficiency, time cost, safety, demand coverage, parking time priority, and peak-valley electricity price economy to train an autonomous charging decision model.

2. The method according to claim 1, characterized in that, The agent model in step (a)S1 includes: MCSV battery capacity The parameters of the charging station include the occupancy status. Charging demand intensity Users' urgent charging needs follow a Gaussian distribution. Parking duration It follows a Weibull distribution; the initial charge follows a truncated normal distribution.

3. The method according to claim 1, characterized in that, The MCMC method in step (b) S2 is specifically as follows: The Metropolis-Hastings algorithm is used to generate parking space occupancy state sequences, and the target distribution and acceptance probability are calculated to update candidate parameters; charging demand intensity is also considered. Pareto distribution perturbations are injected to simulate rare but high-risk events in real-world scenarios.

4. The method according to claim 1, characterized in that, The reward function in step (c)S3 is: ； in, Dynamic optimization through Pareto front analysis; As a reward for energy efficiency, As a time penalty, For safety reasons, To cover the rewards for demand, Parking time is given priority as a reward. This is an economic penalty for peak-valley electricity pricing.

5. The system according to claim 1, characterized in that, The charging strategies are divided into three categories based on parking duration: fast charging strategy, balanced charging strategy, and slow charging strategy.

6. The method according to claim 1, characterized in that, Step S3 also includes: adding lighting noise and sensor errors to the virtual environment, and adjusting the reward weights through Bayesian optimization during the online fine-tuning phase.

7. A mobile charging and storage vehicle autonomous charging optimization system for implementing the method of any one of claims 1-6, characterized in that, include: The method according to claim 1 is characterized in that the method comprises the following modules: a user terminal module, used to collect the user's input of the expected parking time, target battery level, vehicle battery capacity, and current battery level; The environmental sensing module is used to acquire real-time grid load data, time-of-use electricity price information, and charging pile occupancy status. The Monte Carlo data engine module generates a simulated dataset based on a probability distribution model of parking duration. The dataset includes a random combination of parking duration, user arrival time, initial battery level, and charging demand parameters. The strategy optimization model module trains the simulated dataset using a deep learning algorithm and outputs the optimal charging power curve that matches the parking time interval. The dynamic pricing controller module generates differentiated charging prices based on charging strategies, grid load, and user selections.

Images