Mobile charging station behavior decision-making method based on deep reinforcement learning

Abstract

The invention provides a mobile charging station (MCS) service electric vehicle behavior decision-making method based on deep reinforcement learning. The behavior of a mobile charging station is subjected to decision-making optimization through a deep reinforcement learning model. The method can dynamically adjust the charging request response, the self-charging strategy, the moving path and other behaviors of the MCS so as to maximize the service efficiency and minimize the operation cost, and especially when the battery electric quantity is insufficient, the self-charging and moving strategy of the MCS is optimized, and high-cost long-distance movement is reduced.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent charging scheduling and artificial intelligence technology, and specifically relates to a mobile charging station (MCS) behavior decision-making system and method based on deep reinforcement learning (DRL). The system is used to optimize the path planning, charging scheduling, and energy management of mobile charging stations in a dynamic environment, thereby achieving coordinated optimization of service efficiency, operating costs, and power safety. Background Art

[0002] With the increasing popularity of electric vehicles, fixed charging stations (FCS) have become a vital addition to urban energy replenishment systems. However, the development of EV charging infrastructure has far outstripped the growth in the number of EVs. Fixed charging stations are expensive to build and operate, and can lead to queues during peak hours. Therefore, mobile charging stations (MCS), as a flexible charging solution, can complement fixed charging stations and provide more efficient and intelligent charging services.

[0003] Currently, the scheduling problem of MCS involves multiple challenges. Traditional scheduling methods (such as rule engines and heuristic algorithms) cannot adapt to dynamic demands and environmental changes (such as real-time electricity price fluctuations and traffic congestion). Existing solutions make it difficult to balance user service efficiency (response time), operating costs (mobile energy consumption) and the charging needs of the MCS itself. Electricity prices fluctuate between peaks and valleys. Reasonable scheduling needs to consider charging during valley electricity prices and providing services during peak hours to optimize operating costs. The MCS itself requires battery power. While providing charging services, it also needs to balance its own power to avoid running out of power and affecting operations. The MCS needs to cruise or stop in different areas to provide services. Path planning needs to take into account road congestion to improve scheduling efficiency.

[0004] To overcome these issues, the present invention uses Deep Reinforcement Learning (DRL) to enable the MCS to autonomously learn optimal scheduling strategies in a dynamic environment. The PPO algorithm, a leading method in the field of reinforcement learning, can improve the convergence speed of scheduling decisions while ensuring strategy stability. By training the MCS's scheduling strategy through DRL, the present invention enables it to adapt to the distribution of charging demand in different urban environments, improve operational efficiency, and reduce overall energy consumption. Summary of the Invention

[0005] This paper provides a deep reinforcement learning-based intelligent scheduling method for mobile charging stations serving electric vehicles (MCS). This method aims to address the poor adaptability of traditional MCS scheduling methods in dynamic environments, high computational complexity, and unstable operating costs. By employing the Proximal Policy Optimization (PPO) algorithm, the MCS can autonomously learn optimal scheduling strategies in a constantly changing environment, such as charging demand, road traffic, and grid prices, to improve charging efficiency and reduce operating costs. Applicable to urban environments of varying sizes, the system can dynamically adjust operating strategies based on changes in EV charging demand, grid load conditions, and urban traffic conditions, optimizing charging station layout and improving the overall efficiency of the charging network.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows: A method for making decisions on the behavior of mobile charging stations serving electric vehicles based on deep reinforcement learning, comprising the following steps:

[0007] Step 1,construct the state space model and operating environment of the mobile charging station.,The state space includes the remaining power of the MCS, charging orders,,the location of the MCS, real-time electricity price, weather, and,the action space includes selecting the target order and planning the,route, selecting the charging pile and charging amount, and selecting the,regional cruising;

[0008] Step 2: Design an adaptive reward function and train the model based on the operating costs of the mobile charging station, which include battery consumption, self-charging costs, and long-distance travel costs.

[0009] Step 3: Use the PPO deep reinforcement learning algorithm to train the mobile charging station's behavioral decision-making strategy in different environments to optimize the mobile charging station's charging request response, self-charging, and mobile path selection;

[0010] Step 4: Adapt to different urban environments through a strategy that combines transfer learning with online learning.

[0011] As an improvement of the present invention, the state space model is as follows:

[0012] Environment definition: Divide the city into N locations and discretize time into Time∈{1,2,...,T}

[0013] State space S t :

[0014] S t =[Soc,Orders,Location,Gird_price,Weather,Time]

[0015] Among them, Soc is the remaining power of MCS, Orders is the order queue, Location is the region list, Gird_price is the real-time electricity price, Weather is the weather, Time is the time,

[0016] Action space A t :

[0017]

[0018] Among them, Respond_orders selects the target order and plans the route, Self_charge selects the charging pile and charging amount, and Cruise selects the regional cruise.

[0019] Reward function R t design:

[0020] R t =α×R service -β×R Moving -γ×R charge +δ×R safety

[0021] where R service is the service income of the mobile charging station, R Moving is the moving cost, R charge is the charging cost, R safety is the safety reward, α, β, γ, δ are R service , R Moving , R charge , R safety The weight coefficient of .

[0022] As an improvement to the present invention, the reward function design should follow the following rules:

[0023] Positive rewards are given for successful EV charging, negative rewards are given for MCS accepting orders even when the battery level is too low, negative rewards are given for MCS replenishing energy during peak electricity prices, positive rewards are given for replenishing energy during off-peak electricity prices, and negative rewards are given for long-distance cruising to reduce energy consumption.

[0024] As an improvement of the present invention, the PPO algorithm learns the optimal scheduling strategy through policy optimization and dynamically adjusts the policy parameters based on historical data. The PPO algorithm uses policy gradient optimization to calculate the policy loss function by sampling the behavior data of MCS in different states:

[0025] L(θ)=E(t)[min(r t (θ)A t ,clip(r t (θ),1-∈,1+ε)A t )]

[0026] Among them, r t (θ) is the ratio of the new and old strategies, A t is the advantage function, ε is the clipping parameter of PPO, and the clipping function clip(r t (θ),1-∈,1+ε) constrains the strategy update amplitude to prevent drastic fluctuations;

[0027] In order to realize the efficient training of the mobile charging station (MCS) scheduling strategy, the present invention adopts a technical solution combining the experience replay mechanism (ReplayBuffer) with the adaptive learning rate adjustment. The experience replay mechanism is specifically implemented as follows: a priority experience replay buffer is constructed to store the tuple data (S t ,A t ,R t ,S t+1 ), assigns priority to each experience based on the time difference error (TD Error), adopts a proportional priority sampling strategy, prioritizes samples that contribute significantly to the policy update, improves training efficiency, regularly updates the buffer priority, and breaks data correlation through random batch sampling (Batch Sampling) to reduce policy overfitting. The adaptive learning rate adjustment is specifically implemented as follows: dynamically monitor the average reward change rate (ΔR) during training, design a learning rate adjustment rule, when ΔR continues to rise, increase the learning rate (such as multiplying by 1.05) to accelerate policy convergence, when ΔR fluctuates or decreases, reduce the learning rate (such as multiplying by 0.95) to avoid policy oscillation, enhance exploration ability through entropy regularization, introduce the entropy regularization term in the policy loss function of the PPO algorithm, and the calculation formula is:

[0028] L(total)=L clip (θ)-λ entropy ·H(π(·|S t ))

[0029] Where L(total) is the total loss function, H(π(·|S t )) represents the policy entropy, λ entropy The entropy coefficient of the dynamic decay, L clip (θ) represents the clipping policy gradient loss function.

[0030] As an improvement of the present invention, the strategy of combining transfer learning with online learning is as follows: Initialize the transfer learning, and set the reinforcement learning model parameters θ trained in city A to A As the initial parameters of city B The network weights of the state encoding layer (common features such as battery level, order queue, and electricity price) are retained, and only the output layer (action decision) is fine-tuned. The model is trained for a small number of iterations (1,000 steps) using historical data from City B (traffic flow and charging demand distribution) to adjust the feature mapping relationship. The formula is:

[0031]

[0032] in, is the gradient of the objective function of city B, is the transfer learning rate,

[0033] The specific implementation of online learning is as follows: real-time data collection and fusion, real-time acquisition of MCS location, remaining power, charging orders, real-time electricity prices, traffic congestion index and weather, synchronous update every 5 minutes and generate a multi-dimensional state vector; using a dual mode of event-driven (new order receipt, electricity price fluctuation exceeding 10%, power consumption less than 20%) and cycle-driven (every 5 minutes), activate the policy network update, combined with the PPO algorithm to ensure that the policy θ t The smooth transition from exploration to exploitation is θ t+1 The formula is:

[0034]

[0035] Among them, ω is the weight vector, is the gradient of the objective function.

[0036] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the mobile charging station behavior decision method based on deep reinforcement learning is implemented.

[0037] A computer-readable storage medium stores computer instructions, which, when executed by a processor, implement the mobile charging station behavior decision-making method based on deep reinforcement learning.

[0038] Compared with the prior art, the advantages of the present invention are as follows:

[0039] 1. Dynamic Environment Adaptability: By co-optimizing a multi-dimensional state space (real-time electricity prices, weather, and time) with the PPO algorithm, the Mobile Charging Station (MCS) can respond in real time to dynamic charging demands and grid fluctuations. Experiments have shown that this can reduce operating costs by avoiding high electricity prices during peak hours.

[0040] 2. Training efficiency and stability: The experience replay mechanism (Replay Buffer) and adaptive learning rate adjustment are used to improve the model convergence speed. The PPO pruning strategy is combined to prevent sudden changes in policy gradients and improve training stability.

[0041] 3. Exploration and Exploitation Balance: Introducing the Entropy Regularization Term H(π(·|S t )), in the initial stage, diversified action exploration is encouraged (such as cruising in unknown areas), and in the later stage, high-yield actions are gradually favored (such as charging at low electricity prices) to improve service coverage.

[0042] 4. Cross-city migration and online learning: Based on transfer learning, the pre-trained model parameters of city A are migrated to city B. Combined with the online learning mechanism, the real-time update strategy improves the service efficiency of the model deployed in the new city. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Diagram of the reinforcement learning framework for mobile charging stations. DETAILED DESCRIPTION

[0044] In order to deepen the knowledge and understanding of the present invention, the solution is described in detail below with reference to the accompanying drawings and embodiments.

[0045] Embodiment: The present invention provides a method for intelligent decision-making of mobile charging station service electric vehicles (MCS) based on deep reinforcement learning, comprising the following steps:

[0046] Step 1: Build a state space model and set up a reinforcement learning environment;

[0047] Step 2: Design the reward function and train the reinforcement learning model;

[0048] Step 3: Long-term strategy optimization based on the PPO algorithm;

[0049] Step 4: Dynamic adaptation and real-time decision-making across urban environments.

[0050] In the above-mentioned intelligent decision-making method for mobile charging stations serving electric vehicles based on deep reinforcement learning, step 1 is as follows.

[0051] A reinforcement learning environment, built using the Gymnasium framework, was used to model the dispatching behavior of the MCS in various urban environments. This environment primarily consists of the following: State space design: Multidimensional state variables are used to describe the MCS's operating environment, including the MCS's current power level, order quantity and location, grid price levels, EV charging demand distribution, and time information. Action space definition: At each time step, the MCS can perform three basic actions: responding to charging requests, self-charging, and navigating to high-demand areas.

[0052] In the aforementioned intelligent decision-making method for mobile charging stations serving electric vehicles based on deep reinforcement learning, step 2 is specifically as follows.

[0053] The reward function design provides positive rewards for successful charging, negative rewards for charging during peak electricity prices, positive rewards for charging during off-peak electricity prices, penalties for accepting orders with low battery levels, and negative rewards for exceeding long cruising distances. Operating costs are also considered, including battery consumption, self-charging costs, and long-distance travel costs. Model training process: Initialize the policy network (Actor) and the value network (Critic). The policy network outputs action probability distributions, while the value network evaluates state values. Training data is sampled and the simulation environment is run for 100,000 steps, with a priority sampling batch size of 256. Finally, optimization is performed using the Adam optimizer.

[0054] In the aforementioned intelligent decision-making method for mobile charging stations serving electric vehicles based on deep reinforcement learning, step 3 is specifically as follows.

[0055] The PPO algorithm is used to train the intelligent scheduling strategy of MCS so that it can dynamically adapt to the charging needs in different urban environments. The training process includes the following steps: Initialize the policy network and the value network: the policy network is responsible for deciding the optimal action of MCS in different states, and the value network evaluates the expected return of the current strategy. Sampling training data: running the MCS simulation environment, collecting decisions under different states and their corresponding reward values. Calculating the loss function: using the clipping policy gradient loss function for optimization to ensure the stability of the policy update. First, the policy updates the target, by optimizing the policy π θ To maximize the expected reward:

[0056]

[0057] Where J(θ) is τ, which represents the trajectory, γ is the discount factor, and R t is the reward at the time step. This formula indicates that the goal of PPO training is to maximize the cumulative reward, which is used to measure the impact of future rewards on current decisions.

[0058]

[0059] Importance sampling ratio r t (θ) is the probability ratio of the new and old strategies to choose the same action in the same state, which is used to adjust the strategy update amplitude.

[0060] L(θ)=E(t)[min(r t (θ)A t ,clip(r t (θ),1-∈,1+ε)A t )]

[0061] Among them, r t (θ) is the ratio of the new and old strategies, A tis the advantage function, ε is the clipping parameter of PPO, and the loss function constrains the strategy update amplitude through the clipping function to avoid training instability caused by large changes in strategy.

[0062] To further improve the generalization ability of the model, this paper introduces the experience replay (Replay Buffer) optimization strategy. Experience replay is used to store past state-action-reward data and extract samples from it for training to improve sample utilization. Its update method is as follows:

[0063] D←D∪{(S t ,A t ,R t ,S t+1 )}

[0064] In the aforementioned intelligent decision-making method for mobile charging station services for electric vehicles based on deep reinforcement learning, step 4 is as follows.

[0065] In order to make the proposed scheduling method adaptable to different urban environments, the present invention adopts a strategy combining transfer learning and online learning. First, the reinforcement learning model parameters θ trained in a data-rich city (e.g., city A) are A It can be used as the initial model parameters for other cities (such as city B), that is, the initial parameter formula for city B is:

[0066]

[0067] Among them, J B (θ) represents the objective function in the city B environment, α is the learning rate, and the gradient This reflects the guidance of city B's specific data on strategy improvement. In addition, the present invention combines the online learning mechanism to enable the MCS to dynamically adapt to the ever-changing environment during deployment. The online learning strategy is updated as follows:

[0068]

[0069] This update rule enables the model to be continuously optimized based on newly observed data, ensuring that MCS can maintain efficient scheduling in different urban environments.

[0070] Through the organic combination of the above two mechanisms, the dispatching system of the present invention can not only leverage the data advantages of existing cities to accelerate model convergence in new urban environments, but can also dynamically adjust according to real-time data after deployment, thereby ensuring rapid adaptation and efficient operation in different urban environments.

Claims

1. A behavior decision-making method for mobile charging station services for electric vehicles based on deep reinforcement learning, characterized in that: The steps include: Step 1,construct the state space model and operating environment of the mobile charging station.,The state space includes the remaining power of the MCS, charging orders,,the location of the MCS, real-time electricity price, weather, and,the action space includes selecting the target order and planning the,route, selecting the charging pile and charging amount, and selecting the,regional cruising; Step 2: Design an adaptive reward function and train the model based on the operating costs of the mobile charging station, which include battery consumption, self-charging costs, and long-distance travel costs. Step 3: Use the PPO deep reinforcement learning algorithm to train the mobile charging station's behavioral decision-making strategy in different environments to optimize the mobile charging station's charging request response, self-charging, and mobile path selection; Step 4: Adapt to different urban environments through a strategy that combines transfer learning with online learning.

2. The mobile charging station behavior decision-making method based on deep reinforcement learning according to claim 1 is characterized in that: The state space model is as follows: Environment definition: Divide the city into N locations and discretize time into a state space S of Time∈{1,2,...,T} t : S t =[Soc,Orders,Location,Gird_price,Weather,Time] Among them, Soc is the remaining power of MCS, Orders is the order queue, Location is the region list, Gird_price is the real-time electricity price, Weather is the weather, Time is the time, Action space A t : Among them, Respond_orders selects the target order and plans the route, Self_charge selects the charging pile and charging amount, and Cruise selects the regional cruise. Reward function R t design: R t =α×R service -β×R Moving -γ×R charge +δ×R safety where R service is the service income of the mobile charging station, R Moving is the moving cost, R charge is the charging cost, R safety is the safety reward, α, β, γ, δ are R service , R Moving , R charge , R safety The weight coefficient of .

3. The mobile charging station behavior decision-making method based on deep reinforcement learning according to claim 1 is characterized in that: The reward function design should follow the following rules: Positive rewards are given for successful EV charging, negative rewards are given for MCS accepting orders even when the battery level is too low, negative rewards are given for MCS replenishing energy during peak electricity prices, positive rewards are given for replenishing energy during off-peak electricity prices, and negative rewards are given for long-distance cruising to reduce energy consumption.

4. The mobile charging station behavior decision-making method based on deep reinforcement learning according to claim 1, characterized in that: The PPO algorithm learns the optimal scheduling strategy through policy optimization and dynamically adjusts the policy parameters based on historical data. The PPO algorithm uses policy gradient optimization to calculate the policy loss function by sampling the behavior data of MCS in different states: L(θ)=E(t)[min(r t (i)A t ,clip(r t (θ),1-∈,1+ε)A t )] Among them, r t (θ) is the ratio of the new and old strategies, A t is the advantage function, ε is the clipping parameter of PPO, and the clipping function clip(r t (θ),1-∈,1+ε) constrains the strategy update amplitude to prevent drastic fluctuations; The technical solution of combining the experience replay mechanism (Replay Buffer) with the adaptive learning rate adjustment is adopted. The experience replay mechanism is specifically implemented as follows: a priority experience replay buffer is constructed to store the tuple data (S t ,A t ,R t ,S t+1 ), assigns priority to each experience based on the temporal difference error (TD Error), adopts a proportional priority sampling strategy, prioritizes samples that contribute significantly to policy updates, improves training efficiency, regularly updates the buffer priority, and breaks data correlation through random batch sampling (Batch Sampling) to reduce policy overfitting. The adaptive learning rate adjustment is specifically implemented as follows: dynamically monitor the average reward change rate (ΔR) during training, design a learning rate adjustment rule, increase the learning rate when ΔR continues to rise, accelerate policy convergence, and reduce the learning rate when ΔR fluctuates or decreases to avoid policy oscillation. Enhance exploration ability through entropy regularization, introduce an entropy regularization term in the policy loss function of the PPO algorithm, and the calculation formula is: L(total)=L clip (i)-l entropy ·H(π(·|S t )) Where L(total) is the total loss function, H(π(·|S t )) represents the policy entropy, λ entropy The entropy coefficient of the dynamic decay, L clip (θ) represents the clipping policy gradient loss function.

5. The mobile charging station behavior decision-making method based on deep reinforcement learning according to claim 1, characterized in that: The strategy of combining transfer learning with online learning is as follows: Initialize transfer learning and set the reinforcement learning model parameters θ trained in city A to A As the initial parameters of city B The network weights of the state encoding layer (common features such as battery level, order queue, and electricity price) are retained, and only the output layer (action decision) is fine-tuned. The model is trained on a small number of iterations using the historical data of city B (traffic flow, charging demand distribution) to adjust the feature mapping relationship. The formula is: in, is the gradient of the objective function of city B, is the transfer learning rate, The specific implementation of online learning is as follows: real-time data collection and fusion, real-time acquisition of MCS location, remaining power, charging orders, real-time electricity prices, traffic congestion index and weather, synchronous update every 5 minutes and generate multi-dimensional state vector; using the dual mode of event-driven and cycle-driven, activating the policy network update, combined with the PPO algorithm to ensure the policy θ t The smooth transition from exploration to exploitation is θ t+1 The formula is: Among them, ω is the weight vector, is the gradient of the objective function.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the mobile charging station behavior decision method based on deep reinforcement learning as described in any one of claims 1 to 5 is implemented.

7. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the mobile charging station behavior decision method based on deep reinforcement learning as described in any one of claims 1 to 5 is implemented.

Images