Intelligent electric vehicle platoon queue optimization method based on congestion prediction and DRL

Abstract

The application discloses a congestion prediction and DRL-based intelligent electric vehicle fleet queue optimization method, and steps include: (1) modeling the queue position optimization problem of the electric vehicle fleet as a mathematical model with energy balance as the target and subject to traffic environment constraints; (2) solving the mathematical model in step (1) by using a DRL method based on a TRPO algorithm to obtain an optimal queue adjustment strategy of the electric vehicle fleet. In step (2), congestion state information in the external traffic environment is obtained by using an LSTM traffic prediction and an FCM method, and the remaining electric quantity and the cumulative driving distance of each vehicle in the vehicle fleet are used as state inputs of the TRPO algorithm; the TRPO algorithm uses a strategy network and a value network to respectively obtain a strategy for adjusting the queue position and evaluate the expected return under the current state. The TRPO algorithm dynamically adjusts the strategy update step length by constraining the KL divergence of the strategy update, so that the stability and convergence in the strategy update process are ensured.

Description

Technical Field

[0001] This invention relates to the application of computer artificial intelligence technology in intelligent fleet management, specifically to an intelligent electric vehicle fleet queue optimization method that combines traffic congestion prediction with DRL methods (such as the Trust Region Policy Optimization TRPO algorithm). Background Technology

[0002] In recent years, electric vehicle fleets, as an important component of vehicle-road cooperation and autonomous driving, have been increasingly widely used in intelligent transportation systems, effectively improving road traffic efficiency and safety. Intelligent fleets typically consist of multiple vehicles connected via wireless communication technology, collaboratively achieving synchronized acceleration and deceleration control. This allows the fleet to travel in close formation within the same lane with a spacing smaller than the conventional safety interval. Thanks to the "follow-the-car effect" of this close-spaced convoy driving, intelligent fleets can effectively reduce air resistance and fuel or energy loss. Numerous wind tunnel experiments, CFD simulations, and real-vehicle tests have shown that intelligent fleet driving can reduce energy consumption by an average of more than 10%, offering greater energy-saving and environmental advantages compared to solo driving. Aerodynamic studies further reveal that vehicles in different positions within a fleet experience significantly different air resistance forces due to their location. Vehicles at the front of the convoy have to overcome more oncoming airflow, resulting in significantly higher air resistance and energy consumption than vehicles behind them. Vehicles at the rear of the convoy, however, benefit significantly from the "wind-breaking effect" of the vehicles in front, resulting in a substantial reduction in air resistance and energy consumption. Related studies consistently agree that this uneven energy distribution caused by static, fixed platoons is particularly significant, especially in electric vehicle platoons. Because electric vehicles rely more heavily on limited battery capacity than gasoline vehicles, and their range, charging time, and cycle life are all affected by energy distribution, the energy imbalance in a fixed platoon will cause the lead vehicle to run out of power earlier, dragging down the entire platoon's range and travel efficiency. It will also accelerate the aging process of the lead vehicle's battery, impacting the platoon's operational economy and vehicle lifespan.

[0003] To address the uneven energy distribution in convoys caused by fixed queuing mechanisms, a widely accepted and effective method is to dynamically adjust the convoy order during convoy operation. By rationally arranging the convoy structure, each vehicle has the opportunity to be at the rear of the convoy during travel, thus benefiting from the reduced air resistance and energy consumption resulting from the "wind-breaking effect." Theoretically, this dynamic convoy adjustment strategy can effectively balance the energy distribution among vehicles in the convoy, preventing problems such as excessive energy consumption, shortened battery life, and increased charging frequency for the vehicle at the front of the convoy due to continuously experiencing high air resistance. Simultaneously, dynamically adjusting the convoy can also increase the overall convoy mileage, reduce the energy consumption gap between vehicles, optimize battery management, and reduce operating costs, providing a promising technical path for convoy energy consumption optimization.

[0004] However, this approach still faces numerous challenges in practical applications. Firstly, queue reshuffling itself incurs energy costs; excessively high reshuffling frequency may negate some of the energy-saving effects. Secondly, dynamic queue rearrangement typically relies on adjacent empty lanes on the road, but during peak hours or congested conditions, lane resource scarcity often makes queue rearrangement difficult to implement. Furthermore, queue optimization is a combinatorial optimization problem; as the size of the vehicle fleet increases, the number of possible queue arrangements increases exponentially, leading to a rapid increase in computational complexity. Traditional optimization algorithms struggle to provide optimal solutions within a limited timeframe, failing to meet the real-time optimization needs of dynamic traffic environments. Simultaneously, optimization methods relying on fixed rules lack the ability to learn from and adapt to changes in the environment, making them ill-suited for complex and ever-changing traffic scenarios. These real-world factors collectively limit the effectiveness of dynamic queue optimization strategies in practical applications. Summary of the Invention

[0005] Based on the above discussion, this invention proposes an intelligent electric vehicle fleet queue optimization method based on congestion prediction and DRL (Deep Reinforcement Learning). Addressing the uneven energy consumption problem caused by fixed queues, this invention combines traffic congestion prediction with reinforcement learning algorithms (such as the Trust Region Policy Optimization (TRPO) algorithm) to design an adaptive intelligent queue adjustment mechanism in dynamic traffic environments.

[0006] This invention uses a Long Short-Term Memory (LSTM) network model to predict time-series data such as traffic flow, speed, and density at future moments, and uses a fuzzy C-means (FCM) classification method to classify and identify traffic conditions, thereby obtaining the probability of traffic congestion and the congestion threshold at future moments. Both of these are used as environmental constraint information and input into the TRPO reinforcement learning model.

[0007] By combining the remaining battery power and cumulative driving distance of each vehicle in the fleet, the TRPO model dynamically adjusts the queue structure of the electric vehicle fleet to achieve energy consumption balance under different traffic environments and improve overall range. This invention uses the TRPO algorithm, which trains on historical data through a policy network and a value network, ultimately providing the electric vehicle fleet with an optimal dynamic queue rearrangement scheme, reducing energy consumption disparities and improving fleet operating efficiency.

[0008] The steps of the method of the present invention include:

[0009] (i) The problem of optimizing the queue position of electric vehicle fleets is modeled as a mathematical model constrained by the traffic environment and with energy consumption balance as the optimization objective;

[0010] (ii) A deep reinforcement learning method based on trust domain policy optimization is adopted to solve the mathematical model established in step (i) and obtain the dynamic queue optimization strategy of the electric vehicle fleet.

[0011] In step (I), an energy balance mathematical model is established with the optimization objective of minimizing the standard deviation of the remaining battery power of each vehicle when the electric vehicle fleet stops. To dynamically constrain the queue transformation process, a traffic congestion identification and prediction method / module combining LSTM and FCM is adopted. The FCM algorithm, combined with a weighted difference algorithm, optimizes the initial cluster centers. Entropy and AHP methods are used to determine the feature weights such as traffic flow, average speed, and traffic density, classifying traffic states into four categories: free flow, mostly free flow, mild congestion, and severe congestion. The classification criteria, mild congestion threshold, and traffic congestion probability at each time step are output. Simultaneously, the LSTM model performs time-series predictions of traffic flow, speed, and density for future time steps. The prediction results are input into the FCM model to generate the traffic congestion state and probability at each future time step. Finally, the congestion probability and threshold are used as traffic environment constraints and input into the queue optimization model to assist in decision-making during the fleet queue rearrangement process.

[0012] In step (ii), the traffic congestion probability and congestion threshold corresponding to the LSTM prediction time are used as traffic environment constraint parameters and input into the TRPO queue optimization model to assist the subsequent dynamic decision-making process. The TRPO algorithm consists of a policy network and a value network. The policy network receives the current observation state of the queue. The algorithm includes the remaining battery power, cumulative travel distance, and current traffic congestion probability of each vehicle in the fleet, and outputs the optimal queue position adjustment strategy. The value network is used to estimate the long-term reward in the current state to assist in the updating and optimization of the policy network. The TRPO algorithm constrains the update magnitude of the policy network through the trust region mechanism and uses KL divergence constraint to limit the policy update step size, avoiding fluctuations and instability of the policy during the update process, and improving the robustness and convergence of model training.

[0013] In the Markov decision framework for the queue optimization problem, the convoy is treated as a single intelligent agent with a state space of... in The remaining battery power for each vehicle. To calculate the total driving distance, Traffic congestion probability; motion space within the convoy. All possible queue combinations of vehicles; the reward function is a scaled value of the standard deviation of the remaining battery power of each vehicle when the convoy stops moving. In each scene of TRPO training, according to the set time intervals, such as... Every few minutes, the probability of traffic congestion is assessed. If the probability is lower than the preset congestion threshold, the vehicle queue is allowed to rearrange its position. If the probability is higher than the threshold, the current queue remains unchanged until any vehicle runs out of power. At this point, the training session ends and a reward is calculated.

[0014] By collecting and inputting the remaining battery power, cumulative driving distance, and predicted traffic congestion probability of each vehicle in the fleet, the TRPO algorithm dynamically outputs the optimal queue rearrangement strategy. The queue with the largest cumulative reward is the optimal fleet queue scheme in the current time period, thereby achieving dynamic balance of queue energy consumption and range optimization under the premise of meeting traffic environment constraints. Attached Figure Description

[0015] Figure 1 It is a TRPO framework diagram that integrates traffic congestion identification and prediction;

[0016] Figure 2 This is a fitted graph of the LSTM traffic flow prediction results;

[0017] Figure 3 This is a diagram showing the results of FCM traffic state recognition.

[0018] Figure 4 This is a comparison chart of the return curves of the fusion mechanism and the benchmark algorithm;

[0019] Figure 5 This is a comparison chart of the return curves of the fusion mechanism and the unconstrained TRPO.

[0020] Figure 6 This is a comparison chart of the number of queue transformations between the fusion mechanism and the unconstrained TRPO.

[0021] Figure 1 Translation of key Chinese terms:

[0022] FCM Traffic Levels Categorization

[0023] Traffic congestion recognition and prediction module

[0024] LSTM-Based Traffic Feature Forecasting: A traffic feature forecasting process based on LSTM.

[0025] Predicted traffic trends (congestion probability) prediction results.

[0026] Predicted congestion categories

[0027] Deep Reinforcement Learning Decision Module;

[0028] forecasted data

[0029] TRPO-based Reinforcement Learning Agent. Detailed Implementation

[0030] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0031] 1. Overview

[0032] In recent years, with the rapid growth of global car ownership, problems such as traffic congestion, increased energy consumption, traffic safety hazards, and environmental pollution have become increasingly prominent. Road transportation has become one of the most energy-intensive and fastest-growing modes of transportation. To address these challenges, cooperative automated driving technology (CACC) has gradually become an important research direction in the fields of intelligent transportation and autonomous driving. Based on vehicle-to-vehicle (V2V) technology, cooperative automated driving can achieve synchronized acceleration and braking of multiple vehicles in a convoy, reducing vehicle spacing and forming a closely coordinated convoy structure. In this way, the air resistance of the entire convoy can be significantly reduced, improving energy efficiency. Several real-vehicle tests have shown that convoy driving can achieve energy savings of 6% to 10%.

[0033] However, in traditional static queuing mechanisms, the fixed positions of vehicles in the platoon result in vehicles at the front end experiencing greater air resistance over extended periods, leading to significantly higher energy consumption than vehicles at the back. This uneven energy distribution within the platoon, especially pronounced in electric vehicle platoons, exacerbates the problem. Furthermore, the dynamic and uncertainties of real-world traffic environments, such as traffic congestion, bottlenecks, or single-lane conditions, are often overlooked by existing queuing optimization methods. This makes it difficult to effectively guide dynamic adjustments to the platoon, hindering the successful implementation of strategies in complex environments.

[0034] To address the aforementioned issues, this invention proposes a dynamic queue optimization method for electric vehicle fleets based on traffic congestion prediction and deep reinforcement learning. The optimization objective is to minimize the standard deviation of the remaining battery power of each vehicle when the fleet stops. A combined LSTM and FCM traffic congestion identification and prediction method is employed to dynamically identify traffic conditions and provide traffic environment constraint information. Subsequently, a deep reinforcement learning model based on the Trust Region Policy Optimization (TRPO) algorithm is used to adaptively generate the optimal queue adjustment strategy for the electric vehicle fleet, incorporating the fleet's energy consumption status and predicted traffic environment information.

[0035] Simulation results show that this method can effectively balance the energy consumption distribution of the fleet in dynamic traffic environments, reduce unnecessary queue rearrangement operations, and improve the practical feasibility and environmental adaptability of queue optimization strategies.

[0036] 2. Queue planning problem

[0037] This section first defines the queue optimization problem for electric vehicle fleets in a dynamic traffic environment.

[0038] To systematically describe the electric vehicle fleet queue optimization problem in this invention, it is necessary to clarify the problem's input, output, and optimization objective. Consider a... A smart fleet of electric vehicles of the same type, using Differentiate vehicle numbers; use express The car is The current location at any given time;

[0039] Along the planned route, select several time intervals of equal length, each time interval being [length missing]. This allows the convoy to determine whether to perform a queue change at the end of each interval. Indicates the departure time. Indicates the time it takes for the queue to reach its destination;

[0040] Indicates the first The car in The relative positions of the time intervals; construct a matrix. This matrix is ​​called the queue position transformation matrix;

[0041] express The remaining battery power of each vehicle at any given time; express The probability of traffic congestion at any given time. This represents the traffic congestion threshold, where, specifically, it is extracted... As a traffic state prediction module that inputs features into a Long Short-Term Memory (LSTM) network, this module predicts future traffic indicators and obtains a congestion probability sequence for future time periods through a traffic state classification method. Furthermore, the index corresponding to the classification standard for mild congestion is used as the congestion threshold, i.e. ;

[0042] make , Let the minimum upper bound be used. Then the queue planning problem for the electric vehicle platoon can be modeled as follows:

[0043]

[0044] 3. TRPO-based queue optimization algorithm

[0045] This section will introduce how to use the TRPO algorithm to solve the electric vehicle fleet queue optimization problem defined above.

[0046] The TRPO algorithm is a reinforcement learning algorithm based on the policy gradient method. Building upon the Actor-Critic structure, it introduces a trust region mechanism. TRPO addresses the issue of performance instability or even degradation caused by excessively large step sizes in traditional policy gradient methods during policy updates by constraining the policy update step size. The core idea of ​​TRPO is to limit the KL divergence between the old and new policies to a small range during each policy update, ensuring that each update process is as stable and conservative as possible, thereby improving convergence and policy robustness during training.

[0047] The TRPO algorithm's neural network consists of a policy network and a value network. The first few layers are shared feature extraction modules used to encode and extract features from the input environmental observation states. The back-end branches into the policy network and the value network. The policy network outputs the queue rearrangement policy for the current state, and the value network estimates the long-term reward for that state. The two are combined to guide policy optimization.

[0048] In the electric vehicle fleet queue optimization problem proposed in this invention, the intelligent fleet acts as an intelligent agent. The observed state of the fleet is defined in... At any given moment, including the remaining battery power of each vehicle in the convoy. The cumulative driving distance of each vehicle and the probability of traffic congestion. That is, the observation state is: ,in, This represents a hybrid observation space that combines the SoC vectors, cumulative driving distance, and congestion probability of electric vehicles in the queue. The dimension of this observation space is . Dimension; Action space for the team All possible queue sequences that electric vehicles can form; each time a queue passes through one The time step serves as a trigger condition for a queue decision; in a training scenario, it determines the probability of congestion in the current traffic environment. If it is below the preset congestion threshold If the condition is met, the queue position can be changed. If not, the current queue remains unchanged until any vehicle runs out of power, at which point the process terminates and a reward is given. The goal of this method is to minimize the standard deviation of the remaining power when the convoy stops moving. When any vehicle in the convoy runs out of power and enters the termination state, the reward value is calculated based on the standard deviation of the total remaining power of the vehicles. In non-termination states, the reward is zero. Final Reward in, To reward scaling factor, It represents the standard deviation.

[0049] In the TRPO network structure, the data processing steps include:

[0050] Input observation status The This includes the remaining battery power, cumulative driving distance, and probability of traffic congestion for each vehicle in the fleet;

[0051] The policy function is calculated using two neural networks. and value function ;

[0052] Based on the collected trajectory data and value function, the generalized advantage estimation (GAE) method is used to calculate the advantage function for each state-action pair:

[0053]

[0054] in, This represents the estimate of the advantage function calculated using GAE; Indicates the discount factor; This indicates an additional hyperparameter introduced in GAE; This represents the time-series difference, which is the error of the current value function; V represents the immediate reward received; V represents the state value function of the current state; w represents the hyperparameters of the value network.

[0055] Here, trajectory data refers to a complete record of the agent's "interaction process" in the environment, which includes action-state-reward pairs at multiple time steps. For example, trajectory = {(s0,a0,r0), (s1,a1,r1)...}.

[0056] Calculate the gradient of the policy objective function And based on the Hessian matrix The conjugate gradient method is used to solve for the parameter update direction. ;

[0057] Perform a linear search within the trust region to determine the step size that satisfies the KL constraint. Update strategy parameters And update the value network parameters simultaneously:

[0058] .

[0059] 4. Dynamic learning rate adjustment strategy

[0060] In this invention, an exponentially decaying learning rate adjustment strategy is introduced to adapt to dynamically changing traffic conditions and optimize fleet energy allocation strategies. This method ensures the smoothness and stability of the training process by gradually reducing the learning rate, avoiding gradient fluctuations or model divergence caused by excessively high learning rates. Simultaneously, using a higher learning rate in the early stages of training accelerates the exploration of the state space, effectively improving the model's adaptability to complex traffic environments; while gradually reducing the learning rate in the later stages of training helps the model to more finely optimize the optimal strategy, thereby achieving better energy management and formation adjustment strategies. The learning rate adjustment follows the formula:

[0061]

[0062] in, This represents the updated learning rate. The initial learning rate, As the attenuation factor, This indicates the current training round.

[0063] 5. Performance Analysis

[0064] To verify the effectiveness of the proposed electric vehicle fleet queue optimization method based on traffic congestion state prediction and TRPO optimization, this paper designs a series of simulation experiments based on real traffic data, focusing on evaluating the optimization performance and environmental adaptability of the method under dynamic traffic environments. The experimental traffic data comes from the California PeMS traffic monitoring system, selecting historical data from typical road segments, covering traffic operation indicators such as traffic flow, average speed, and traffic density, with a 5-minute sampling period. Based on this data, an LSTM model is used to predict traffic indicators for future periods, and the prediction results are input into an FCM clustering model to dynamically classify traffic state levels, generating a traffic congestion probability sequence and corresponding mild congestion thresholds, providing traffic environment constraints for the subsequent queue rearrangement process.

[0065] The electric vehicle fleet consists of four fully charged Teslas, each with a battery capacity of 85kWh, averaging 340Wh of energy consumption per 100 miles. The queue energy consumption matrix takes into account the wind resistance differences of vehicles at different queue positions, setting positional energy consumption reduction rates of 4.3%, 10%, 14%, and 14%, corresponding to the positions from the front to the back of the queue, respectively, expressed as follows: Each vehicle starts with a full charge, meaning... .

[0066] During the simulation experiment, the convoy travels along a preset route, with a fixed queue rearrangement judgment period of 20 minutes. Combining the traffic state predicted by the LSTM model and the traffic congestion probability output by the FCM, it is determined whether to allow the queue to rearrange. If the congestion probability at the current time is lower than the mild congestion discrimination threshold, the TRPO policy network is allowed to output a queue rearrangement policy; otherwise, the existing formation is maintained until the next judgment time.

[0067] To comprehensively evaluate the performance of the proposed intelligent fleet queue optimization method based on LSTM prediction and TRPO optimization, this paper designs four sets of comparative experiments under the same experimental environment and energy consumption parameter settings: Genetic Algorithm (GA), Q-Learning Algorithm, Monte Carlo (MC) method, and standard TRPO algorithm without traffic congestion constraints. These are uniformly applied to the same dynamic traffic scenario and fleet optimization task. Regarding performance metrics, this paper adopts two evaluation criteria: first, the cumulative reward performance of each algorithm under the same environment, to measure the energy efficiency balance capability of different optimization strategies in dynamic traffic environments; a higher cumulative reward value indicates a better queue optimization scheme and a more balanced energy consumption distribution; second, comparing the changes in the number of queue reordering processes under conditions with and without traffic congestion constraints, to evaluate the inhibitory effect of different algorithms on queue reordering strategies and their effective avoidance of redundant reordering in dynamic traffic environments.

[0068] The algorithm proposed in this invention was developed using Python and implemented based on the PyTorch 1.13.1 framework and the OpenAI Gym environment. In the designed reinforcement learning experiments, the initial learning rate was consistently used throughout the training process. Learning rate decay factor Fixed learning rate Discount factor dominance function smoothing factor The TRPO optimization algorithm employs a trust region mechanism and linearly searches for coefficients. =0.5. Furthermore, in the traffic state recognition and classification section, the FCM algorithm is used to determine the traffic state level and the number of clusters. Fuzzy weight parameters .

[0069] All experiments were conducted on a workstation equipped with an NVIDIA GeForce RTX 4050 (16GB GDDR6X VRAM). The experimental platform was Python, and the core algorithm framework was based on PyTorch 1.13.1 and OpenAI Gym. The number of reinforcement learning training sets was set to 10,000. A smoothing window technique was used to process the reward curve during training, with a window size of 31. Training terminated when the cumulative number of training epochs reached a preset limit. All comparison algorithms were run under the same experimental environment and hardware platform to ensure the fairness and comparability of the experimental results.

[0070] Table 1 shows the initial cluster centers of traffic states identified using the improved FCM algorithm. The results indicate that as traffic conditions worsen, vehicle speed gradually decreases, while traffic density and flow rate significantly increase, fully reflecting the typical congestion characteristics of "low speed and high density" in traffic operations. These classification results provide a scientific basis for subsequent targeted fleet rearrangement and energy consumption optimization under different traffic conditions.

[0071] surface Initial cluster centers

[0072]

[0073] Figure 2 The fitting curves of time-series prediction results for traffic flow, speed, and density based on the LSTM model are shown. Figure 3 This displays the traffic state classification results achieved using the FCM model. Figure 2 and Figure 3 It is evident that the LSTM model can accurately depict the dynamic changes in traffic indicators, and the FCM model can clearly and effectively classify traffic states at future times, providing reliable traffic environment constraint information for queue rearrangement strategies.

[0074] Figure 4 The cumulative reward curves of the queue optimization method with the joint mechanism proposed in this invention and three benchmark algorithms are shown during the training process. As can be seen from the figure, the proposed method exhibits a higher reward value throughout the training process and a faster convergence speed. The fluctuation range during training is significantly smaller than that of other algorithms, verifying the good convergence performance and stability of the proposed method in dynamic traffic environments. In contrast, the genetic algorithm performs reasonably well in the early stages, but its overall convergence performance is insufficient; the Q-Learning algorithm, limited by its exploration ability, suffers from large fluctuations in training reward values ​​and poor stability, making it difficult to adapt to the queue rearrangement requirements in complex environments. This further demonstrates the advantages of the proposed method in terms of energy efficiency balance and the robustness of the optimization strategy.

[0075] Figure 5 and Figure 6This paper demonstrates the performance difference between the proposed fusion mechanism and the standard TRPO algorithm without congestion constraints in queue optimization tasks. Results show that the proposed fusion mechanism combines congestion prediction with deep reinforcement learning optimization, employing TRPO and introducing congestion constraints to reduce unnecessary queue adjustments. In contrast, the optimization algorithm without congestion constraints, due to its lack of consideration for traffic flow constraints, leads to frequent reorganizations, achieving higher rewards but lacking practical feasibility. Experimental results demonstrate that the fusion mechanism maintains system stability while achieving a dynamic balance between traffic state and queue configuration, exhibiting stronger environmental adaptability and robustness.

[0076] In summary, the proposed method integrates traffic congestion identification and prediction mechanisms, outperforming traditional methods in optimizing energy consumption distribution and improving strategy feasibility. Furthermore, it is more consistent with actual traffic environments than the standard TRPO algorithm, demonstrating better adaptability and practicality.

[0077] 6. Summary

[0078] This invention proposes an intelligent electric vehicle fleet queue optimization method based on congestion prediction and deep reinforcement learning (RDL). The steps include: (i) modeling the queue position optimization problem of the electric vehicle fleet as a mathematical model with energy balance as the objective and constrained by the traffic environment; (ii) using a deep reinforcement learning method based on the Trust Region Policy Optimization (TRPO) algorithm to solve the mathematical model of step (i) and obtain the optimal queue adjustment strategy for the electric vehicle fleet. In step (ii), the congestion state information in the external traffic environment is obtained through Long Short-Term Memory (LSTM) traffic prediction and fuzzy C-means clustering (FCM) method, and is used together with the remaining battery power and cumulative driving distance of each vehicle in the electric vehicle fleet as the state input of the TRPO algorithm. The TRPO algorithm uses a policy network and a value network. The policy network is used to output the strategy for adjusting the queue position, and the value network is used to evaluate the expected return in the current state. The TRPO algorithm dynamically adjusts the policy update step size by constraining the KL divergence of the policy update, ensuring the stability and convergence of the policy update process, and reducing unnecessary queue changes in the dynamic traffic environment.

Claims

1. A method for optimizing the queues of intelligent electric vehicle fleets based on congestion prediction and DRL, characterized by the following steps: include: (i) The problem of optimizing the queue position of electric vehicle fleets is abstracted into a mathematical model with energy balance as the goal and constrained by the traffic environment; (ii) The mathematical model obtained in step (i) is solved by using the deep reinforcement learning (DRL) method to obtain the optimal queue optimization strategy for the intelligent electric vehicle fleet to execute; In step (1), a mathematical model constrained by the traffic environment is established as the queue optimization model with the goal of minimizing the standard deviation of the remaining battery power of each vehicle when the electric vehicle fleet stops moving. A traffic congestion identification and prediction module combining LSTM and FCM is used to dynamically obtain traffic environment constraints. The method is as follows: First, the initial cluster centers are optimized by using the fuzzy C-means clustering algorithm (FCM) combined with the weighted difference algorithm. Then, the feature weights of traffic flow, speed, and density are determined using the entropy method and the analytic hierarchy process (AHP). Based on the feature weights, the traffic conditions are classified into categories with different levels of congestion. The traffic condition classification criteria, congestion thresholds, and congestion probabilities at each time point are obtained. Then, the Long Short-Term Memory (LSTM) network model is used to make time-series predictions of traffic flow, speed, and density at future time steps; the prediction results are input into the FCM model to generate the traffic congestion probability at each future time step. Ultimately, the congestion probability and congestion threshold are used as traffic environment constraints and input into the queue optimization model to assist in decision-making during the queue rearrangement process. In step (ii), the DRL method is the Trust Region Policy Optimization TRPO algorithm; the TRPO algorithm is used to solve the queue optimization model; The TRPO algorithm's policy network receives the current observation state of the queue and outputs the optimal queue position adjustment policy; the TRPO algorithm's value network estimates the long-term reward in the current state, which is used to assist in the updating and optimization of the policy network. In the Markov decision framework for the queue optimization problem: the agent represents the queuing team; The observed status is the remaining battery power, cumulative travel distance, and current congestion probability of each vehicle in the convoy; the action is the total number of queue sequences that can be formed by all vehicles in the convoy; the reward is the scaled value of the standard deviation of the remaining battery power of each vehicle when the convoy ends. In each scene of the TRPO algorithm training, the congestion probability of the traffic state is judged at fixed intervals: if the congestion probability is lower than the congestion threshold, the queue is allowed to change positions; if the congestion probability is higher than the congestion threshold, the original queue remains unchanged until any car runs out of power, at which point the scene is considered to end and a reward is given. The TRPO algorithm outputs the queue with the highest cumulative reward as the optimal fleet queue scheme for the current time period, and uses this as the optimal queue optimization strategy.

2. The intelligent electric vehicle fleet queue optimization method based on congestion prediction and DRL as described in claim 1, characterized in that... In step (one), a... In a smart fleet of electric vehicles, using Differentiate vehicle numbers; use express The car is The current location at any given time; Along the planned route, select several time intervals of equal length, each time interval being [length missing]. This allows the convoy to determine whether to perform a queue change at the end of each interval. Indicates the departure time. Indicates the time when the queue terminated its operation; Indicates the first The car in The relative positions of the time intervals; construct a matrix. This matrix is ​​called the queue position transformation matrix; express The remaining battery power of each vehicle at any given time; express The probability of congestion at any given time. Indicates the congestion threshold; By extraction The traffic features are input into the traffic congestion identification and prediction module to predict future traffic conditions, ultimately obtaining a congestion probability sequence for future time periods. Flow(t), Speed(t), and Density(t) represent the traffic flow, speed, and density at time t, respectively. Traffic conditions are categorized into four types: free-flow, mostly free-flow, mild congestion, and severe congestion. The congestion threshold is determined by the index corresponding to the classification criteria for mild congestion. ; make , Denotes the least upper bound. express If the remaining battery power of the i-th vehicle at time i is given, then the queue planning problem for the electric vehicle platoon can be modeled as follows: ， ， ， ， , ， ， ， ， in, The standard deviation of the total remaining battery power (SoC) of the fleet when it terminates operation; Indicates that vehicle i is in the time period Electricity consumed; This represents the average power consumption rate per unit distance. Indicates the vehicle's maximum battery capacity; This indicates that vehicle i was in the previous time period. Location, in position The rate of reduction in electricity consumption during the period.

3. The intelligent electric vehicle fleet queue optimization method based on congestion prediction and DRL as described in claim 2, characterized in that... In step (two), The fleet acts as an intelligent agent; The observation status of the convoy is defined in At any given moment, the observed status includes the remaining battery power of each vehicle in the convoy. The cumulative driving distance of each vehicle and the probability of traffic congestion. That is, the observed state is represented as: ,in, This represents a hybrid observation space that combines the SoC vectors, cumulative driving distance, and congestion probability of electric vehicles in the queue. The dimension of this observation space is . dimension; Action space for the team All possible queue sequences that electric vehicles can form; In each scene of TRPO training, after each... The time step serves as a trigger condition for a queue decision; in a training scenario, it determines the probability of congestion in the current traffic environment. If it is below the preset congestion threshold If the condition is met, the queue position can be changed. If the condition is not met, the current queue remains unchanged until any car runs out of power, at which point the scene is terminated and a reward is given. At the end of a training session, the reward function provides immediate feedback to the vehicle platoon after each action, guiding the learning process to optimize the balanced distribution of vehicle energy. The ultimate goal is to minimize the standard deviation of the remaining battery power when the platoon stops. When any vehicle in the platoon runs out of battery power and enters a terminated state, the reward value is calculated based on the standard deviation of the total remaining battery power of the vehicles. In non-terminating states, the reward is zero. , To reward scaling factor, It represents the standard deviation.

4. The intelligent electric vehicle fleet queue optimization method based on congestion prediction and DRL as described in claim 3, characterized in that: The TRPO algorithm process is as follows: Step 1) Input the observation status S, including the remaining battery power, cumulative driving distance, and congestion probability of each vehicle in the fleet; Step 2) Calculate the policy function using the policy network and value network respectively. and value function ; Step 3) Based on the value function and the collected trajectory data, the generalized advantage estimation (GAE) method is used to calculate the advantage function corresponding to each state action: ; in, This represents the estimate of the advantage function calculated using GAE; Indicates the discount factor; This indicates an additional hyperparameter introduced in GAE; This represents the time-series difference, which is the error of the current value function; V represents the immediate reward received; V represents the state value function of the current state; and w represents the hyperparameters of the value network. Step 4) Calculate the gradient of the policy objective function. And based on the Hessian matrix The conjugate gradient method is used to solve for the parameter update direction. ; Step 5) Perform a linear search within the trust region to determine the step size that satisfies the KL constraint. Update strategy parameters And update the value network parameters simultaneously: ， In the formula, Indicates the policy network parameters, This represents the step size factor for the i-th linear search. Let g represent the time-series error, g represent the gradient vector of the policy objective function, and H represent the Hessian matrix.

5. The intelligent electric vehicle fleet queue optimization method based on congestion prediction and DRL as described in claim 4, characterized in that: The TRPO algorithm's dynamic learning rate adjustment strategy employs an exponential decay learning rate adjustment method, with the learning rate adjusted according to the following formula: ， in, This represents the updated learning rate. The initial learning rate, As the attenuation factor, This indicates the current training round.

Images