Intelligent electric vehicle queue optimization method based on congestion prediction and DRL

Abstract

The invention discloses an intelligent electric vehicle queue optimization method based on congestion prediction and DRL. The method comprises the following steps: (1) modeling a queue position optimization problem of an electric vehicle queue into a mathematical model which takes energy balance as a target and is constrained by a traffic environment; and (2) solving the mathematical model in the step (1) by adopting a DRL method based on a TRPO algorithm to obtain an optimal queue adjustment strategy of the electric vehicle queue. In the step (2), congestion state information in an external traffic environment is obtained through an LSTM traffic prediction and FCM method, and the congestion state information, the remaining electric quantity of each vehicle in the motorcade and the accumulated driving distance are jointly used as state input of a TRPO algorithm; according to the TRPO algorithm, a strategy network and a value network are adopted to respectively obtain a strategy for adjusting the queue position, and the return expectation in the current state is evaluated. The TRPO algorithm dynamically adjusts the updating step length of the strategy through the KL divergence updated by the constraint strategy, and guarantees the stability and convergence in the strategy updating process.

Description

Technical Field

[0001] The present invention relates to the application of computer artificial intelligence technology in intelligent fleet management, and specifically to an intelligent electric vehicle fleet queue optimization method that combines traffic congestion prediction with a DRL method (such as the Trust Region Policy Optimization (TRPO) algorithm). Background Art

[0002] In recent years, electric platooning, as a key component of vehicle-road collaboration and autonomous driving, has seen increasing adoption in smart transportation and transport systems, effectively improving road efficiency and safety. Smart platoons typically consist of multiple vehicles using wireless communication technology to coordinate synchronized acceleration and deceleration, enabling them to operate in close formation within the same lane with spacing closer than conventional safety distances. Thanks to the "following effect" created by closely spaced platooning, smart platoons can effectively reduce air resistance and minimize fuel or energy losses. Extensive wind tunnel experiments, CFD simulations, and field tests have demonstrated that smart platooning can reduce energy consumption by an average of over 10%, offering significant energy and environmental advantages compared to single-vehicle operation. Aerodynamic research further reveals that vehicles in different positions within a platoon experience significant differences in air resistance. Vehicles at the front of the platoon experience significantly higher air resistance due to the increased oncoming airflow, resulting in significantly higher energy consumption than those at the rear. Vehicles at the rear of the platoon, on the other hand, benefit significantly from the "wind-breaking effect" of the preceding vehicle, significantly reducing air resistance and energy consumption. Related research consistently agrees that the uneven energy consumption distribution caused by static, fixed platoons is particularly significant, especially in electric vehicle fleets. Because electric vehicles rely more heavily on limited battery capacity than fuel-powered vehicles, and their range, charging time, and cycle life are all affected by energy consumption distribution, the uneven energy consumption in a fixed platoon will cause the lead vehicle to deplete its battery earlier, hindering the range and travel efficiency of the entire fleet. It also accelerates the aging of the lead vehicle's battery, impacting the fleet's operating economy and vehicle lifespan.

[0003] To address the uneven energy consumption distribution problem in fleets caused by fixed queue mechanisms, a widely recognized effective method is to dynamically adjust the queue order of vehicles while the fleet is driving. By properly arranging the queue structure, each vehicle has the opportunity to be at the end of the queue during driving, thereby benefiting from the reduced air resistance and energy consumption brought about by the "wind-breaking effect." In theory, this dynamic queue adjustment strategy can effectively balance the energy consumption distribution of each vehicle in the fleet, avoiding problems such as excessive energy consumption, shortened battery life, and increased charging frequency caused by the head vehicle being subjected to continuous high air resistance. At the same time, dynamic queue adjustment can also improve the overall mileage of the fleet, reduce the energy consumption gap between vehicles, optimize battery management, and reduce operating costs, providing a technical path with practical application prospects for fleet energy consumption optimization.

[0004] However, this approach still faces numerous challenges in practical application. For one thing, queue changes inherently incur energy costs, and if they occur too frequently, they can offset some of the energy savings. Furthermore, dynamic rearrangement of vehicle queues typically relies on adjacent free lanes on the road. During peak hours or in congested conditions, queue rearrangement is often difficult to implement due to limited lane resources. Furthermore, queue optimization is a combinatorial optimization problem. As the size of the fleet increases, the number of possible queue arrangements increases exponentially, and the computational complexity rapidly increases. Traditional optimization algorithms struggle to provide optimal solutions within a limited timeframe, making them ill-suited for the real-time optimization demands of dynamic traffic environments. Furthermore, optimization methods that rely on fixed rules lack the ability to learn and adapt to changes in the environment, making them difficult to handle intricate and volatile traffic scenarios. These practical factors collectively limit the effectiveness of dynamic queue optimization strategies in real-world scenarios. Summary of the Invention

[0005] Based on the above discussion, this paper proposes a method for optimizing the platooning of intelligent electric vehicle fleets based on congestion prediction and Deep Reinforcement Learning (DRL). To address the uneven energy consumption caused by fixed platoons, this paper combines traffic congestion prediction with reinforcement learning algorithms (such as the Trust Region Policy Optimization (TRPO) algorithm) to design an intelligent platooning adjustment mechanism that is adaptive in dynamic traffic environments.

[0006] The present invention predicts time series data such as traffic flow, speed and density at future moments through the long short-term memory network (LSTM) model, and classifies traffic status based on the fuzzy C-means (FCM) classification method to obtain the traffic congestion probability and congestion threshold at future moments. Both are input into the TRPO reinforcement learning model as environmental constraint information.

[0007] The TRPO model dynamically adjusts the fleet's platoon structure by combining the remaining battery life and accumulated distance traveled of each vehicle in the fleet, achieving balanced energy consumption in varying traffic conditions and improving overall endurance. This paper employs the TRPO algorithm, training historical data through a policy network and a value network, ultimately providing an optimal dynamic platoon rearrangement solution for the fleet, reducing energy consumption disparities and improving fleet operational efficiency.

[0008] The steps of the method of the present invention include:

[0009] (1) Modeling the platoon position optimization problem of an electric vehicle fleet as a mathematical model subject to traffic environment constraints and with energy consumption balance as the optimization goal;

[0010] (2) Using a deep reinforcement learning method based on the trust region strategy optimization TRPO, we solve the mathematical model established in step (1) and obtain the dynamic platoon optimization strategy for the electric vehicle fleet.

[0011] In step (1), an energy balance mathematical model is established with the optimization objective of minimizing the standard deviation of the remaining charge of each vehicle in the electric fleet when stopped. To dynamically constrain the queue transformation process, a traffic congestion identification and prediction method / module combining LSTM and FCM is employed. The FCM algorithm, combined with a weighted difference algorithm, optimizes the initial cluster centers. The entropy method and the AHP method are used to determine the weights of features such as traffic flow, average speed, and traffic density. Traffic conditions are classified into four categories: free flow, basic free flow, mild congestion, and severe congestion. The model outputs traffic state classification criteria, mild congestion thresholds, and the probability of traffic congestion at each moment. Simultaneously, the LSTM model performs time-series predictions of traffic flow, speed, and density for future time steps. These predictions are fed into the FCM model to generate traffic congestion states and probabilities for each future moment. Finally, the congestion probability and threshold, as traffic environment constraints, are input into the queue optimization model to assist in decision-making during the fleet's queue rearrangement process.

[0012] In step (2), the traffic congestion probability and congestion threshold corresponding to the LSTM prediction time are input into the TRPO queue optimization model as traffic environment constraint parameters to assist the subsequent dynamic decision-making process. The TRPO algorithm consists of a policy network and a value network. The policy network receives the current observation state of the queue. , including the remaining power of each vehicle in the fleet, the cumulative driving distance and the traffic congestion probability at the current moment, and outputs the optimal queue position adjustment strategy; the value network is used to estimate the long-term return under the current state to assist in the update and optimization of the policy network; the TRPO algorithm constrains the update amplitude of the policy network through the trust region mechanism and uses the KL divergence constraint to limit the policy update step size, avoiding fluctuations and instability of the policy during the update process, and improving the robustness and convergence effect of model training.

[0013] In the Markov decision framework of the platoon optimization problem, the fleet is a single agent with a state space of in is the remaining power of each vehicle, is the cumulative driving distance, is the probability of traffic congestion; the action space is the The reward function is the scaled value of the standard deviation of the remaining power of each vehicle when the convoy stops. In each scene of TRPO training, according to the set time interval, The congestion probability of the traffic environment is judged in turn every minute. If it is lower than the preset congestion threshold, the fleet is allowed to rearrange the queue position. If it is higher than the threshold, the current queue remains unchanged until any car runs out of power. At this point, the training is deemed to be over and the reward is calculated.

[0014] By collecting and inputting the remaining battery life, cumulative driving distance, and predicted traffic congestion probability of each vehicle in the fleet, the TRPO algorithm dynamically outputs the optimal queue rearrangement strategy. The queue with the largest cumulative reward is the optimal fleet queue solution for the current period, thereby achieving dynamic balancing of queue energy consumption and endurance optimization while meeting traffic environment constraints. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a TRPO framework diagram that integrates traffic congestion identification and prediction;

[0016] Figure 2 This is the fitting diagram of the LSTM traffic flow prediction results;

[0017] Figure 3 This is the FCM traffic state recognition result diagram;

[0018] Figure 4 This is a comparison chart of the return curves of the fusion mechanism and the benchmark algorithm;

[0019] Figure 5 This is a comparison of the return curves of the fusion mechanism and TRPO under no constraints;

[0020] Figure 6 This is a comparison chart of the number of queue changes between the fusion mechanism and TRPO without constraints.

[0021] Figure 1 Translation of main terms in:

[0022] FCM Traffic Levels Categorization FCM Traffic Levels Categorization,

[0023] Traffic congestion recognition and prediction module,

[0024] LSTM-Based traffic feature forecasting LSTM-based traffic feature forecasting process,

[0025] Predicted traffic trends Traffic trends (congestion probability) prediction results,

[0026] Predicted congestion categories Congestion category prediction results,

[0027] Deep Reinforcement Learning decision module Deep reinforcement learning decision module;

[0028] forecasted data

[0029] TRPO-based Reinforcement Learning Agent TRPO-based reinforcement learning agent. DETAILED DESCRIPTION

[0030] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0031] 1. Overview

[0032] In recent years, with the rapid growth of the global car population, traffic congestion, increased energy consumption, traffic safety hazards, and environmental pollution have become increasingly prominent. Road transport has become one of the most energy-intensive and fastest-growing modes of transportation in the transportation system. To address these challenges, cooperative autonomous driving (CACC) has gradually become a key research direction in the fields of smart transportation and autonomous driving. Based on vehicle-to-vehicle (V2V) technology, cooperative autonomous driving enables synchronized acceleration and braking of multiple vehicles in a platoon, reducing the distance between vehicles and forming a tightly coordinated platoon structure. This significantly reduces aerodynamic drag throughout the platoon and improves energy efficiency. Numerous field tests have demonstrated that platooning can achieve energy savings of 6% to 10%.

[0033] However, under traditional static platooning mechanisms, the positions of vehicles in a platoon are fixed, resulting in vehicles at the front end experiencing greater air resistance over extended periods, leading to significantly higher energy consumption than those at the rear. This leads to uneven energy distribution within the platoon, a situation particularly acute in electric vehicle fleets. Furthermore, existing platooning optimization methods often overlook the dynamics and uncertainties of real-world traffic environments, such as traffic congestion, bottlenecks, or single-lane situations. This makes it difficult to effectively guide dynamic adjustments to the platoon, making the strategies difficult to implement in complex environments.

[0034] To address these issues, this paper proposes a dynamic platoon optimization method for electric vehicle fleets based on traffic congestion prediction and deep reinforcement learning. Minimizing the standard deviation of the remaining battery charge of each vehicle when the fleet stops is the optimization objective. A traffic congestion identification and prediction method combining LSTM and FCM is employed to dynamically identify traffic conditions and provide information on traffic environment constraints. Subsequently, a deep reinforcement learning model constructed using the Trust Region Policy Optimization (TRPO) algorithm is used to adaptively generate the optimal platoon adjustment strategy for the electric vehicle fleet, combining the fleet's energy consumption status with the predicted traffic environment.

[0035] The simulation results show that this method can effectively balance the energy consumption distribution of the fleet in a dynamic traffic environment, reduce unnecessary queue rearrangement operations, and improve the practical feasibility and environmental adaptability of the queue optimization strategy.

[0036] 2. Queue planning issues

[0037] This section first defines the platoon optimization problem of electric vehicle fleets in a dynamic traffic environment.

[0038] In order to systematically describe the electric vehicle fleet optimization problem in this invention, it is necessary to clarify the input, output and optimization goal of the problem. A smart fleet of electric vehicles of the same type, To distinguish vehicle numbers; express A car in The location reached at the time;

[0039] On the planned route, select several time intervals of equal length, each time interval is , allowing the fleet to decide whether to change the queue at the end of each interval, Indicates the departure time. Indicates the time when the queue arrives at the destination;

[0040] Indicates the Car in the The relative positions of time intervals; construct a matrix , this matrix is ​​called the queue position transformation matrix;

[0041] express The remaining power of each vehicle at all times; express The probability of traffic congestion at the moment, represents the traffic congestion threshold, where, by extracting The feature is input into the traffic state prediction module of the long short-term memory network LSTM, which predicts future traffic indicators and obtains the congestion probability sequence at future moments through the traffic state classification method. , and the corresponding index of the classification standard corresponding to mild congestion is used as the congestion threshold, that is, ;

[0042] make , represents the minimum upper bound, then the platoon planning problem of the electric vehicle fleet is modeled as:

[0043]

[0044] 3. Queue optimization algorithm based on TRPO

[0045] This section will introduce how to use the TRPO algorithm to solve the electric vehicle fleet optimization problem defined above.

[0046] The TRPO algorithm is a reinforcement learning algorithm based on the policy gradient method. It incorporates a trust region mechanism based on the actor-critic architecture. By constraining the step size of policy updates, TRPO addresses the problem of traditional policy gradient methods where excessively large step sizes can lead to unstable or even degraded performance. The core idea of ​​TRPO is to constrain the KL divergence between the old and new policies to a small range during each policy update, ensuring that each update is as stable and conservative as possible, thereby improving convergence and policy robustness during training.

[0047] The TRPO algorithm's neural network consists of a policy network and a value network. The first few layers are shared feature extraction modules, which encode and extract features from the input environmental observations. The backend branches into a policy network and a value network. The policy network outputs the queue reordering strategy for the current state, while the value network estimates the long-term reward for that state. The two networks are combined to guide policy optimization.

[0048] In the electric vehicle fleet optimization problem proposed in this paper, the intelligent fleet is used as an intelligent agent. The observation state of the fleet is defined as At this moment, including the remaining power of each vehicle in the fleet , the cumulative driving distance of each vehicle , and the congestion probability of the traffic environment , that is, the observation state is: ,in, represents the mixed observation space combining the SoC vectors, cumulative driving distances and congestion probabilities of the electric vehicles in the queue, The dimension of the observation space is dimension; the action space is the fleet All queue sequences that can be formed by electric vehicles; each time a Time step, as a queue decision triggering condition; in one scene training, judge the congestion probability of the current traffic environment , if it is lower than the preset congestion threshold , the queue position is allowed to change. If the conditions are not met, the current queue remains unchanged until any vehicle runs out of power, the scene is judged to be terminated, and a reward is given. The goal of this method is to minimize the standard deviation of the remaining power when the team stops. When any vehicle in the team runs out of power and enters the termination state, the reward value is calculated based on the standard deviation of the remaining power of the entire vehicle. In the non-termination state, the reward is zero. Final reward in, is the reward scaling factor, Represents standard deviation.

[0049] In the TRPO network structure, the data processing steps include:

[0050] Input observation status , Including the remaining power, cumulative driving distance and traffic congestion probability of each vehicle in the fleet;

[0051] Calculate the policy function separately through two neural networks and value function ;

[0052] Based on the collected trajectory data and value function, the generalized advantage estimation (GAE) method is used to calculate the advantage function of each state-action pair:

[0053]

[0054] in, represents the advantage function estimate calculated using GAE; represents the discount factor; Indicates an additional hyperparameter introduced in GAE; represents the temporal difference, which is the error of the current value function; Represents the immediate reward obtained; V represents the state value function of the current state; w represents the hyperparameter of the value network;

[0055] Here, trajectory data refers to the complete record of the agent's "interaction process" in the environment, which contains action-state-reward pairs for multiple time steps, for example, trajectory = {(s0, a0, r0), (s1, a1, r1)...}.

[0056] Calculate the gradient of the policy objective function , and based on the Hessian matrix , the conjugate gradient method is used to solve the parameter update direction ;

[0057] Perform a linear search in the trust region to determine the step size that satisfies the KL constraint , update the policy parameters , and update the value network parameters synchronously:

[0058] .

[0059] 4. Learning rate dynamic adjustment strategy

[0060] In this invention, in order to adapt to dynamically changing traffic conditions and optimize the fleet energy allocation strategy, an exponential decay learning rate adjustment strategy is introduced. This method ensures the smoothness and stability of the training process by gradually reducing the learning rate, avoiding gradient fluctuations or model divergence problems caused by excessively high learning rates. At the same time, using a higher learning rate in the early stages of training can accelerate the exploration of the state space and effectively improve the model's adaptability to complex traffic environments; while in the later stages of training, gradually reducing the learning rate helps the model to perform more refined optimization of the optimal strategy, thereby achieving better energy management and formation adjustment strategies. The adjustment of the learning rate follows the following formula:

[0061]

[0062] in, represents the updated learning rate, is the initial learning rate, is the attenuation factor, Indicates the current training round.

[0063] 5. Performance Analysis

[0064] To validate the effectiveness of the proposed electric vehicle fleet optimization method based on traffic congestion prediction and TRPO optimization, this paper designed a series of simulation experiments based on actual traffic data, focusing on evaluating the method's optimization performance and environmental adaptability in dynamic traffic environments. The experimental traffic data was obtained from the California PeMS traffic monitoring system. Historical data from typical road sections was selected, covering traffic operation indicators such as traffic flow, average speed, and traffic density, with a sampling period of 5 minutes. Based on this data, an LSTM model was used to predict traffic indicators for future time periods. The prediction results were input into the FCM clustering model, which dynamically classified traffic status levels and generated a traffic congestion probability sequence and corresponding light congestion thresholds, providing traffic environment constraints for the subsequent queue rearrangement process.

[0065] The electric vehicle fleet consists of 4 fully charged Teslas, each with a battery capacity of 85kWh and an average energy consumption of 340Wh per 100 miles. The queue energy consumption matrix refers to the wind resistance difference of vehicles at different queue positions, and sets the position energy consumption reduction rate of 4.3%, 10%, 14% and 14%, corresponding to the position from the head to the tail of the queue, respectively, and is expressed as ; Each vehicle is initially fully charged, i.e. .

[0066] During the simulation, the fleet traveled along a pre-set route, with a fixed queue reshuffling cycle of 20 minutes. The LSTM model combined traffic conditions predicted by the model with the congestion probability output by the FCM to determine whether to allow queue reshuffling. If the current congestion probability was below the threshold for mild congestion, the TRPO strategy network would output a queue reshuffling strategy. Otherwise, the existing queue would be maintained until the next reshuffling decision.

[0067] To comprehensively evaluate the performance of the proposed intelligent fleet platoon optimization method based on LSTM prediction and TRPO optimization, this paper designed four sets of comparative experiments under the same experimental environment and energy consumption parameter settings. These experiments involved a genetic algorithm (GA), a Q-Learning algorithm, a Monte Carlo (MC) method, and a standard TRPO algorithm without traffic congestion constraints. These experiments were uniformly applied to the same dynamic traffic scenario and fleet optimization task. Regarding performance metrics, this paper employed two evaluation criteria: first, the cumulative reward performance of each algorithm under the same environment was used to measure the energy efficiency balance of different optimization strategies in a dynamic traffic environment. Higher cumulative reward values ​​indicate better platoon optimization solutions and more balanced energy consumption distribution. Second, the number of formation changes during platoon rearrangement was compared with and without traffic congestion constraints to assess the different algorithms' ability to suppress platoon change strategies and effectively avoid redundant rearrangements in a dynamic traffic environment.

[0068] The algorithm proposed in this paper is developed in Python and implemented based on the PyTorch 1.13.1 framework and the OpenAI Gym environment. In the designed reinforcement learning experiment, the training process uniformly adopts the initial learning rate , learning rate decay factor , fixed learning rate , discount factor , advantage function smoothing factor The TRPO optimization algorithm uses a trust region mechanism and a linear search coefficient =0.5. In addition, in the traffic state recognition and classification part, the FCM algorithm is used to determine the traffic state level and the number of clusters. , fuzzy weight parameter .

[0069] All experiments were conducted on a workstation equipped with an NVIDIA GeForce RTX 4050 (16GB GDDR6X video memory). The experimental platform was Python, and the core algorithm framework was based on PyTorch 1.13.1 and OpenAI Gym. The number of reinforcement learning training episodes was set to 10,000. The reward curve during training was processed using a smoothing window technique with a window size of 31. Training was terminated when the cumulative number of training epochs reached a preset upper limit. All compared algorithms were run in the same experimental environment and hardware platform to ensure fair and comparable experimental results.

[0070] Table 1 shows the initial cluster centers of traffic conditions identified using the improved FCM algorithm. The results show that as traffic conditions deteriorate, vehicle speeds gradually decrease, while traffic density and flow rate significantly increase, fully reflecting the typical congestion characteristics of "low speed, high density" in traffic operation. This classification provides a scientific basis for subsequent targeted fleet rearrangement and energy efficiency optimization under different traffic conditions.

[0071] surface Initial cluster centers

[0072]

[0073] Figure 2 The fitting curve of the time series prediction results of traffic flow, speed and density based on the LSTM model is shown. Figure 3 The traffic status classification results completed by the FCM model are shown in Figure 2. Figure 2 and Figure 3 It can be seen that the LSTM model can accurately depict the dynamic change trend of traffic indicators, and the FCM model clearly and effectively divides the traffic status at future moments, which can provide reliable traffic environment constraint information for queue rescheduling strategies.

[0074] Figure 4 The cumulative reward comparison curves of the queue optimization method of the joint mechanism proposed in this invention and three groups of benchmark algorithms during the training process are shown. As can be seen from the figure, the proposed method shows a higher reward value throughout the training process, and the convergence speed is faster. The fluctuation amplitude during the training process is significantly smaller than that of other algorithms, verifying the good convergence performance and stability of the proposed method in a dynamic traffic environment. In comparison, the genetic algorithm has a good reward performance in the early stages, but the overall convergence performance is insufficient; the Q-Learning algorithm is limited by its exploration ability, and the training reward value is highly volatile and has poor stability, making it difficult to adapt to the queue rearrangement needs in complex environments. This further demonstrates the advantages of this method in terms of energy efficiency balance and optimization strategy robustness.

[0075] Figure 5 and Figure 6The performance difference between the proposed fusion mechanism and the standard TRPO algorithm without congestion constraints in a queue optimization task was demonstrated. The results show that the proposed fusion mechanism combines congestion prediction with deep reinforcement learning optimization, employing TRPO and introducing congestion constraints to reduce unnecessary queue adjustments. The optimization algorithm without congestion constraints, however, fails to consider traffic flow constraints, resulting in frequent reorganizations. While this yields higher returns, it lacks practical feasibility. Experimental results demonstrate that the fusion mechanism achieves a dynamic balance between traffic conditions and queue configuration while maintaining system stability, demonstrating enhanced environmental adaptability and robustness.

[0076] In summary, the proposed method integrates traffic congestion identification and prediction mechanisms, and is superior to traditional methods in optimizing energy consumption distribution and improving strategy feasibility. Compared with the standard TRPO algorithm, it is more in line with the actual traffic environment and has better adaptability and practicality.

[0077] 6. Summary

[0078] The present invention proposes a method for optimizing the queues of intelligent electric vehicle fleets based on congestion prediction and deep reinforcement learning (RDL). The method comprises the following steps: (1) modeling the queue position optimization problem of the electric vehicle fleet as a mathematical model with energy balance as the goal and subject to traffic environment constraints; (2) solving the mathematical model of step (1) using a deep reinforcement learning method based on the Trust Region Policy Optimization (TRPO) algorithm to obtain the optimal queue adjustment strategy for the electric vehicle fleet. In step (2), congestion status information in the external traffic environment is obtained through long short-term memory (LSTM) traffic prediction and fuzzy C-means clustering (FCM) methods. This information, along with the remaining power and accumulated travel distance of each vehicle in the electric vehicle fleet, serves as the state input of the TRPO algorithm. The TRPO algorithm employs a policy network and a value network. The policy network is used to output a strategy for adjusting the queue position, and the value network is used to evaluate the expected return under the current state. The TRPO algorithm dynamically adjusts the policy update step size by constraining the KL divergence of the policy update, ensuring stability and convergence during the policy update process, thereby reducing unnecessary queue changes in a dynamic traffic environment.

Claims

1. A method for optimizing the queue of an intelligent electric vehicle fleet based on congestion prediction and DRL, characterized by the following steps: include: (1) The problem of optimizing the platoon position of electric vehicle fleets is abstracted into a mathematical model with energy balance as the goal and subject to traffic environment constraints; (2) Using deep reinforcement learning (DRL) to solve the mathematical model obtained in step (1) to obtain the optimal platoon optimization strategy for the smart electric vehicle fleet to execute; In step (1), with the goal of minimizing the standard deviation of the remaining power of each vehicle in the electric fleet when it stops, a mathematical model constrained by the traffic environment is established as a platoon optimization model; The traffic congestion identification and prediction module combined with LSTM and FCM is used to dynamically obtain traffic environment constraints. The method is as follows: First, the fuzzy C-means clustering algorithm (FCM) was used in combination with the weighted difference algorithm to optimize the initial cluster centers. The entropy method and the analytic hierarchy process (AHP) were then used to determine the characteristic weights of traffic flow, speed, and density. Traffic conditions were then divided into different congestion levels based on the characteristic weights. The traffic condition classification criteria, congestion threshold, and congestion probability at each moment were obtained. Then, a long short-term memory network model (LSTM) is used to perform time series predictions on traffic flow, speed, and density at future time steps. The prediction results are input into the FCM model to generate the probability of traffic congestion at each future moment. Finally, the congestion probability and congestion threshold are used as traffic environment constraints and input into the queue optimization model to assist in the decision-making process of the fleet queue rearrangement; In step (2), the DRL method is the Trust Region Policy Optimization (TRPO) algorithm; the TRPO algorithm is used to solve the queue optimization model; The TRPO algorithm's policy network receives the current observed state of the queue and outputs the optimal queue position adjustment strategy; the TRPO algorithm's value network estimates the long-term return under the current state to assist in updating and optimizing the policy network. In the Markov decision framework for platoon optimization problems: the agent is the fleet; The observation state is the remaining battery power, cumulative travel distance, and congestion probability of each vehicle in the convoy; the action is the complete queue sequence that all vehicles in the convoy can form; the reward is the scaled value of the standard deviation of the remaining battery power of each vehicle at the end of the convoy. In each TRPO algorithm training episode, the congestion probability of the traffic state is determined at fixed intervals: if the congestion probability is below the congestion threshold, the queue is allowed to change positions; if the congestion probability is above the congestion threshold, the original queue remains unchanged until any vehicle runs out of power, at which point the episode is considered terminated and a reward is given. The TRPO algorithm outputs the queue with the largest cumulative reward as the optimal fleet queue plan in the current period, and uses it as the optimal queue optimization strategy.

2. The intelligent electric vehicle fleet queue optimization method based on congestion prediction and DRL according to claim 1 is characterized in that In step (1), a In a smart fleet of electric vehicles, To distinguish vehicle numbers; express A car in The location reached at the time; On the planned route, select several time intervals of equal length, each time interval is , allowing the fleet to decide whether to change the queue at the end of each interval, Indicates the departure time. Indicates the time when the queue terminates; Indicates the Car in the The relative positions of time intervals; construct a matrix , this matrix is ​​called the queue position transformation matrix; express The remaining power of each vehicle at all times; express The probability of congestion at a given moment, represents the congestion threshold; By extracting As traffic characteristics, it is input into the traffic congestion identification and prediction module to predict the future traffic status, and finally obtain the congestion probability sequence at the future moment. Flow(t), Speed(t) and Density(t) represent the traffic flow, speed and density at time t respectively; Traffic status is divided into four categories: free flow, basic free flow, mild congestion and severe congestion. The corresponding index of the classification standard corresponding to mild congestion is used as the congestion threshold, that is, ; make , represents the least upper bound, express The remaining power of the i-th vehicle at the moment, the platoon planning problem of the electric vehicle fleet is modeled as: ， ， ， ， , ， ， ， ， in, Indicates the standard deviation of the overall remaining power SoC when the fleet ends operation; Indicates that vehicle i is in the time period The amount of electricity consumed; Indicates the average power consumption rate per unit distance, Indicates the vehicle's maximum battery capacity; Indicates that vehicle i is in the previous time period at, in position The power consumption reduction rate.

3. The intelligent electric vehicle fleet queue optimization method based on congestion prediction and DRL according to claim 2 is characterized in that In step (2), The fleet as an intelligent agent; The observed state of the fleet is defined in At time t, the observed state includes the remaining power of each vehicle in the fleet , the cumulative driving distance of each vehicle , and the congestion probability of the traffic environment , that is, the observation state is expressed as: ,in, represents the mixed observation space combining the SoC vectors, cumulative driving distances and congestion probabilities of the electric vehicles in the queue, The dimension of the observation space is dimension; Action space for the fleet All queue sequences that can be formed by electric vehicles; In each scene of TRPO training, each time Time step, as a queue decision triggering condition; in one scene training, judge the congestion probability of the current traffic environment , if it is lower than the preset congestion threshold , the queue position is allowed to change. If the conditions are not met, the current queue remains unchanged until any car runs out of power. The scene is considered terminated and a reward is given. At the end of a training episode, the reward function provides immediate feedback to the vehicle fleet after executing an action, guiding the learning process to optimize the balanced distribution of vehicle energy; the ultimate goal is to minimize the standard deviation of the remaining power when the fleet stops driving; when any vehicle in the fleet runs out of power and enters the termination state, the reward value is calculated based on the standard deviation of the remaining power of the entire vehicle. In the non-termination state, the reward is zero; the final reward , is the reward scaling factor, Represents standard deviation.

4. The intelligent electric vehicle fleet queue optimization method based on congestion prediction and DRL according to claim 3 is characterized by The process of TRPO algorithm is: Step 1) Input the observation state S, including the remaining power, cumulative driving distance and congestion probability of each vehicle in the fleet; Step 2) Calculate the policy function through the policy network and value network respectively and value function ; Step 3) Based on the value function and the collected trajectory data, the generalized advantage estimation (GAE) method is used to calculate the advantage function corresponding to each state action: ; in, represents the advantage function estimate calculated using GAE; represents the discount factor; Represents an additional hyperparameter introduced in GAE; represents the temporal difference, which is the error of the current value function; Represents the immediate reward obtained; V represents the state value function of the current state, and w represents the hyperparameter of the value network; Step 4) Calculate the gradient of the policy objective function , and based on the Hessian matrix Use the conjugate gradient method to solve the parameter update direction ; Step 5) Perform a linear search within the trust region to determine the step size that satisfies the KL constraint , update the policy parameters , and update the value network parameters synchronously: ， In the formula, represents the policy network parameters, represents the step size factor of the i-th linear search, represents the timing error, g represents the gradient vector of the policy objective function, and H represents the Hessian matrix.

5. The intelligent electric vehicle fleet queue optimization method based on congestion prediction and DRL according to claim 4 is characterized by: The dynamic learning rate adjustment strategy of the TRPO algorithm adopts the exponential decay learning rate adjustment method. The adjustment of the learning rate follows the following formula: ， in, represents the updated learning rate, is the initial learning rate, is the attenuation factor, Indicates the current training round.

Images