Ecological driving control method based on queue length under low permeability mixed traffic flow

Abstract

The application relates to the field of intelligent traffic and automatic driving technology and discloses an ecological driving control method based on queue length under a low-permeability mixed traffic flow.The application is aimed at solving the problem of low traffic efficiency and high automobile energy consumption caused by poor control performance of the existing ecological driving control method.The application comprises the following steps: acquiring signal timing information of a signal intersection and running state data of CAVs in a road network;acquiring a space-time trajectory of HDVs by using the running state data of the CAVs in the road network;acquiring a queue length upstream of the intersection based on the space-time trajectory of the HDVs and a CAV trajectory;constructing an action buffer queue;constructing an augmented state vector by using the action buffer queue, the running state data of the vehicles, the queue length upstream of the intersection and the signal timing information of the signal intersection;inputting the augmented state vector into a trained DA-TD3 model; and obtaining a required longitudinal acceleration of the vehicles.The application is used for controlling the acceleration of the vehicles so as to shorten the queue time of the vehicles.

Description

Technical Field

[0001] This invention relates to the fields of intelligent transportation and autonomous driving technology, and in particular to an eco-driving control method based on queue length in low-penetration mixed traffic flows. Background Technology

[0002] With the development of connected and autonomous vehicle (CAV) technology, road traffic systems are transitioning from traditional human-driven vehicles (HDVs) to fully intelligent and connected systems. During this phase, the road environment is characterized by low-penetration mixed traffic flow, meaning that only a small number of CAVs coexist with a large number of HDVs. "Low penetration" refers to a situation where connected and autonomous vehicles (CAVs) account for less than 40% of all vehicles.

[0003] Existing eco-driving strategies primarily focus on speed guidance and energy-saving control at signalized intersections. These methods assume real-time and reliable vehicle communication; however, in real-world low-penetration environments, wireless communication is susceptible to interference and typically suffers from communication latency (usually 0-1 seconds). This latency leads to delays in the acquisition of information such as the preceding vehicle's status and traffic light phases by the vehicle-to-vehicle (CAV), causing delayed decision-making and resulting in increased speed fluctuations in the vehicle queue, amplified disturbances, and decreased traffic flow stability. Ultimately, this leads to inaccurate eco-driving control and increased vehicle energy consumption. Furthermore, existing methods either rely on an ideal full-observation assumption or use simple probabilistic models to estimate the queue length upstream of the intersection. However, in low-penetration environments, only a small amount of CAV trajectory data is available in real time, while the operational status (position, speed, acceleration) of a large number of high-speed vehicles (HDVs) is missing. Under sparse trajectory data conditions, it is difficult to accurately reconstruct vehicle trajectories and estimate queue lengths. This makes eco-driving strategies prone to inefficient behaviors such as sudden deceleration and frequent stop-and-go operations when facing real-world congestion queues, further increasing vehicle energy consumption. Traditional model-based control methods or basic reinforcement learning methods do not consider the two core constraints of "communication latency" and "queue length" that are coupled in low-penetration environments. Therefore, when faced with real-world scenarios with latency and information gaps, the control strategies provided by existing methods have poor control performance, resulting in low traffic efficiency and high vehicle energy consumption. Summary of the Invention

[0004] This invention addresses the problems of low traffic efficiency and high vehicle energy consumption caused by poor control performance in existing eco-driving control methods, and proposes an eco-driving control method based on queue length under low-penetration mixed traffic flow.

[0005] An eco-driving control method based on queue length for low-permeability mixed traffic flows includes the following steps:

[0006] S1. Obtain signal timing information of signalized intersections and operating status data of CAVs in the road network;

[0007] The signal timing information at the signalized intersection includes: the current phase and the remaining time;

[0008] The CAV's operational status data includes: vehicle ID, location, speed, and acceleration;

[0009] S2. Obtain the spatiotemporal trajectory of the HDV using the operational status data of the CAV in the road network, and obtain the upstream queue length of the intersection based on the spatiotemporal trajectory of the HDV and the CAV trajectory. ;

[0010] S3. Construct an action buffer queue, utilizing the action buffer queue, vehicle operating status data, and upstream queue length at the intersection. The signal timing information of the signalized intersection is used to construct an augmented state vector. The augmented state vector is then input into the trained DA-TD3 model to obtain the longitudinal acceleration required by the vehicle.

[0011] Furthermore, in step S2, the spatiotemporal trajectory of the HDV is obtained using the operating status data of the CAV in the road network, and the upstream queue length of the intersection is obtained based on the spatiotemporal trajectory of the HDV and the CAV trajectory. Specifically:

[0012] S201. Obtain the historical speed difference value, historical acceleration difference value, and historical longitudinal displacement increment of CAVs in the queue using the operating status data of CAVs in the road network. Obtain the longitudinal displacement increment of each HDV using the historical speed difference value, historical acceleration difference value, and historical longitudinal displacement increment of CAVs in the queue.

[0013] The longitudinal displacement increment of each HDV is obtained using the historical velocity difference, historical acceleration difference, and historical longitudinal displacement increment of the CAVs in the queue. Specifically:

[0014] Obtain the CAV closest to HDV. If HDV is before the CAV closest to HDV, perform forward reconstruction to obtain the vertical distance increment of HDV. If HDV is after the CAV closest to HDV, perform backward reconstruction to obtain the vertical distance increment of HDV.

[0015] S202. Obtain the absolute position and speed of each vehicle using the longitudinal displacement increment and CAV trajectory. Determine all vehicles with speeds below a preset threshold as queued vehicles. Use the distance between the queued vehicle furthest from the stop line and the stop line as the current queue length upstream of the intersection. .

[0016] Furthermore, if the HDV is ahead of the nearest CAV, forward reconstruction is performed to obtain the vertical distance increment of the HDV, specifically:

[0017] If the CAV closest to the HDV is the first CAV in the current queue, then the number of HDVs N' between the CAV and the stop line is obtained using the distance from the CAV to the stop line, the preset safety distance, and the preset vehicle length. If the CAV closest to the HDV is not the first CAV in the current queue, then the distance between the current CAV and the previous CAV in front of the queue is obtained, and the number of HDVs N' between the two CAVs is obtained using the distance between the two CAVs, the preset safety distance, and the preset vehicle length.

[0018] Let CAV be denoted as vehicle N, and the nearest HDV in front of CAV be denoted as vehicle N-1. Input the historical speed difference, historical acceleration difference, and historical longitudinal displacement increment of vehicle N into the TCN-Transformer cascade model to obtain the longitudinal displacement increment of vehicle N-1. Repeat the prediction N' times to obtain the longitudinal distance increment of HDV behind CAV.

[0019] Furthermore, if the HDV is located after the nearest CAV, backward reconstruction is performed to obtain the vertical distance increment of the HDV, specifically:

[0020] If the current CAV is not the last CAV in the current queue, then obtain the distance between the current CAV and the next CAV after it in the queue. Using the distance between the two CAVs, the preset safety distance, and the preset vehicle length, obtain the number of HDVs N' between the two CAVs. If the current CAV is the last CAV in the current queue, then obtain the number of HDVs N' after the CAV.

[0021] Let CAV be denoted as vehicle N, and the nearest HDV behind CAV be denoted as vehicle N+1. Input the historical velocity difference, historical acceleration difference, and historical longitudinal displacement increment of vehicle N into the TCN-Transformer cascade model to obtain the longitudinal displacement increment of vehicle N+1. Repeat the prediction N' times to obtain the longitudinal distance increment of HDV behind CAV.

[0022] Furthermore, in S202, the absolute position and velocity of each vehicle are obtained using the longitudinal displacement increment and CAV trajectory of each vehicle, specifically as follows:

[0023] First, the absolute position of the CAV at each time step is obtained using the CAV trajectory; then, based on the distance between two adjacent vehicles, the initial absolute position of each HDV is obtained; finally, based on the initial position of the HDV, the longitudinal displacement increment at each time step is superimposed to obtain the absolute position of the HDV at each time step.

[0024] Then, the absolute position difference of HDV between every two adjacent time points is obtained, and the HDV velocity at each time point is obtained by dividing the absolute position difference of HDV by the sampling interval between adjacent time points.

[0025] Furthermore, in S3, the action buffer queue is constructed using the action buffer queue, vehicle operating status data, and upstream queue length at the intersection. The signal timing information of the signalized intersection is used to construct an augmented state vector. This augmented state vector is then input into the trained DA-TD3 model to obtain the required longitudinal acceleration for the vehicle. Specifically:

[0026] S301. Discretize the continuous time to obtain the time delay step. The initial length is First-in-first-out action buffer queue ;

[0027] In the initial state Each element in the array is 0. It is the first in the action buffer queue One action;

[0028] S302, utilizing action buffer queues, vehicle operating status, signal timing information at signalized intersections, and upstream queue lengths at intersections. Composition of augmented state vectors ,Will Input the trained DA-TD3 model to obtain the required longitudinal acceleration of the vehicle, store the required longitudinal acceleration at the end of the action buffer queue, remove it from the action buffer queue, and execute the action. This keeps the action buffer queue length constant. Reorder the updated action buffer queue elements from 1 to... Label;

[0029] in, It is a vehicle exist The speed of time, It is a vehicle exist acceleration at any moment It is the current distance of the vehicle to the stop line. It is a vehicle Distance from the front of the vehicle in front, It is a vehicle Relative speed to the vehicle in front, yes The phase of the signal at any moment, yes Remaining time for phase at any given moment It is an action buffer queue.

[0030] Furthermore, in step S301, the continuous time is discretized to obtain the time delay step. Specifically:

[0031]

[0032] in, It is the time delay step. It controls the step size. It is a continuous delay.

[0033] Furthermore, the trained DA-TD3 model is obtained in the following way:

[0034] Constructing augmented state vectors , augment the state vector Input the DA-TD3 model for reinforcement learning to obtain a trained DA-TD3 model, specifically:

[0035] A1. Construct an action buffer queue, and build an augmented state vector based on the action buffer queue. Then construct the DA-TD3 model;

[0036] The DA-TD3 model includes: an Actor network. and Critic network;

[0037] The Actor network Used based on the augmented state vector Output acceleration ;

[0038] The Critic network includes: a value network and value network ;

[0039] The Critic network is used to evaluate the value of state-action pairs;

[0040] A2. Augment the state vector Input the DA-TD3 model to obtain the required longitudinal acceleration of the vehicle. Perform acceleration action Obtain the state at the next moment. and rewards , experience Store the experience playback, and then based on the vehicle's required longitudinal acceleration. Update the action buffer queue.

[0041] Furthermore, the reward Specifically:

[0042]

[0043]

[0044]

[0045]

[0046]

[0047]

[0048] in, , , , These are weighting coefficients. It is a security reward value. It's the green light passage bonus value. It is a moment Time required to reach the stop line This is the remaining passable time for the current phase. It is a comfort indicator. It is a vehicle exist acceleration at any moment It is a comfort bonus value. It is the penalty coefficient. It is the maximum value of jerk. It is a vehicle exist acceleration at any moment It is the absolute value symbol. It is a traffic efficiency bonus value. That is the maximum speed. It is the minimum speed. It is the expected speed. It is normalization bias.

[0049] Furthermore, security reward value Specifically:

[0050]

[0051]

[0052]

[0053]

[0054] in, It's a penalty value. It's a strong penalty value. , It is a constant. It is a safety threshold. yes The remaining time until the collision. It is a vehicle Distance from the front of the vehicle in front, It is a vehicle exist The speed of time, It is a vehicle exist The speed of time, It is a function that takes the maximum value.

[0055] The beneficial effects of this invention are as follows:

[0056] This invention addresses the challenges of low-penetration mixed traffic scenarios, where only a small number of CAV trajectories are observable while a large number of HDV trajectories are missing, compounded by observation lag caused by communication latency. It proposes an eco-driving control method based on queue length for low-penetration mixed traffic flows. This method extracts local spatiotemporal features of vehicle motion using a TCN-Transformer cascaded model, capturing short-term velocity and acceleration variations, and then models long-term dependencies to complete the missing trajectories in both forward and backward directions. Even with limited CAV trajectory data, it accurately completes the spatiotemporal trajectories of HDVs, obtaining reconstructed trajectories. Based on these reconstructed trajectories, it accurately estimates the queue length at intersections. This invention overcomes the bottleneck of insufficient trajectory completion accuracy in low-penetration, latency-coupled scenarios, providing reliable queue state input for subsequent eco-driving control. Furthermore, this invention constructs an action buffer queue to augment the state space, successfully incorporating the impact of communication latency into decision-making considerations and restoring the Markov property of the process. This invention takes into account both communication delay and queue length, and uses the DA-TD3 model to obtain the acceleration required for vehicle passage, avoiding frequent vehicle starts and stops, improving the control performance of the control strategy, thereby improving traffic efficiency, reducing vehicle energy consumption, and enhancing driving comfort. Attached Figure Description

[0057] Figure 1 This is a flowchart illustrating the overall technical route of the present invention;

[0058] Figure 2 A trajectory reconstruction architecture diagram for the TCN-Transformer cascaded model;

[0059] Figure 3 Here is a diagram of the ecological driving strategy structure of the DA-TD3 model;

[0060] Figure 4(a) is a schematic diagram comparing the spatiotemporal trajectories of vehicles using the method of the present invention and the DA-DQN method in the same scenario;

[0061] Figure 4(b) is a schematic diagram comparing the spatiotemporal trajectories of vehicles using the method of the present invention and the DA-DDPG method in the same scenario;

[0062] Figure 4(c) is a schematic diagram comparing the spatiotemporal trajectories of vehicles in the same scenario with the method of the present invention and the DA-TD3 method. Detailed Implementation

[0063] Specific implementation method one: as follows Figure 1 As shown, the specific process of the ecological driving control method based on queue length under low-permeability mixed traffic flow in this embodiment is as follows:

[0064] S1. Obtain the signal timing information of the signalized intersection and the operating status data of the CAV in the road network, specifically:

[0065] First, signal timing information of signalized intersections is acquired at a fixed frequency through roadside units (RSU) and on-board units (OBU);

[0066] The signal timing information at the signalized intersection includes: the current phase and the remaining time;

[0067] Then, acquire the operational status data of a small number of observable CAVs in the road network;

[0068] The CAV's operational status data includes: vehicle ID, location, speed, and acceleration;

[0069] S2. Obtain the spatiotemporal trajectory of unobservable HDVs using the operational status data of CAVs in the road network, and obtain the upstream queue length of the intersection based on the spatiotemporal trajectory of the HDVs and the CAV trajectories. Specifically:

[0070] S201. Obtain the longitudinal displacement increment of each HDV using the historical velocity difference value, historical acceleration difference value, and historical longitudinal displacement increment of the CAVs in the queue, specifically:

[0071] Obtain the CAV closest to HDV. If HDV is before the CAV closest to HDV, perform forward reconstruction to obtain the vertical distance increment of HDV. If HDV is after the CAV closest to HDV, perform backward reconstruction to obtain the vertical distance increment of HDV.

[0072] If the HDV is ahead of the nearest CAV, then forward reconstruction is performed to obtain the vertical distance increment of the HDV, specifically:

[0073] If the CAV closest to the HDV is the first CAV in the current queue, then the number of HDVs N' between the CAV and the stop line is obtained using the distance from the CAV to the stop line, the preset safety distance, and the preset vehicle length. If the CAV closest to the HDV is not the first CAV in the current queue, then the distance between the current CAV and the previous CAV in front of the queue is obtained, and the number of HDVs N' between the two CAVs is obtained using the distance between the two CAVs, the preset safety distance, and the preset vehicle length.

[0074] Let CAV be the Nth vehicle, and the nearest HDV in front of CAV be the N-1th vehicle. Input the historical velocity difference, historical acceleration difference, and historical longitudinal displacement increment of the Nth vehicle into the TCN-Transformer cascade model to obtain the longitudinal displacement increment of the N-1th vehicle. Repeat this process N' times to obtain the longitudinal distance increment of the HDV behind CAV.

[0075] The historical velocity difference, historical acceleration difference, and historical longitudinal displacement increment of vehicle N are input into the TCN-Transformer cascade model to obtain the longitudinal displacement increment of vehicle N-1, specifically:

[0076] The historical velocity difference and historical acceleration difference of vehicle N are input into the temporal convolutional network (TCN) module in the TCN-Transformer cascaded model to obtain the velocity difference and acceleration difference of vehicle N-1. The velocity difference, acceleration difference, and historical longitudinal displacement increment of vehicle N-1 are input into the Transformer encoder-decoder module to obtain the longitudinal displacement increment of vehicle N-1.

[0077] If the HDV is after the nearest CAV, then backward reconstruction is performed to obtain the vertical distance increment of the HDV, specifically:

[0078] If the current CAV is not the last CAV in the current queue, then obtain the distance between the current CAV and the next CAV after it in the queue. Using the distance between the two CAVs, the preset safety distance, and the preset vehicle length, obtain the number of HDVs N' between the two CAVs. If the current CAV is the last CAV in the current queue, then obtain the number of HDVs N' after the CAV.

[0079] Let CAV be the Nth vehicle, and the nearest HDV behind CAV be the N+1th vehicle. Input the historical velocity difference, historical acceleration difference, and historical longitudinal displacement increment of the Nth vehicle into the TCN-Transformer cascade model to obtain the longitudinal displacement increment of the N+1th vehicle. Repeat this process N' times to obtain the longitudinal distance increment of the HDV behind CAV.

[0080] The process involves inputting the historical velocity difference, historical acceleration difference, and historical longitudinal displacement increment of the Nth vehicle into the TCN-Transformer cascade model to obtain the longitudinal displacement increment of the (N+1)th vehicle. Specifically:

[0081] The historical velocity difference and historical acceleration difference of vehicle N are input into the temporal convolutional network (TCN) module in the TCN-Transformer cascaded model to obtain the velocity difference and acceleration difference of vehicle N+1. The velocity difference, acceleration difference, and historical longitudinal displacement increment of vehicle N+1 are input into the Transformer encoder-decoder module to obtain the longitudinal displacement increment of vehicle N+1.

[0082] The estimated number N' of HDV after CAV is achieved in any of the following ways:

[0083] a. Obtain the total number of vehicles on the road and the number of CAVs within a preset time period. Divide the number of CAVs by the total number of vehicles to obtain the CAV penetration rate. Then, obtain the number of CAVs in the queue. Divide the number of CAVs by the CAV penetration rate to obtain the total number of HDVs in the queue. Subtract the number of HDVs before CAVs in the queue from the total number of HDVs to obtain the number of HDVs after CAVs.

[0084] b. Input the historical traffic state time series matrix into the trained queue length prediction model to obtain the current queue length prediction value. Subtract the distance between the CAV and the stop line from the current queue length prediction value to obtain the HDV queue length. Divide the HDV queue length by the sum of the vehicle preset length and preset spacing to obtain the number of HDVs after the CAV.

[0085] The historical traffic state time series matrix is ​​obtained through the following method:

[0086] b1. Using the road stop line as a reference point, the fixed space before and after the single lane where the CAV is located is taken as the collection interval. The historical traffic state feature data of the collection interval is obtained. A fixed length sliding time window is set, and the continuous historical traffic state feature data is divided frame by frame along the time axis. The historical traffic state feature data in each sliding event window is taken as a sample.

[0087] Historical traffic condition data includes vehicle-level features and road segment-level features.

[0088] The vehicle-level features include: the speed, acceleration, headway, and spatial position changes of the CAV and its preceding and following vehicles;

[0089] The road segment-level features include information such as traffic flow density, average speed, queue occupancy, and signal phase status.

[0090] b2. Stack the samples in chronological order to obtain the traffic state time series matrix;

[0091] The trained queue length prediction model is obtained through the following method:

[0092] History The traffic state time series matrix corresponding to each window and The queue lengths corresponding to the windows form a training set. The training set is used to train a nonlinear model to obtain a well-trained queue length prediction model.

[0093] like Figure 2 As shown, the TCN-Transformer cascaded model includes: a Temporal Convolutional Network (TCN) module and a Transformer encoder-decoder module;

[0094] The Temporal Convolutional Network (TCN) module uses dilated convolution to extract short-term local dynamic features of vehicle motion, capturing detailed information such as second-level speed fluctuations and sudden acceleration / deceleration changes to obtain local features;

[0095] The input to the Transformer encoder-decoder module is the local features output by the Temporal Convolutional Network (TCN) module. It uses a self-attention mechanism to model the global dependencies of long-term series and generates the future displacement increment sequence within the prediction window in an autoregressive manner.

[0096] In this step, the present invention employs a cascaded structure of TCN and Transformer, rather than a single model or other combinations such as CNN-LSTM. This is because: TCN excels at extracting short-term local dynamic features of vehicle motion, but has limited ability to model global dependencies over long time series; Transformer excels at capturing long-term dependencies, but is insensitive to subtle local motion features and is prone to overfitting in low-penetration, small-sample, sparse data. The present invention, through a cascaded design, allows the local features extracted by TCN to be directly used as input to the Transformer encoder, forming a hierarchical feature extraction chain from "local" to "global," preserving short-term motion details while achieving accurate completion of long-term missing trajectories. In coupled scenarios with low penetration rates of 10%–30%, sparse trajectories, and communication delays, this structure significantly outperforms a single TCN or single Transformer model in completion accuracy.

[0097] S202. Obtain the absolute position and speed of each vehicle using the longitudinal displacement increment and CAV trajectory. Identify all vehicles with speeds below a preset threshold (e.g., 5 km / h) as queued vehicles. Use the distance between the queued vehicle furthest from the stop line and the stop line as the current queue length upstream of the intersection. Specifically:

[0098] First, the absolute position of the CAV at each moment is obtained using the CAV trajectory; then, the initial absolute position of the adjacent HDV is obtained by combining the distance between the two vehicles; finally, based on the initial position of the HDV, the longitudinal displacement increment at each moment is superimposed to obtain the absolute position of the HDV at each moment.

[0099] Then, the absolute position difference of HDV between every two adjacent time points is obtained, and the velocity of HDV at each time point is obtained by dividing the absolute position difference of HDV by the sampling interval between adjacent time points; the position and velocity of CAV can be obtained directly.

[0100] Finally, all vehicles with speeds below a preset threshold (e.g., 5 km / h) are identified as queuing vehicles, and the distance between the queuing vehicle furthest from the stop line and the stop line is taken as the current upstream queue length at the intersection. ;

[0101] The speed threshold in this step can be dynamically adjusted based on road speed limits, traffic flow conditions, or historical data. For example, it can be set to 10% to 20% of the free-flow speed, rather than fixed at 5 km / h, to adapt to queuing determination requirements in different scenarios.

[0102] S3. Construct an action buffer queue, utilizing the action buffer queue, vehicle operating status data, and upstream queue length at the intersection. The signal timing information of the signalized intersection is used to construct an augmented state vector. This augmented state vector is then input into the trained DA-TD3 model to obtain the required longitudinal acceleration for the vehicle. Specifically:

[0103] S301. Discretize the continuous time to obtain the time delay step. The initial length is First-in-first-out action buffer queue ;

[0104] In the initial state Each element in the array is 0;

[0105] The action buffer queue is used to store action sequences that have not yet been executed by the environment and are yet to take effect, representing the currently active control actions. The first element of the queue The step size for obtaining the time delay is obtained by discretizing the continuous time. Specifically:

[0106]

[0107] in, It is the time delay step. It controls the step size. It is a continuous delay. It is the first in the action buffer queue One action;

[0108] S302, utilizing action buffer queues, vehicle operating status, signal timing information at signalized intersections, and upstream queue lengths at intersections. Composition of augmented state vectors ,Will Input the trained DA-TD3 model to obtain the required longitudinal acceleration of the vehicle and execute the action. Update the action buffer queue based on the longitudinal acceleration required by the vehicle;

[0109] in, It is a vehicle exist The speed of time, It is a vehicle exist acceleration at any moment It is the current distance of the vehicle to the stop line. It is a vehicle Distance from the front of the vehicle in front, It is a vehicle Relative speed to the vehicle in front, yes The phase of the signal at any moment, yes Remaining time for phase at any given moment It is an action buffer queue;

[0110] The action buffer queue is updated based on the vehicle's required longitudinal acceleration, specifically as follows:

[0111] Will Remove from the motion buffer queue, store the required longitudinal acceleration of the vehicle at the end of the motion buffer queue, and maintain the length of the motion buffer queue at [value missing]. Reorder the updated action buffer queue elements from 1 to... The label, that is, the first element of the new queue is the second element of the original queue, and the subsequent elements move forward in sequence, with the tail of the queue being the longitudinal acceleration required by the vehicle.

[0112] Existing methods for addressing communication latency mainly fall into three categories: directly ignoring latency and using outdated information as the current state, which leads to control lag and oscillations; fixed latency compensation, which simply advances the observed state by a fixed step, but suffers from severe error accumulation under time-varying or random latency; and compensation based on prediction models, which uses neural networks to predict the current true state, but the prediction error amplifies with the prediction length and is difficult to optimize jointly with downstream policies. Existing technologies do not fundamentally solve the problem that latency violates the "fully observable state" assumption of Markov Decision Processes (MDPs). Therefore, this invention proposes an explicit modeling method using an action buffer queue and augmented states. Instead of attempting to predict or compensate for latency, it treats historical actions with latency as part of the state, allowing the decision process to re-satisfy Markov properties in the augmented state space. This method has unique advantages such as independence from the underlying prediction model, insensitivity to latency distribution, and the ability to be jointly trained with reinforcement learning policies.

[0113] Specific implementation method two: such as Figure 3 As shown, this implementation differs from specific implementation one in that the trained DA-TD3 model is obtained in the following way:

[0114] Constructing augmented state vectors , augment the state vector Input the DA-TD3 model for reinforcement learning to obtain a trained DA-TD3 model, specifically:

[0115] A1. Construct an action buffer queue, and build an augmented state vector based on the action buffer queue. Then, the DA-TD3 model is constructed, specifically as follows:

[0116] A1.1. Discretize the continuous time to obtain the time delay step. The initial length is First-in-first-out action buffer queue ;

[0117] A1.2 Construct an augmented state vector using the vehicle's speed, acceleration, distance from the current vehicle to the stop line, distance between the vehicle and the vehicle in front, relative speed between the vehicle and the vehicle in front, signal phase, phase remainder time, and action buffer queue. ;

[0118] A1.3 Construct the DA-TD3 model;

[0119] The DA-TD3 model includes: an Actor network. and Critic network;

[0120] The Actor network Used based on the augmented state vector Output acceleration ;

[0121] The objective function of the Actor network is:

[0122]

[0123] in, It is the objective function value of the Actor network. It is an experience replay pool;

[0124] The Critic network includes: a value network and value network The Critic network is used to evaluate the value of state-action pairs and take the minimum of the two to suppress overestimation.

[0125] To reduce the instability caused by the fluctuation of the Critic target value during parameter iteration, DA-TD3 adopts a soft update mechanism to smoothly evolve the target network parameters. The update rule is as follows:

[0126]

[0127] in, It is a value network identifier. It is the first The parameters of a value network It is a smoothing parameter. These are the parameters of the Actor network. These are the target network parameters for the Actor;

[0128] A2. Augment the state vector Input the DA-TD3 model to obtain the required longitudinal acceleration of the vehicle. Perform acceleration action Obtain the state at the next moment. and rewards , experience Store the experience playback, and then based on the vehicle's required longitudinal acceleration. Update the action buffer queue;

[0129] The required longitudinal acceleration of the vehicle Updating the action buffer queue specifically involves: adding actions to the end of the action buffer queue, and... Remove from the action buffer queue, so that the number of elements in the action buffer queue is [number missing]. Reorder the elements in the new action buffer queue according to 1 to 2. Label;

[0130] In this implementation, the Critic network is updated by minimizing the temporal difference error, while the Actor network and its target network employ a delayed update strategy to improve training stability. This invention constructs a Delay-Aware Markov Decision Process (DAMDP), which explicitly models the impact of communication delay on the system state by introducing an action buffer queue. This overcomes the assumption of "fully observable environment" in traditional MDP models, achieving robustness of decision-making under time-delay conditions. In the augmented state space, the Double-Delay Deep Deterministic Policy Gradient (TD3) algorithm is used for policy learning to generate continuous control actions.

[0131] The other steps and parameters are the same as in Specific Implementation Method 1.

[0132] Specific Implementation Method Three: This implementation method differs from Specific Implementation Method One or Two in that the reward... Obtained through the following methods:

[0133] B1. Obtain security reward points :

[0134] B1.1 Obtain the remaining time to collision (TTC), specifically:

[0135] Following vehicles requires prioritizing safety and avoiding collisions. Therefore, Time to Collision (TTC) serves as a safety metric. TTC represents the remaining time required for a rear-end collision to occur between the following and preceding vehicles, assuming the current speed remains constant. TTC has a meaningful finite value only when the following vehicle's speed is greater than the preceding vehicle's speed, indicating a potential risk of a rear-end collision. When the following vehicle's speed is no greater than the preceding vehicle's speed, the distance between the two vehicles either increases or remains constant, eliminating the risk of a rear-end collision; therefore, TTC is defined as positive infinity. Consequently, statistics on the proportion of vehicles with a TTC < 4 seconds only apply to those meeting the safety standard. The time step is the moment when there is a potential risk of rear-end collision. This proportion reflects the percentage of times a situation is in a dangerous state out of all moments with potential collision risk. The formula is:

[0136]

[0137] in, yes The remaining time until the collision. It is a vehicle Distance from the front of the vehicle in front, It is a vehicle exist The speed of time, It is a vehicle exist The speed of time;

[0138] B1.2, Utilize the remaining time TTC after the collision to obtain the penalty value. and strong penalty value Specifically:

[0139]

[0140]

[0141] in, It is a safety threshold. It's a penalty value. It's a strong penalty value. , Take 1.0 and 5.0 respectively. It is a function that takes the maximum value;

[0142] B1.3, Utilizing Penalty Values and strong penalty value Obtain security reward points:

[0143]

[0144] in, It is a security reward value.

[0145] B2. Obtaining Green Light Passage Bonus Value :

[0146] To guide vehicles through the intersection with minimal stops within the permitted signal phase time window, a reward is constructed based on the phase-allowable window. Positive rewards will be given when the time is right, and penalties will be imposed otherwise, as follows:

[0147]

[0148] in, Indicates at time The time required to reach the stop line is predicted based on the current state. Indicates the remaining passable time for the current phase. This is the reward value for passing through a green light.

[0149] In this step, the current spatial distance between the vehicle and the stop line is divided by the current instantaneous speed to obtain the predicted time required to reach the stop line.

[0150] B3. Obtain comfort reward points, specifically:

[0151] First, in actual driving, frequent or drastic acceleration changes not only cause vehicle jolting but may also lead to passenger discomfort or even motion sickness. Therefore, the derivative of acceleration, jerk, is set as a comfort index, and its calculation formula is:

[0152]

[0153] in, It is a comfort indicator. It is a vehicle exist Acceleration at any moment;

[0154] Then, the comfort reward value is obtained using the comfort index, specifically as follows:

[0155]

[0156] in, It is a comfort bonus value. This is the penalty coefficient, set to 1.0. It is the maximum value of jerk. It is the absolute value symbol;

[0157] B4. Obtain traffic efficiency bonus points, specifically:

[0158] First, speed-based traffic efficiency rewards. The design aims to incentivize CAVs to pass through signalized intersections at the highest possible speed without stopping, while ensuring safety. A desired speed is set to maintain reasonable driving efficiency. And penalize the degree of deviation from the expected speed. Obtain the normalized bias:

[0159]

[0160] in, That is the maximum speed. It is the minimum speed. It is the expected speed. This is normalization bias;

[0161] Then, using the normalized bias Obtain traffic efficiency bonus points:

[0162]

[0163] in, It is the traffic efficiency bonus value;

[0164] B5. Obtain rewards using safety bonus points, green light passability bonus points, comfort bonus points, and traffic efficiency bonus points. Specifically:

[0165]

[0166] in, , , , The weighting coefficients for each objective are 0.5, 0.5, 0.3, and 0.8, respectively.

[0167] The other steps and parameters are the same as in specific implementation method one or two.

[0168] Example: To verify the beneficial effects of the present invention, the following experiments were conducted:

[0169] Implementation scenario setup:

[0170] A simulation platform was built using SUMO microscopic traffic simulation software and Python. The simulation scenario was a typical two-way, two-lane signalized intersection, 500 meters long, with a speed limit of 50 km / h and a signal cycle of 100 seconds (40 seconds for green, 55 seconds for red, and 5 seconds for yellow). The CAV penetration rate was set to 30%, and the communication delay was fixed at 0.4 seconds. It should be noted that this invention does not rely on the assumption of fixed delay. For scenarios with random or time-varying delays, the length of the action buffer queue is simply set to the step size corresponding to the maximum possible delay, and the actual observed delay distribution is implicitly expressed through the queue state. For example, when the delay varies randomly between 0 and 0.6 seconds, K=6 (control step size 0.1 seconds) is sufficient for adaptation. Simulation verification shows that this invention can maintain stable control performance under random delays. (Control step size...) Set it to 0.1 seconds.

[0171] Implementation steps:

[0172] 1. Initialization: Load the road network and traffic demand (e.g., 900 veh / h) into SUMO, and randomly distribute 30% of the CAV. Initialize all network parameters of the TCN-Transformer model and DA-TD3 model in Python, and initialize the experience replay pool capacity to 300,000.

[0173] 2. Trajectory Reconstruction and Queue Length Estimation:

[0174] Within a simulation step, acquire the ID, position, and velocity information of all CAVs.

[0175] Suppose the first vehicle in a queue is an HDV, and the system identifies the first observable CAV. The velocity difference and acceleration difference data of this CAV over the past second (10 steps) are input into the pre-trained TCN-Transformer model.

[0176] The forward reconstruction branch of the model outputs the displacement increment of the HDV in front of the CAV in the next second (10 steps). By accumulating the displacement increment, the complete spatiotemporal trajectory of the HDV in front is recovered.

[0177] Based on the position and speed of all vehicles (observed CAV + reconstructed HDV), vehicles with speeds below 1.39 m / s (5 km / h) are identified as queuing vehicles, and the queue length is calculated. .

[0178] 3. Construct augmenting states:

[0179] Get the controlled CAV's own state: current speed acceleration Distance to the stop line Distance from the front of the vehicle relative speed .

[0180] Signal status: Current phase is green, remaining time .

[0181] Obtain the current queue length from step S2. .

[0182] Constructing an action buffer queue: Assuming a delay step size Buffer queue (unit ).

[0183] Combine all the above data into a multidimensional vector. And normalize.

[0184] 4. DA-TD3 Decision and Control (Specific Example of Step S4):

[0185] Action generation: augmenting state The input is fed into the Actor network, which outputs a value between [-1, 1], which is then linearly transformed to the acceleration range. Assuming the output action .

[0186] Action in effect: The valid action at the current moment. The first element of the buffer queue is 0.1. The environment executes this acceleration, and the CAV state is updated. Simultaneously, new actions are... Push it to the end of the queue, and the queue is updated. Reward Calculation: After the environment performs an action, the reward is calculated based on the new state. Assuming no collision occurs, It is predicted that vehicles can pass through when the light is green. ;Calculate if the jerk value is less than the threshold, The current speed is close to the desired speed. Assuming all weights are 1, then the total reward is... .

[0187] Experience storage and learning: ( , , , The samples are stored in the replay pool. A batch (e.g., 256 samples) is randomly sampled from the replay pool, and the Critic network is updated using gradient descent. After every two Critic updates, the Actor network and its target network are updated again (soft update, coefficients...). ).

[0188] Implementation results:

[0189] After thousands of episodes of training, the deployed DA-TD3 strategy exhibits superior performance. Figures 4(a)-4(c) show that, under the same traffic scenario, the DA-TD3 method, compared to DA-DDPG and DA-DQN, demonstrates a smoother vehicle spatiotemporal trajectory upstream of the intersection, with no significant abrupt deceleration or long queues. Performance metrics show that, compared to strategies that do not consider queue length, the present invention, by incorporating queue length information, increases the average vehicle speed by approximately 8.05%, reduces average delay by approximately 3.07%, and reduces fuel consumption by approximately 2.16%. The present invention maintains good control performance under different communication delays (0-1s) and different traffic flows (600-1200 veh / h), demonstrating its robustness and generalization ability.

[0190] Experiments show that the method of this invention can work stably under CAV penetration rates of 10%–30% and communication latency of 0–1s. Compared with traditional methods, the trajectory reconstruction error is reduced by 10%–24%, and the mean absolute percentage error (MAPE) of queue length estimation is reduced from 25.69% to 9.77%. After introducing queue length, the average vehicle speed increases by 4.33%–12.09%, the average delay decreases by 2.35%–3.07%, fuel consumption decreases by 0.80%–2.16%, and the root mean square acceleration decreases by 5.43%–6.66%, significantly improving overall ecological driving performance. Under different penetration rates (10%–30%) and communication latency (0–1s), the DA-TD3 strategy of this invention can effectively suppress speed fluctuations compared with existing DDPG and DQN methods. For example, at a 30% penetration rate and a 0.4s latency, the root mean square acceleration of the vehicle is reduced by more than 15%, and fuel consumption is reduced by 2–10%, demonstrating stronger anti-latency interference capability. Compared to traditional 1D-CNN-LSTM models, this invention reduces the root mean square error (RMSE) of trajectory reconstruction by 10%-24% and the mean absolute percentage error (MAPE) of queue length estimation from 25.69% (10% penetration rate) to 9.77% (30% penetration rate). By incorporating the estimated queue length into the state space, vehicles can "sense" unseen congestion queues ahead, thus adjusting their speed in advance and smoothly. In typical signalized intersection scenarios, using this method increases the average vehicle speed by 4%-12%, reduces average delay by 2%-4%, and reduces fuel consumption by 1%-2.5% (the effect is even better after considering queue length information). Furthermore, the speed and acceleration distribution is more concentrated, and the acceleration and deceleration processes are smoother, significantly improving the driving experience. Under different traffic flow rates (600-1200 veh / h), this method demonstrates superior trajectory smoothness and operational continuity compared to existing technologies, exhibiting strong scenario generalization ability.

Claims

1. An eco-driving control method based on queue length under low-permeability mixed traffic flow, characterized in that... The specific process of the method is as follows: S1. Obtain signal timing information of signalized intersections and operating status data of CAVs in the road network; The signal timing information at the signalized intersection includes: the current phase and the remaining time; The CAV's operational status data includes: vehicle ID, location, speed, and acceleration; S2. Obtain the spatiotemporal trajectory of the HDV using the operational status data of the CAV in the road network, and obtain the upstream queue length of the intersection based on the spatiotemporal trajectory of the HDV and the CAV trajectory. ; S3. Construct an action buffer queue, utilizing the action buffer queue, vehicle operating status data, and upstream queue length at the intersection. The signal timing information of the signalized intersection is used to construct an augmented state vector. The augmented state vector is then input into the trained DA-TD3 model to obtain the longitudinal acceleration required by the vehicle.

2. The ecological driving control method based on queue length under low-permeability mixed traffic flow according to claim 1, characterized in that: In step S2, the spatiotemporal trajectory of the HDV is obtained using the operating status data of the CAV in the road network, and the upstream queue length of the intersection is obtained based on the spatiotemporal trajectory of the HDV and the CAV trajectory. Specifically: S201. Obtain the historical speed difference value, historical acceleration difference value, and historical longitudinal displacement increment of CAVs in the queue using the operating status data of CAVs in the road network. Obtain the longitudinal displacement increment of each HDV using the historical speed difference value, historical acceleration difference value, and historical longitudinal displacement increment of CAVs in the queue. The longitudinal displacement increment of each HDV is obtained using the historical velocity difference, historical acceleration difference, and historical longitudinal displacement increment of the CAVs in the queue. Specifically: Obtain the CAV closest to HDV. If HDV is before the CAV closest to HDV, perform forward reconstruction to obtain the vertical distance increment of HDV. If HDV is after the CAV closest to HDV, perform backward reconstruction to obtain the vertical distance increment of HDV. S202. Obtain the absolute position and speed of each vehicle using the longitudinal displacement increment and CAV trajectory. Determine all vehicles with speeds below a preset threshold as queued vehicles. Use the distance between the queued vehicle furthest from the stop line and the stop line as the current queue length upstream of the intersection. .

3. The ecological driving control method based on queue length under low-permeability mixed traffic flow according to claim 2, characterized in that: If the HDV is ahead of the nearest CAV, then forward reconstruction is performed to obtain the vertical distance increment of the HDV, specifically: If the CAV closest to the HDV is the first CAV in the current queue, then the number of HDVs N' between the CAV and the stop line is obtained using the distance from the CAV to the stop line, the preset safety distance, and the preset vehicle length. If the CAV closest to the HDV is not the first CAV in the current queue, then the distance between the current CAV and the previous CAV in front of the queue is obtained, and the number of HDVs N' between the two CAVs is obtained using the distance between the two CAVs, the preset safety distance, and the preset vehicle length. Let CAV be denoted as vehicle N, and the nearest HDV in front of CAV be denoted as vehicle N-1. Input the historical speed difference, historical acceleration difference, and historical longitudinal displacement increment of vehicle N into the TCN-Transformer cascade model to obtain the longitudinal displacement increment of vehicle N-1. Repeat the prediction N' times to obtain the longitudinal distance increment of HDV behind CAV.

4. The ecological driving control method based on queue length under low-permeability mixed traffic flow according to claim 3, characterized in that: If the HDV is after the nearest CAV, then backward reconstruction is performed to obtain the vertical distance increment of the HDV, specifically: If the current CAV is not the last CAV in the current queue, then obtain the distance between the current CAV and the next CAV after it in the queue. Using the distance between the two CAVs, the preset safety distance, and the preset vehicle length, obtain the number of HDVs N' between the two CAVs. If the current CAV is the last CAV in the current queue, then obtain the number of HDVs N' after the CAV. Let CAV be denoted as vehicle N, and the nearest HDV behind CAV be denoted as vehicle N+1. Input the historical velocity difference, historical acceleration difference, and historical longitudinal displacement increment of vehicle N into the TCN-Transformer cascade model to obtain the longitudinal displacement increment of vehicle N+1. Repeat the prediction N' times to obtain the longitudinal distance increment of HDV behind CAV.

5. The ecological driving control method based on queue length under low-permeability mixed traffic flow according to claim 4, characterized in that: In step S202, the absolute position and velocity of each vehicle are obtained using the longitudinal displacement increment and CAV trajectory of each vehicle. Specifically: First, the absolute position of the CAV at each time step is obtained using the CAV trajectory; then, based on the distance between two adjacent vehicles, the initial absolute position of each HDV is obtained; finally, based on the initial position of the HDV, the longitudinal displacement increment at each time step is superimposed to obtain the absolute position of the HDV at each time step. Then, the absolute position difference of HDV between every two adjacent time points is obtained, and the HDV velocity at each time point is obtained by dividing the absolute position difference of HDV by the sampling interval between adjacent time points.

6. The ecological driving control method based on queue length under low-permeability mixed traffic flow according to claim 5, characterized in that: The action buffer queue constructed in S3 utilizes the action buffer queue, vehicle operating status data, and upstream queue length at the intersection. The signal timing information of the signalized intersection is used to construct an augmented state vector. This augmented state vector is then input into the trained DA-TD3 model to obtain the required longitudinal acceleration for the vehicle. Specifically: S301. Discretize the continuous time to obtain the time delay step. The initial length is First-in-first-out action buffer queue ; In the initial state Each element in the array is 0. It is the first in the action buffer queue One action; S302, utilizing action buffer queues, vehicle operating status, signal timing information at signalized intersections, and upstream queue lengths at intersections. Composition of augmented state vectors ,Will Input the trained DA-TD3 model to obtain the required longitudinal acceleration of the vehicle, store the required longitudinal acceleration at the end of the action buffer queue, remove it from the action buffer queue, and execute the action. This keeps the action buffer queue length constant. Reorder the updated action buffer queue elements from 1 to... Label; in, It is a vehicle exist The speed of time, It is a vehicle exist acceleration at any moment It is the current distance of the vehicle to the stop line. It is a vehicle Distance from the front of the vehicle in front, It is a vehicle Relative speed to the vehicle in front, yes The phase of the signal at any moment, yes Remaining time for phase at any given moment It is an action buffer queue.

7. The ecological driving control method based on queue length under low-permeability mixed traffic flow according to claim 6, characterized in that: In step S301, the continuous time is discretized to obtain the time delay step. Specifically: in, It is the time delay step. It controls the step size. It is a continuous delay.

8. The ecological driving control method based on queue length under low-permeability mixed traffic flow according to claim 7, characterized in that: The trained DA-TD3 model is obtained through the following method: Constructing augmented state vectors , augment the state vector Input the DA-TD3 model for reinforcement learning to obtain a trained DA-TD3 model, specifically: A1. Construct an action buffer queue, and build an augmented state vector based on the action buffer queue. Then construct the DA-TD3 model; The DA-TD3 model includes: an Actor network. and Critic network; The Actor network Used based on the augmented state vector Output acceleration ; The Critic network includes: a value network and value network ; The Critic network is used to evaluate the value of state-action pairs; A2. Augment the state vector Input the DA-TD3 model to obtain the required longitudinal acceleration of the vehicle. Perform acceleration action Obtain the state at the next moment. and rewards , experience Store the experience playback, and then based on the vehicle's required longitudinal acceleration. Update the action buffer queue.

9. The ecological driving control method based on queue length under low-permeability mixed traffic flow according to claim 8, characterized in that: The reward Specifically: in, , , , These are weighting coefficients. It is a security reward value. It's the green light passage bonus value. It is a moment Time required to reach the stop line This is the remaining passable time for the current phase. It is a comfort indicator. It is a vehicle exist acceleration at any moment It's a comfort bonus value. It is the penalty coefficient. It is the maximum value of jerk. It is a vehicle exist acceleration at any moment It is the absolute value symbol. It is a traffic efficiency bonus value. That is the maximum speed. It is the minimum speed. It is the expected speed. It is normalization bias.

10. The ecological driving control method based on queue length under low-permeability mixed traffic flow according to claim 9, characterized in that: Security Bonus Value Specifically: in, It's a penalty value. It's a strong penalty value. , It is a constant. It is a safety threshold. yes The remaining time until the collision. It is a vehicle Distance from the front of the vehicle in front, It is a vehicle exist The speed of time, It is a vehicle exist The speed of time, It is a function that takes the maximum value.

Images