An Intention-Driven Hierarchical Reinforcement Learning Approach for Hybrid Traffic Cooperative Trajectory Optimization

Abstract

This invention proposes an intent-driven hierarchical reinforcement learning method for hybrid traffic cooperative trajectory optimization, addressing the efficiency and safety issues arising from the unpredictability of human driving behavior at mixed traffic intersections. To address the limitations of traditional traffic signal control and single-layer trajectory planning methods in effectively handling dynamic changes in mixed traffic, and the dimensionality explosion problem inherent in fully decentralized multi-agent systems, this invention constructs a dual-timescale cooperative control architecture between a Roadside Unit (RSU) edge computing server and a Connected Autonomous Vehicle (CAV), decoupling the hybrid traffic coordination problem into two interactive decision-making processes. This invention designs a hierarchical reinforcement learning algorithm based on a competitive dual deep Q-network (D3QN). At the macro level, the RSU acts as a meta-controller dynamically broadcasting traffic intentions, while at the micro level, the CAV acts as a sub-controller executing precise acceleration and lane-changing maneuvers. A rule-based action masking mechanism is introduced to transform physical driving boundaries into hard constraints. This invention can preemptively eliminate stop-and-go traffic shockwaves through top-down intent broadcasting, significantly reducing average vehicle delay and emergency braking events while maintaining high intersection throughput, thus improving driving safety and comfort in mixed traffic environments.

Description

Technical Field

[0001] This invention relates to the fields of intelligent transportation, autonomous driving, and deep reinforcement learning, specifically to an intent-driven hierarchical reinforcement learning method for optimizing hybrid traffic cooperative trajectories, applicable to scenarios such as urban mixed traffic intersections where traffic congestion needs to be alleviated and safety and comfort improved. Background Technology

[0002] Building efficient, comfortable, and safe intelligent transportation systems is a core challenge in achieving urban sustainable development goals, and urban intersections naturally become major bottlenecks in these road networks. Mixed-traffic intersections in cities face significant challenges in terms of efficiency and safety; in unstable stop-and-go traffic conditions, these problems can further translate into reduced driving comfort. With the transition to the era of hybrid autonomous driving, connected autonomous vehicles (CAVs) must share roads with human-driven vehicles (HVs). Because human driving is inherently random, traditional control methods relying on fixed rules or deterministic optimization often fail to achieve globally optimal results in such unpredictable environments.

[0003] To alleviate intersection congestion, existing research mainly focuses on two independent directions, but both have obvious limitations:

[0004] 1) Infrastructure-based macro-control (such as adaptive traffic signal timing): These systems fundamentally treat vehicles as passive entities and cannot directly mitigate the micro-level stop-and-go traffic shockwaves caused by imperfect human driving behavior.

[0005] 2) Vehicle-based micro-trajectory optimization: Pure micro-controllers are limited by local field of view and lack global intersection perception. They often sacrifice intersection-level throughput in order to improve local mobility.

[0006] In recent years, deep reinforcement learning (DRL) has become a powerful tool for coupling these two paradigms. However, directly deploying flat or fully decentralized DRLs to mixed traffic intersections remains extremely challenging. Single-layer agents are susceptible to the curse of dimensionality and severe overestimation bias. Furthermore, fully decentralized coordination cannot proactively publish intentions across the entire intersection, making traffic flow prone to stalemates. Existing hierarchical frameworks typically reduce the top-level controller to a simple path routing or phase selection, lacking deep coupling with vehicle-level maneuverability.

[0007] Therefore, there is an urgent need for a collaborative trajectory optimization method that can proactively break this dichotomy, decouple intention-level decisions from execution-level actions within a unified framework, and simultaneously ensure microscopic physical security boundaries. Summary of the Invention

[0008] The purpose of this invention is to provide an intent-driven hierarchical reinforcement learning method for hybrid traffic cooperative trajectory optimization, which solves the problems of dimensional explosion, macro-micro decision-making separation, and insufficient safety due to lack of physical hard constraints in existing technologies in hybrid traffic scenarios. It achieves joint optimization of intersection throughput and vehicle waiting time, and actively dissipates traffic shock waves.

[0009] This invention considers a typical urban signalized intersection where a roadside unit (RSU)-based hierarchical coordinator replaces traditional phase logic. The intersection contains four symmetrical entrances (north, south, east, and west). Each entrance lane includes a dedicated left-turn lane, a dedicated through lane, and a shared through-and-right-turn lane. The traffic flow across the intersection is a heterogeneous mixture of CAV and HV.

[0010] The vehicle dynamics model, car-following model, and problem optimization objective are modeled as follows.

[0011] Dynamics model: For any vehicle, its longitudinal kinematics update follows the standard discrete-time equations of motion:

[0012]

[0013]

[0014] in, , and These represent the vehicle's time. The longitudinal position, velocity, and physical acceleration of the vehicle. Acceleration is strictly limited by the vehicle's physical limits, i.e. ,in and These represent maximum deceleration and maximum acceleration capabilities, respectively. Human-driven vehicle (HV) car-following behavior model: The longitudinal behavior of the HV is controlled by the Krauss car-following model, designed to maintain a safe speed with the vehicle in front. To prevent rear-end collisions:

[0015]

[0016] in, It is the current speed of the vehicle in front. The bumper-to-bumper clearance between the two vehicles is defined. This represents the inherent reaction time of a human driver. This indicates the average speed of the vehicle. It is the maximum deceleration constant of the vehicle in front. The expected speed will be randomly perturbed to reflect human imperfections.

[0017] Optimization objective: Define cumulative cross-section throughput Total vehicle delay The optimization objective is to find the optimal parameter set to maximize the expected global utility. :

[0018]

[0019] Constraints include the shortest physical safety boundary for collision avoidance. Vehicle speed limits and acceleration limits.

[0020] Compared with the prior art, the beneficial effects of this invention are:

[0021] 1) Intent-driven hierarchical collaborative architecture: A dual-timescale framework coupling macro-level intersection management and micro-level vehicle execution is proposed. By broadcasting intents from top to bottom through RSUs, the feasible trajectory space of CAVs is dynamically constrained, effectively mitigating the dimensionality explosion problem associated with fully decentralized multi-agent systems;

[0022] 2) Rule-based action masking to ensure operational safety: A dynamic action masking mechanism is introduced, directly transforming strict physical driving boundaries into mathematical hard constraints before the neural network inference stage. Compared with traditional soft reward penalties, this method improves operational safety by reducing the selection of inactive actions during exploration in a highly stochastic hybrid autonomous driving environment.

[0023] 3) Quantitatively Improved Explainable Shockwave Dissipation: This method dissipates stop-and-go shockwaves upstream by broadcasting advanced intent. Compared to state-of-the-art baseline methods, this approach effectively reduces average vehicle wait times and emergency braking events while maintaining high intersection throughput, directly improving driving comfort. Attached Figure Description

[0024] Figure 1 A diagram illustrating the collaborative control architecture connecting macro-level intersection management and micro-level CAV trajectory execution.

[0025] Figure 2 This is a flowchart of the intention-driven hierarchical reinforcement learning (iHRL) decision-making process. Detailed Implementation

[0026] An intent-driven hierarchical reinforcement learning method for hybrid traffic cooperative trajectory optimization includes the following steps:

[0027] (I) Constructing a macro- and micro-scale dual-time-scale collaborative control architecture. Deploying a macro-scale meta-controller located at the edge server of the roadside unit (RSU) and micro-scale sub-controllers embedded in each CAV. The macro-scale meta-controller operates at a macro-scale time resolution. Under the following operation, the microscopic sub-controller operates at each atom simulation step. Next update.

[0028] (II) Definition of Hierarchical Markov Decision Process (H-MDP). For the macroscopic meta-controller, its state space... The data comprises intersection metrics continuously collected by the RSU wide-area perception system, including normalized cumulative throughput, average waiting time, and global vehicle density. Macroscopic reward function. The calculation formula is: This includes a weighted tradeoff between throughput increment and average latency. For the micro-controller, its local state space... This includes the CAV's self-perception (position, speed, acceleration, distance to the intersection), meso-level environmental perception (average speed and density of surrounding lanes), and the RSU intent vector encoded using one-hot encoding. The micro-level intrinsic reward function... It consists of basic rewards and intentional rewards, and transforms macroscopic intentions into specific kinematic incentives through physical acceleration and controlled actions.

[0029] (III) Constructing a rule-based dynamic action masking mechanism. By evaluating real-time kinematic limits and adjacent lane clearances, safety logic is designed to explicitly filter dangerous agent actions (e.g., initiating lane changes or accelerating beyond speed limits when clearances are insufficient). This is achieved by using mask vectors... The action value function is applied to network estimation to determine the final executable actions. .

[0030] (iv) Optimization algorithm based on competitive dual deep Q-network (D3QN). At the macro level, the meta-controller uses a standard experience replay mechanism to map continuous global states to discrete intention spaces; at the micro level, the sub-controller utilizes the D3QN architecture to separate state value streams and advantage streams, effectively learning the intrinsic value of meso-level traffic states, and mitigating value overestimation bias in highly random mixed traffic environments by separating action selection and action evaluation through the dual DQN paradigm.

[0031] (v) Generate the optimal trajectory decision. The macroscopic meta-controller evaluates experience and issues intent guidance every N environmental steps to prevent high-frequency jitter in policy scheduling; the microscopic sub-controller operates at each atomic step, and applies a dynamic safety mask to limit the exploration process within the physical driving limits before executing micro actions, and finally outputs the longitudinal acceleration and lateral lane change command of the CAV.

[0032] Example:

[0033] Example 1:

[0034] In this embodiment, the parameter configurations for the dual-timescale architecture and algorithm are as follows:

[0035] Simulated step size Set to 0.1 seconds, meta controller update interval Set to 10 steps (i.e., 1 second).

[0036] Each episode lasts 180 seconds and has a maximum speed limit of 15 m / s.

[0037] In vehicle kinematic constraints, maximum acceleration for Maximum deceleration for The vehicle is 5.0 meters long and has a minimum safety clearance of 2.5 meters.

[0038] In the D3QN hyperparameters, the discount factor The learning rate is set to 0.99. The experience replay buffer capacity is 50,000, the batch size is 64, and the target network hard update frequency is 100 steps.

[0039] The CAV penetration rate is set at 80%, and the HV penetration rate is set at 20%. The specific execution steps of the control method in this embodiment are as follows:

[0040] (1) The RSU meta controller senses the intersection status. If it detects that congestion is about to occur upstream, it selects a macro-intention aimed at stabilizing traffic. .

[0041] (2) Receiving intent of each CAV micro sub-controller By combining its own micro and meso states, a Q-value distribution is generated.

[0042] (3) The safety masking mechanism calculates the distance and speed between the vehicle and the vehicles in front and behind in real time. If a lane-changing action may cause a collision, the masking will set the Q value of the action to negative infinity.

[0043] (4) CAV selects the optimal action from the set of safety actions filtered by the mask and performs smooth deceleration, thereby absorbing the downstream delay at the upstream end before the congestion occurs and realizing "shock wave dissipation".

[0044] Example 2:

[0045] An intent-driven hierarchical system for hybrid traffic cooperative trajectory optimization includes:

[0046] (1) Global perception and macro decision-making module: Deployed on the edge computing server, it is used to obtain global traffic throughput and average delay status, and output discrete high-level traffic guidance intentions using the meta-network model.

[0047] (2) Local perception and micro-decision module: Deployed on the CAV terminal, it is used to combine the status of the vehicle and the status of surrounding vehicles, and use the sub-network model to output the value assessment of trajectory planning actions.

[0048] (3) Safety constraint execution module: Deployed in the CAV terminal, it generates a dynamic mask using physical kinematics to filter unsafe actions output by the sub-network, and sends the final safe action command to the vehicle actuator to complete longitudinal acceleration and deceleration and lateral lane change.

[0049] The beneficial effects of this embodiment are as follows:

[0050] This embodiment verifies the superiority of the present invention by comparing it with baseline algorithms such as Fixed-Time Traffic Signal (FT), Single-Agent DQN (SA-DQN), Flow, CoTV, and HONEST-CAV:

[0051] (1) Convergence performance: During the training phase, the proposed iHRL framework achieves the fastest convergence speed and the lowest steady-state time difference error (TD-Error). By decoupling macroscopic intent from microscopic execution, the action space is effectively limited, enabling the sub-controller to quickly master safe cooperative maneuvers without suffering severe overestimation bias.

[0052] (2) Macroeconomic efficiency and shock wave dissipation: Under heavy traffic demand, iHRL completely eliminates congestion shock waves. Under the guidance of the meta controller, CAV actively absorbs downstream delays by smoothly decelerating in the early stage of the upstream road segment, achieving stagnation-free fluid crossing of the intersection. The average throughput reaches 408 vehicles / set, and the average waiting time is reduced to 21.5 seconds, which is better than all the comparison algorithms.

[0053] (3) Microscopic safety and comfort: In the longitudinal acceleration distribution assessment, FT and SA-DQN produce frequent stop-and-go traffic, resulting in severe deceleration curves (extending to) (The following is an excerpt from the original text). The iHRL invention, through rule-based security masks, rigorously limits severe deceleration, concentrating acceleration highly within... Nearby, the danger of emergency braking was eliminated, ensuring micro-level safety and superior passenger comfort.

[0054] (4) Robustness and scalability: Under different CAV penetration rates from 0% to 100%, SA-DQN suffers severe performance degradation due to the curse of dimensionality when the CAV penetration rate exceeds 60%; the iHRL framework of this invention achieves robust near-linear improvement in intersection throughput, effectively protecting the meta-controller from the effects of dimensional explosion. Ablation experiments further confirm that removing the safety mask leads to a collision rate increase of 14.5%, removing the macroscopic intent leads to an average speed decrease of 8.5 m / s, while the complete iHRL architecture maximizes traffic flow (average speed 12.4 m / s) while suppressing the collision rate to near zero (0.1%).

Claims

1. An intent-driven hierarchical reinforcement learning method for hybrid traffic cooperative trajectory optimization, characterized in that... It includes the following steps: (i) Constructing a dual-time-scale collaborative control architecture for roadside units (RSUs) and connected automated vehicles (CAVs). This architecture includes an RSU edge server acting as a macroscopic meta-controller, and individual CAVs acting as microscopic sub-controllers; (ii) Define a hierarchical Markov decision process to decouple the hybrid traffic coordination problem. The process includes defining a macroscopic state space, an intention-action space, a macroscopic reward function, and a microscopic local state space, a microscopic physical action space, and an intrinsic sub-reward function; (iii) Construct a rule-based dynamic safety action masking mechanism to transform the physical driving boundary of the vehicle into mathematical hard constraints in order to filter dangerous actions output by the micro sub-controller; (iv) Design of a hierarchical reinforcement learning optimization algorithm based on a competitive dual deep Q-network (D3QN). The macroscopic meta-controller dynamically broadcasts traffic intentions, and the microscopic sub-controller combines the broadcast intentions with the dynamic safety action mask to generate safe longitudinal acceleration and optimal lateral lane-changing maneuvers, thus completing the cooperative trajectory optimization of mixed traffic.

2. The intention-driven hierarchical reinforcement learning method for hybrid traffic cooperative trajectory optimization as described in claim 1, characterized in that, In step (ii), the macroscopic state space Defined as: ； in, This represents the normalized cumulative throughput of the intersection. This represents the average waiting time for all vehicles. Indicates the overall vehicle density; The macro reward function The calculation formula is: ； in, Represents the cumulative cross-port throughput. and For the corresponding scaling weights, A dynamic clipping threshold to prevent gradient accumulation.

3. The intention-driven hierarchical reinforcement learning method for hybrid traffic cooperative trajectory optimization as described in claim 1, characterized in that, In step (ii), the intrinsic sub-reward received by the micro-controller The calculation formula is: ； in The basic reward is to encourage agents to remain active. This is a dynamic intent reward system used to transform macro-intentions. This translates into specific kinematic excitations for the actual vehicle's physical acceleration and selected actions.

4. The intention-driven hierarchical reinforcement learning method for hybrid traffic cooperative trajectory optimization as described in claim 1, characterized in that, In step (ii), the microscopic local state space Defined as: ； in, Includes normalized position, velocity, acceleration, and distance from the intersection from the CAV. Aggregating the average speed and density of surrounding lanes to provide spatial awareness. This is the macroscopic intent vector encoded using one-hot encoding.

5. The intention-driven hierarchical reinforcement learning method for hybrid traffic cooperative trajectory optimization as described in claim 4, characterized in that, In step (iii), the rule-based dynamic safety action masking mechanism explicitly filters dangerous actions to generate mask vectors by evaluating real-time kinematic constraints and adjacent lane clearances. Microscopic executable actions The following formula is used to determine safety: 。 6. The intention-driven hierarchical reinforcement learning method for hybrid traffic cooperative trajectory optimization as described in claim 1, characterized in that, In step (iv), the competitive dual-depth Q-network architecturally separates the state value stream and the state-dependent advantage stream to address the value overestimation bias. Simultaneously, the macroscopic meta-controller... The micro-controller updates its decision every environmental step to capture macroscopic traffic flow changes, with each atomic simulation step updating the decision. The system is used to control the vehicle's kinematic dynamics.

7. An intent-driven hierarchical system for hybrid traffic cooperative trajectory optimization, employing the method described in any one of claims 1-6, characterized in that, include: (i) The global perception and macro decision-making module is deployed on the RSU edge server to obtain global traffic throughput, average waiting time and global vehicle density status, and output and broadcast discrete macro traffic intentions using the meta-network model. (ii) Local perception and micro-decision module, deployed on each CAV terminal, is used to combine the vehicle status, the status of surrounding vehicles and the received macro traffic intentions, and use the sub-network model to calculate the evaluation value of each maneuver. (iii) Safety constraint execution module, deployed on CAV terminal, is used to generate dynamic safety action mask by using physical kinematic limits and adjacent lane gap rules, filter dangerous actions output by the sub-network, and select the optimal safety action to control the longitudinal acceleration and deceleration and lateral lane change of the vehicle; (iv) Dual reward and state update module, used to calculate the sub-reward and update the sub-controller network parameters at the micro step size, and calculate the macro reward and update the meta-controller network parameters at the macro step size.

Images