Automatic driving decision control method based on hierarchical strategy network
By employing a hierarchical policy network architecture and digital twin-driven training, the balance between safety and high performance in autonomous driving systems is addressed, enhancing adaptability to extreme scenarios and making it suitable for L4 and higher level autonomous driving systems.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHENYANG AEROSPACE UNIVERSITY
- Filing Date
- 2026-01-13
- Publication Date
- 2026-05-05
AI Technical Summary
Existing autonomous driving decision-making architectures struggle to balance safety and high performance, especially in extreme long-tail scenarios where processing capacity is insufficient, modular methods lack versatility, and end-to-end methods raise safety concerns.
By adopting a hierarchical policy network architecture, and through hierarchical design of driving primitives, arbitrator-expert policy collaboration mechanism and digital twin-driven training, an autonomous driving decision-making and control method based on hierarchical policy networks is constructed to ensure the transparency of decision logic and improve the adaptability to extreme scenarios.
It achieves safe and efficient decision-making and control in both normal and extreme scenarios, combining the interpretability of modular methods with the high performance of end-to-end methods, and is suitable for autonomous driving systems at Level 4 and above.
Smart Images
Figure CN121978920A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of autonomous driving and artificial intelligence technology, specifically relating to an autonomous driving decision control method based on hierarchical policy networks. Background Technology
[0002] The core objective of autonomous driving systems is to achieve safe and efficient autonomous navigation. Existing technologies mainly fall into two architectural categories: modular approaches and end-to-end approaches.
[0003] The modular approach breaks down driving tasks into independent sub-modules such as perception, prediction, planning, and control, with each sub-module collaborating according to a standardized process. The perception module processes sensor data and identifies surrounding targets, the prediction module infers the future positions of targets, the planning module generates the desired trajectory, and the control module translates the trajectory into physical control commands. This approach has advantages such as strong interpretability, easy implementation of safety rules, and support for parallel development, but it also suffers from problems such as inter-module error accumulation, high system fragility, and suboptimal overall performance, making it difficult to cope with complex and dynamic driving environments.
[0004] End-to-end methods directly map sensor inputs to driving control commands via a single neural network, eliminating the need for intermediate sub-modules. This approach can learn complex environmental relationships that are difficult for human programming to cover, possessing the potential to surpass human performance and reducing manual engineering costs. However, it also suffers from key drawbacks such as a "black box" problem (lack of transparency in internal working mechanisms), poor adaptability to long-tail scenarios (insufficient data in rare scenarios leading to poor performance), and unverifiable safety (inability to implement hard safety rules).
[0005] Both existing architectures have significant limitations: while modular approaches are reliable, they lack versatility; end-to-end approaches offer superior performance, but their security is questionable. This makes it difficult for them to simultaneously meet the dual core requirements of autonomous driving: safety and efficiency. Especially in extreme long-tail scenarios (such as emergency braking, hazard avoidance steering, and navigating complex intersections), existing methods either lead to decision-making errors due to the accumulation of module errors or fail to formulate effective response strategies due to insufficient data, becoming a key bottleneck restricting the deployment of autonomous driving technology.
[0006] In summary, providing a hierarchical policy network method based on digital twins and designed for extreme scenarios in autonomous driving is a problem that urgently needs to be solved. Summary of the Invention
[0007] Therefore, the purpose of this invention is to provide an autonomous driving decision control method based on hierarchical policy networks, so as to solve the problems of existing autonomous driving decision architectures in balancing safety and high performance, as well as insufficient ability to handle extreme long-tail scenarios.
[0008] The technical solution provided by this invention is: an autonomous driving decision control method based on hierarchical policy networks, comprising:
[0009] Collect core data for autonomous driving;
[0010] Create driving scenarios;
[0011] Based on the hierarchical design of driving primitives and the collaborative mechanism of arbitrator-expert policy, a hierarchical policy network architecture is constructed:
[0012] The hierarchical policy network is trained and validated using digital twins; the mature model is then deployed to a real vehicle system and further fine-tuned using real vehicle data feedback.
[0013] Preferably, the hierarchical design of driving primitives includes:
[0014] The principles for classifying driving primitives are established, including: the principle of mutual exclusion: each primitive corresponds to a unique driving scenario; and the principle of completeness: covering all possible driving scenarios.
[0015] Based on scenario frequency and priority, driving primitives are divided into four layers:
[0016] Layer 0 - System State Primitives: In non-driving scenarios, these include system shutdown, standby, and fault states, and have the highest priority.
[0017] Level 1 - Safety Coverage Primitives: Extreme emergency scenarios, including emergency braking and evasive steering, account for less than 1% of driving time, and have the second highest priority.
[0018] Level 2 - Tactical Transition Primitives -: Short-term transition scenarios, including lane changes, merging, and intersection passage, accounting for about 1% of driving time, with medium priority;
[0019] 3-Steady-State Driving Primitives-: Common default scenarios, including lane keeping, following other vehicles, and low-speed crawling, accounting for about 99% of driving time, with the lowest priority;
[0020] Set hierarchical preemption rules, with higher-level primitives overriding lower-level primitives: level 0 covers all levels, level 1 covers levels 2-3, level 2 covers level 3, ensuring security response priority in extreme scenarios.
[0021] Preferably, the construction of the hierarchical strategy network architecture includes:
[0022] Construct a shared sensing backbone network, integrate data from cameras, lidar, radar, GPS, and vehicle dynamics to generate a unified bird's-eye view of the environment, serving as a global environmental information source for the system;
[0023] The arbitrator module is designed, which uses a high-level neural network classifier as input, takes the BEV environment representation and navigation target as input, and outputs a "candidate expert strategy" adapted to the current scenario.
[0024] Deploy a policy gateway and validator module, embed preset security rules and traffic regulations, and perform security verification on candidate expert policies;
[0025] Construct an expert policy group and train an exclusive end-to-end model for each driving primitive. The expert policy only processes the control command output for the corresponding scenario.
[0026] A state transition mechanism is established, which abstracts the hierarchical policy network into a finite state machine, and the arbitrator controls the smooth transition between driving primitives.
[0027] Preferably, the digital twin-driven training and verification includes:
[0028] Construct virtual training environments with diverse scenarios;
[0029] Generating scenario datasets based on a virtual training environment;
[0030] The network is trained in modules. First, the shared perception backbone network is trained, then each expert policy is trained separately to adapt to the corresponding scenario, and finally the arbitrator is trained.
[0031] Closed-loop verification is performed using a digital twin platform to simulate various driving scenarios, test the system's decision-making accuracy and safety performance, and iteratively optimize network parameters.
[0032] The autonomous driving decision-making and control method based on hierarchical policy networks provided by this invention balances safety and high performance: it integrates the interpretability of modular methods with the learning capabilities of end-to-end methods, and adopts an arbitrator-expert policy architecture to ensure the transparency of decision logic while achieving optimal control in complex scenarios through end-to-end training. By designing driving primitives hierarchically for each scenario, each expert policy focuses on training for a specific scenario. Combined with rich extreme scenario data generated by digital twin simulation, the adaptability to long-tail scenarios is significantly improved.
[0033] This method integrates the interpretability of modular approaches with the high performance advantages of end-to-end approaches. By leveraging the hierarchical design of driving primitives and digital twin simulation training, it achieves accurate processing of both conventional driving scenarios and extreme long-tail scenarios, making it suitable for decision control modules in L4 and higher level autonomous driving systems. Attached Figure Description
[0034] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0035] Figure 1 A flowchart illustrating the construction process of the hierarchical strategy network architecture provided by this invention;
[0036] Figure 2A flowchart of an autonomous driving control decision-making method provided in an embodiment of the present invention;
[0037] Figure 3 This invention provides a diagram of driving scenario types.
[0038] Figure 4 A diagram of a hierarchical strategy network model provided in an embodiment of the present invention. Detailed Implementation
[0039] The present invention will be further explained below with reference to specific implementation schemes, but this explanation does not limit the scope of the invention.
[0040] To address the difficulty in balancing safety and high performance in existing autonomous driving decision-making architectures, and their insufficient ability to handle extreme long-tail scenarios, this paper proposes an autonomous driving control decision-making method based on digital twin-driven, hierarchical policy networks, specifically designed for extreme scenarios. The method includes:
[0041] First, a hierarchical policy network architecture is constructed. By leveraging the hierarchical design of driving primitives and the collaborative mechanism of arbitrator-expert policy, the interpretability of modular methods and the high performance of end-to-end methods are integrated to achieve "trustworthy and high-performance" decision output.
[0042] The hierarchical design of driving primitives includes:
[0043] S11: Establish the principles for dividing driving primitives, including mutual exclusivity (each primitive corresponds to a unique driving scenario) and completeness (covering all possible driving scenarios).
[0044] S12: Based on scenario frequency and priority, driving primitives are divided into four layers:
[0045] Layer 0 (System State Primitives): Non-driving scenarios, covering system shutdown, stationary standby, and fault status, with the highest priority;
[0046] Layer 1 (Safety Coverage Primitives): Extreme emergency scenarios, including emergency braking and evasive steering, accounting for less than 1% of driving time, with the second highest priority;
[0047] Level 2 (Tactical Transition Primitives): Short-term transition scenarios, including lane changes, merging, and intersection passage, accounting for about 1% of driving time, with medium priority;
[0048] Layer 3 (Steady-state driving primitives): The default scenarios include lane keeping, following other vehicles, and low-speed crawling, accounting for about 99% of driving time, and have the lowest priority.
[0049] S13: Set hierarchical preemption rules, where higher-level primitives can override lower-level primitives (level 0 overrides all levels, level 1 overrides levels 2-3, and level 2 overrides level 3), ensuring security response priority in extreme scenarios.
[0050] The construction process of a layered strategy network architecture, such as Figure 1 As shown.
[0051] S21: Construct a shared perception backbone network, integrate camera, lidar, radar, GPS and vehicle dynamics data to generate a unified bird's-eye view (BEV) environmental representation as a global environmental information source for the system;
[0052] S22: Design an arbitrator module that uses a high-level neural network classifier, takes the BEV environment representation and navigation target as input, and outputs a "candidate expert strategy" adapted to the current scenario.
[0053] S23: Deploy the policy gateway and validator module, embed preset safety rules and traffic regulations (such as speed limits, yielding rules, collision avoidance conditions, etc.), and perform security verification on candidate expert policies;
[0054] S24: Build an expert policy group and train an end-to-end model for each driving primitive. The expert policy only processes the control command output of the corresponding scenario (e.g., the lane keeping expert is responsible for maintaining the lane center position and target speed).
[0055] S25: Establish a state transition mechanism, abstract the hierarchical policy network into a finite state machine, and use an arbitrator to control the smooth transition between driving primitives to ensure driving stability when switching between different scenarios.
[0056] Based on the frequency and priority of driving scenarios, the hierarchy of driving primitives is divided, and dedicated expert strategies are trained for high-frequency routine scenarios and low-frequency extreme scenarios, thereby overcoming the performance bottleneck caused by the lack of data in long-tail scenarios.
[0057] Furthermore, a policy gateway and validator module are introduced to embed hard safety rules and traffic regulations to ensure that the driving behavior output by the expert policy meets safety standards, thus solving the problem of unverifiable safety in end-to-end methods.
[0058] In addition, a high-fidelity training environment is constructed using digital twin simulation technology to generate rich extreme scenario data, providing sufficient samples for expert strategy training and enhancing the system's adaptability to extreme scenarios.
[0059] Among them, digital twin-driven training and validation:
[0060] S31: Select a high-fidelity digital twin simulation platform (such as CARLA or AirSim) to build a virtual training environment that includes diverse scenarios such as urban roads, highways, and complex intersections;
[0061] S32: Generate scene datasets, covering high-frequency routine scenes and low-frequency extreme scenes (such as sudden obstacles, severe weather, traffic violations, etc.), to solve the problem of insufficient data on real-world extreme scenes;
[0062] S33: Train network components in modules. First, train the shared perception backbone network to ensure the accuracy of environmental representation. Then, train each expert policy separately to adapt it to the corresponding scenario. Finally, train the arbitrator to achieve accurate matching of scenario and expert policy.
[0063] Based on the state transition mechanism of finite state machines, the continuity of control commands during the switching between driving primitives is ensured, avoiding driving jitter or sudden decision changes caused by module switching.
[0064] It supports parallel development (different teams can train different expert strategies at the same time), and the optimization of a single expert strategy will not affect the overall architecture, significantly reducing the cost of system iteration and maintenance.
[0065] By designing driving primitives in a scenario-layered manner, each expert strategy focuses on training for a specific scenario. Combined with rich extreme scenario data generated by digital twin simulation, the adaptability to long-tail scenarios is greatly improved.
[0066] S34: Conduct closed-loop verification through a digital twin platform, simulate various driving scenarios (especially extreme long-tail scenarios), test the system's decision-making accuracy and safety performance, and iteratively optimize network parameters;
[0067] By embedding hard security rules into the policy gateway and validator, the policies of candidate experts are subject to mandatory verification, thereby avoiding dangerous driving behaviors from a mechanism perspective and effectively solving the "black box" security risks of end-to-end methods.
[0068] S35: Deploy the well-trained model to the real vehicle system, and further fine-tune it through feedback from real vehicle data to ensure that the model trained in the virtual environment can adapt to real driving scenarios.
[0069] Digital twin simulations are used to generate targeted scene data, reducing reliance on real road data while avoiding the safety risks and high costs associated with training in real-world scenarios.
[0070] Finally, by sharing a perception backbone network to achieve multi-sensor data fusion and unified environmental representation, we can ensure that the arbitrator and expert strategies obtain consistent global environmental information, reduce decision-making bias caused by information asymmetry, and ultimately achieve safe, efficient and stable operation of the autonomous driving system in various scenarios.
[0071] Example 1
[0072] like Figure 2As shown, this embodiment takes urban road autonomous driving scenario as the application object, and verifies the decision control performance of hierarchical policy network (HPN) in normal scenario and extreme long-tail scenario by integrating digital twin simulation training technology. The specific steps are as follows:
[0073] S1: Collect core data for autonomous driving
[0074] S11: Collects multi-source sensor data, including camera images, LiDAR point clouds, radar detection data, GPS positioning data, and vehicle dynamics parameters.
[0075] S12: Perform time resolution and alignment processing;
[0076] S2: Constructing Driving Scenarios
[0077] S21: Based on open-source high-fidelity simulators such as CARLA, it restores the real urban road topology, covering static environmental elements such as road layout, traffic signs and markings, and building distribution.
[0078] S22: Import sensor data and vehicle dynamics models to reproduce dynamic driving scenarios in the simulator, supporting flexible configuration of environmental parameters such as weather conditions (sunny, rainy, snowy), light intensity, and traffic flow.
[0079] S23: Establish a mapping relationship between scenarios and data, enabling digital twins to accurately simulate sensor feedback and vehicle response in real driving, and achieve scenario reproducibility and controllability.
[0080] S3: Defining Driving Primitives and Constructing Hierarchical Structures
[0081] S31: Classify driving scenario types into four categories of driving primitives: non-driving primitives (Tier 0), safety coverage primitives (Tier 1), transitional primitives (Tier 2), and steady-state primitives (Tier 3), ensuring that the primitives are mutually exclusive and cover all driving scenarios, such as... Figure 3 As shown.
[0082] S32: Define the core parameters of each primitive, including the activation condition, execution logic, and termination criteria. For example, the "emergency braking" primitive uses the time to collision (TTC) as the activation threshold, and the "lane keeping" primitive uses the lane centerline offset as the control target.
[0083] S33: Establish a primitive priority mechanism and set preemption logic according to the rule of Tier 0 > Tier 1 > Tier 2 > Tier 3 to ensure that high-priority primitives are executed first in extreme scenarios.
[0084] S4: The raw data from multiple sensors are fused and processed through a shared perception backbone network to generate an enriched bird's-eye view (BEV) world representation, which serves as the single source of truth for the system's environmental perception; a dataset is constructed according to driving primitives, and the expert model corresponding to each primitive is trained using only the exclusive data related to its scene.
[0085] S5: Construct a hierarchical policy network (HPN) model, as shown in Figure 4;
[0086] S51: Construct a shared sensing backbone network to fuse multi-source sensor data into a high-dimensional bird's-eye view (BEV) world representation, serving as the unified data input source for the system.
[0087] S52: Train the Arbiter model, based on BEV representation and navigation target, and use a neural network classifier to select candidate expert models that are suitable for the current scenario.
[0088] S53: Train expert model clusters and configure dedicated end-to-end models for each driving primitive. For example, train special control models for transition primitives such as "lane change" and "merge" to ensure optimized performance in various scenarios.
[0089] S54: Integrated Policy Gateway and Verifier (PGV), which verifies the expert model recommended by the arbitrator based on hard-coded security rules, and outputs physical control commands such as steering, throttle, and brake after verification.
[0090] S6: Model Validation and Scenario Testing
[0091] S61: Conduct full-coverage testing in digital twins, including 99% high-frequency steady-state scenarios (such as lane keeping and following other vehicles) and 1% long-tail extreme scenarios (such as emergency avoidance and sudden failure).
[0092] S62: Verify the model's response speed and decision accuracy, focusing on testing the smoothness of primitive switching and the timely execution of high-priority primitives. For example, the startup delay of the emergency braking primitive in extreme collision scenarios should be less than 0.2 seconds.
[0093] S63: Iteratively optimize model parameters, adjust expert model training data and arbitrator decision logic based on problems found in testing, and improve the robustness of the model in extreme scenarios.
[0094] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.
[0095] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.
Claims
1. An autonomous driving decision-making and control method based on hierarchical policy networks, characterized in that, include: Collect core data for autonomous driving; Create driving scenarios; Based on the hierarchical design of driving primitives and the collaborative mechanism of arbitrator-expert policy, a hierarchical policy network architecture is constructed: The hierarchical policy network is trained and validated using digital twins; the mature model is then deployed to a real vehicle system and further fine-tuned using real vehicle data feedback.
2. The autonomous driving decision control method based on hierarchical policy networks according to claim 1, characterized in that, The hierarchical design of driving primitives includes: The principles for classifying driving primitives are established, including: the principle of mutual exclusion: each primitive corresponds to a unique driving scenario; and the principle of completeness: covering all possible driving scenarios. Based on scenario frequency and priority, driving primitives are divided into four layers: Layer 0 - System State Primitives: In non-driving scenarios, these include system shutdown, standby, and fault states, and have the highest priority. Level 1 - Safety Coverage Primitives: Extreme emergency scenarios, including emergency braking and evasive steering, account for less than 1% of driving time, and have the second highest priority. Level 2 - Tactical Transition Primitives -: Short-term transition scenarios, including lane changes, merging, and intersection passage, accounting for about 1% of driving time, with medium priority; 3-Steady-State Driving Primitives-: Common default scenarios, including lane keeping, following other vehicles, and low-speed crawling, accounting for about 99% of driving time, with the lowest priority; Set hierarchical preemption rules, with higher-level primitives overriding lower-level primitives: level 0 covers all levels, level 1 covers levels 2-3, level 2 covers level 3, ensuring security response priority in extreme scenarios.
3. The autonomous driving decision-making and control method based on hierarchical policy networks according to claim 1, characterized in that, The construction of the hierarchical strategy network architecture includes: Construct a shared sensing backbone network, integrate data from cameras, lidar, radar, GPS, and vehicle dynamics to generate a unified bird's-eye view of the environment, serving as a global environmental information source for the system; The arbitrator module is designed, which uses a high-level neural network classifier as input, takes the BEV environment representation and navigation target as input, and outputs a "candidate expert strategy" adapted to the current scenario. Deploy a policy gateway and validator module, embed preset security rules and traffic regulations, and perform security verification on candidate expert policies; Construct an expert policy group and train an exclusive end-to-end model for each driving primitive. The expert policy only processes the control command output for the corresponding scenario. A state transition mechanism is established, which abstracts the hierarchical policy network into a finite state machine, and the arbitrator controls the smooth transition between driving primitives.
4. The autonomous driving decision-making and control method based on hierarchical policy networks according to claim 1, characterized in that, The digital twin-driven training and validation includes: Construct virtual training environments with diverse scenarios; Generating scenario datasets based on a virtual training environment; The network is trained in modules. First, the shared perception backbone network is trained, then each expert policy is trained separately to adapt to the corresponding scenario, and finally the arbitrator is trained. Closed-loop verification is performed through a digital twin platform to simulate various driving scenarios, test the system's decision-making accuracy and safety performance, and iteratively optimize network parameters.