A method, system, and device for constructing autonomous driving evaluation benchmarks

Abstract

This invention discloses a method, system, and device for constructing an autonomous driving evaluation benchmark, belonging to the field of autonomous driving evaluation technology. The method includes parsing original traffic regulations into atomic rules to generate candidate rule pairs with hierarchical labels; verifying the spatiotemporal coexistence of the candidate rule pairs to construct a hierarchical rule set; converting the hierarchical rule set into a textual scene description and generating a structured semantic representation, then mapping it to an autonomous driving simulator to render and capture multi-view image sequences, generating high-fidelity visual scene data; constructing multi-choice question-answer pairs based on the generated high-fidelity visual scene data to form an evaluation dataset containing different cognitive levels; using the evaluation dataset to train or evaluate an autonomous driving model, and optimizing the model's planning strategy based on the evaluation results. This invention solves the technical problem of autonomous driving model evaluation.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving evaluation technology, and in particular to a method, system and device for constructing autonomous driving evaluation benchmarks. Background Technology

[0002] With the rapid development of autonomous driving technology, multimodal large language models, due to their powerful semantic understanding and logical reasoning capabilities, are gradually being integrated into the decision-making and planning layers of autonomous driving systems. To verify the safety and compliance of these models in handling complex traffic environments, constructing scientific and rigorous evaluation benchmarks has become a key requirement in the industry.

[0003] However, existing multimodal evaluation benchmarks in the field of autonomous driving still have significant limitations when dealing with complex traffic rule reasoning: First, most existing benchmarks only focus on understanding a single traffic rule, such as simple traffic sign recognition or a single right-of-way judgment. In real driving scenarios, there are often complex situations where multiple rules are concurrent or even conflict with each other, and existing benchmarks cannot cover such scenarios of multi-rule combination reasoning. Second, the evaluation datasets rely heavily on 2D static images and lack the support of high-fidelity dynamic 3D driving scenarios. Such static data cannot simulate the temporal interaction and dynamic environmental changes in the actual driving process, resulting in a deviation between the evaluation results and real driving performance. Third, a hierarchical cognitive evaluation system has not yet been established, making it impossible to systematically quantify the model's reasoning level from basic perception to complex logical decision-making. This means that the model may perform well in simple tasks, but when faced with long-tail scenarios or rule priority conflicts, it is very prone to decision-making errors or violations of traffic rules, making it difficult to meet the core requirements of safety and compliance of autonomous driving systems.

[0004] Therefore, how to construct a hierarchical cognitive evaluation benchmark that supports multimodal input, enables the resolution of multi-rule conflicts from atomic rules, and achieves the automated transformation of text rules into high-fidelity dynamic simulation scenarios has become an urgent technical problem to be solved in the field of autonomous driving model evaluation. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method, system, and device for constructing autonomous driving evaluation benchmarks, aiming to solve the problems of existing technologies.

[0006] To achieve the above objectives, this application provides the following solution:

[0007] Firstly, this application provides a method for constructing an autonomous driving evaluation benchmark, comprising: parsing original traffic regulations into atomic rules and generating candidate rule pairs with hierarchical labels based on semantic structuring; verifying the spatiotemporal coexistence of the candidate rule pairs to construct a hierarchical rule set from basic atomic rules to complex conflicting rules; converting the hierarchical rule set into a text scene description and generating a structured semantic representation, then mapping it to an autonomous driving simulator to render and capture multi-view image sequences to generate high-fidelity visual scene data; constructing multi-choice question-answer pairs based on the generated high-fidelity visual scene data to form an evaluation dataset containing different cognitive levels; using the evaluation dataset to train or evaluate an autonomous driving model, and optimizing the model's planning strategy based on the evaluation results.

[0008] Secondly, this application provides an autonomous driving evaluation benchmark construction system, including an atomic rule extraction module for parsing original traffic regulations into atomic rules and generating candidate rule pairs with hierarchical labels based on semantic structuring; a rule-scene transformation module for verifying the spatiotemporal coexistence of the candidate rule pairs and constructing a hierarchical rule set from basic atomic rules to complex conflict rules; transforming the hierarchical rule set into a text scene description and generating a structured semantic representation, then mapping it to an autonomous driving simulator to render and capture multi-view image sequences to generate high-fidelity visual scene data; and a multi-level question construction module for constructing multi-choice question-answer pairs based on the generated high-fidelity visual scene data to form an evaluation dataset containing different cognitive levels. A model evaluation module is also provided for training or evaluating an autonomous driving model using the evaluation dataset and optimizing the model's planning strategy based on the evaluation results.

[0009] Thirdly, this application provides a computer device, including: a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0010] Through the above technical solutions, the beneficial effects of this invention are as follows: by establishing a five-level cognitive ladder evaluation system, from single-rule understanding (L1) to multi-rule conflict resolution (L5), a hierarchical and accurate evaluation of model reasoning ability is achieved, comprehensively covering the cognitive needs of complex driving scenarios; the automatic conversion of text rules to dynamic 3D driving scenarios is realized, ensuring the semantic consistency between rules and scenarios, and improving the realism and diversity of evaluation scenarios; multimodal input evaluation is supported, and text single-modal evaluation variants can be derived to adapt to different model evaluation scenarios. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention, and the embodiments in the accompanying drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart illustrating a method for constructing an autonomous driving evaluation benchmark, as provided in an embodiment of this application;

[0013] Figure 2 An effect diagram illustrating an autonomous driving evaluation benchmark construction method provided in an embodiment of this application;

[0014] Figure 3 A structural diagram of an autonomous driving evaluation benchmark construction system provided in one embodiment of this application;

[0015] Figure 4 This is a schematic diagram of the structure of a computer device provided in one embodiment of this application;

[0016] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0017] It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of the application. Rather, these embodiments are provided to make the disclosure more thorough and complete, and to fully convey the scope of the disclosure to those skilled in the art.

[0018] The foregoing and other technical contents, features and effects of the present invention are described in conjunction with the appendix below. Figure 1-4 The detailed description of the embodiments will make this clear. All structural details mentioned in the following embodiments are based on the accompanying drawings.

[0019] Exemplary embodiments of the present invention will now be described with reference to the accompanying drawings.

[0020] In one exemplary embodiment, such as Figure 1 As shown, a method for constructing an autonomous driving evaluation benchmark is provided, including:

[0021] S101 parses the original traffic regulations into atomic rules and generates candidate rule pairs with hierarchical labels based on semantic structuring.

[0022] In step S101, official driver's manuals and traffic regulations from multiple countries are first acquired as the raw data source. Since legal texts expressed in natural language are often ambiguous and difficult to execute directly by a computer simulation system, they need to be parsed using a rule-making module (such as the Rule Crafter submodule mentioned below). Specifically, natural language processing techniques or large language models are used to assist syntactic analysis, breaking down complex, compound traffic rules into smaller atomic rules. Each atomic rule defines a specific context and a corresponding action mapping.

[0023] Building upon this foundation, to evaluate the model's ability to handle multiple tasks, these atomic rules are combined in pairs to generate candidate rule pairs. Simultaneously, these rule pairs undergo preliminary semantic structuring, analyzing the logical relationships between rules, such as whether they belong to the same type of action or whether there are normative conflicts, and assigning hierarchical labels accordingly. These labels preliminarily define whether the rule combination is a simple superposition of single rules or involves complex priority judgments.

[0024] S102, perform spatiotemporal coexistence verification on the candidate rule pairs, and construct a hierarchical rule set from basic atomic rules to complex conflict rules.

[0025] Not all rule combinations are valid in the physical world. In step S102, invalid combinations that cannot occur simultaneously in physical space or temporal logic need to be eliminated. A coexistence verification mechanism based on physical and temporal contexts is introduced to verify the coexistence of candidate rule pairs. Only rule pairs that pass the verification are retained.

[0026] Based on the selected rule pairs, a hierarchical rule set is constructed. This rule set is not merely a collection of rules, but rather arranged in a hierarchical manner according to cognitive difficulty: from understanding single atomic rules at the lowest level, to processing multiple rules in parallel, and finally reaching the highest level of resolving complex conflicts. This hierarchical design ensures that the subsequently generated evaluation benchmarks can comprehensively cover various driving scenarios from simple to difficult.

[0027] S103, the hierarchical rule set is transformed into a text scene description and a structured semantic representation is generated. Then, it is mapped to the autonomous driving simulator to render and capture multi-view image sequences to generate high-fidelity visual scene data.

[0028] The purpose of step S103 is to transform the abstract text of the aforementioned rule set into a concrete scene, primarily executed by the scene weaving module (e.g., the Scene Weaver submodule mentioned below in this application). First, the abstract rules in the hierarchical rule set are transformed into a text scene described in natural language, and further converted into a computer-executable structured semantic representation. Subsequently, this structured data is mapped into the asset library of an autonomous driving simulator (such as CARLA).

[0029] In the simulation environment, road grids are automatically generated, the initial states of traffic participants are configured, and dynamic driving trajectories are generated according to rule logic. Finally, using the simulator's rendering engine, continuous multi-view RGB image sequences are captured from the perspective of the autonomous vehicle. These image sequences not only possess high-fidelity visual details but also strictly adhere to the aforementioned defined traffic rule logic, thus ensuring consistency between visual data and rule semantics.

[0030] S104. Based on the generated high-fidelity visual scene data, construct multiple-choice question-answer pairs to form an evaluation dataset containing different cognitive levels.

[0031] To quantitatively evaluate the cognitive capabilities of autonomous driving models, simple video data is insufficient; supplementary test questions are also required. In step S104, standardized multiple-choice question-answer pairs are constructed based on the generated visual scene data. For each generated scene, test questions are generated, including a scene description, question stem, and multiple options, based on its corresponding rule level. Correct options correspond to driving behaviors that comply with traffic regulations or, in conflict scenarios, meet the highest priority rules, while distractor options contain deceptive violations. This forms a complete evaluation dataset encompassing different cognitive levels. This dataset includes both visual and textual modalities.

[0032] S105, Use the evaluation dataset to train or evaluate the autonomous driving model, and optimize the model's planning strategy based on the evaluation results.

[0033] In step S105, the constructed dataset is practically applied to the development process of autonomous driving algorithms. On one hand, this dataset can be used as an evaluation benchmark to test existing multimodal large language models or end-to-end autonomous driving models under zero-shot or other settings, calculating their accuracy on different levels of tasks, thereby accurately identifying the model's shortcomings in long-tail scenarios or complex rule reasoning. On the other hand, this dataset can also be used as training material (e.g., for supervised fine-tuning of SFT), injecting the rich traffic rule knowledge it contains into the model. Experiments show that models fine-tuned with such data can significantly improve trajectory planning accuracy in real driving environments, thereby optimizing the model's planning strategy and improving the safety and compliance of autonomous driving systems.

[0034] It is easy to see that, compared with the existing technology, this application can effectively reveal the performance bottleneck of the model in complex rule reasoning, and fine-tuning the model based on this benchmark can significantly improve the accuracy of downstream autonomous driving planning tasks, reduce trajectory deviation, and improve the safety and compliance of autonomous driving systems.

[0035] In one specific embodiment, parsing the original traffic regulations into atomic rules includes: decomposing the complex traffic regulations into a set of atomic rules through large language model-assisted syntactic analysis and manual verification, with each atomic rule defining a specific context and action mapping; and performing quadruple modeling on each atomic rule, with the quadruple including rule content, perception type, action type, and normative type.

[0036] We collected official driver's manuals and traffic regulations from multiple countries, and used a large language model to assist in syntactic analysis and manual verification to decompose complex rules into sets of atomic rules. Each atomic rule A clear context and action mapping is defined.

[0037] Each atomic rule is semantically structured and represented as a quadruple. .in, This indicates the content of the rule. Indicates the type of perception. Indicates the action type. Indicates the specification type.

[0038] The three state spaces are specifically defined as: perception space (i.e., static and dynamic); normative space (i.e., permission, obligation, prohibition); space of action It covers real-world driving behavior decisions, specifically including three categories: Category 1: Driving Control: including overtaking, left turn, right turn, U-turn, lane change, merging into the main road, entering ramps, acceleration, deceleration, reversing, and emergency lane use. Category 2: Lighting and Signaling: including left turn signal, right turn signal, low beam headlights, high beam headlights, flashing headlights, hazard lights, fog lights, side marker lights, and horn. Category 3: Parking and Yielding: including temporary parking, pulling over to the side of the road, and yielding.

[0039] In one specific embodiment, generating candidate rule pairs with hierarchical labels includes: generating a set of candidate rule pairs with consistent action types, and assigning a perceptual combination type label, a normative relationship type label, and a hierarchical level label for defining cognitive difficulty to each rule pair; wherein the hierarchical level label includes five levels from single rule comprehension L1 to multi-rule conflict resolution L5.

[0040] To capture the semantic and normative relationships between rules, a set of candidate rule pairs with consistent action types is generated. For regular rule pairs, the calculation formula is:

[0041]

[0042] This represents the complete set of atomic rules obtained by parsing and modeling in steps S101 and S102; Indicates the generated first There are 1 rule pair, each rule pair consisting of two distinct atomic rules. composition; This represents the total number of candidate rule pairs generated in the final set.

[0043] Constraints middle This represents the action space component of the atomic rule quadruple. This condition requires the selection of two atomic rules. and The actions must be of the same type. By enforcing action consistency, it is possible to capture scenarios where the same driving intention is met with different environmental constraints or regulatory restrictions. For example, when... and When both actions are deceleration, the rules It may originate from speed limit signs, but the rules... Possibly stemming from ongoing construction, this pairing method forces the evaluation model to handle multiple semantic conflicts or complements under the same action guidance, thereby effectively exploring the decision-making depth of the model under complex specifications.

[0044] For each rule pair Assign three derived labels: perceptual composition type Standardized relation types and hierarchical level .

[0045] Perceptual Combination Type The calculation formula is as follows:

[0046] in, and Representing rule pairs The original perception types of the two atomic rules. According to the perception space defined in this application, perception types include static and dynamic. The final output rule pair perception combination label is used to identify whether the rule combination belongs to dual static, dual dynamic or mixed perception mode.

[0047] Dual-static combination mode: If both rules involved in the combination originate from static perception elements (e.g., a combination of a speed limit sign and a U-turn prohibition sign), then this type is defined. This means the model only needs to process the inherent road infrastructure information in the scene, making perception relatively easy.

[0048] Dual-Dynamic Combination Mode: This type is defined as follows: if both rules involved in the combination involve dynamic perception elements (e.g., traffic lights and yield rules for pedestrians ahead). This indicates that there are multiple interactive objects with temporal changes in the scene, requiring a very high level of dynamic spatiotemporal understanding from the model.

[0049] Hybrid perception mode: When a rule pair contains both a static element and a dynamic element (e.g., a speed limit sign combined with a rule for yielding to vehicles ahead), it is classified as hybrid perception. This requires the model to be able to process both fixed regulatory constraints and instantaneous dynamic interaction information in the scene simultaneously.

[0050] Using this formula, the system can automatically identify the richness of a scene's perceptual modalities. This is not only for determining the final cognitive level. The key variables can also help developers analyze why the model fails to reason under specific combinations of perceptions. It achieves automated mapping from atomic features to rule-to-complexity, ensuring consistency and objectivity in labeling for large-scale evaluation benchmarks and reducing the workload of manual labeling.

[0051] Canonical relation types The calculation formula is as follows:

[0052] in, and Representing rule pairs The original specification types of the two atomic rules are as follows. According to the specification space defined in this application, the specification types of atomic rules include: Permissive, Obligatory (i.e., must be executed), and Forbidden. The final output of the normative relation labels is divided into two categories: norm conflict and norm harmony.

[0053] Rule conflict determination: A rule is considered to have a logical rule conflict if and only if one rule requires that an action must be performed, while another rule prohibits that action. Example: Rules When the green light is on, you must proceed, and the rules... The instruction is to prohibit passage. Since the two instructions conflict within the same action space, the model must make a decision through a higher-order priority decision.

[0054] Normative harmonization determination: All combinations other than the aforementioned conflict between obligations and prohibitions are deemed normatively harmonized. This includes situations where both rules are permissive, or one is permissive and the other is prohibitive, where no absolute command conflict arises.

[0055] Normative conflict labels are key features for identifying high-difficulty assessment samples. Rule pairs judged as normatively conflicting are typically core elements constituting the highest cognitive level (L5). Using this formula, the system can automatically filter out long-tail scenarios with decision-making challenges from massive combinations of legal texts, avoiding the subjectivity of manual selection and ensuring the depth of the assessment benchmark's examination of the model's logical reasoning ability.

[0056] Hierarchical level The formula used to define the cognitive difficulty of a scene is as follows:

[0057] Layered Decision Logic (L2-L5): L2 level: When a rule pair is "dual-static" and logically "coordinated," it is classified as L2. This means the model only needs to handle two non-conflicting fixed facility rules. L3 level: When a rule pair is "dual-dynamic" and logically "coordinated," it is classified as L3. At this level, the scenario involves predicting the behavior of multiple dynamic targets (such as a non-conflicting combination of traffic lights and pedestrians), further increasing the difficulty. L4 level: When the perception type is "Hybrid" and logically "coordinated," it is classified as L4. This requires the model to understand static traffic facility constraints while responding in real-time to changes in dynamic traffic flow. L5 level: This level has the highest priority decision authority. As long as a rule pair has "normative conflict," regardless of the perception type, it is directly classified as L5. This is the most challenging part of the evaluation system, requiring the model to make logical decisions based on specific priority principles (such as pedestrian safety being higher than traffic lights). All individual, uncombined atomic rules are uniformly marked as... This forms a complete and continuous cognitive ladder assessment system from L1 basic perception to L5 higher-order logical decision-making.

[0058] This formula establishes a metric for the cognitive capabilities of autonomous driving models, enabling evaluations to move beyond single-task limitations and quantify performance bottlenecks under varying logical complexities and perceptual stresses. The automatic labeling of scene difficulty through mathematical logic ensures the scientific rigor and objectivity of the large-scale DriveCTR dataset, providing precise data support for subsequent targeted model fine-tuning and performance optimization.

[0059] In one specific embodiment, the method further includes: for speed limit rules, adopting a pairing mechanism based on the intersection of speed intervals; mapping the rules to feasible speed intervals; if the intervals of two speed limit rules do not intersect, it is determined that there is a conflict and directly assigned to the highest conflict level L5.

[0060] For speed-limit rules, a special pairing mechanism based on the intersection of speed intervals is employed. First, atomic rules with speed-limit action type are selected and mapped to feasible speed intervals. and For such rule pairs, the calculation of their derived labels involves velocity specification relationships. The definition is as follows:

[0061] Among them, all conflicting speed limit rule pairs (i.e., no intersection between intervals) are directly assigned to the highest conflict level (L5).

[0062] In one specific embodiment, the spatiotemporal coexistence verification of the candidate rule pairs includes: extracting road type, agent state and environmental attributes from the rules using a large language model, determining whether the two rules are compatible in physical space and temporal context using dedicated instructions, and retaining only the valid rule pairs into the hierarchical rule set.

[0063] Using large language models to analyze candidate rules Perform coexistence verification of physical and temporal contexts. Using a dedicated Prompt command, extract road type, agent state, and environment attributes from the rules to determine their spatial and temporal compatibility. This indicates that rules can coexist. Ultimately, all valid rule pairs are retained. And obtain the final hierarchical rule set. .

[0064] The system inputs specialized instructions into the large language model, guiding it to determine whether two rules in a rule pair can coexist in the same space and time period, based on physical common sense, traffic logic, and temporal sequence. Spatial compatibility judgment includes verifying whether the road environments required by the two rules match. For example, if rule A is limited to an intersection with traffic lights, while rule B is limited to a roundabout without traffic lights, then they are determined to be spatially incompatible. Temporal / state compatibility judgment includes verifying whether the actions required by the rules conflict on the timeline. For example, the same agent vehicle cannot simultaneously perform emergency braking and acceleration for overtaking. In this process, the large language model outputs a binary decision result. When the model determines that the two rules can occur simultaneously in the physical world, it indicates that the rule pair can coexist. The system only retains these valid rule pairs that pass the verification.

[0065] Through the aforementioned spatiotemporal coexistence verification, the system can automatically filter out scenario logic with practical evaluation significance from a massive number of combinations, and obtain the final hierarchical rule set based on this. This not only ensures that the subsequently generated dynamic simulation scenarios conform to common sense about driving, but also lays a reliable data foundation for evaluating the reasoning ability of multimodal large models in real complex environments.

[0066] In one specific embodiment, generating high-fidelity visual scene data includes: using a self-checking alignment pipeline to convert elements in the hierarchical rule set into text scene descriptions and structured semantic representations; mapping the structured semantic representations to the simulator's asset library; generating a static initial scene through position validity checks and collision detection; generating temporal trajectories of each entity based on road topology and behavioral semantics, encoding them into simulation action structures for rendering, and capturing RGB image sequences.

[0067] Elements in the rule set Input a large language model and generate a text scene description. And further transformed into structured semantic representation. The process employs a generation-self-checking-alignment pipeline, utilizing a large language model to transform abstract rules into detailed natural language scene descriptions, including environmental constraints and agent behavior requirements. The generated descriptions undergo logical verification and are further translated into structured semantic representations. These structured representations use a domain-specific language to precisely define entity attributes, spatial relationships, and environmental settings (such as weather and lighting) within the scene.

[0068] Static initial scene construction includes: structuring semantics The asset library is mapped to the simulator. By querying the road topology, extracting lane centerlines and coordinate transformations, the initial position, orientation, and speed of each traffic participant are calculated, and weather parameters are parsed to generate OpenSCENARIO files. This stage includes position validity checks and collision detection. By querying the lane centerline and coordinate transformation, the initial position and orientation of each traffic participant are calculated, and position validity checks and collision detection are performed to ensure that there is no spatial overlap between static objects, generating a static initial scene file that conforms to physical specifications.

[0069] Dynamic scene construction and trajectory generation include: based on a static initial state, and combining road topology and behavioral semantics (such as driving along the centerline, changing lanes, following other vehicles, and interactive approach), generating a temporal trajectory for each entity. These trajectories are encoded into OpenSCENARIO action structures, which are then rendered and captured. Frame RGB image sequence .

[0070] In one specific embodiment, the method further includes, when constructing multiple-choice question-answer pairs, determining the correct action for the L5 conflict level based on the following priority principles: pedestrian safety > yielding to emergency vehicles > on-site command > traffic lights > traffic signs > road markings > interactive right-of-way > defensive driving > emergency exceptions.

[0071] Generate a set of questions for levels L1 to L5, including a scenario description, a question stem, and four options. The first step is determining the correct action: for levels L1 to L4, the correct action... To simultaneously satisfy the semantic and normative constraints of the rules, for Level 5 (conflict level), when the rules generate contradictory instructions, the correct action is determined according to the following priority principle: pedestrian safety > yielding to emergency vehicles > on-site command > traffic lights > traffic signs > road markings > right-of-way interaction > defensive driving > emergency exceptions.

[0072] The second step is option design: each question contains one correct option that conforms to the specifications. And three distractor options. These distractor options are generated and filtered by the main language model and include priority obfuscation, minor violations, or incorrect yielding behaviors. The third step is the generation and validation process: a complete question-and-answer sample is generated using the main language model (such as Claude Sonnet 4.5) in a fixed JSON format; then, independent binary validation (Valid / Invalid) is performed using multiple secondary language models (such as Gemini 2.5 Pro, GPT-5 Pro, etc.). Only when all validation models determine it to be valid is the question retained, and 5% are further sampled for human expert verification.

[0073] In one specific embodiment, the method further includes a quality scoring mechanism, which is used to independently score the generated results using multiple large language models at each stage of scene generation, and if the average score is lower than a preset threshold, manual intervention verification is triggered.

[0074] At each core stage of scene generation, the system independently evaluates the output of that stage by calling multiple heterogeneous large language models. Each model provides a score based on preset evaluation dimensions (such as logical consistency, syntactic correctness, and physical plausibility), with scores ranging from 0 to 1. The system aggregates the scores from multiple evaluation models in real time and calculates their average score. The system has a preset quality and safety threshold and compares the average score to this threshold. If the average score is higher than or equal to the preset threshold, the output quality of that stage is deemed acceptable, and the process automatically proceeds to the next stage. If the average score is lower than the preset threshold, the system automatically triggers an alert and marks the current sample as pending verification, requesting human expert intervention for manual verification or correction. Only after manual verification and confirmation can the data continue to participate in subsequent simulation rendering and benchmark construction processes. By introducing this quality control mechanism based on multi-model scoring, this invention can effectively filter out potential logical errors while maintaining a high degree of automation, significantly improving the rigor and reliability of the evaluation benchmark dataset.

[0075] In one specific embodiment, the autonomous driving model is trained or evaluated using the evaluation dataset, specifically including: systematically evaluating 14 multimodal large language models (MLLMs) covering different architectures and parameter scales, and constructing an experimental evaluation environment and model selection. The proprietary models include the GPT-5 series (nano, mini, pro), Gemini-2.5-Flash, Gemini-2.5-Pro, and Claude-Sonnet-4.5; the open-source models include the Gemma-3 series, Llama-3.2, Qwen-3-VL series, and GLM-4.5V.

[0076] All atomic traffic rules and 80% of the driving test samples were used to construct the training set, while the remaining 20% ​​was used as the evaluation set. Based on the same test set, a DriveCTR-Text variant was constructed by replacing visual frames with corresponding text scene descriptions to isolate and evaluate pure text reasoning capabilities without visual input interference. For each multiple-choice question (MCQ), four RGB images were captured as visual input.

[0077] The specific evaluation process included: using accuracy as the primary evaluation metric. All evaluations were conducted in a zero-shot setting. To ensure robustness and statistical significance, each model was run independently three times on the same test set. Furthermore, human drivers were invited to complete randomly sampled test questions, achieving an accuracy rate exceeding 98%, thus validating the reliability of the benchmark.

[0078] Benchmark tests were performed and the main results were analyzed. Table 1 shows the performance of each multimodal large model on the proposed hierarchical traffic rule reasoning-based autonomous driving system. The results show that proprietary models generally outperform open-source models, but as the task difficulty increases from L1 (atomic rules) to L5 (priority conflict adjudication), the accuracy of all models decreases. Especially at L5 level, the accuracy of the best model is only about 44%, which is far below human performance.

[0079]

[0080] Table 1. Performance of MLLM on the autonomous driving benchmark DriveCTR

[0081] Table 2 shows the model's performance on the DriveCTR-Text variant (text input only). Compared to the visual version, all models showed a modest improvement in accuracy, indicating that removing visual perception pressure contributes to clearer text reasoning, while also revealing semantic loss and alignment issues that still exist in the visual understanding stage. Even in pure text mode, the best model only achieved an accuracy of 47.42% on the L5 task.

[0082]

[0083] Table 2. Performance of MLLM on the Autonomous Driving Benchmark Text Variant DriveCTR-Text

[0084] The effectiveness of reasoning enhancement strategies was evaluated by validating three optimization strategies: Chained Thinking (CoT), Retrieval Augmentation (RAG), and Supervised Fine-Tuning (SFT). CoT represents applying a fixed template containing "step-by-step thinking." RAG represents constructing a retrieval corpus containing only the original traffic rulebook and retrieving the top 5 relevant paragraphs. SFT represents fine-tuning the model using the LoRA strategy (Rank=8) based on the LLaMA-Factory framework.

[0085] As shown in Table 3, SFT outperforms CoT and RAG, significantly improving the inference accuracy of multi-rule combination tasks. For example, Llama-3.2 achieved a 29.7% accuracy improvement on DriveCTR after fine-tuning, demonstrating that this dataset can effectively inject transferable traffic rule knowledge into MLLM. However, even the best model after fine-tuning (Qwen3-VL-8B) still only achieved a 60.2% accuracy on the L5 task, far below human levels, highlighting the limitations of traditional optimization methods in complex rule reasoning.

[0086]

[0087] Table 3. Performance of using DriveCTR, an autonomous driving benchmark data, as a knowledge injector

[0088] The end-to-end trajectory planning performance was evaluated, as shown in Table 4. The model fine-tuned using the DriveCTR dataset of this invention exhibited significantly lower L2 trajectory error on the nuScenes validation set compared to the model fine-tuned using the DriveQA dataset. This indicates that this dataset has more efficient transfer value from simulation to the real world.

[0089]

[0090] Table 4. End-to-end trajectory planning results on the nuScenes validation set (L2 error, in meters)

[0091] Analyzing the complexity of multi-rule combination reasoning, the accuracy of all models significantly decreased as the number of rules (#Rules) involved in the scenario increased. Specifically, when the number of rules increased to 4 or 5, even high-performance models (such as GPT-5 pro) experienced an accuracy drop of more than 20 percentage points. This evaluation result validates the effectiveness of the hierarchical cognitive design proposed in this application, and can systematically quantify and reveal the limitations of the model in handling high-dimensional traffic semantics.

[0092] like Figure 2The results show the performance of MLLM in multi-rule combined reasoning. "#Rules" indicates the number of traffic rules included in each scenario. The results were obtained under a zero-sample setting. Since L1 is a single-rule setting, only the results for L2–L5 are shown here.

[0093] Based on the same inventive concept, this application also provides a system. The solution provided by this system is similar to the solution described in the above method. Therefore, the specific limitations in one or more method embodiments provided below can be found in the limitations described above, and will not be repeated here.

[0094] In one exemplary embodiment, such as Figure 3 As shown, a system is provided, including:

[0095] The atomic rule extraction module is used to parse the original traffic regulations into atomic rules and generate candidate rule pairs with hierarchical labels based on semantic structuring.

[0096] The rule-scene transformation module is used to verify the spatiotemporal coexistence of the candidate rule pairs, construct a hierarchical rule set from basic atomic rules to complex conflict rules, transform the hierarchical rule set into a text scene description and generate a structured semantic representation, and then map it into an autonomous driving simulator to render and capture multi-view image sequences to generate high-fidelity visual scene data.

[0097] A multi-level question construction module is used to construct multiple-choice question-answer pairs based on the generated high-fidelity visual scene data, forming an evaluation dataset containing different cognitive levels;

[0098] The model evaluation module is used to train or evaluate the autonomous driving model using the evaluation dataset, and optimize the model's planning strategy based on the evaluation results.

[0099] The Rule2Scene Agent module, which is part of the Rule Scene Conversion module, includes the Rule Crafter and Scene Weaver sub-modules.

[0100] The atomic rule extraction module outputs a standardized set of atomic rules to the Rule2Scene Agent module; the RuleCrafter submodule performs semantic structuring and rule pair generation on the atomic rules, outputting a hierarchical rule set to the SceneWeaver submodule; the SceneWeaver submodule converts the rules into dynamic 3D scene data, combines it with the test questions generated by the multi-level question building module, and inputs them into the model evaluation module; the model evaluation module outputs evaluation results for model optimization.

[0101] The atomic rule extraction module standardizes rules through LLM-assisted syntactic analysis and manual verification; the RuleCrafter submodule completes rule layering through semantic structuring algorithms, rule pair generation algorithms, and spatiotemporal coexistence verification algorithms; the Scene Weaver submodule transforms rules into scenes through scene description generation, structured semantic transformation, CARLA simulation mapping, and image rendering algorithms; the multi-level problem construction module generates test problems of corresponding difficulty based on the hierarchical rule set; and the model evaluation module completes model performance evaluation through accuracy calculation and multi-round verification logic.

[0102] By using Rule2Scene Agent, a closed loop of automated generation of traffic rules from text to visual scenes was achieved. The constructed DriveCTR benchmark not only covers the full cognitive levels from L1 to L5, but also demonstrates through experiments its significant technical effect in improving the reasoning ability and trajectory planning accuracy of autonomous driving models in long-tail scenarios.

[0103] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 4 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores underwater dynamic mud level measurement data based on a rotary encoder. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements an underwater dynamic mud level measurement method based on a rotary encoder.

[0104] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0105] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0106] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0107] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0108] It should be noted that the engineering cost investment information and data involved in this application (including but not limited to data used for analysis, data stored, data displayed, etc.) are all information and data authorized by the client or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0109] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (Read-Only Memory). Memory includes ROM, magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0110] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0111] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0112] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for constructing an autonomous driving evaluation benchmark, characterized in that, The method includes: By using a large language model to assist syntactic analysis and manual verification, the complex traffic regulations in the original traffic regulations are decomposed into a set of atomic rules. Each atomic rule defines a specific context and action mapping. A four-tuple model is then performed on each atomic rule, including rule content, perception type, action type, and normative type, completing the atomic rule parsing of the original traffic regulations. Based on semantic structuring, candidate rule pairs with hierarchical labels are generated, resulting in a set of candidate rule pairs with consistent action types. Each rule pair is assigned a perception combination type label, a normative relation type label, and a hierarchical level label to define cognitive difficulty. The hierarchical level labels include five levels, from single-rule comprehension (L1) to multi-rule conflict resolution (L5). The candidate rule pairs are verified for spatiotemporal coexistence. The road type, agent state and environmental attributes in the rules are extracted using a large language model. The compatibility of two rules in physical space and temporal context is determined by special instructions. Only valid rule pairs are retained, and a hierarchical rule set from basic atomic rules to complex conflict rules is constructed. A self-checking alignment pipeline is adopted to transform the hierarchical rule set into a text scene description and generate a structured semantic representation. The structured semantic representation is mapped to the asset library of the autonomous driving simulator, and a static initial scene is generated through position legality check and collision detection. The temporal trajectory of each entity is generated based on road topology and behavioral semantics, and encoded into a simulation action structure for rendering. Multi-view RGB image sequences are captured to generate high-fidelity visual scene data. Based on the generated high-fidelity visual scene data, multiple-choice question-and-answer pairs are constructed to form an evaluation dataset containing different cognitive levels. Among them, when constructing multiple-choice question-and-answer pairs for the L5 conflict level, the correct action is determined according to the following priority principle: pedestrian safety > avoidance of special vehicles > on-site command > traffic lights > traffic signs > road markings > interactive right-of-way > defensive driving > emergency exceptions. The autonomous driving model is trained or evaluated using the evaluation dataset, and the planning strategy of the model is optimized based on the evaluation results.

2. The method according to claim 1, characterized in that, The method further includes: for speed limit rules, a pairing mechanism based on the intersection of speed intervals is adopted; the rule is mapped to a feasible speed interval; if the intervals of two speed limit rules do not intersect, it is determined that there is a conflict and it is directly assigned to the highest conflict level L5.

3. The method according to claim 1, characterized in that, The method further includes: The quality scoring mechanism is used to independently score the generated results using multiple large language models at each stage of scene generation. If the average score is lower than a preset threshold, manual intervention is triggered for verification.

4. An autonomous driving evaluation benchmark construction system, used to execute the method as described in any one of claims 1 to 3, characterized in that, The system includes: The atomic rule extraction module is used to parse the original traffic regulations into atomic rules and generate candidate rule pairs with hierarchical labels based on semantic structuring. The rule-scene transformation module is used to verify the spatiotemporal coexistence of the candidate rule pairs, construct a hierarchical rule set from basic atomic rules to complex conflict rules, transform the hierarchical rule set into a text scene description and generate a structured semantic representation, and then map it into an autonomous driving simulator to render and capture multi-view image sequences to generate high-fidelity visual scene data. A multi-level question construction module is used to construct multiple-choice question-answer pairs based on the generated high-fidelity visual scene data, forming an evaluation dataset containing different cognitive levels; The model evaluation module is used to train or evaluate the autonomous driving model using the evaluation dataset, and optimize the model's planning strategy based on the evaluation results.

5. A computer device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method as described in any one of claims 1 to 3.

Images