Automatic driving evaluation benchmark construction method, system and equipment

Abstract

The invention discloses an automatic driving evaluation benchmark construction method, system and device, and belongs to the technical field of automatic driving evaluation. The method comprises the following steps: analyzing original traffic regulations into atomic rules, and generating candidate rule pairs with hierarchical labels; performing space-time coexistence verification on the candidate rule pair, and constructing a hierarchical rule set; converting the hierarchical rule set into a text scene description and generating a structured semantic representation, then mapping the structured semantic representation into an automatic driving simulator, rendering and capturing a multi-view image sequence, and generating high-fidelity visual scene data; constructing a multi-choice question and answer pair based on the generated high-fidelity visual scene data, and forming an evaluation data set containing different cognitive levels; and training or evaluating an automatic driving model by using the evaluation data set, and optimizing a planning strategy of the model according to an evaluation result. The technical problem of automatic driving model evaluation is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving evaluation, in particular to an automatic driving evaluation benchmark construction method, system and device. BACKGROUND

[0002] With the rapid development of automatic driving technology, multi-modal large language models are gradually being integrated into the decision-making and planning layer of automatic driving systems due to their strong semantic understanding and logical reasoning capabilities. In order to verify the safety and compliance of these models in handling complex traffic environments, it has become a key requirement in the industry to construct a scientific and rigorous evaluation benchmark.

[0003] However, existing multi-modal evaluation benchmarks in the field of automatic driving still have significant limitations in dealing with complex traffic rule reasoning: first, most existing benchmarks only focus on the understanding of a single traffic rule, such as simple traffic sign recognition or single right-of-way judgment. In real driving scenarios, there are often complex situations where multiple rules occur simultaneously, or even conflict with each other. Existing benchmarks are difficult to cover such multi-rule combination reasoning scenarios; second, evaluation datasets rely heavily on 2D static images, lacking support for high-fidelity dynamic 3D driving scenarios. Such static data cannot simulate the timing interaction and dynamic environmental changes in the actual driving process, leading to deviations in evaluation results and real driving performance; third, there is no hierarchical cognitive evaluation system established, which cannot systematically quantify the reasoning level of models from basic perception to complex logical decision-making. This leads to the possibility that models may perform well in simple tasks, but when faced with long-tail scenarios or rule priority conflicts, they are prone to decision-making errors or violations of traffic rules, making it difficult to meet the core needs of automatic driving system safety and compliance.

[0004] Therefore, how to construct a set of multi-modal input supported, hierarchical cognitive evaluation benchmark that can realize atomic rule to multi-rule conflict resolution, and realize the automatic conversion of text rules to high-fidelity dynamic simulation scenarios, has become a technical problem to be solved in the field of automatic driving model evaluation. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application provides an automatic driving evaluation benchmark construction method, system and device, which aims to solve the problems of the prior art.

[0006] To achieve the above-mentioned purpose, the present application provides the following solutions: In a first aspect, the application provides a method for constructing an automatic driving evaluation benchmark, comprising: parsing original traffic regulations into atomic rules, and generating candidate rule pairs with hierarchical labels based on semantic structuring; verifying the spatiotemporal coexistence of the candidate rule pairs, and constructing a hierarchical rule set from basic atomic rules to complex conflict rules; converting the hierarchical rule set into a text scene description and generating a structured semantic representation, which is then mapped into an automatic driving simulator to render and capture a multi-view image sequence, and generate high-fidelity visual scene data; constructing multiple-choice question and answer pairs based on the generated high-fidelity visual scene data, and forming an evaluation dataset containing different cognitive levels; training or evaluating an automatic driving model using the evaluation dataset, and optimizing the planning strategy of the model according to the evaluation results.

[0007] In a second aspect, the application provides a system for constructing an automatic driving evaluation benchmark, comprising: an atomic rule extraction module for parsing original traffic regulations into atomic rules, and generating candidate rule pairs with hierarchical labels based on semantic structuring; a rule scene conversion module for verifying the spatiotemporal coexistence of the candidate rule pairs, and constructing a hierarchical rule set from basic atomic rules to complex conflict rules; converting the hierarchical rule set into a text scene description and generating a structured semantic representation, which is then mapped into an automatic driving simulator to render and capture a multi-view image sequence, and generate high-fidelity visual scene data; a multi-level question construction module for constructing multiple-choice question and answer pairs based on the generated high-fidelity visual scene data, and forming an evaluation dataset containing different cognitive levels; and a model evaluation module for training or evaluating an automatic driving model using the evaluation dataset, and optimizing the planning strategy of the model according to the evaluation results.

[0008] In a third aspect, the application provides a computer device, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program. Through the above technical solutions, the application has the following beneficial effects: by establishing a five-level cognitive ladder evaluation system, from single rule understanding L1 to multi-rule conflict resolution L5, the hierarchical and accurate evaluation of model reasoning ability is realized, and the cognitive needs of complex driving scenarios are fully covered; the automatic conversion from text rules to dynamic 3D driving scenes is realized, the semantic consistency of rules and scenes is ensured, and the authenticity and diversity of the evaluation scene are improved; multi-modal input evaluation is supported, and text single-modal evaluation variants can be derived, which is suitable for different model evaluation scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments description will be briefly introduced. Obviously, the drawings in the following description only constitute some embodiments of the present application, and the embodiments in the drawings do not constitute any limitation to the present application. Other drawings can also be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0010] Figure 1 A flow chart of an automatic driving evaluation benchmark construction method provided by an embodiment of the present application is shown in FIG. 1. Figure 2 An effect diagram of an automatic driving evaluation benchmark construction method provided by an embodiment of the present application is shown in FIG. 2. Figure 3 A structural diagram of an automatic driving evaluation benchmark construction system provided by an embodiment of the present application is shown in FIG. 3. Figure 4 A structural diagram of a computer device provided by an embodiment of the present application is shown in FIG. 4. The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0011] It should be understood that the specific embodiments described herein are intended to explain the present application, but not to limit the present application. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0012] The foregoing and other technical contents, features and effects of the present application will be described below in conjunction with the accompanying drawings. Figures 1-4 The detailed description of the embodiments will be clearly presented. The structural contents mentioned in the following embodiments are all referred to the drawings.

[0013] The exemplary embodiments of the present application will be described below with reference to the accompanying drawings.

[0014] In one exemplary embodiment, as shown in FIG. 1, an automatic driving evaluation benchmark construction method is provided, comprising: Figure 1 S101, parsing the original traffic regulations into atomic rules, and generating candidate rule pairs with hierarchical labels based on semantic structuring. S101, parsing the original traffic regulations into atomic rules, and generating candidate rule pairs with hierarchical labels based on semantic structuring.

[0015] In step S101, first, the official driving manuals and traffic regulations of multiple countries are acquired as the original data source. Since the legal text expressed in natural language is ambiguous and difficult to be directly executed by the computer simulation system, it needs to be parsed by a rule-making module (such as the Rule Crafter sub-module mentioned below). Specifically, the complex composite traffic rules are decomposed into small atomic rules by using natural language processing technology or large language model assisted syntax analysis. Each atomic rule defines a specific context and corresponding action mapping.

[0016] On this basis, in order to evaluate the ability of the model to handle multiple tasks, the atomic rules are combined two by two to generate candidate rule pairs. At the same time, these rule pairs are preliminarily semantically structured and processed to analyze the logical relationship between the rules, such as whether they belong to the same type of action, whether there is a standard opposition, etc., and accordingly labeled with a hierarchical tag. These tags preliminarily define whether the rule combination belongs to a simple single rule superposition or involves a complex priority judgment.

[0017] S102, the spatiotemporal coexistence of the candidate rule pairs is verified, and a hierarchical rule set from the basic atomic rules to the complex conflict rules is constructed.

[0018] Not all rule combinations are reasonable in the physical world. In step S102, invalid combinations that cannot occur simultaneously in physical space or time logic need to be removed. By introducing a verification mechanism of physical and temporal context, the coexistence of the candidate rule pairs is verified. Only the rule pairs that pass the verification will be retained.

[0019] According to the filtered rule pairs, a hierarchical rule set is constructed. The rule set is not just a stack of rules, but is arranged in a hierarchical manner according to the cognitive difficulty: from the understanding of the simplest single atomic rule at the bottom, to the parallel processing of multiple rules, and finally to the solution of the most complex conflict rules at the highest level. This hierarchical design ensures that the evaluation benchmarks generated subsequently can fully cover various driving scenarios from simple to difficult.

[0020] S103, the hierarchical rule set is converted into a text scene description and a structured semantic representation is generated, which is then mapped to an autonomous driving simulator, rendering and capturing multiple view image sequences, and generating high-fidelity visual scene data.

[0021] The purpose of step S103 is to transform the abstract text of the aforementioned rule set into a concrete scene, primarily executed by the scene weaving module (e.g., the Scene Weaver submodule mentioned below in this application). First, the abstract rules in the hierarchical rule set are transformed into a text scene described in natural language, and further converted into a computer-executable structured semantic representation. Subsequently, this structured data is mapped into the asset library of an autonomous driving simulator (such as CARLA).

[0022] In the simulation environment, road grids are automatically generated, the initial states of traffic participants are configured, and dynamic driving trajectories are generated according to rule logic. Finally, using the simulator's rendering engine, continuous multi-view RGB image sequences are captured from the perspective of the autonomous vehicle. These image sequences not only possess high-fidelity visual details but also strictly adhere to the aforementioned defined traffic rule logic, thus ensuring consistency between visual data and rule semantics.

[0023] S104. Based on the generated high-fidelity visual scene data, construct multiple-choice question-answer pairs to form an evaluation dataset containing different cognitive levels.

[0024] To quantitatively evaluate the cognitive capabilities of autonomous driving models, simple video data is insufficient; supplementary test questions are also required. In step S104, standardized multiple-choice question-answer pairs are constructed based on the generated visual scene data. For each generated scene, test questions are generated, including a scene description, question stem, and multiple options, based on its corresponding rule level. Correct options correspond to driving behaviors that comply with traffic regulations or, in conflict scenarios, meet the highest priority rules, while distractor options contain deceptive violations. This forms a complete evaluation dataset encompassing different cognitive levels. This dataset includes both visual and textual modalities.

[0025] S105, Use the evaluation dataset to train or evaluate the autonomous driving model, and optimize the model's planning strategy based on the evaluation results.

[0026] In step S105, the constructed dataset is practically applied to the development process of autonomous driving algorithms. On one hand, this dataset can be used as an evaluation foundation to test existing multimodal large language models or end-to-end autonomous driving models under zero-shot or other settings, calculating their accuracy on different levels of tasks, thereby accurately identifying the model's shortcomings in long-tail scenarios or complex rule reasoning. On the other hand, this dataset can also be used as training material (e.g., for supervised fine-tuning of SFT), injecting the rich traffic rule knowledge it contains into the model. Experiments show that models fine-tuned with such data can significantly improve trajectory planning accuracy in real driving environments, thereby optimizing the model's planning strategy and improving the safety and compliance of autonomous driving systems.

[0027] It is easy to see that, compared with the existing technology, this application can effectively reveal the performance bottleneck of the model in complex rule reasoning, and fine-tuning the model based on this benchmark can significantly improve the accuracy of downstream autonomous driving planning tasks, reduce trajectory deviation, and improve the safety and compliance of autonomous driving systems.

[0028] In one specific embodiment, parsing the original traffic regulations into atomic rules includes: decomposing the complex traffic regulations into a set of atomic rules through large language model-assisted syntactic analysis and manual verification, with each atomic rule defining a specific context and action mapping; and performing quadruple modeling on each atomic rule, with the quadruple including rule content, perception type, action type, and normative type.

[0029] We collected official driver's manuals and traffic regulations from multiple countries, and used a large language model to assist in syntactic analysis and manual verification to decompose complex rules into sets of atomic rules. Each atomic rule A clear context and action mapping is defined.

[0030] Each atomic rule is semantically structured and represented as a quadruple. .in, This indicates the content of the rule. Indicates the type of perception. Indicates the action type. Indicates the specification type.

[0031] The three state spaces are specifically defined as: perception space (i.e., static and dynamic); normative space (i.e., permission, obligation, prohibition); space of action It covers real-world driving behavior decisions, specifically including three categories: Category 1: Driving Control: including overtaking, left turn, right turn, U-turn, lane change, merging into the main road, entering ramps, acceleration, deceleration, reversing, and emergency lane use. Category 2: Lighting and Signaling: including left turn signal, right turn signal, low beam headlights, high beam headlights, flashing headlights, hazard lights, fog lights, side marker lights, and horn. Category 3: Parking and Yielding: including temporary parking, pulling over to the side of the road, and yielding.

[0032] In one specific embodiment, generating candidate rule pairs with hierarchical labels includes: generating a set of candidate rule pairs with consistent action types, and assigning a perceptual combination type label, a normative relationship type label, and a hierarchical level label for defining cognitive difficulty to each rule pair; wherein the hierarchical level label includes five levels from single rule comprehension L1 to multi-rule conflict resolution L5.

[0033] To capture the semantic and normative relationships between rules, a set of candidate rule pairs with consistent action types is generated. For regular rule pairs, the calculation formula is:

[0034] This represents the complete set of atomic rules obtained by parsing and modeling in steps S101 and S102; Indicates the generated first There are 1 rule pair, each rule pair consisting of two distinct atomic rules. composition; This represents the total number of candidate rule pairs generated in the final set.

[0035] Constraints middle This represents the action space component of the atomic rule quadruple. This condition requires the selection of two atomic rules. and The actions must be of the same type. By enforcing action consistency, it is possible to capture scenarios where the same driving intention is met with different environmental constraints or regulatory restrictions. For example, when... and When both actions are deceleration, the rules It may originate from speed limit signs, but the rules... Possibly stemming from ongoing construction, this pairing method forces the evaluation model to handle multiple semantic conflicts or complements under the same action guidance, thereby effectively exploring the decision-making depth of the model under complex specifications.

[0036] For each rule pair Assign three derived labels: perceptual composition type Standardized relation types and hierarchical level .

[0037] Perceptual Combination Type The calculation formula is as follows: in, and Representing rule pairs The original perception types of the two atomic rules. According to the perception space defined in this application, perception types include static and dynamic. The final output rule pair perception combination label is used to identify whether the rule combination belongs to dual static, dual dynamic or mixed perception mode.

[0038] Dual-static combination mode: If both rules involved in the combination originate from static perception elements (e.g., a combination of a speed limit sign and a U-turn prohibition sign), then this type is defined. This means the model only needs to process the inherent road infrastructure information in the scene, making perception relatively easy.

[0039] Dual-Dynamic Combination Mode: This type is defined as follows: if both rules involved in the combination involve dynamic perception elements (e.g., traffic lights and yield rules for pedestrians ahead). This indicates that there are multiple interactive objects with temporal changes in the scene, requiring a very high level of dynamic spatiotemporal understanding from the model.

[0040] Hybrid perception mode: When a rule pair contains both a static element and a dynamic element (e.g., a speed limit sign combined with a rule for yielding to vehicles ahead), it is classified as hybrid perception. This requires the model to be able to process both fixed regulatory constraints and instantaneous dynamic interaction information in the scene simultaneously.

[0041] Using this formula, the system can automatically identify the richness of a scene's perceptual modalities. This is not only for determining the final cognitive level. The key variables can also help developers analyze why the model fails to reason under specific combinations of perceptions. It achieves automated mapping from atomic features to rule-to-complexity, ensuring consistency and objectivity in labeling for large-scale evaluation benchmarks and reducing the workload of manual labeling.

[0042] Standardized relation types The calculation formula is as follows: in, and Representing rule pairs The original specification types of the two atomic rules are as follows. According to the specification space defined in this application, the specification types of atomic rules include: Permissive, Obligatory (i.e., must be executed), and Forbidden. The final output of the normative relation labels is divided into two categories: norm conflict and norm harmony.

[0043] Rule conflict determination: A rule is considered to have a logical rule conflict if and only if one rule requires that an action must be performed, while another rule prohibits that action. Example: Rules When the green light is on, you must proceed, and the rules... The instruction is to prohibit passage. Since the two instructions conflict within the same action space, the model must make a decision through a higher-order priority decision.

[0044] Normative harmonization determination: All combinations other than the aforementioned conflict between obligations and prohibitions are deemed normatively harmonized. This includes situations where both rules are permissive, or one is permissive and the other is prohibitive, where no absolute command conflict arises.

[0045] Normative conflict labels are key features for identifying high-difficulty assessment samples. Rule pairs judged as normatively conflicting are typically core elements constituting the highest cognitive level (L5). Using this formula, the system can automatically filter out long-tail scenarios with decision-making challenges from massive combinations of legal texts, avoiding the subjectivity of manual selection and ensuring the depth of the assessment benchmark's examination of the model's logical reasoning ability.

[0046] Hierarchical level The formula used to define the cognitive difficulty of a scene is as follows: Layered Decision Logic (L2-L5): L2 level: When a rule pair is "dual-static" and logically "coordinated," it is classified as L2. This means the model only needs to handle two non-conflicting fixed facility rules. L3 level: When a rule pair is "dual-dynamic" and logically "coordinated," it is classified as L3. At this level, the scenario involves predicting the behavior of multiple dynamic targets (such as a non-conflicting combination of traffic lights and pedestrians), further increasing the difficulty. L4 level: When the perception type is "Hybrid" and logically "coordinated," it is classified as L4. This requires the model to understand static traffic facility constraints while responding in real-time to changes in dynamic traffic flow. L5 level: This level has the highest priority decision authority. As long as a rule pair has "normative conflict," regardless of the perception type, it is directly classified as L5. This is the most challenging part of the evaluation system, requiring the model to make logical decisions based on specific priority principles (such as pedestrian safety being higher than traffic lights). All individual, uncombined atomic rules are uniformly marked as... This forms a complete and continuous cognitive ladder assessment system from L1 basic perception to L5 higher-order logical decision-making.

[0047] This formula establishes a metric for the cognitive capabilities of autonomous driving models, enabling evaluations to move beyond single-task limitations and quantify performance bottlenecks under varying logical complexities and perceptual stresses. The automatic labeling of scene difficulty through mathematical logic ensures the scientific rigor and objectivity of the large-scale DriveCTR dataset, providing precise data support for subsequent targeted model fine-tuning and performance optimization.

[0048] In one specific embodiment, the method further includes: for speed limit rules, adopting a pairing mechanism based on the intersection of speed intervals; mapping the rules to feasible speed intervals; if the intervals of two speed limit rules do not intersect, it is determined that there is a conflict and directly assigned to the highest conflict level L5.

[0049] For speed-limit rules, a special pairing mechanism based on the intersection of speed intervals is employed. First, atomic rules with speed-limit action type are selected and mapped to feasible speed intervals. and For such rule pairs, the calculation of their derived labels involves velocity specification relationships. The definition is as follows: Among them, all conflicting speed limit rule pairs (i.e., no intersection between intervals) are directly assigned to the highest conflict level (L5).

[0050] In one specific embodiment, the spatiotemporal coexistence verification of the candidate rule pairs includes: extracting road type, agent state and environmental attributes from the rules using a large language model, determining whether the two rules are compatible in physical space and temporal context using dedicated instructions, and retaining only the valid rule pairs into the hierarchical rule set.

[0051] Using large language models to analyze candidate rules Perform coexistence verification of physical and temporal contexts. Using a dedicated Prompt command, extract road type, agent state, and environment attributes from the rules to determine their spatial and temporal compatibility. This indicates that rules can coexist. Ultimately, all valid rule pairs are retained. And obtain the final hierarchical rule set. .

[0052] The system inputs specialized instructions into the large language model, guiding it to determine whether two rules in a rule pair can coexist in the same space and time period, based on physical common sense, traffic logic, and temporal sequence. Spatial compatibility judgment includes verifying whether the road environments required by the two rules match. For example, if rule A is limited to an intersection with traffic lights, while rule B is limited to a roundabout without traffic lights, then they are determined to be spatially incompatible. Temporal / state compatibility judgment includes verifying whether the actions required by the rules conflict on the timeline. For example, the same agent vehicle cannot simultaneously perform emergency braking and acceleration for overtaking. In this process, the large language model outputs a binary decision result. When the model determines that the two rules can occur simultaneously in the physical world, it indicates that the rule pair can coexist. The system only retains these valid rule pairs that pass the verification.

[0053] Through the aforementioned spatiotemporal coexistence verification, the system can automatically filter out scenario logic with practical evaluation significance from a massive number of combinations, and obtain the final hierarchical rule set based on this. This not only ensures that the subsequently generated dynamic simulation scenarios conform to common sense about driving, but also lays a reliable data foundation for evaluating the reasoning ability of multimodal large models in real complex environments.

[0054] In one specific embodiment, generating high-fidelity visual scene data includes: using a self-checking alignment pipeline to convert elements in the hierarchical rule set into text scene descriptions and structured semantic representations; mapping the structured semantic representations to the simulator's asset library; generating a static initial scene through position validity checks and collision detection; generating temporal trajectories of each entity based on road topology and behavioral semantics, encoding them into simulation action structures for rendering, and capturing RGB image sequences.

[0055] Elements in the rule set Input a large language model and generate a text scene description. And further transformed into structured semantic representation. The process employs a generation-self-checking-alignment pipeline, utilizing a large language model to transform abstract rules into detailed natural language scene descriptions, including environmental constraints and agent behavior requirements. The generated descriptions undergo logical verification and are further translated into structured semantic representations. These structured representations use a domain-specific language to precisely define entity attributes, spatial relationships, and environmental settings (such as weather and lighting) within the scene.

[0056] Static initial scene construction includes: structuring semantics The asset library is mapped to the simulator. By querying the road topology, extracting lane centerlines and coordinate transformations, the initial position, orientation, and speed of each traffic participant are calculated, and weather parameters are parsed to generate OpenSCENARIO files. This stage includes position validity checks and collision detection. By querying the lane centerline and coordinate transformation, the initial position and orientation of each traffic participant are calculated, and position validity checks and collision detection are performed to ensure that there is no spatial overlap between static objects, generating a static initial scene file that conforms to physical specifications. Dynamic scene construction and trajectory generation include: based on a static initial state, and combining road topology and behavioral semantics (such as driving along the centerline, changing lanes, following other vehicles, and interactive approach), generating a temporal trajectory for each entity. These trajectories are encoded into OpenSCENARIO action structures, which are then rendered and captured. Frame RGB image sequence .

[0057] In one specific embodiment, the method further includes, when constructing multiple-choice question-answer pairs, determining the correct action for the L5 conflict level based on the following priority principles: pedestrian safety > yielding to emergency vehicles > on-site command > traffic lights > traffic signs > road markings > interactive right-of-way > defensive driving > emergency exceptions.

[0058] Generate a set of questions for levels L1 to L5, including a scenario description, a question stem, and four options. The first step is determining the correct action: for levels L1 to L4, the correct action... To simultaneously satisfy the semantic and normative constraints of the rules, for Level 5 (conflict level), when the rules generate contradictory instructions, the correct action is determined according to the following priority principle: pedestrian safety > yielding to emergency vehicles > on-site command > traffic lights > traffic signs > road markings > right-of-way interaction > defensive driving > emergency exceptions.

[0059] The second step is option design: each question contains one correct option that conforms to the specifications. And three distractor options. These distractor options are generated and filtered by the main language model and include priority obfuscation, minor violations, or incorrect yielding behaviors. The third step is the generation and validation process: a complete question-and-answer sample is generated using the main language model (such as Claude Sonnet 4.5) in a fixed JSON format; then, independent binary validation (Valid / Invalid) is performed using multiple secondary language models (such as Gemini 2.5 Pro, GPT-5 Pro, etc.). Only when all validation models determine it to be valid is the question retained, and 5% are further sampled for human expert verification.

[0060] In one specific embodiment, the method further includes a quality scoring mechanism, which is used to independently score the generated results using multiple large language models at each stage of scene generation, and if the average score is lower than a preset threshold, manual intervention verification is triggered.

[0061] At each core stage of scene generation, the system independently evaluates the output of that stage by calling multiple heterogeneous large language models. Each model provides a score based on preset evaluation dimensions (such as logical consistency, syntactic correctness, and physical plausibility), with scores ranging from 0 to 1. The system aggregates the scores from multiple evaluation models in real time and calculates their average score. The system has a preset quality and safety threshold and compares the average score to this threshold. If the average score is higher than or equal to the preset threshold, the output quality of that stage is deemed acceptable, and the process automatically proceeds to the next stage. If the average score is lower than the preset threshold, the system automatically triggers an alert and marks the current sample as pending verification, requesting human expert intervention for manual verification or correction. Only after manual verification and confirmation can the data continue to participate in subsequent simulation rendering and benchmark construction processes. By introducing this quality control mechanism based on multi-model scoring, this invention can effectively filter out potential logical errors while maintaining a high degree of automation, significantly improving the rigor and reliability of the evaluation benchmark dataset.

[0062] In one specific embodiment, the autonomous driving model is trained or evaluated using the evaluation dataset, specifically including: systematically evaluating 14 multimodal large language models (MLLMs) covering different architectures and parameter scales, and constructing an experimental evaluation environment and model selection. The proprietary models include the GPT-5 series (nano, mini, pro), Gemini-2.5-Flash, Gemini-2.5-Pro, and Claude-Sonnet-4.5; the open-source models include the Gemma-3 series, Llama-3.2, Qwen-3-VL series, and GLM-4.5V.

[0063] All atomic traffic rules and 80% of the driving test samples were used to construct the training set, while the remaining 20% ​​was used as the evaluation set. Based on the same test set, a DriveCTR-Text variant was constructed by replacing visual frames with corresponding text scene descriptions to isolate and evaluate pure text reasoning capabilities without visual input interference. For each multiple-choice question (MCQ), four RGB images were captured as visual input.

[0064] The specific evaluation process included: using accuracy as the primary evaluation metric. All evaluations were conducted in a zero-shot setting. To ensure robustness and statistical significance, each model was run independently three times on the same test set. Furthermore, human drivers were invited to complete randomly sampled test questions, achieving an accuracy rate exceeding 98%, thus validating the reliability of the benchmark.

[0065] Benchmark tests were performed and the main results were analyzed. Table 1 shows the performance of each multimodal large model on the proposed hierarchical traffic rule reasoning-based autonomous driving system. The results show that proprietary models generally outperform open-source models, but as the task difficulty increases from L1 (atomic rules) to L5 (priority conflict adjudication), the accuracy of all models decreases. Especially at L5 level, the accuracy of the best model is only about 44%, which is far below human performance.

[0066]

[0067] Table 1. Performance of MLLM on the autonomous driving benchmark DriveCTR Table 2 shows the model's performance on the DriveCTR-Text variant (text input only). Compared to the visual version, all models showed a modest improvement in accuracy, indicating that removing visual perception pressure contributes to clearer text reasoning, while also revealing semantic loss and alignment issues that still exist in the visual understanding stage. Even in pure text mode, the best model only achieved an accuracy of 47.42% on the L5 task.

[0068]

[0069] Table 2. Performance of MLLM on the Autonomous Driving Benchmark Text Variant DriveCTR-Text The effectiveness of reasoning enhancement strategies was evaluated by validating three optimization strategies: Chained Thinking (CoT), Retrieval Augmentation (RAG), and Supervised Fine-Tuning (SFT). CoT represents applying a fixed template containing "step-by-step thinking." RAG represents constructing a retrieval corpus containing only the original traffic rulebook and retrieving the top 5 relevant paragraphs. SFT represents fine-tuning the model using the LoRA strategy (Rank=8) based on the LLaMA-Factory framework.

[0070] As shown in Table 3, SFT outperforms CoT and RAG, significantly improving the inference accuracy of multi-rule combination tasks. For example, Llama-3.2 achieved a 29.7% accuracy improvement on DriveCTR after fine-tuning, demonstrating that this dataset can effectively inject transferable traffic rule knowledge into MLLM. However, even the best model after fine-tuning (Qwen3-VL-8B) still only achieved a 60.2% accuracy on the L5 task, far below human levels, highlighting the limitations of traditional optimization methods in complex rule reasoning.

[0071]

[0072] Table 3. Performance of using DriveCTR, an autonomous driving benchmark data, as a knowledge injector The end-to-end trajectory planning performance was evaluated, as shown in Table 4. The model fine-tuned using the DriveCTR dataset of this invention exhibited significantly lower L2 trajectory error on the nuScenes validation set compared to the model fine-tuned using the DriveQA dataset. This indicates that this dataset has more efficient transfer value from simulation to the real world.

[0073]

[0074] Table 4. End-to-end trajectory planning results on the nuScenes validation set (L2 error, in meters) Analyzing the complexity of multi-rule combination reasoning, the accuracy of all models significantly decreased as the number of rules (#Rules) involved in the scenario increased. Specifically, when the number of rules increased to 4 or 5, even high-performance models (such as GPT-5 pro) experienced an accuracy drop of more than 20 percentage points. This evaluation result validates the effectiveness of the hierarchical cognitive design proposed in this application, and can systematically quantify and reveal the limitations of the model in handling high-dimensional traffic semantics.

[0075] like Figure 2 The results show the performance of MLLM in multi-rule combined reasoning. "#Rules" indicates the number of traffic rules included in each scenario. The results were obtained under a zero-sample setting. Since L1 is a single-rule setting, only the results for L2–L5 are shown here.

[0076] Based on the same inventive concept, this application also provides a system. The solution provided by this system is similar to the solution described in the above method. Therefore, the specific limitations in one or more method embodiments provided below can be found in the limitations described above, and will not be repeated here.

[0077] In one exemplary embodiment, such as Figure 3 As shown, a system is provided, including: The atomic rule extraction module is used to parse the original traffic regulations into atomic rules and generate candidate rule pairs with hierarchical labels based on semantic structuring. The rule-scene transformation module is used to verify the spatiotemporal coexistence of the candidate rule pairs, construct a hierarchical rule set from basic atomic rules to complex conflict rules, transform the hierarchical rule set into a text scene description and generate a structured semantic representation, and then map it into an autonomous driving simulator to render and capture multi-view image sequences to generate high-fidelity visual scene data. A multi-level question construction module is used to construct multiple-choice question-answer pairs based on the generated high-fidelity visual scene data, forming an evaluation dataset containing different cognitive levels; The model evaluation module is used to train or evaluate the autonomous driving model using the evaluation dataset, and optimize the model's planning strategy based on the evaluation results. The Rule2Scene Agent module, which is part of the Rule Scene Conversion module, includes the Rule Crafter and Scene Weaver sub-modules.

[0078] The atomic rule extraction module outputs a standardized set of atomic rules to the Rule2Scene Agent module; the RuleCrafter submodule performs semantic structuring and rule pair generation on the atomic rules, outputting a hierarchical rule set to the SceneWeaver submodule; the SceneWeaver submodule converts the rules into dynamic 3D scene data, combines it with the test questions generated by the multi-level question building module, and inputs them into the model evaluation module; the model evaluation module outputs evaluation results for model optimization.

[0079] The atomic rule extraction module standardizes rules through LLM-assisted syntactic analysis and manual verification; the RuleCrafter submodule completes rule layering through semantic structuring algorithms, rule pair generation algorithms, and spatiotemporal coexistence verification algorithms; the Scene Weaver submodule transforms rules into scenes through scene description generation, structured semantic transformation, CARLA simulation mapping, and image rendering algorithms; the multi-level problem construction module generates test problems of corresponding difficulty based on the hierarchical rule set; and the model evaluation module completes model performance evaluation through accuracy calculation and multi-round verification logic.

[0080] By using Rule2Scene Agent, a closed loop of automated generation of traffic rules from text to visual scenes was achieved. The constructed DriveCTR benchmark not only covers the full cognitive levels from L1 to L5, but also demonstrates through experiments its significant technical effect in improving the reasoning ability and trajectory planning accuracy of autonomous driving models in long-tail scenarios.

[0081] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 4As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores underwater dynamic mud level measurement data based on a rotary encoder. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements an underwater dynamic mud level measurement method based on a rotary encoder.

[0082] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0083] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0084] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0085] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0086] It should be noted that the engineering cost investment information and data involved in this application (including but not limited to data used for analysis, data stored, data displayed, etc.) are all information and data authorized by the client or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0087] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (Read-Only Memory). Memory includes ROM, magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0088] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0089] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0090] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An automatic driving evaluation benchmark construction method, characterized by, The method includes: The original traffic regulations are parsed into atomic rules, and candidate rule pairs with hierarchical labels are generated based on semantic structuring. The candidate rule pairs are subjected to spatiotemporal coexistence verification to construct a hierarchical rule set from basic atomic rules to complex conflict rules; The hierarchical rule set is transformed into a text scene description and a structured semantic representation is generated. Then, it is mapped into an autonomous driving simulator to render and capture multi-view image sequences, generating high-fidelity visual scene data. Based on the generated high-fidelity visual scene data, multiple-choice question-answer pairs are constructed to form an evaluation dataset containing different cognitive levels; The autonomous driving model is trained or evaluated using the evaluation dataset, and the planning strategy of the model is optimized based on the evaluation results.

2. The method of claim 1, wherein, The process of parsing original traffic regulations into atomic rules includes: By using a large language model to assist in syntactic analysis and manual verification, the complex traffic regulations are decomposed into a set of atomic rules, each of which defines a specific context and action mapping. Each atomic rule is modeled as a quadruple, which includes rule content, perception type, action type, and specification type.

3. The method of claim 2, wherein, The generation of candidate rule pairs with hierarchical labels includes: Generate a set of candidate rule pairs with consistent action types, and assign a perceptual combination type label, a canonical relation type label, and a hierarchical level label for defining cognitive difficulty to each rule pair; The hierarchical labels include five levels, from single-rule understanding (L1) to multi-rule conflict resolution (L5).

4. The method of claim 3, wherein, The method further includes: for speed limit rules, a pairing mechanism based on the intersection of speed intervals is adopted; the rule is mapped to a feasible speed interval; if the intervals of two speed limit rules do not intersect, it is determined that there is a conflict and it is directly assigned to the highest conflict level L5.

5. The method of claim 1, wherein, The spatiotemporal coexistence verification of the candidate rule pairs includes: The road type, agent state, and environment attributes in the rules are extracted using a large language model. Special instructions are used to determine whether two rules are compatible in physical space and temporal context. Only valid rule pairs are retained to enter the hierarchical rule set.

6. The method according to any one of claims 1 to 5, characterized in that, The generated high-fidelity visual scene data includes: A self-checking alignment pipeline is adopted to transform the elements in the hierarchical rule set into text scene descriptions and structured semantic representations; The structured semantic representation is mapped to the simulator's asset library, and a static initial scene is generated through position validity checks and collision detection; The temporal trajectories of each entity are generated based on road topology and behavioral semantics, and encoded into simulated action structures for rendering, capturing RGB image sequences.

7. The method of claim 3, wherein, The method also includes determining the correct action based on the following priority principle when constructing multiple-choice question-answer pairs for the L5 conflict level: Pedestrian safety > Yielding to emergency vehicles > On-site command > Traffic lights > Traffic signs > Road markings > Right-of-way interaction > Defensive driving > Emergency exceptions.

8. The method of claim 6, wherein, The method further includes: The quality scoring mechanism is used to independently score the generated results using multiple large language models at each stage of scene generation. If the average score is lower than a preset threshold, manual intervention is triggered for verification.

9. An automated driving evaluation benchmark building system for performing the method according to any one of claims 1 to 8, characterized in that The system includes: The atomic rule extraction module is configured to parse original traffic laws into atomic rules and generate candidate rule pairs with hierarchical labels based on semantic structuring; The rule scenario conversion module is configured to perform spatiotemporal coexistence verification on the candidate rule pairs, construct a hierarchical rule set from basic atomic rules to complex conflict rules, convert the hierarchical rule set into a text scenario description and generate a structured semantic representation, and then map the structured semantic representation to an autonomous driving simulator, render and capture a multi-view image sequence, and generate high-fidelity visual scene data; The multi-level question construction module is configured to construct multiple-choice question and answer pairs based on the generated high-fidelity visual scene data, and form an evaluation data set containing different cognitive levels; The model evaluation module is configured to train or evaluate an autonomous driving model using the evaluation data set, and optimize the planning strategy of the model according to the evaluation result.

10. A computer device, comprising: The computer program product comprises a memory and a processor, wherein the memory stores the computer program, and the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 8. The computer program product comprises a memory and a processor, wherein the memory stores the computer program, and the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 8.

Images