A perception test method and system based on an automatic driving system
By utilizing the operational data of autonomous vehicles to evaluate the continuity and regularity of perception modules, and combining this with downstream module evaluation, the problems of high cost and insufficient evaluation in existing technologies are solved, achieving low-cost and efficient perception module testing and problem localization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HANGZHOU FABU TECH CO LTD
- Filing Date
- 2026-03-10
- Publication Date
- 2026-06-02
AI Technical Summary
Existing testing methods for autonomous driving perception modules rely on manually labeled data, which is costly and makes it difficult to effectively assess the continuity of perception results in real-world scenarios and their actual impact on downstream modules.
By acquiring daily operational data from autonomous vehicles, preprocessing it, evaluating the performance of the perception module based on the continuity and regularity of the perception results, inputting the perception results into downstream modules for evaluation, and automatically marking abnormal data by comparing with reference modules.
It enables low-cost and efficient testing of perception modules, comprehensively evaluates the stability of perception results and their impact on downstream modules, automatically locates problems, and provides difficult example datasets.
Smart Images
Figure CN122131255A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of autonomous driving, and specifically relates to a perception testing method and system based on an autonomous driving system. Background Technology
[0002] The perception module of an autonomous driving system is responsible for acquiring environmental information through sensors (such as cameras, LiDAR, and millimeter-wave radar) and identifying and tracking dynamic and static targets in the surrounding environment. The accuracy and stability of the perception results are fundamental to ensuring the safe and reliable operation of autonomous vehicles.
[0003] Currently, the testing and evaluation of perception modules mainly rely on large-scale, finely labeled datasets (e.g., datasets containing precise bounding boxes and category labels for each target in every frame of an image). This approach has significant drawbacks: First, data labeling requires substantial human, time, and financial resources; second, the scenario coverage of labeled data is always limited, making it difficult to exhaustively represent long-tail scenarios and complex working conditions in the real world; finally, traditional methods typically evaluate the static metrics (such as accuracy and recall) of perception modules on closed test sets, making it difficult to effectively assess the continuity and rationality of the perception module's output over time, as well as its actual impact on downstream planning and decision-making modules. For example, a perception module that performs well on the test set may experience unexpected sudden braking or swaying in the downstream planning module due to occasional jumps in target attributes (such as speed or position), affecting ride comfort and safety.
[0004] Therefore, there is an urgent need for a testing method and system that does not rely on manual annotation, can utilize actual vehicle operation data, and can evaluate the performance of perception modules from the perspective of actual system interaction and physical laws. Summary of the Invention
[0005] To address the problems existing in the background technology, this invention provides a perception testing method and system based on an autonomous driving system. It solves the technical problems of existing perception testing methods relying on labeled data, which leads to high costs, low efficiency, and difficulty in evaluating the continuity, stability, and actual impact of perception results on downstream modules in real-world scenarios.
[0006] The technical solution adopted in this invention is: I. A perception testing method based on autonomous driving systems: S1. Obtain the raw data collected by the target vehicle during its daily autonomous driving operation as test data.
[0007] S2. Input the test data into the perception module to be tested for processing to obtain the perception results.
[0008] S3. Evaluate the performance of the sensing module based on the continuity or regularity of the sensing results.
[0009] The evaluation based on the continuity of the perception results includes: determining whether the rate of change of the monitored quantity of the same target in consecutive frames exceeds a preset physical constraint threshold.
[0010] The evaluation based on the regularity of the perception results includes: statistically analyzing the patterns of fixed targets appearing in the perception results according to the periodic operation characteristics of daily operations, and calculating the false positive rate and / or false negative rate based on this.
[0011] Before step S2, a step of preprocessing the test data is also included.
[0012] The preprocessing includes at least one of the following operations: filtering by target distance, filtering by target type, filtering by target speed, filtering by target location, filtering by target direction, and filtering by target detection box size.
[0013] The preprocessing step is configured to make the distribution of the number of each type of target relatively uniform in the processed test data.
[0014] In step S3, the evaluation is performed based on the continuity of the perception results, including: Determine whether the rate of change of velocity of the same obstacle in consecutive frames exceeds the first physical constraint threshold in the perception results; determine whether the rate of change of orientation of the same obstacle in consecutive frames exceeds the second physical constraint threshold in the perception results.
[0015] The method further includes step S4, inputting the perception result into the downstream planning module for processing to obtain the planning output result.
[0016] Step S3 further includes: evaluating the performance of the perception module based on the planning output results, specifically analyzing the impact of the perception module on the output of the planning module.
[0017] The downstream planning module includes a path planning module and a decision-making module; the evaluation is based on the planning output results, specifically including: statistically analyzing changes in the planned trajectory or changes in the number of emergency braking operations.
[0018] The method also includes a comparative testing step: Test data is simultaneously input to both the sensing module under test and a reference sensing module; the performance of the sensing module under test is evaluated by comparing the differences in the sensing results output by the sensing module under test and the reference sensing module.
[0019] The method further includes: marking the test data or corresponding data segments that produce differences based on the differences in the perception results output by the perception module to be tested and the reference perception module.
[0020] The method further includes: when the perception result is determined to contain a false positive target or a false negative target, marking the test data that generated the perception result or its corresponding data segment.
[0021] II. A perception testing system based on an autonomous driving system: The data acquisition module is used to acquire raw data collected by the target vehicle during its daily autonomous driving operations, which is then used as test data.
[0022] The perception processing module is used to input test data into the perception algorithm under test for processing and to obtain perception results.
[0023] An evaluation module is used to evaluate the performance of the sensing module based on the continuity or regularity of the sensing results. The evaluation module is configured to: evaluate based on the continuity of the sensing results, including: determining whether the rate of change or amount of change of the monitored quantity of the same target in consecutive frames exceeds a preset physical constraint threshold; or evaluate based on the regularity of the sensing results, including: statistically analyzing the regularity of the occurrence of fixed targets in the sensing results according to the periodic operation characteristics of daily operations, and calculating the false positive rate or false negative rate based on this.
[0024] The beneficial effects of this invention are: 1. No labeling, low cost, and high efficiency: The test directly utilizes the massive amount of raw data generated from the daily operation of vehicles, completely eliminating the expensive and time-consuming manual data labeling process, greatly reducing testing costs and improving testing iteration efficiency.
[0025] 2. More comprehensive and realistic evaluation dimensions: It not only evaluates the detection accuracy of a single frame, but also evaluates the rationality and stability of the perception results from two dimensions: temporal continuity (in accordance with physical laws) and scene regularity (in accordance with prior knowledge), which can better reflect the performance of the perception module in real dynamic environments.
[0026] 3. Effectively assess the impact on downstream modules: By inputting the perception results into the actual downstream planning and control modules and observing the system-level behavioral outputs (such as trajectories and decision commands), it is possible to discover perception problems that are not obvious in traditional indicators but have a substantial impact on the overall performance and safety of the system.
[0027] 4. Automated Problem Localization and Data Mining: It can automatically identify anomaly detection results and backtrack the corresponding original data segments, providing algorithm engineers with an efficient automated tool for quickly locating problems and building difficult example datasets. Attached Figure Description
[0028] Figure 1 This is a flowchart of the method of the present invention.
[0029] Figure 2 This is a block diagram of the system of the present invention. Detailed Implementation
[0030] The present invention will now be described in more detail with reference to the accompanying drawings and embodiments. However, the present invention is not limited thereto. For those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention. Contents not described in detail in this specification are prior art known to those skilled in the art.
[0031] like Figure 1 As shown, the perception testing method in this embodiment includes the following steps: S1. Obtain the raw data collected by the target vehicle during its daily autonomous driving operation as test data.
[0032] S2. Preprocess the test data, and use the preprocessed test data as the final test data.
[0033] Preprocessing includes at least one of the following operations: filtering by target distance, filtering by target type, filtering by target speed, filtering by target location, filtering by target direction, and filtering by target detection box size, so as to make the test data more focused on a specific test scenario or to make the target distribution more uniform.
[0034] S3. Input the test data into the perception module to be tested for processing to obtain the perception results.
[0035] S4. Evaluate the performance of the sensing module based on the continuity or regularity of the sensing results.
[0036] The evaluation based on the continuity of the perception results includes: determining whether the rate of change of the monitored quantity of the same target in consecutive frames exceeds a preset physical constraint threshold.
[0037] The evaluation is based on the continuity of the perception results, including: determining whether the rate of change of velocity of the same obstacle in consecutive frames exceeds a first physical constraint threshold; and determining whether the rate of change of orientation of the same obstacle in consecutive frames exceeds a second physical constraint threshold.
[0038] In practice, evaluation is based on the continuity of perception results. For example, it is determined whether the rate of change of velocity (i.e., acceleration) of the same obstacle in consecutive frames exceeds the normal acceleration limit of the vehicle (e.g., no more than 2 m / s² in a port scenario).2 Alternatively, it can be determined whether the rate of change of its direction exceeds the reasonable limit of angular velocity.
[0039] The evaluation based on the regularity of the perception results includes: statistically analyzing the patterns of fixed targets appearing in the perception results according to the periodic operation characteristics of daily operations, and calculating the false positive rate and / or false negative rate based on this.
[0040] S5. Input the perception results into the downstream planning module for processing to obtain the planning output results.
[0041] The downstream planning module includes a path planning module and a decision-making module.
[0042] S6. Evaluate the performance of the perception module based on the planning output results, specifically by analyzing the impact of the perception module on the output of the planning module.
[0043] Evaluation is based on the planning output, specifically including: statistically analyzing changes in the planned trajectory or the number of emergency braking events. For example, analyzing changes in the smoothness of the planned trajectory and abnormal increases or decreases in the number of emergency braking events can help determine whether instability has been introduced into the perception results.
[0044] S7. Comparison Test: Test data is simultaneously input to the perception module under test and a reference perception module (e.g., a perception module from a previous stable version); the performance of the perception module under test is evaluated by comparing the differences in the perception results output by the perception module under test and the reference perception module.
[0045] S8. Based on the differences in the perception results output by the perception module to be tested and the reference perception module (e.g., continuity violation, regularity violation, or significant difference from the reference module), mark the test data or corresponding data segments that produce the differences.
[0046] S9. When the perception result is determined to contain a false positive target or a false negative target, the test data that generated the perception result or its corresponding data segment shall be marked.
[0047] like Figure 2As shown, this embodiment also provides a perception testing system based on an autonomous driving system, including: a data acquisition module for acquiring raw data collected by the target vehicle during daily autonomous driving operations as test data; a perception processing module for inputting the test data into the perception algorithm to be tested for processing to obtain perception results; and an evaluation module for evaluating the performance of the perception module based on the continuity or regularity of the perception results. The evaluation module is configured to: evaluate based on the continuity of the perception results, including: determining whether the rate of change or amount of change of the monitored quantity of the same target in consecutive frames exceeds a preset physical constraint threshold; and evaluate based on the regularity of the perception results, including: statistically analyzing the regularity of the occurrence of fixed targets in the perception results according to the periodic operation characteristics of daily operations, and calculating the false positive rate or false negative rate accordingly.
[0048] Furthermore, this embodiment takes the perception test of an autonomous driving system for container transshipment trucks within a port as an example to describe in detail the implementation of the present invention, including the following steps: S101: Acquire and preprocess test data.
[0049] The target vehicle (autonomous container truck) performs periodic round-trip transfer operations between port areas A and B. Raw data streams, synchronously collected by the onboard forward-facing camera, LiDAR, and millimeter-wave radar during its normal operation over the past month, are collected as a test data pool.
[0050] First, the raw data is preprocessed: Distance filtering: Only target data within 0-80 meters in front of the sensing module are retained, as this distance is the main focus of port planning decisions.
[0051] Scene filtering: Based on GPS location information, filter out data segments where the vehicle is located on the main road (fixed route) inside the port.
[0052] Target type balance: Analysis of the filtered data revealed that the number of targets such as "container trailers" and "personnel" far exceeded that of smaller targets such as "cones". By appropriately oversampling the data segments containing "cones", the frequency distribution of the main target types in the final test dataset was made relatively uniform, avoiding test bias.
[0053] After preprocessing, the test dataset used for this test is obtained.
[0054] S102: Perception processing, obtaining perception results.
[0055] The test dataset is input into the new version of the perception module to be evaluated (denoted as "Perception Module A"). This module processes each frame of data and outputs a list of perception results containing the following information: [Target ID, Target Type, Position (X,Y,Z), Velocity (Vx,Vy), Bounding Box Size, Confidence Score]. The system also records the tracking correlation information of each target across consecutive frames.
[0056] S103: Continuity-based evaluation.
[0057] This step assesses whether the perception results conform to the continuity of the physical world in the time dimension. The core is to check whether the abrupt changes in the state variables of the same tracked target between adjacent frames are reasonable.
[0058] The specific steps are as follows: For any target that is being tracked stably (such as a container trailer with ID=100), calculate its velocity change between two adjacent frames (time interval Δt=0.1s): acceleration a=(current frame velocity - previous frame velocity) / time interval.
[0059] In the scenario of roads within the port, the physical constraint threshold for acceleration is set to ±2 m / s². 2 (This value is based on the port's vehicle safety operation regulations). If the calculated |a| > 2 m / s 2 If the system detects an anomaly, it determines that the target's speed perception result at the current moment has a continuity anomaly. The system records this anomaly event, the target ID, the time of occurrence, and the corresponding raw data segment.
[0060] Similarly, the change in angular velocity in the direction of the target's motion can be calculated, and a reasonable angular velocity threshold can be set for judgment.
[0061] S104: Evaluation based on regularity.
[0062] Because the transshipment routes within the port are fixed, there are several fixed static reference points along the way, such as Lighthouse No. 1, Height Restriction Sign No. 2, and Fixed Traffic Sign No. 3. Based on prior historical knowledge, when a vehicle travels from A to B, the order in which these fixed targets are perceived should be [Lighthouse No. 1] -> [Height Restriction Sign No. 2] -> [Traffic Sign No. 3], and their location range is relatively fixed.
[0063] Analyze the output of "Perception Module A" on the entire test dataset: False positive analysis: In 95% of the data segments, the perceived order of static targets conformed to the above pattern. However, in one data segment, the perception order changed to [Lighthouse No. 1] -> [Unknown Safe Island] -> [Height Restriction Sign No. 2] -> [Indicator Sign No. 3]. The "Unknown Safe Island" did not exist at that location in the historical prior model. Therefore, the system determined the "Unknown Safe Island" to be a false positive target.
[0064] False negative analysis: In another data segment, the perception order was [Lighthouse No. 1] -> [Sign No. 3], missing the [Height Restriction Sign No. 2] that should have been present. Considering that historical data shows a 99% detection rate for Height Restriction Sign No. 2 at this location, the system determined that a false negative missed detection of "Height Restriction Sign No. 2" occurred here.
[0065] Based on the occurrence of fixed targets in all data segments, the system calculates the false positive rate and false negative rate of the sensing module for various fixed reference objects, which serve as important indicators of its stability and reliability.
[0066] S105: Assessment of the impact of downstream modules (optional enhancement step).
[0067] To further evaluate the "real-world" impact of the perception results, the perception results obtained from S102 are input into the actual planning and control module used by the port trucks in real time.
[0068] Comparative experiment: At the same time, the same test dataset was input into a baseline system (consisting of the old stable perception module "Perception Module B" and the corresponding planning module).
[0069] Metric monitoring: During the parallel processing of test data by the two systems, record and compare the following downstream output metrics: Smoothness of the planned trajectory: Calculate the rate of change of curvature of the planned trajectory. If the trajectory experiences significantly more frequent and minor jitters when using "Perception Module A" than the baseline system, it indicates that its perception results (such as target position) may contain high-frequency noise.
[0070] Emergency braking trigger count: This counts the number of emergency braking commands issued by the downstream decision-making module. If the number of emergency braking triggers by "Perception Module A" increases abnormally, it may indicate false detection (misclassifying harmless objects as dangerous) or unstable speed estimation (leading to sudden changes in collision time prediction); conversely, if the number of triggers decreases abnormally, it may indicate a risk of missed detections.
[0071] By analyzing changes in these system-level metrics, the performance of the sensing module can be indirectly but effectively assessed from a functional safety perspective.
[0072] S106: Problem Data Location and Marking.
[0073] Based on the detection results of S103, S104, and S105, the system automatically marks all time points or data segments where anomalies occur (such as continuity violations, missed / false detections of fixed targets, and causes of downstream planning anomalies). For example, the raw sensor data (images + point clouds) frames containing the abnormal acceleration jumps in S103 are labeled "unstable velocity perception" and saved to a dedicated database.
[0074] These automatically labeled "difficult case" data segments provide valuable and targeted material for subsequent analysis of the root causes of problems and optimization of the perception model.
[0075] Furthermore, in the test platform, this invention also deploys two processing chains in parallel: Link A (to be tested): Test data -> Sensing module A -> Planning module A.
[0076] Link B (Reference Baseline): Test data -> Reference Sensing Module B (such as a stable version already used in existing vehicles) -> Reference Planning Module B (a version that matches Sensing Module B).
[0077] The system maintains strict synchronization with the input data. The evaluation process is as follows: Intermediate result comparison: Directly compare the perception results output by "Perception Module A" and "Reference Perception Module B" for the same frame of data, such as the number of targets, positional deviation, and class confidence differences. If the difference exceeds a set threshold (e.g., positional deviation greater than 1 meter), it is recorded.
[0078] Final Output Comparison: Compare the trajectories output by "Planning Module A" and "Reference Planning Module B". Calculate the point-to-point distance between the two trajectories. If the average or maximum distance exceeds the safety tolerance (e.g., trajectory deviation greater than 0.5 meters in a critical area at an intersection), it is determined that there is a significant difference in the perception-planning joint output of the test link, which may introduce risks.
[0079] Difference localization: When a difference in the final output is detected, the system automatically traces back to the starting point of the difference and examines the differences in the intermediate sensing results of the two links near that point. This helps to quickly pinpoint which type of output change in the sensing module (such as a newly detected target, the disappearance of a target, or a sudden change in the position of a target) ultimately caused the change in system behavior.
[0080] Data labeling: Label the original test data segments corresponding to all output difference results (whether intermediate result differences or final output differences) as "AB difference data" for subsequent analysis.
[0081] The method of this invention does not rely on labeled data. By utilizing the raw data generated by the daily operation of vehicles, it can efficiently and cost-effectively evaluate the performance and stability of the perception module from multiple dimensions, such as the continuity of the inherent physical laws of the perception results and the consistency between the perception results and the laws of known operation scenarios. It can also effectively discover the actual impact of perception problems on downstream modules.
[0082] The above embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention. The scope of protection of the present invention is defined by the claims.
Claims
1. A perception testing method based on an autonomous driving system, characterized in that, Includes the following steps: S1. Obtain the raw data collected by the target vehicle during its daily autonomous driving process as test data; S2. Input the test data into the perception module to be tested for processing to obtain the perception results; S3. Evaluate the performance of the sensing module based on the continuity or regularity of the sensing results; Among them, the evaluation based on the continuity of the perception results includes: determining whether the rate of change of the monitored quantity of the same target in consecutive frames exceeds the preset physical constraint threshold. The evaluation based on the regularity of the perception results includes: statistically analyzing the patterns of fixed targets appearing in the perception results according to the periodic operation characteristics of daily operations, and calculating the false positive rate and / or false negative rate based on this.
2. The perception testing method according to claim 1, characterized in that, Before step S2, a step of preprocessing the test data is also included; The preprocessing includes at least one of the following operations: filtering by target distance, filtering by target type, filtering by target speed, filtering by target location, filtering by target direction, and filtering by target detection box size; The preprocessing step is configured to make the distribution of the number of each type of target relatively uniform in the processed test data.
3. The perception testing method according to claim 1, characterized in that, In step S3, the evaluation is performed based on the continuity of the perception results, including: Determine whether the rate of change of velocity of the same obstacle in consecutive frames exceeds the first physical constraint threshold in the perception results; determine whether the rate of change of orientation of the same obstacle in consecutive frames exceeds the second physical constraint threshold in the perception results.
4. The perception testing method according to claim 1, characterized in that: The method further includes step S4, inputting the perception result into the downstream planning module for processing to obtain the planning output result; Step S3 further includes: evaluating the performance of the perception module based on the planning output results, specifically analyzing the impact of the perception module on the output of the planning module.
5. The perception testing method according to claim 4, characterized in that: The downstream planning module includes a path planning module and a decision-making module; the evaluation is based on the planning output results, specifically including: statistically analyzing changes in the planned trajectory or changes in the number of emergency braking operations.
6. The perception testing method according to claim 1, characterized in that, The method also includes a comparative testing step: Test data is simultaneously input to both the sensing module under test and a reference sensing module; the performance of the sensing module under test is evaluated by comparing the differences in the sensing results output by the sensing module under test and the reference sensing module.
7. The perception testing method according to claim 6, characterized in that, The method further includes: Based on the difference between the perception results output by the perception module to be tested and the reference perception module, the test data or corresponding data segments that produce the difference are marked.
8. The perception testing method according to claim 1, characterized in that, The method further includes: When the perception result is determined to contain false positive or false negative targets, the test data that generated the perception result or its corresponding data segment is marked.
9. A perception testing system based on an autonomous driving system, employing the method described in any one of claims 1-8, characterized in that, include: The data acquisition module is used to acquire raw data collected by the target vehicle during its daily autonomous driving operation as test data. The perception processing module is used to input test data into the perception algorithm under test for processing and to obtain perception results. The evaluation module is used to assess the performance of the sensing module based on the continuity or regularity of the sensing results; the evaluation module is configured as follows: Evaluation based on the continuity of perception results includes: determining whether the rate of change or amount of change of the monitored quantity of the same target in consecutive frames exceeds a preset physical constraint threshold. Alternatively, assessments can be based on the regularity of the perception results, including: statistically analyzing the patterns of fixed targets appearing in the perception results according to the periodic operational characteristics of daily operations, and calculating the false positive rate or false negative rate based on this.