Real vehicle 3D detection and perception performance evaluation method and system based on labeled true value data

Abstract

The application relates to the technical field of intelligent networked vehicles, in particular to a real vehicle 3D detection and perception performance evaluation method and system based on labeled true value data, which comprises the following steps: obtaining real vehicle scene original labeling data and performing true value preprocessing to obtain labeled true value data; collecting real vehicle multi-source data and synchronously storing the same to generate multi-source collected data; dynamically aligning the labeled true value data and the multi-source collected data in time and space to obtain synchronized labeled true value data and perception data; performing multi-dimensional perception performance evaluation according to the synchronized labeled true value data and the perception data to generate evaluation parameters and process control signals; generating visual verification results and quantitative evaluation results according to the evaluation parameters and automatically controlling according to the process control signals. The application can reduce the influence of labeling differences and multi-source data time and space dislocation on the evaluation results, and improve the accuracy and reliability of real vehicle 3D detection and perception performance evaluation.

Description

Technical Field

[0001] This application relates to the field of intelligent connected vehicle technology, and in particular to a method and system for evaluating the perception performance of a real vehicle based on labeled ground truth data in 3D detection. Background Technology

[0002] With the development of intelligent connected vehicle technology, in-vehicle 3D detection and perception systems have been widely used for the identification and localization of road targets such as vehicles, pedestrians, non-motorized vehicles, and road cones. These systems typically collect real-world road environment data based on sensors such as LiDAR and vision cameras, and output detection results including target category, 3D position, target size, and attitude angle. To assess the detection performance of in-vehicle 3D detection and perception systems in actual road environments, it is usually necessary to use ground truth data generated by annotation tools or manual annotation as a reference to quantitatively evaluate the real-world perception output. The evaluation results directly affect the reliability of perception algorithm iteration, factory testing, road test optimization, and performance verification.

[0003] Currently, evaluation methods for 3D detection and perception performance typically use labeled datasets, real-vehicle data, or simulated scene data as the evaluation basis, comparing the detection results output by the perception system with ground truth or reference targets. Specifically, the evaluation end generally acquires data such as target category, target location, target size, attitude angle, detection box, and confidence score, and matches the ground truth target with the perceived target based on the temporal and spatial information in the data. After matching, indicators such as recognition accuracy, target false detection rate, target false detection rate, position error, and size error are calculated. Some evaluation methods also combine scene type, sensor type, or target category to statistically summarize the indicators, forming an evaluation result that characterizes the detection capability of the perception system.

[0004] However, in real-world 3D vehicle inspection and perception evaluation scenarios, the evaluation object is not a static sample or a single sensor output, but rather multi-frame, multi-source, dynamic data from the actual vehicle operation process. Target occlusion, target motion, vehicle posture changes, differences in sensor sampling frequencies, and variations in manual annotations in the real-world road environment all affect the correspondence between the ground truth target and the perceived target. Existing evaluation methods, which primarily rely on the temporal information inherent in the data, fixed spatial relationships, and conventional statistical indicators for comparison, are easily affected by the consistency of labeled data, differences in data acquisition times, and changes in the spatial state of the real vehicle. This results in calculated perception performance indicators that fail to accurately reflect the true performance of the onboard 3D inspection and perception system in real-world vehicle scenarios. Summary of the Invention

[0005] This application provides a method and system for evaluating the perception performance of real-vehicle 3D detection based on labeled ground truth data. This method reduces the impact of annotation differences and spatiotemporal misalignment of multi-source data on the evaluation results, thereby improving the accuracy and reliability of real-vehicle 3D detection perception performance evaluation. The technical solution provided in this application is as follows: Firstly, this application provides a method for evaluating the perception performance of a real vehicle in 3D detection based on labeled ground truth data, the method comprising: Obtain the original labeled data corresponding to the real vehicle scene, and perform label ground value preprocessing on the original labeled data to obtain the labeled ground value data corresponding to the real vehicle scene; Collect real vehicle multi-source data corresponding to the real vehicle scenario, and synchronously associate and store the real vehicle multi-source data to generate multi-source collected data corresponding to the real vehicle scenario; The labeled ground truth data and the multi-source acquired data are dynamically spatiotemporally aligned to obtain synchronized labeled ground truth data and sensed data. Based on the synchronized labeled true data and perception data, a multi-dimensional perception performance evaluation is performed to generate real vehicle 3D detection perception performance evaluation parameters and process control signals. Based on the aforementioned real vehicle 3D detection and perception performance evaluation parameters, visualized verification results and quantitative evaluation results are generated; The process control signals are used to automatically control the real vehicle 3D detection and perception performance evaluation process corresponding to the real vehicle scene.

[0006] In one specific implementation scheme, the step of performing ground truth preprocessing on the original labeled data to obtain labeled ground truth data corresponding to the real vehicle scenario includes: The intersection-union ratio (IU) of the target bounding boxes is calculated for the labeled targets in the original labeled data. Multiple labeled targets with the same target category and an IU greater than a preset duplication threshold are identified as duplicate labeled targets, and the duplicate labeled targets are removed. Based on the occlusion rate of the labeled target, the occluded targets that meet the completion conditions are determined, and the target size and attitude angle of the occluded targets are completed by using the same type of target feature interpolation method. The target categories in the original labeled data are uniformly encoded according to the preset category mapping relationship, and the labeled data after category unification is converted into a preset format to obtain the labeled true value data.

[0007] In a specific feasible implementation, the step of collecting real-vehicle multi-source data corresponding to the real-vehicle scenario, and synchronously associating and storing the real-vehicle multi-source data to generate multi-source collected data corresponding to the real-vehicle scenario includes: The system collects environmental data, obstacle perception data, and attitude data corresponding to the actual vehicle scene using vehicle-mounted LiDAR, vehicle-mounted vision camera, and inertial measurement unit, and obtains sensor calibration data corresponding to the actual vehicle scene. The environmental data, the perceived obstacle data, the attitude data, and the sensor calibration data are associated and stored according to the scene identifier and timestamp; The data after association and storage is verified for integrity, and the multi-source collected data is generated based on the integrity verification results.

[0008] In a specific feasible implementation, the step of dynamically aligning the labeled ground truth data and the multi-source acquired data in a spatiotemporal manner to obtain synchronized labeled ground truth data and sensed data includes: Vehicle speed data and perception data are extracted from the multi-source collected data, and the timestamps of the labeled true data and the perception data are converted into a unified time unit. A time matching threshold is determined based on the vehicle speed data. A first time matching threshold is used when the vehicle speed data is greater than a preset speed threshold, and a second time matching threshold is used when the vehicle speed data is less than or equal to the preset speed threshold. The first time matching threshold is less than the second time matching threshold. The labeled ground truth data and the perceived data are matched with the nearest neighbor time according to the time matching threshold to obtain time-synchronized labeled ground truth data and perceived data.

[0009] In one specific implementation scheme, the step of dynamically aligning the labeled ground truth data and the multi-source acquired data in a spatiotemporal manner to obtain synchronized labeled ground truth data and sensed data further includes: Attitude data and sensor calibration data are extracted from the multi-source acquired data; Based on the attitude data, attitude compensation is performed on the rotation and translation parameters in the sensor calibration data to obtain dynamic calibration parameters; According to the dynamic calibration parameters, coordinate transformation is performed on the time-synchronized labeled ground truth data and perceived data to make the labeled ground truth target and perceived target in a unified coordinate system, thus obtaining the synchronized labeled ground truth data and perceived data.

[0010] In one specific implementation scheme, the step of performing multi-dimensional perception performance evaluation based on the synchronized labeled ground truth data and perception data, and generating real vehicle 3D detection perception performance evaluation parameters and process control signals includes: Based on the target category and the 3D spatial matching relationship, target association is performed between the labeled ground value targets in the synchronized labeled ground value data and the perceived targets in the perceived data; Based on the target association results, basic evaluation indicators and real-vehicle robustness evaluation indicators are calculated. The basic evaluation indicators include position error, size error and recognition accuracy. The real-vehicle robustness evaluation indicators include occlusion adaptability, dynamic target tracking accuracy and illumination adaptability. Based on the basic evaluation indicators, the real vehicle robustness evaluation indicators, and the target category distribution, the real vehicle 3D detection and perception performance evaluation parameters are generated, and the process control signals are generated based on the data integrity status and evaluation error status.

[0011] In a specific implementation scheme, the step of generating visual verification results and quantitative evaluation results based on the actual vehicle 3D detection and perception performance evaluation parameters, and automating the actual vehicle 3D detection and perception performance evaluation process corresponding to the actual vehicle scene according to the process control signals, includes: Based on the real vehicle 3D detection perception performance evaluation parameters, the system generates the labeled true value target display result, the perception target display result, and the superimposed comparison display result of the labeled true value target and the perception target, and generates a quantitative evaluation result that includes basic evaluation indicators, real vehicle robustness evaluation indicators, and target category evaluation results. If the process control signal indicates that the evaluation is normal, the visual verification result and the quantitative evaluation result will be output. If the process control signal indicates abnormal data or abnormal evaluation error, an error message will be triggered and the evaluation process corresponding to the current real vehicle scenario will be paused.

[0012] Secondly, this application provides a real vehicle 3D detection and perception performance evaluation system based on labeled ground truth data, which adopts the following technical solution: A real-vehicle 3D detection and perception performance evaluation system based on labeled ground truth data includes: The annotation truth preprocessing module is used to obtain the original annotation data corresponding to the real vehicle scene, and to perform annotation truth preprocessing on the original annotation data to obtain the annotation truth data corresponding to the real vehicle scene. The real vehicle multi-source data acquisition module is used to acquire real vehicle multi-source data corresponding to the real vehicle scenario, and synchronously associate and store the real vehicle multi-source data to generate multi-source acquisition data corresponding to the real vehicle scenario. The dynamic spatiotemporal alignment module is used to dynamically align the labeled ground value data and the multi-source acquired data in spatiotemporal mode to obtain synchronized labeled ground value data and sensed data. The multi-dimensional evaluation module is used to perform multi-dimensional perception performance evaluation based on the synchronized labeled true data and perception data, and generate real vehicle 3D detection perception performance evaluation parameters and process control signals. The visualization verification module is used to generate visualization verification results and quantitative evaluation results based on the actual vehicle 3D detection and perception performance evaluation parameters. An automated control module is used to automatically control the real vehicle 3D detection and perception performance evaluation process corresponding to the real vehicle scene according to the process control signal.

[0013] Thirdly, this application provides an electronic device, the device including a processor and a memory; the memory stores a program, the program being loaded and executed by the processor to implement a real vehicle 3D detection perception performance evaluation method based on labeled ground truth data as described in the first aspect.

[0014] Fourthly, this application provides a computer-readable storage medium storing a program that, when executed by a processor, is used to implement a method for evaluating the perception performance of a real vehicle based on labeled ground truth data, as described in the first aspect.

[0015] By acquiring the original labeled data corresponding to the real vehicle scene and performing labeled ground truth preprocessing, labeled ground truth data corresponding to the real vehicle scene is obtained; by collecting real vehicle multi-source data corresponding to the real vehicle scene and storing it synchronously and in association, multi-source acquisition data corresponding to the real vehicle scene is generated; then, the labeled ground truth data and multi-source acquisition data are dynamically spatiotemporally aligned to obtain synchronized labeled ground truth data and perception data; then, multi-dimensional perception performance evaluation is performed based on the synchronized labeled ground truth data and perception data, generating real vehicle 3D detection perception performance evaluation parameters and process control signals, and generating visualized verification results and quantitative evaluation results based on the real vehicle 3D detection perception performance evaluation parameters, while automatically controlling the real vehicle 3D detection perception performance evaluation process corresponding to the real vehicle scene according to the process control signals. Therefore, the original labeled data can form a unified evaluation reference after being preprocessed with labeled ground truth. The multi-source collected data can maintain the correspondence between different data in the real vehicle scenario after being synchronously associated and stored. Dynamic spatiotemporal alignment enables the labeled ground truth data and perception data to be compared under a consistent data benchmark. Multi-dimensional perception performance evaluation enables the evaluation parameters to more comprehensively characterize the perception performance of real vehicle 3D detection. Visualized verification results and quantitative evaluation results can intuitively present and numerically express the evaluation results. Process control signals enable the evaluation process to be automatically executed according to the evaluation state, thereby improving the accuracy, reliability and execution efficiency of the evaluation results of real vehicle 3D detection perception performance.

[0016] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, the preferred embodiments of this application are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the method for evaluating the perception performance of a real vehicle based on labeled ground truth data in this application.

[0018] Figure 2 This is a comparison diagram of the perception results and the labeled true values ​​in the embodiments of this application.

[0019] Figure 3 This is a schematic diagram of the overall process of the vehicle 3D detection and perception performance evaluation method based on labeled true value data in the embodiments of this application.

[0020] Figure 4 This is a structural block diagram of the real vehicle 3D detection and perception performance evaluation system based on labeled true value data in the embodiments of this application.

[0021] Figure 5 This is a block diagram of an electronic device for evaluating the 3D detection and perception performance of a real vehicle based on labeled ground truth data, as described in this application. Detailed Implementation

[0022] The specific embodiments of this application will be described in further detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate this application, but are not intended to limit the scope of this application.

[0023] Optionally, this application uses the real vehicle 3D detection perception performance evaluation method based on labeled truth data provided in various embodiments as an example for application in an electronic device. The electronic device is a terminal or server. The terminal can be a computer, tablet computer, etc. This embodiment does not limit the type of electronic device.

[0024] Reference Figure 1 This is a flowchart illustrating a method for evaluating the perception performance of a real vehicle based on labeled ground truth data, according to an embodiment of this application. The method includes at least the following steps: Step S101: Obtain the original labeled data corresponding to the real vehicle scene, and perform label ground value preprocessing on the original labeled data to obtain the labeled ground value data corresponding to the real vehicle scene.

[0025] In step S101, the original annotation data corresponding to the real vehicle scene is obtained, and the original annotation data is preprocessed to obtain the annotation ground truth data corresponding to the real vehicle scene. The original annotation data can be output by an external annotation tool. The original annotation data includes the target category, 3D position, target size, attitude angle, target bounding box, and occlusion rate of each annotated target in the real vehicle scene. Since there are many targets and complex target occlusion situations in the real vehicle scene, the data output by manual annotation or annotation tools may contain duplicate annotations, incomplete occluded target information, and inconsistent target category expressions. Therefore, it is necessary to preprocess the original annotation data to obtain the annotation ground truth data, so that the obtained annotation ground truth data can serve as a unified reference for evaluating the 3D detection and perception performance of the real vehicle.

[0026] Specifically, the intersection-union ratio (IU) of the bounding boxes is calculated for the labeled targets in the original labeled data. Multiple labeled targets with the same target category and an IU greater than a preset duplication threshold are identified as duplicate labeled targets and are then removed. The preset duplication threshold can be set to 0.8. When two labeled targets have the same target category and an IU greater than 0.8, it is determined that the two labeled targets correspond to the same real-world vehicle scene target. One of the duplicate labeled targets is removed, and one labeled target is retained as a valid labeled target.

[0027] For occluded labeled targets, the occlusion rate is used to determine the occluded targets that meet the completion criteria. Then, the target size and attitude angle of the occluded targets are completed using interpolation of similar target features. The occlusion rate can be determined based on the occlusion markers or the proportion of the visible area output by the annotation tool. When the occlusion rate of a labeled target is less than a preset occlusion completion threshold, the labeled target is determined to be an occluded target that meets the completion criteria. The preset occlusion completion threshold can be set to 30%. For occluded targets with an occlusion rate less than 30%, the target size and attitude angle are completed using interpolation of similar target features; for occluded targets with an occlusion rate greater than or equal to 30%, a manual verification marker is added to the occluded target to bring it into the manual verification range.

[0028] After completing the duplicate annotation removal and occluded target information completion, the target categories in the original annotation data are uniformly encoded according to the preset category mapping relationship. The category-unified annotation data is then converted into a preset format to obtain the annotation ground truth data. The preset category mapping relationship is used to uniformly convert target category names used by different annotation tools or different annotators into standard category codes. For example, the `obj_type_to_class_id` mapping relationship can be used to uniformly convert category names such as "car" and "vehicle" into the category code corresponding to "car", "truck" into the category code corresponding to "truck", and "road cone" into the category code corresponding to "cone". The standard category code can use category numbers from 0 to 9, where 0 represents "car", 1 represents "truck", 8 represents "pedestrian", and 9 represents "cone". The remaining category numbers are defined according to the preset target category table.

[0029] When converting the standardized annotation data to a preset format, the original JSON format annotation data can be converted to TXT format annotation data. Each row in the TXT format annotation data corresponds to an annotation target, and each row includes at least class_id, x, y, z, width, length, height, and yaw. Here, class_id represents the target category code, x, y, and z represent the 3D position of the annotation target in the real vehicle scene coordinates, width represents the width of the annotation target, length represents the length of the annotation target, height represents the height of the annotation target, and yaw represents the yaw angle of the annotation target. Through the above annotation ground value preprocessing, the impact of duplicate annotations, incomplete occluded target information, and inconsistent category representations on the evaluation reference data can be reduced, ensuring that the obtained annotation ground value data has a unified data representation in terms of target category, target position, target size, and attitude angle.

[0030] Step S102: Collect real vehicle multi-source data corresponding to the real vehicle scenario, and synchronously associate and store the real vehicle multi-source data to generate multi-source collected data corresponding to the real vehicle scenario.

[0031] In step S102, real-vehicle multi-source data corresponding to the real-vehicle scene is collected, and the real-vehicle multi-source data is synchronously associated and stored to generate multi-source collected data corresponding to the real-vehicle scene. The real-vehicle multi-source data comes from various vehicle-mounted devices and processing results in the real-vehicle road environment, including environmental data, obstacle perception data, attitude data, and sensor calibration data. Since the evaluation of the real-vehicle 3D detection and perception performance requires comparing the labeled ground truth data with the real-vehicle perception results, if there is a lack of unified scene identifiers, time identifiers, and data association relationships among the real-vehicle multi-source data, it will be difficult to accurately correspond between different data sources. Therefore, it is necessary to synchronously associate and store the real-vehicle multi-source data during the collection stage so that the multi-source data within the same real-vehicle scene forms a traceable data set.

[0032] Specifically, environmental data, obstacle perception data, and attitude data corresponding to the real-world vehicle scene are collected through an onboard LiDAR, an onboard vision camera, and an inertial measurement unit (IMU), and sensor calibration data corresponding to the real-world vehicle scene is obtained. The onboard LiDAR is used to collect point cloud environmental data in the real-world vehicle scene, the onboard vision camera is used to collect image environmental data in the real-world vehicle scene, the IMU is used to collect attitude data during vehicle operation, and the onboard 3D detection and perception system outputs obstacle perception data. Obstacle perception data may include the target category, 3D position, target size, attitude angle, detection box, and confidence level of the perceived target. Attitude data may include the vehicle's pitch angle, roll angle, and yaw angle. Sensor calibration data may include joint calibration parameters between the LiDAR and the vision camera, as well as the corresponding calibration file for the LiDAR.

[0033] In one implementation, the vehicle can be equipped with three LiDARs, six high-definition forward-looking vision cameras, and one high-precision inertial measurement unit (IMU). The LiDARs have a detection range greater than or equal to 100m and a point cloud frequency of 10Hz; the high-definition forward-looking vision cameras have a resolution greater than or equal to 1920×1080 and a frame rate of 30fps; and the IMU's attitude measurement accuracy is less than or equal to 0.1°. The LiDARs and vision cameras undergo joint calibration, and the IMU synchronizes data with the vehicle's powertrain system. The vehicle terminal can be equipped with sensor data acquisition software, vehicle attitude calculation software, and structured data storage software. The sensor data acquisition software controls the onboard LiDARs, vision cameras, and IMU to acquire data synchronously. The vehicle attitude calculation software performs real-time calculations on the data output by the IMU, and the structured data storage software performs hierarchical storage and associated management of data from different sources.

[0034] When collecting multi-source real-vehicle data, real-vehicle road scenarios such as urban main roads, intersections, and residential roads can be selected, covering different time periods such as morning rush hour, off-peak hour, evening rush hour, and nighttime; different weather conditions such as sunny, cloudy, and light rain; and different road conditions such as going straight, turning, following other vehicles, and meeting oncoming traffic. The collection time for each real-vehicle scenario can be greater than or equal to 1 minute. Through the above collection methods, multi-source real-vehicle data containing multiple types of traffic targets, various lighting conditions, and various vehicle motion states can be obtained.

[0035] Environmental data, obstacle perception data, attitude data, and sensor calibration data are stored in a linked manner based on scene identifiers and timestamps. Scene identifiers distinguish different real-vehicle scenarios, while timestamps indicate the acquisition or output time of various data types. During linked storage, a hierarchical directory can be established based on scene codes and timestamps, allowing point cloud environmental data, image environmental data, obstacle perception data, attitude data, and sensor calibration data within the same real-vehicle scenario to be retrieved using the same scene code and corresponding timestamp. Obstacle perception data can be stored in record format, attitude data in JSON format, and LiDAR calibration data in TXT format. Establishing data associations through scene identifiers and timestamps ensures that data from different sources maintains a corresponding relationship within the same real-vehicle scenario.

[0036] After completing the associated storage, the data is subjected to integrity verification, and multi-source acquisition data is generated based on the integrity verification results. Integrity verification includes checking for the existence of point cloud environment data, image environment data, obstacle perception data, attitude data, and sensor calibration data under the same scene identifier; checking for missing time periods from different data sources within the same time range; and checking whether data files can be read normally. When any data source has missing data, corrupted files, or abnormal timestamps, the abnormal data is marked. The generated multi-source acquisition data includes environmental data, obstacle perception data, attitude data, sensor calibration data associated with scene identifiers and timestamps, as well as integrity verification markers. Through synchronous acquisition, associated storage, and integrity verification, the multi-source data from the real vehicle can have a clear data source, time correspondence, and integrity status, reducing the impact of missing or mismatched data on the evaluation of the real vehicle's 3D detection and perception performance.

[0037] Step S103: Dynamically align the labeled ground truth data and multi-source collected data in time and space to obtain synchronized labeled ground truth data and sensed data.

[0038] In step S103, the labeled ground truth data and multi-source acquired data are dynamically spatiotemporally aligned to obtain synchronized labeled ground truth data and sensed data. The multi-source acquired data includes data generated by different sensors and processing stages; the acquisition frequency, output time, and coordinate reference of various data types are not entirely consistent. Simultaneously, the actual vehicle undergoes changes in motion states such as acceleration, braking, turning, and bumping during operation, and these changes in vehicle attitude affect the spatial relationships of the sensors. To enable the labeled ground truth data and sensed data to be compared under the same time and spatial reference, dynamic spatiotemporal alignment of the labeled ground truth data and multi-source acquired data is necessary.

[0039] Specifically, vehicle speed data and perception data are extracted from multi-source collected data, and the timestamps of the labeled ground truth data and perception data are converted into a unified time unit. The timestamp of the labeled ground truth data is used to represent the moment of the actual vehicle scene corresponding to the labeled target, and the timestamp of the perception data is used to represent the moment when the on-board 3D detection and perception system outputs the perceived target. Since different data sources may use different time units, unifying the timestamps can convert nanosecond-level timestamps into microsecond-level standard timestamps, enabling the labeled ground truth data and perception data to be matched on the same time scale.

[0040] A time matching threshold is determined based on vehicle speed data. A first time matching threshold is used when the vehicle speed is greater than a preset threshold, and a second time matching threshold is used when the vehicle speed is less than or equal to the preset threshold. The first time matching threshold is less than the second time matching threshold. The preset speed threshold can be set to 60 km / h, the first time matching threshold to 50 ms, and the second time matching threshold to 100 ms. When the vehicle speed is greater than 60 km / h, the displacement change of the actual vehicle is more significant in a shorter time, so 50 ms is used as the time matching threshold. When the vehicle speed is less than or equal to 60 km / h, the displacement change of the actual vehicle within the same time range is relatively smaller, so 100 ms is used as the time matching threshold. The labeled ground truth data and perceived data are then matched to their nearest neighbor times according to the time matching thresholds to obtain time-synchronized labeled ground truth data and perceived data. Nearest neighbor time matching refers to selecting the sensing data with the smallest time difference for each frame of labeled ground truth data within the time matching threshold range, or selecting the labeled ground truth data with the smallest time difference for each frame of sensing data; if the time difference exceeds the time matching threshold, a time synchronization relationship is not established, and an unmatched mark is added to the data of that frame.

[0041] After time synchronization is completed, attitude data and sensor calibration data are extracted from the multi-source acquired data. Attitude data may include the vehicle's pitch, roll, and yaw angles, while sensor calibration data may include rotation and translation parameters between the LiDAR and the vision camera. Attitude compensation is performed on the rotation and translation parameters in the sensor calibration data based on the attitude data to obtain dynamic calibration parameters. During attitude compensation, the rotation quaternions, translation parameters, or the extrinsic parameter relationship between the LiDAR and the vision camera are updated according to the pitch, roll, and yaw angles corresponding to the current vehicle scene, generating dynamic calibration parameters adapted to the current vehicle attitude. Dynamic calibration parameters reflect the current state of the sensor spatial relationships under changing vehicle motion conditions, avoiding spatial reference deviations caused by using only fixed calibration parameters.

[0042] According to the dynamic calibration parameters, coordinate transformation is performed on the time-synchronized labeled ground truth data and perceived data to place the labeled ground truth target and the perceived target in a unified coordinate system, resulting in synchronized labeled ground truth data and perceived data. Coordinate transformation can be performed sequentially using local coordinates, radar coordinates, and world coordinates. For labeled ground truth targets, the target's 3D position, size, and attitude angles in the labeled ground truth data are transformed to a unified coordinate system; for perceived targets, the target's 3D position, size, and attitude angles in the perceived data are transformed to the same unified coordinate system. The unified coordinate system can be either the radar coordinate system or the world coordinate system; the specific coordinate system type can be determined based on the coordinate reference of the output data from the actual vehicle 3D detection and perception system. After time synchronization and coordinate transformation, synchronized labeled ground truth data and perceived data are obtained, making the labeled ground truth target and perceived target comparable under the same time and spatial reference.

[0043] Through the above dynamic spatiotemporal alignment processing, the impact of differences in sensor sampling frequency, differences in the output time of perception results, changes in vehicle speed, and changes in actual vehicle posture on data matching relationships can be reduced, making the correspondence between labeled ground truth data and perception data more accurate, thereby improving the reliability of target association and index calculation in the perception performance evaluation of actual vehicle 3D detection.

[0044] Step S104: Perform multi-dimensional perception performance evaluation based on the synchronized labeled true data and perception data, and generate real vehicle 3D detection perception performance evaluation parameters and process control signals.

[0045] In step S104, a multi-dimensional perception performance evaluation is performed based on the synchronized labeled ground truth data and perceived data, generating real vehicle 3D detection perception performance evaluation parameters and process control signals. The synchronized labeled ground truth data and perceived data are already under the same time and spatial reference, enabling the determination of the correspondence between the labeled ground truth target and the perceived target. Since the real vehicle 3D detection perception performance is reflected not only in whether the target is correctly identified, but also in the target position, target size, target posture, and stability under complex real vehicle scenarios, this step evaluates both basic detection accuracy and real vehicle robustness, ensuring that the generated real vehicle 3D detection perception performance evaluation parameters reflect the comprehensive performance of the onboard 3D detection perception system in real vehicle scenarios.

[0046] Specifically, firstly, based on target categories and 3D spatial matching relationships, target association is performed between the labeled ground truth targets in the synchronized labeled ground truth data and the perceived targets in the perceived data. During target association, the labeled ground truth targets are used as reference objects to find perceived targets that are in the same category as the labeled ground truth targets or meet the category matching requirements under the same synchronization relationship at the same time. The corresponding perceived targets are then determined by combining 3D spatial positional relationships, target bounding box overlap relationships, or target center distance relationships. When a labeled ground truth target can match a perceived target, the two are identified as a successfully associated target pair; when a labeled ground truth target fails to match a corresponding perceived target, the labeled ground truth target is identified as a missed detection target; when a perceived target fails to match a corresponding labeled ground truth target, the perceived target is identified as a false detection target. Through the above target association, target matching results, missed detection results, and false detection results can be generated for index calculation.

[0047] After target association is completed, basic evaluation metrics and real-vehicle robustness evaluation metrics are calculated based on the target association results. Basic evaluation metrics include positional error, dimensional error, and recognition accuracy. Positional error represents the deviation between the 3D position of the perceived target and the 3D position of the labeled ground truth target in a successfully associated target pair; it can be represented by the Euclidean distance between the 3D positions. Dimensional error represents the deviation between the size of the perceived target and the size of the labeled ground truth target; it can be determined based on the relative errors in the target's width, length, and height. Recognition accuracy represents the proportion of perceived target categories that match the labeled ground truth target categories, and it can be combined with the statistical results of missed and false detections to characterize target recognition performance.

[0048] Real-world robustness evaluation metrics include occlusion adaptability, dynamic target tracking accuracy, and illumination adaptability. Occlusion adaptability represents the vehicle-mounted 3D detection and perception system's ability to recognize targets with different occlusion rates; it can be calculated by statistically analyzing the target recognition accuracy within the occlusion rate range of the labeled ground truth target. Dynamic target tracking accuracy represents the stability of the vehicle-mounted 3D detection and perception system in detecting continuously moving targets; it can be determined based on the average position error of the same dynamic target within five consecutive frames. Illumination adaptability represents the perception performance of the vehicle-mounted 3D detection and perception system under different lighting conditions; it can be calculated by statistically analyzing the comprehensive evaluation scores under strong light, weak light, and normal lighting conditions. By using occlusion adaptability, dynamic target tracking accuracy, and illumination adaptability, real-world interference factors such as target occlusion, target motion, and changes in lighting can be included in the perception performance evaluation scope.

[0049] Furthermore, based on the basic evaluation metrics, the vehicle robustness evaluation metrics, and the target category distribution, performance evaluation parameters for real-vehicle 3D detection are generated. The target category distribution represents the proportion of ground truth targets labeled in different categories within the current real-vehicle scenario or batch real-vehicle scenario. For target categories with a labeling proportion of less than 5%, these categories can be designated as niche categories and assigned differentiated weights when calculating category evaluation results or overall evaluation results. For example, niche categories such as pedestrians and road cones with a labeling proportion of less than 5% are assigned a double evaluation weight, ensuring that the overall evaluation results are not masked by the large number of high-frequency categories such as vehicles. Performance evaluation parameters for real-vehicle 3D detection can include basic evaluation metrics, vehicle robustness evaluation metrics, evaluation results for each target category, overall evaluation score, and information on scene nodes with significant errors.

[0050] Simultaneously, process control signals are generated based on the data integrity status and evaluation error status. The data integrity status can be determined based on the integrity verification mark formed in step S102, and the evaluation error status can be determined based on the comparison results between position error, size error, recognition accuracy, real-vehicle robustness evaluation index, or overall evaluation score and a preset threshold. When the data missing rate is less than or equal to the preset missing rate threshold and the core evaluation error is within a preset reasonable range, a process control signal indicating normal evaluation is generated; when the data missing rate is greater than the preset missing rate threshold or the core evaluation error exceeds the preset reasonable range, a process control signal indicating abnormal data or abnormal evaluation error is generated. The preset missing rate threshold can be set to 10%, and the core evaluation error can include position error. In this way, the process control signal can reflect the evaluation process status of the current real-vehicle scenario and serve as the basis for automated control of the evaluation process.

[0051] Through the above multi-dimensional perception performance evaluation, we can simultaneously obtain comprehensive evaluation results on target recognition, positional deviation, size deviation, occlusion scene adaptability, dynamic target detection stability, lighting environment adaptability, and category distribution influence, making the real vehicle 3D detection perception performance evaluation parameters more closely match the real perception performance in the real vehicle road environment.

[0052] Step S105: Generate visual verification results and quantitative evaluation results based on the actual vehicle 3D detection perception performance evaluation parameters.

[0053] In step S105, visual verification results and quantitative evaluation results are generated based on the real-vehicle 3D detection perception performance evaluation parameters. These parameters include basic evaluation indicators, real-vehicle robustness evaluation indicators, evaluation results for each target category, overall evaluation score, and information on scene nodes with significant errors. Since real-vehicle 3D detection perception performance evaluation not only requires the generation of numerical indicators but also needs to visually demonstrate the differences in position, size, pose, and category between the labeled ground truth target and the perceived target, this step generates visual verification results simultaneously with the quantitative evaluation results, enabling the evaluation results to be presented in the form of images, videos, and reports.

[0054] Specifically, based on the performance evaluation parameters of 3D vehicle detection, the system generates labeled ground truth target display results, perceived target display results, and overlay comparison display results of labeled ground truth targets and perceived targets. The labeled ground truth target display results show the labeled ground truth targets in the labeled ground truth data, the perceived target display results show the perceived targets in the perceived data, and the overlay comparison display results show both labeled ground truth targets and perceived targets simultaneously in the same real-vehicle scene. When displaying targets, different display styles can be configured for different types of targets according to their categories, enabling the differentiation of target categories such as vehicles, trucks, pedestrians, and road cones in the image. Labeled ground truth targets and perceived targets can be displayed using 2D bounding boxes, 3D bounding box projections, or target bounding box outlines, and the overlay comparison display results show target position deviations, target size deviations, and attitude angle deviations.

[0055] In one implementation, a three-screen comparison chart can be generated based on the labeled true target display results, the perceived target display results, and the overlay comparison display results. (See reference...) Figure 2 , Figure 2 To compare the perceived results with the labeled ground truth, a three-screen comparison chart is provided, including a labeled ground truth target view, a perceived target view, and an overlay comparison view. The labeled ground truth target view is used to display the labeled ground truth target separately, the perceived target view is used to display the perceived target output by the vehicle-mounted 3D detection and perception system separately, and the overlay comparison view is used to overlay the labeled ground truth target and the perceived target in the same real vehicle scene to observe the deviations of the same target in terms of category, position, size, and attitude angle. Figure 2 The left-hand view is the labeled true value target view, the middle view is the perceived target view, and the right-hand view is the overlay comparison view. The three-screen comparison chart can intuitively present the differences between the labeled true value data and the perceived data.

[0056] Furthermore, the performance evaluation parameters of real-vehicle 3D detection and perception can be overlaid and fused with the real-vehicle driving video stream to generate a real-vehicle scene reconstruction video. This video can display target bounding boxes, target categories, positional errors, size errors, pose errors, scene identifiers, and timestamps. For targets or scene nodes with large errors, they can be prominently marked in the reconstruction video, enabling evaluators to locate time segments and road scenes with significant perception deviations. The reconstruction video can be combined with a three-screen comparison image to form a visual verification result.

[0057] Simultaneously, based on the real-vehicle 3D detection perception performance evaluation parameters, quantitative evaluation results are generated, including basic evaluation indicators, real-vehicle robustness evaluation indicators, and target category evaluation results. The quantitative evaluation results can include scene encoding, evaluation time range, number of labeled ground truth targets, number of perceived targets, number of associated targets, number of missed targets, number of falsely detected targets, positional error, size error, recognition accuracy, occlusion adaptability, dynamic target tracking accuracy, illumination adaptability, evaluation results for each target category, overall evaluation score, and error analysis. For evaluation tasks involving multiple real-vehicle scenarios, the quantitative evaluation results can also include the evaluation score for each real-vehicle scenario, perception performance fluctuations, and typical error scenario analysis.

[0058] When generating quantitative evaluation results, the evaluation parameters can be organized according to the dimensions of real-vehicle scenario, target category, time period, and indicator type. The real-vehicle scenario dimension represents the perception performance under different road scenarios, the target category dimension represents the detection effect of different target categories, the time period dimension represents the performance changes at different collection times within the same real-vehicle scenario, and the indicator type dimension distinguishes between basic evaluation indicators and real-vehicle robustness evaluation indicators. The generated quantitative evaluation results can be output in the form of a standardized evaluation report, which records the values ​​of each indicator, corresponding scenario information, and explanations of abnormal errors.

[0059] After generating the visual verification results and quantitative evaluation results, the three-screen comparison chart, the real-vehicle scene reconstruction video, and the standardized evaluation report are stored in a designated directory, and a result index is established using scene identifiers and timestamps. The result index is used to associate the visual verification results and quantitative evaluation results with the corresponding real-vehicle scene, multi-source acquired data, synchronized labeled ground truth data, and perception data. Through this process, the performance evaluation results of the real-vehicle 3D inspection and perception system possess both intuitive display and quantitative analysis capabilities, facilitating the verification and analysis of the performance of the in-vehicle 3D inspection and perception system in real-vehicle scenarios.

[0060] Step S106: Automated control of the real vehicle 3D detection and perception performance evaluation process corresponding to the real vehicle scene according to the process control signal.

[0061] In step S106, the real vehicle 3D detection and perception performance evaluation process corresponding to the real vehicle scenario is automatically controlled according to the process control signal. The process control signal is generated in step S104 based on the data integrity status and evaluation error status, and is used to characterize whether the current evaluation process of the real vehicle scenario meets the control conditions for continuing to output results, triggering exception handling, or entering the scene loop evaluation. By automating the evaluation process through the process control signal, the manual item-by-item judgment and manual switching processing links can be reduced, and a continuous automated evaluation process can be formed between the annotation truth preprocessing, real vehicle multi-source data acquisition, dynamic spatiotemporal alignment, multi-dimensional perception performance evaluation, visualization verification result generation, and quantitative evaluation result generation.

[0062] Specifically, when the process control signal indicates that the evaluation is normal, the system outputs both visual verification results and quantitative evaluation results. A normal evaluation means that the data integrity status of the current real-vehicle scenario meets preset requirements, and the core evaluation errors are within a preset reasonable range. For example, if the data missing rate is less than or equal to 10%, and core evaluation errors such as positional errors do not exceed preset reasonable thresholds, the evaluation process of the current real-vehicle scenario is determined to be normal. At this point, the three-screen comparison chart, real-vehicle scenario reconstruction video, and standardized evaluation report generated in step S105 are output to a designated directory, and the storage path of the output results is recorded according to the scenario identifier and timestamp, enabling the evaluation results of the current real-vehicle scenario to be retrieved and verified.

[0063] When process control signals indicate data anomalies or evaluation error anomalies, an anomaly alert is triggered, and the evaluation process corresponding to the current real-vehicle scenario is paused. Data anomalies include missing point cloud environment data, image environment data, obstacle perception data, attitude data, or sensor calibration data under the same scene identifier, or data files that cannot be read normally, or timestamp anomalies. Evaluation error anomalies include position errors, size errors, recognition accuracy, real-vehicle robustness evaluation indicators, or overall evaluation scores exceeding a preset reasonable range. When an anomaly alert is triggered, an alarm message containing a scene identifier, abnormal data type, abnormal timestamp, abnormal indicator name, and abnormal indicator value can be generated, and the evaluation process corresponding to the current real-vehicle scenario can be paused. After pause, abnormal data can be manually verified, or relevant parameters can be adjusted. After data verification or parameter adjustment is completed, the evaluation process corresponding to the current real-vehicle scenario can be restarted.

[0064] For batch evaluation tasks involving multiple real-vehicle scenarios, after the visualization verification results and quantitative evaluation results for the current real-vehicle scenario are output, it is determined whether there is any unprocessed real-vehicle scenario data. If unprocessed real-vehicle scenario data exists, the evaluation process corresponding to the next real-vehicle scenario is initiated according to the scenario identification order or the preset scenario processing order. This process sequentially executes labeled ground truth preprocessing, real-vehicle multi-source data acquisition, dynamic spatiotemporal alignment, multi-dimensional perception performance evaluation, visualization verification result generation, and quantitative evaluation result generation for the next real-vehicle scenario. If no unprocessed real-vehicle scenario data exists, a summary evaluation report is generated based on the quantitative evaluation results corresponding to each real-vehicle scenario. The summary evaluation report may include the evaluation score for each real-vehicle scenario, perception performance fluctuations, typical error scenario analysis, and comprehensive evaluation results for different target categories.

[0065] During automated control, the evaluation status of each real-vehicle scenario can be recorded. Evaluation statuses include: not started, processing, evaluation normal, data abnormal, evaluation error abnormal, paused, and completed. Each status is associated with a corresponding scenario identifier, timestamp, process control signal, and result storage path. By recording the evaluation status, it is possible to clearly determine which evaluation stage each real-vehicle scenario is in and whether any abnormalities exist.

[0066] Through the above-mentioned automated control methods, the real vehicle 3D inspection and perception performance evaluation process can automatically complete result output, anomaly prompts, pause processing, restart and batch scene loop evaluation according to the process control signals, reduce manual intervention, improve the processing efficiency of real vehicle scene evaluation tasks, and ensure that abnormal data and abnormal evaluation results can be identified in a timely manner.

[0067] In summary, combining Figure 3This application provides a method and system for evaluating the perception performance of real vehicle 3D detection based on labeled ground truth data. First, the original labeled data corresponding to the real vehicle scene is acquired, and labeled ground truth data corresponding to the real vehicle scene is generated through labeled ground truth preprocessing. Then, multi-source real vehicle data corresponding to the real vehicle scene is collected and synchronously associated with storage to form multi-source acquired data. Next, the labeled ground truth data and multi-source acquired data are dynamically spatiotemporally aligned to obtain synchronized labeled ground truth data and perception data. Based on this, multi-dimensional perception performance evaluation is performed according to the synchronized labeled ground truth data and perception data, generating real vehicle 3D detection perception performance evaluation parameters and process control signals. Then, based on the real vehicle 3D detection perception performance evaluation parameters, visual verification results and quantitative evaluation results are generated, and the real vehicle 3D detection perception performance evaluation process corresponding to the real vehicle scene is automatically controlled according to the process control signals.

[0068] Through the above scheme, the annotation ground truth preprocessing can form a unified expression of the original annotation data in terms of target category, target size, target pose and data format, reducing the impact of annotation differences on the evaluation reference data; synchronous associated storage can maintain clear scene correspondence and time correspondence between real vehicle multi-source data; dynamic spatiotemporal alignment can compare the annotation ground truth data and perception data under the same time and space reference, reducing the matching deviation caused by changes in vehicle speed, attitude and sensor data time differences; multi-dimensional perception performance evaluation can reflect perception performance from the aspects of basic detection accuracy and real vehicle robustness; visualization verification results, quantitative evaluation results and automated control can improve the efficiency of evaluation result verification and batch evaluation processing, thereby improving the accuracy, reliability and applicability of real vehicle 3D detection perception performance evaluation.

[0069] Figure 4 This is a structural block diagram of a real vehicle 3D detection and perception performance evaluation system based on labeled ground truth data, provided in one embodiment of this application. The system includes at least the following modules: The annotation truth preprocessing module is used to obtain the original annotation data corresponding to the real vehicle scenario, and to perform annotation truth preprocessing on the original annotation data to obtain the annotation truth data corresponding to the real vehicle scenario. The real vehicle multi-source data acquisition module is used to collect real vehicle multi-source data corresponding to real vehicle scenarios, and to synchronously associate and store the real vehicle multi-source data to generate multi-source acquisition data corresponding to the real vehicle scenarios. The dynamic spatiotemporal alignment module is used to dynamically align labeled ground truth data and multi-source acquired data in spatiotemporal space to obtain synchronized labeled ground truth data and sensed data. The multi-dimensional evaluation module is used to perform multi-dimensional perception performance evaluation based on synchronized labeled true data and perception data, and generate real vehicle 3D detection perception performance evaluation parameters and process control signals. The visualization verification module is used to generate visualization verification results and quantitative evaluation results based on the 3D detection and perception performance evaluation parameters of the actual vehicle. The automation control module is used to automatically control the real vehicle 3D detection and perception performance evaluation process corresponding to the real vehicle scenario according to the process control signals.

[0070] For relevant details, please refer to the above method implementation examples.

[0071] Figure 5 This is a block diagram of an electronic device provided in one embodiment of this application. The electronic device includes at least a processor 501 and a memory 502. The electronic device can be an in-vehicle computing terminal, a server, an industrial computer, or a processing terminal for evaluating the 3D detection and perception performance of a real vehicle, and is capable of executing the 3D detection and perception performance evaluation method based on labeled ground truth data provided in this application embodiment.

[0072] Processor 501 may include one or more processing cores, such as a 4-core processor or an 8-core processor. Processor 501 may be implemented using at least one hardware form of DSP, FPGA, or PLA, where DSP is a digital signal processor, FPGA is a field-programmable gate array, and PLA is a programmable logic array. Processor 501 may also include a main processor and a coprocessor. The main processor is used to perform calculations on data processing tasks such as labeled ground truth preprocessing, synchronous association and storage of multi-source real vehicle data, dynamic spatiotemporal alignment, multi-dimensional perception performance evaluation, and process control. The coprocessor is used to handle data caching, image rendering, or data management tasks in low-power states. In some embodiments, processor 501 may integrate a GPU, which is an image processor used to render and draw the graphical content in the visualization verification results, real vehicle scene reconstruction images, and quantitative evaluation results. In some embodiments, processor 501 may also include an AI processor, which is an artificial intelligence processor used to handle computational tasks related to target recognition result analysis, batch evaluation data processing, or evaluation result statistics.

[0073] The memory 502 may include one or more computer-readable storage media, which may be non-transitory. The memory 502 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. The memory 502 is used to store raw labeled data, labeled ground truth data, multi-source real vehicle data, multi-source acquired data, synchronized labeled ground truth data and perceived data, real vehicle 3D detection perception performance evaluation parameters, visualization verification results, and quantitative evaluation results. In some embodiments, the non-transitory computer-readable storage medium in the memory 502 is used to store at least one instruction, which is executed by the processor 501 to implement the real vehicle 3D detection perception performance evaluation method based on labeled ground truth data provided in the embodiments of this application.

[0074] Optionally, this application also provides a computer-readable storage medium storing a program that is loaded and executed by a processor to implement the real vehicle 3D detection perception performance evaluation method based on labeled ground truth data in the above method embodiments.

[0075] Optionally, this application also provides a computer product including a computer-readable storage medium storing a program, which is loaded and executed by a processor to implement the real vehicle 3D detection perception performance evaluation method based on labeled ground truth data in the above method embodiments.

[0076] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0077] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for evaluating the perception performance of a real vehicle in 3D detection based on labeled ground truth data, characterized in that, The method includes: Obtain the original labeled data corresponding to the real vehicle scene, and perform label ground value preprocessing on the original labeled data to obtain the labeled ground value data corresponding to the real vehicle scene; Collect real vehicle multi-source data corresponding to the real vehicle scenario, and synchronously associate and store the real vehicle multi-source data to generate multi-source collected data corresponding to the real vehicle scenario; The labeled ground truth data and the multi-source acquired data are dynamically spatiotemporally aligned to obtain synchronized labeled ground truth data and sensed data. Based on the synchronized labeled true data and perception data, a multi-dimensional perception performance evaluation is performed to generate real vehicle 3D detection perception performance evaluation parameters and process control signals. Based on the aforementioned real vehicle 3D detection and perception performance evaluation parameters, visualized verification results and quantitative evaluation results are generated; The process control signals are used to automatically control the real vehicle 3D detection and perception performance evaluation process corresponding to the real vehicle scene.

2. The method for evaluating the perception performance of a real vehicle based on labeled ground truth data according to claim 1, characterized in that, The step of performing ground truth preprocessing on the original labeled data to obtain labeled ground truth data corresponding to the real vehicle scene includes: The intersection-union ratio (IU) of the target bounding boxes is calculated for the labeled targets in the original labeled data. Multiple labeled targets with the same target category and an IU greater than a preset duplication threshold are identified as duplicate labeled targets, and the duplicate labeled targets are removed. Based on the occlusion rate of the labeled target, the occluded targets that meet the completion conditions are determined, and the target size and attitude angle of the occluded targets are completed by using the same type of target feature interpolation method. The target categories in the original labeled data are uniformly encoded according to the preset category mapping relationship, and the labeled data after category unification is converted into a preset format to obtain the labeled true value data.

3. The method for evaluating the perception performance of a real vehicle based on labeled ground truth data according to claim 1, characterized in that, The process of collecting real-vehicle multi-source data corresponding to the real-vehicle scenario, and synchronously associating and storing the real-vehicle multi-source data to generate multi-source collected data corresponding to the real-vehicle scenario includes: The system collects environmental data, obstacle perception data, and attitude data corresponding to the actual vehicle scene using vehicle-mounted LiDAR, vehicle-mounted vision camera, and inertial measurement unit, and obtains sensor calibration data corresponding to the actual vehicle scene. The environmental data, the perceived obstacle data, the attitude data, and the sensor calibration data are associated and stored according to the scene identifier and timestamp; The data after association and storage is verified for integrity, and the multi-source collected data is generated based on the integrity verification results.

4. The method for evaluating the perception performance of a real vehicle based on labeled ground truth data according to claim 1, characterized in that, The step of dynamically aligning the labeled ground truth data and the multi-source acquired data in a spatiotemporal manner to obtain synchronized labeled ground truth data and sensed data includes: Vehicle speed data and perception data are extracted from the multi-source collected data, and the timestamps of the labeled true data and the perception data are converted into a unified time unit. A time matching threshold is determined based on the vehicle speed data. A first time matching threshold is used when the vehicle speed data is greater than a preset speed threshold, and a second time matching threshold is used when the vehicle speed data is less than or equal to the preset speed threshold. The first time matching threshold is less than the second time matching threshold. The labeled ground truth data and the perceived data are matched with the nearest neighbor time according to the time matching threshold to obtain time-synchronized labeled ground truth data and perceived data.

5. The method for evaluating the perception performance of a real vehicle based on labeled ground truth data according to claim 4, characterized in that, The step of dynamically aligning the labeled ground truth data and the multi-source acquired data in a spatiotemporal manner to obtain synchronized labeled ground truth data and sensed data further includes: Attitude data and sensor calibration data are extracted from the multi-source acquired data; Based on the attitude data, attitude compensation is performed on the rotation and translation parameters in the sensor calibration data to obtain dynamic calibration parameters; According to the dynamic calibration parameters, coordinate transformation is performed on the time-synchronized labeled ground truth data and perceived data to make the labeled ground truth target and perceived target in a unified coordinate system, thus obtaining the synchronized labeled ground truth data and perceived data.

6. The method for evaluating the perception performance of a real vehicle based on labeled ground truth data according to claim 1, characterized in that, The step of performing multi-dimensional perception performance evaluation based on the synchronized labeled ground truth data and perception data, and generating real vehicle 3D detection perception performance evaluation parameters and process control signals includes: Based on the target category and the 3D spatial matching relationship, target association is performed between the labeled ground value targets in the synchronized labeled ground value data and the perceived targets in the perceived data; Based on the target association results, basic evaluation indicators and real-vehicle robustness evaluation indicators are calculated. The basic evaluation indicators include position error, size error and recognition accuracy. The real-vehicle robustness evaluation indicators include occlusion adaptability, dynamic target tracking accuracy and illumination adaptability. Based on the basic evaluation indicators, the real vehicle robustness evaluation indicators, and the target category distribution, the real vehicle 3D detection and perception performance evaluation parameters are generated, and the process control signals are generated based on the data integrity status and evaluation error status.

7. The method for evaluating the perception performance of a real vehicle based on labeled ground truth data according to claim 1, characterized in that, The step of generating visual verification results and quantitative evaluation results based on the real vehicle 3D detection and perception performance evaluation parameters, and automating the real vehicle 3D detection and perception performance evaluation process corresponding to the real vehicle scene according to the process control signals, includes: Based on the real vehicle 3D detection perception performance evaluation parameters, the system generates the labeled true value target display result, the perception target display result, and the superimposed comparison display result of the labeled true value target and the perception target, and generates a quantitative evaluation result that includes basic evaluation indicators, real vehicle robustness evaluation indicators, and target category evaluation results. If the process control signal indicates that the evaluation is normal, the visual verification result and the quantitative evaluation result will be output. If the process control signal indicates abnormal data or abnormal evaluation error, an error message will be triggered and the evaluation process corresponding to the current real vehicle scenario will be paused.

8. A real-vehicle 3D inspection and perception performance evaluation system based on labeled ground truth data, characterized in that, include: The annotation truth preprocessing module is used to obtain the original annotation data corresponding to the real vehicle scene, and to perform annotation truth preprocessing on the original annotation data to obtain the annotation truth data corresponding to the real vehicle scene. The real vehicle multi-source data acquisition module is used to acquire real vehicle multi-source data corresponding to the real vehicle scenario, and synchronously associate and store the real vehicle multi-source data to generate multi-source acquisition data corresponding to the real vehicle scenario. The dynamic spatiotemporal alignment module is used to dynamically align the labeled ground value data and the multi-source acquired data in spatiotemporal mode to obtain synchronized labeled ground value data and sensed data. The multi-dimensional evaluation module is used to perform multi-dimensional perception performance evaluation based on the synchronized labeled true data and perception data, and generate real vehicle 3D detection perception performance evaluation parameters and process control signals. The visualization verification module is used to generate visualization verification results and quantitative evaluation results based on the actual vehicle 3D detection and perception performance evaluation parameters. An automated control module is used to automatically control the real vehicle 3D detection and perception performance evaluation process corresponding to the real vehicle scene according to the process control signal.

9. An electronic device, characterized in that, The device includes a processor and a memory; the memory stores a program, which is loaded and executed by the processor to implement a real vehicle 3D detection perception performance evaluation method based on labeled ground truth data as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a program that, when executed by a processor, is used to implement a method for evaluating the perception performance of a real vehicle based on labeled ground truth data, as described in any one of claims 1 to 7.

Images