Method and apparatus for evaluating obstacle visual perception algorithm, device, medium, and product

Abstract

A method and apparatus for evaluating an obstacle visual perception algorithm, a device, a medium, and a product. The method comprises: acquiring three-dimensional point cloud data acquired by a laser radar and prediction algorithm data obtained by a visual perception algorithm processing acquired road data, and using the three-dimensional point cloud data as ground truth data (S101); preprocessing the ground truth data and the prediction algorithm data to obtain corresponding target ground truth data and target prediction algorithm data, the preprocessing comprising timestamp alignment, spatial alignment, and key frame extraction (S102); on the basis of the target ground truth data and the target prediction algorithm data and in view of a preset adaptive evaluation rule, determining an initial evaluation index of the visual perception algorithm (S103); and, on the basis of the initial evaluation index, determining a target evaluation index of the visual perception algorithm (S104).

Description

Evaluation methods, devices, equipment, media, and products for obstacle visual perception algorithms

[0001] This application claims priority to Chinese Patent Application No. 202411032572.8, filed with the Chinese Patent Office on July 30, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of autonomous driving information perception technology, such as an obstacle visual perception algorithm evaluation method, device, equipment, medium and product. Background Technology

[0003] Effective identification and assessment of obstacles in the surrounding environment are crucial for ensuring safe and efficient autonomous driving in autonomous driving systems. Evaluating visual perception algorithms can effectively assess their perception and prediction performance. When evaluating visual perception algorithms, high-precision 3D point cloud data provided by LiDAR is typically used as the ground truth for obstacles, making the underlying data more convincing. Reliable evaluation rules further increase the confidence level of the visual perception algorithm. However, the difference in timestamps between visual perception algorithms and LiDAR outputs makes it difficult to match the predicted algorithm data with the ground truth data. Furthermore, using a single evaluation rule is unreasonable for obstacles with large volume deviations, thus affecting the evaluation of the predicted algorithm data and leading to inaccurate evaluation results. Summary of the Invention

[0004] This application provides a method, apparatus, device, medium, and product for evaluating obstacle visual perception algorithms, which improves the efficiency and accuracy of analyzing the performance of visual perception algorithms.

[0005] Firstly, this embodiment provides an obstacle visual perception algorithm evaluation method, which includes:

[0006] The three-dimensional point cloud data collected by the lidar and the prediction algorithm data obtained by processing the road data by the visual perception algorithm are acquired, and the three-dimensional point cloud data is used as the ground truth data.

[0007] The ground value data and the prediction algorithm data are preprocessed to obtain the corresponding target ground value data and target prediction algorithm data. The preprocessing includes timestamp alignment, spatial alignment and keyframe extraction.

[0008] Based on the target true value data and the target prediction algorithm data, and combined with the preset adaptive evaluation rules, the initial evaluation index of the visual perception algorithm is determined. The adaptive evaluation rules include a three-dimensional evaluation calculation method and a two-dimensional evaluation calculation method.

[0009] Based on the initial evaluation index, the target evaluation index of the visual perception algorithm is determined.

[0010] Secondly, this embodiment provides an obstacle visual perception algorithm evaluation device, the device comprising:

[0011] The data acquisition module is configured to acquire the three-dimensional point cloud data collected by the lidar and the prediction algorithm data obtained by the visual perception algorithm processing the road data, and to use the three-dimensional point cloud data as the ground truth data.

[0012] The preprocessing module is configured to preprocess the ground value data and the prediction algorithm data to obtain the corresponding target ground value data and target prediction algorithm data. The preprocessing includes timestamp alignment, spatial alignment, and keyframe extraction.

[0013] The initial evaluation module is configured to determine the initial evaluation index of the visual perception algorithm based on the target ground truth data and the target prediction algorithm data, combined with a preset adaptive evaluation rule, wherein the adaptive evaluation rule is related to the size of the obstacle;

[0014] The target evaluation module is configured to determine the target evaluation index of the visual perception algorithm based on the initial evaluation index.

[0015] Thirdly, this embodiment provides an electronic device, including:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the obstacle visual perception algorithm evaluation method according to any embodiment of this application.

[0019] Fourthly, this embodiment provides a computer-readable storage medium, wherein the computer program is executed by the at least one processor to enable the at least one processor to perform the obstacle visual perception algorithm evaluation method according to any embodiment of this application.

[0020] Fifthly, embodiments of this application also provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the obstacle visual perception algorithm evaluation method as described in the first aspect embodiment.

[0021] This application provides an obstacle visual perception algorithm evaluation method, apparatus, device, medium, and product. The method includes: firstly, acquiring three-dimensional point cloud data collected by a lidar and prediction algorithm data obtained by processing the road data using a visual perception algorithm, and using the three-dimensional point cloud data as ground truth data; secondly, preprocessing the ground truth data and the prediction algorithm data to obtain corresponding target ground truth data and target prediction algorithm data, wherein the preprocessing includes timestamp alignment, spatial alignment, and keyframe extraction; then, based on the target ground truth data and the target prediction algorithm data, and in conjunction with preset adaptive evaluation rules, determining the initial evaluation index of the visual perception algorithm, wherein the adaptive evaluation rules include a three-dimensional evaluation calculation method and a two-dimensional evaluation calculation method; finally, determining the target evaluation index of the visual perception algorithm based on the initial evaluation index. Unlike related technologies where the timestamps of visual perception algorithms and LiDAR outputs differ, and where a single evaluation rule can affect the evaluation of prediction algorithm data, leading to inaccurate evaluation results, the above technical solution first aligns the timestamps and spatially of the ground truth data and prediction algorithm data to ensure they match. Then, it performs keyframe extraction on the aligned ground truth and prediction algorithm data, effectively reducing the amount of data that needs to be processed and stored, thus lowering computational and storage costs. Simultaneously, it combines three-dimensional and two-dimensional evaluation calculation methods, adaptively adjusting the evaluation rules to improve the accuracy and efficiency of evaluating visual perception algorithms.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 is a flowchart illustrating an obstacle visual perception algorithm evaluation method provided in Embodiment 1 of this application;

[0025] Figure 2 is a flowchart illustrating another obstacle visual perception algorithm evaluation method provided in Embodiment 2 of this application;

[0026] Figure 3 is a schematic diagram of an obstacle visual perception algorithm evaluation device provided in Embodiment 3 of this application;

[0027] Figure 4 is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of this application. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "original," "target," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any other variations, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] Example 1

[0031] Figure 1 is a flowchart illustrating an obstacle visual perception algorithm evaluation method provided in Embodiment 1 of this application. This method is applicable to the evaluation of obstacle visual perception algorithms. This method can be executed by an obstacle visual perception algorithm evaluation device, which can be implemented in hardware and / or software and is generally integrated into an electronic device.

[0032] As shown in Figure 1, the obstacle visual perception algorithm evaluation method provided in this embodiment may specifically include the following steps:

[0033] S101. Acquire the 3D point cloud data collected by the lidar and the prediction algorithm data obtained by processing the road data using the visual perception algorithm, and use the 3D point cloud data as the ground truth data.

[0034] The application scenario of this embodiment can be described as follows: Based on a vehicle-mounted LiDAR and a visual perception algorithm, obstacles in the road environment are identified. The LiDAR identification result is used as the ground truth value of the obstacle, and the visual perception algorithm's identification result is used as the predicted value. The predicted value is evaluated based on the ground truth value, thus evaluating the performance of the visual perception algorithm. Obstacles can include large vehicles, small vehicles, pedestrians, two-wheeled vehicles, three-wheeled vehicles, water-filled barriers, and traffic cones in the road environment. Obstacle information can include the obstacle's size, shape, and for dynamic obstacles, its speed or heading angle can also be obtained.

[0035] In this embodiment, a LiDAR installed on the vehicle collects 3D point cloud data of the road environment in real time, thereby obtaining information on various obstacles in the road environment. Road data acquisition devices (such as cameras) installed on the vehicle collect image and video data of the road environment in real time, which are recorded as road data. The road data is processed by the visual perception algorithm integrated into the vehicle's visual perception system to obtain information on various obstacles in the road environment. This step is used to acquire the 3D point cloud data collected by the LiDAR and the prediction algorithm data obtained by processing the road data using the visual perception algorithm, and uses the 3D point cloud data as ground truth data. For example, the ground truth data and prediction algorithm data can be stored in a file. The prediction algorithm data and the LiDAR point cloud ground truth data can be read using Python, and both can be formed into a series using the pandas library, processing them into the same format for convenient subsequent data calculation and processing. Here, Python is a programming language, pandas is a tool created to solve data analysis tasks, and a series is a data structure.

[0036] S102. Preprocess the true data and prediction algorithm data to obtain the corresponding target true data and target prediction algorithm data.

[0037] The preprocessing includes timestamp alignment, spatial alignment, and keyframe extraction.

[0038] Given the incomplete synchronization of timestamps output by LiDAR and visual perception algorithms, comparing 3D point cloud data and prediction algorithm data leads to data asynchrony on the timeline, hindering effective evaluation. Furthermore, the different installation locations of LiDAR and road survey equipment result in differences in viewing angle and field of view when comparing 3D point cloud data and prediction perception data, complicating spatial data alignment. Therefore, high-precision time synchronization and spatial alignment are needed to ensure that 3D point cloud data and prediction perception data from the same time stamp and space are compared, improving data accuracy and reliability.

[0039] Meanwhile, due to the massive amount of data generated by LiDAR and road survey equipment, directly processing all the data would incur high computational and storage costs. Furthermore, in autonomous driving systems, computational resources are limited, necessitating efficient resource utilization for data processing. Therefore, in this embodiment, after temporal and spatial alignment of the 3D point cloud data and prediction algorithm data, keyframe extraction is performed on the aligned data. Subsequently, the visual perception algorithm is evaluated based on the extracted data, thereby reducing computational and storage costs and achieving efficient data processing.

[0040] Specifically, the ground truth data and prediction algorithm data are aligned in terms of both timestamp and space to obtain aligned ground truth data and prediction algorithm data, i.e., ground truth data and prediction algorithm data under the same timestamp and space. Then, keyframe extraction is performed on the aligned ground truth data and prediction algorithm data, and the corresponding ground truth data and prediction algorithm data after the frame extraction are recorded as the corresponding target ground truth data and target prediction algorithm data.

[0041] It should be noted that the corresponding target ground truth data and target prediction algorithm data can be considered as data obtained from the same frame after acquisition and processing. Therefore, the target prediction algorithm data can be evaluated based on the target ground truth data. For example, ground truth data and prediction algorithm data over a period of time can be acquired, then the ground truth data and prediction algorithm data can be aligned in time and space, and then keyframe extraction can be performed to obtain the target ground truth data and target prediction algorithm data corresponding to several frames within this period. This is equivalent to obtaining the target ground truth data and target prediction algorithm data corresponding to several frames.

[0042] S103. Based on the target true value data and target prediction algorithm data, and combined with the preset adaptive evaluation rules, determine the initial evaluation index of the visual perception algorithm.

[0043] The adaptive evaluation rules include both three-dimensional and two-dimensional evaluation calculation methods.

[0044] Considering that a single evaluation rule is unreasonable for obstacles with large volume deviations, thus affecting the accuracy of the predicted data evaluation, larger obstacles have larger bounding boxes and are more likely to overlap. Smaller objects have smaller bounding boxes and are more difficult to locate accurately. Therefore, the positive (TP), false positive (FP), and false negative (FN) values ​​for different types of obstacles need to be calculated using appropriate methods. In this embodiment, a combination of three-dimensional and two-dimensional evaluation calculation methods is used, which is more scientific and reasonable, thus ensuring more accurate evaluation results. In this embodiment, the ground truth obstacles contained in the target ground truth data and the predicted obstacles contained in the target prediction algorithm data are obtained. For larger obstacles, the three-dimensional intersection-union ratio (IUU) of the ground truth obstacles and predicted obstacles can be used to evaluate whether the predicted algorithm data is a positive detection. However, for smaller obstacles, evaluation using the three-dimensional IUU method will lead to a large error in the results. Therefore, for smaller obstacles, the two-dimensional Euclidean distance between the ground truth obstacles and predicted obstacles can be used to evaluate whether the predicted algorithm data is a positive detection.

[0045] In this embodiment, ground truth obstacles are used to evaluate the accuracy of predicted obstacles, such as whether the prediction result is a positive detection or a false detection. These positive or false detection results are used as initial evaluation indicators. Specifically, if the ground truth obstacle is larger than a set obstacle threshold, the intersection-union ratio (IUR) between the ground truth obstacle and the predicted obstacle is calculated using a three-dimensional calculation method, and the initial evaluation indicator is determined based on the IUR. If the ground truth obstacle is smaller than or equal to the set obstacle size, the Euclidean distance between the ground truth obstacle and the predicted obstacle is calculated using a two-dimensional calculation method, and the initial evaluation indicator is determined based on the Euclidean distance.

[0046] In this embodiment, an adaptive evaluation rule is employed. This rule integrates two-dimensional and three-dimensional evaluation rules to adapt to the visual perception needs of different types of obstacles. This multi-dimensional evaluation rule design not only improves the system's versatility but also enables more accurate identification and evaluation of different types of obstacles.

[0047] S104. Based on the initial evaluation indicators, determine the target evaluation indicators for the visual perception algorithm.

[0048] In this embodiment, the initial evaluation metrics include evaluating the prediction algorithm data for positive detection, false positives, and false negatives, equivalent to using TP, FP, and FN. Precision and recall are calculated for different types of obstacles using TP, FP, and FN. Simultaneously, based on ground truth data and using the prediction algorithm data and ground truth data within the same frame, the confidence and performance of the visual perception algorithm data are measured frame-by-frame using metrics such as lateral and longitudinal errors, velocity errors, heading angle errors, lateral average relative errors, and longitudinal average relative errors. These metrics are then used as the target evaluation metrics for the visual perception algorithm.

[0049] This application provides an obstacle visual perception algorithm evaluation method. Unlike related technologies where the timestamps of the visual perception algorithm and the LiDAR output differ, and where a single evaluation rule affects the evaluation of the prediction algorithm data, leading to inaccurate evaluation results, the above technical solution first aligns the timestamps and spatially of the ground truth data and the prediction algorithm data to ensure they match. Then, it performs keyframe extraction on the aligned ground truth data and prediction algorithm data, effectively reducing the amount of data that needs to be processed and stored, thus lowering computational and storage costs. Simultaneously, it combines three-dimensional and two-dimensional evaluation calculation methods, adaptively adjusting the evaluation rules to improve the accuracy and efficiency of visual perception algorithm evaluation.

[0050] As a first optional embodiment of the present application, based on the above embodiments, the first optional embodiment can optimize the method by further comprising: displaying the target evaluation index determined based on all obstacles within the perception range in a non-segmented manner on the human-computer interaction interface; and / or displaying the target evaluation index determined based on obstacles at intervals of a set distance within the perception range in a segmented manner on the human-computer interaction interface.

[0051] In this embodiment, the aforementioned target calculation indicators are stored in two forms: segmented and unsegmented, such as in JSON format, and displayed on the human-computer interaction interface in either segmented or unsegmented manner. The unsegmented method can be understood as displaying the target evaluation indicators of all obstacles within the perception range in a summarized manner. The segmented method can be understood as displaying the target evaluation indicators of obstacles within a set distance from the visual perception system in segments. For example, the segmentation rule is that within 0-100m, each segment is output every 10m; that is, the target evaluation indicators of obstacles within 0-10m are displayed as one segment, the target evaluation indicators of obstacles within 10-20m are displayed as another segment, and so on.

[0052] It's important to note that by displaying the evaluation results in segments, we can identify which distance segments yield more reliable results, allowing us to focus on those segments and optimize the perception algorithm accordingly. For example, obstacles closer to the visual perception system result in more accurate performance evaluations, while obstacles farther away are less accurate. Over a period of time, many obstacles may be perceived, perhaps hundreds, for evaluation. By assessing the distance between these obstacles and the vehicle, we can see that the accuracy of predictions decreases with distance. Displaying the results in segments allows users to see that the visual perception algorithm's accuracy may drop at greater distances, enabling us to focus on improving the algorithm in this area.

[0053] This optional embodiment displays the target evaluation indicators in both summarized and segmented forms, making the evaluation results of the visual perception algorithm more scientific and reliable. Through efficient data processing methods and by displaying the data on the platform, the performance of the perception algorithm is clearly demonstrated, providing assurance for downstream algorithms and meeting the high data requirements and safety needs of autonomous driving systems. The final evaluation results are displayed using a combination of segmented and unsegmented distance information. This method provides detailed segmented distance information, helping users or the system understand the distribution of obstacles within different distance ranges, while the unsegmented display provides an overall evaluation result, allowing users to quickly grasp the overall situation.

[0054] Example 2

[0055] Figure 2 is a flowchart illustrating an obstacle visual perception algorithm evaluation method according to Embodiment 2 of this application. This embodiment is a further optimization of the above embodiment. In this embodiment, the limitation optimization of "preprocessing the ground truth data and the prediction algorithm data to obtain corresponding target ground truth data and target prediction algorithm data" is further optimized, and the limitation optimization of "determining the initial evaluation index of the visual perception algorithm based on the target ground truth data and the target prediction algorithm data, combined with the preset adaptive evaluation rules" is further optimized, and the limitation optimization of "determining the target evaluation index of the visual perception algorithm based on the initial evaluation index" is further optimized.

[0056] As shown in Figure 2, this embodiment 2 provides an obstacle visual perception algorithm evaluation method, which specifically includes the following steps:

[0057] S201. Acquire the 3D point cloud data collected by the lidar and the prediction algorithm data obtained by processing the road data using the visual perception algorithm, and use the 3D point cloud data as the ground truth data.

[0058] S202. Based on the time synchronization algorithm, the true data and the prediction algorithm data are timestamped to obtain the timestamped true data and prediction algorithm data.

[0059] In this embodiment, each frame of data can be considered to correspond to a road scene, and the ground truth data and prediction algorithm data corresponding to a road scene need to be aligned. A time synchronization algorithm combining exact matching and fuzzy matching is used to timestamp-synchronize the prediction algorithm data with the 3D point cloud data, which serves as ground truth data, thereby obtaining timestamp-aligned ground truth data and prediction algorithm data.

[0060] As a specific implementation method, the steps of aligning the timestamps of ground truth data and prediction algorithm data based on a time synchronization algorithm to obtain timestamp-aligned ground truth data and prediction algorithm data can be optimized, including:

[0061] a1) Obtain the first timestamp of the truth data for each frame and the second timestamp of the prediction algorithm data for each frame.

[0062] Specifically, this step involves acquiring the files storing ground truth data and prediction algorithm data. Each frame of the image corresponds to one frame of ground truth data and one frame of prediction algorithm data. Over a period of time, several frames of images will be captured, resulting in several frames of ground truth data and several frames of prediction algorithm data. This step is used to obtain the timestamp of each frame of ground truth data, denoted as the first timestamp, and simultaneously obtain the timestamp of each frame of prediction algorithm data, denoted as the second timestamp.

[0063] b1) Determine if the first timestamp is equal to the second timestamp.

[0064] Specifically, iterate through the first timestamp of each frame of truth data and the second timestamp of each frame of prediction algorithm data, and determine whether the first timestamp and the second timestamp are equal.

[0065] c1) If the first timestamp and the second timestamp are equal, then align the truth data corresponding to the first timestamp with the prediction algorithm data corresponding to the second timestamp.

[0066] Specifically, if the first and second timestamps are equal, the ground truth data corresponding to the first timestamp is aligned with the prediction algorithm data corresponding to the second timestamp. Matching based on completely equal timestamps can be considered an exact matching method.

[0067] d1) If the first timestamp and the second timestamp are not equal, then determine whether the difference between the first timestamp and the second timestamp is less than the set time threshold.

[0068] In this embodiment, if the first timestamp and the second timestamp are not equal, a fuzzy matching method can be used to determine whether the first timestamp and the second timestamp match. Specifically, the difference between the first timestamp and the second timestamp is calculated, and it is determined whether the difference is less than a set time threshold. The set time threshold can be set based on empirical values; for example, the set time threshold can be set to 50ms.

[0069] e1) If the difference is less than the set time threshold, then align the true value data corresponding to the first timestamp with the prediction algorithm data corresponding to the second timestamp.

[0070] Specifically, if the difference between the first timestamp and the second timestamp is less than a set time threshold, then the first timestamp and the second timestamp are considered to correspond to the same time point, and the true value data corresponding to the first timestamp is aligned with the prediction algorithm data corresponding to the second timestamp.

[0071] f1) If the difference is greater than or equal to the set time threshold, then it is determined that the true value data corresponding to the first timestamp is not aligned with the prediction algorithm data corresponding to the second timestamp.

[0072] Specifically, if the difference between the first timestamp and the second timestamp is greater than or equal to a set time threshold, it is considered that the first timestamp and the second timestamp are not at the same time point, and the true value data corresponding to the first timestamp is not aligned with the prediction algorithm data corresponding to the second timestamp.

[0073] S203. Spatially align the timestamp-aligned ground truth data and prediction algorithm data to obtain spatially aligned ground truth data and prediction algorithm data.

[0074] In this embodiment, after aligning the ground truth data and prediction algorithm data with timestamps, spatial alignment is also required. Because the LiDAR and camera are installed in different locations, there are differences in viewing angle and field of view when fusing LiDAR point cloud data and camera image data, leading to complex spatial data alignment. Therefore, in this embodiment, the relative position and angle information of the LiDAR and camera are obtained through a calibration algorithm. A spatial transformation matrix can be used to transform the LiDAR point cloud data into the camera's coordinate system, achieving spatial alignment and ensuring they correspond to the same image.

[0075] S204. Based on a preset time interval or event triggering mechanism, extract keyframes from the timestamp-aligned and spatially aligned ground truth data and prediction algorithm data to obtain the corresponding target ground truth data and target prediction algorithm data.

[0076] Specifically, based on preset time intervals or event triggering mechanisms, keyframes are automatically extracted from the aligned video stream and point cloud data to obtain corresponding target ground truth data and target prediction algorithm data. The selection criteria for keyframes can be based on algorithms such as motion detection and object recognition to ensure the representativeness and importance of the extracted frames. For example, a single second may contain many frames, but the road image captured within that second remains almost unchanged, making it meaningless to evaluate consecutive frames and wasting computational resources. For instance, frames can be extracted every 8 seconds as the frame data to be detected.

[0077] S205. Obtain the true obstacle contained in the target true data and the predicted obstacle contained in the target prediction algorithm data.

[0078] Specifically, the target ground truth data packet contains obstacles and related information; obstacles obtained from the target ground truth data are recorded as ground truth obstacles. The target prediction algorithm data also contains obstacles and related information; obstacles obtained from the target prediction algorithm data are recorded as predicted obstacles. It should be noted that obstacle extraction is performed on both the aligned target ground truth data and target prediction data for each frame. Each frame may contain one or more obstacles.

[0079] S206. Perform initial screening of ground truth obstacles and predicted obstacles to obtain candidate matching ground truth obstacles and predicted obstacles.

[0080] Considering that a single frame may contain several obstacles, and given the limited computing resources in an autonomous driving system, efficient data processing is crucial. To avoid having to calculate the correctness of each predicted obstacle against every ground truth obstacle, a preliminary matching of ground truth and predicted obstacles within the same frame can be performed to obtain potentially matching ground truth and predicted obstacles. In this step, initial screening of ground truth and predicted obstacles yields matching ground truth and predicted obstacles, denoted as candidate matching ground truth and predicted obstacles. For example, suppose the obstacles extracted from the ground truth data corresponding to this frame are A, B, C, D, and E, and the predicted obstacles from this frame are A1, B1, C1, D1, and E1. After extracting the ground truth obstacles A, B, C, D, and E, and the predicted obstacles A1, B1, C1, D1, and E1, the two-dimensional Euclidean distance between each ground truth obstacle and each predicted obstacle is calculated. The predicted obstacle with the smallest Euclidean distance to each ground truth obstacle is selected as its candidate matching obstacle. This results in A and A1 being the same obstacle, B and B1 being the same obstacle, C and C1 being the same obstacle, D and D1 being the same obstacle, and E and E1 being the same obstacle. Subsequent evaluation only requires evaluating the predicted obstacles against the candidate matching ground truth obstacles.

[0081] As a specific implementation, an initial screening of ground truth obstacles and predicted obstacles is performed to obtain candidate matching ground truth obstacles and predicted obstacles, including:

[0082] a2) Calculate the Euclidean distance between the true obstacle and the predicted obstacle based on the two-dimensional evaluation calculation method.

[0083] Specifically, the two-dimensional Euclidean distance between the ground truth obstacle and the predicted obstacle is calculated. The specific calculation method for the Euclidean distance will not be elaborated here.

[0084] b2) If the Euclidean distance is greater than the first set Euclidean distance, then the true obstacle and the predicted obstacle are determined to be different obstacles.

[0085] The first set Euclidean distance is set based on empirical values ​​and is used to distinguish whether the ground truth obstacle and the predicted obstacle are the same obstacle. Specifically, for each obstacle in the same frame, if the Euclidean distance between the ground truth obstacle and the predicted obstacle is greater than the first set Euclidean distance, then it is determined that the ground truth obstacle and the predicted obstacle are not the same obstacle.

[0086] c2) If the Euclidean distance is less than or equal to the first set Euclidean distance, then the ground truth obstacle and the predicted obstacle are determined to be the same obstacle, and are used as candidate ground truth obstacles and predicted obstacles for matching.

[0087] Specifically, for each obstacle in the same frame, if the Euclidean distance between the ground truth obstacle and the predicted obstacle is less than or equal to the first set Euclidean distance, then the ground truth obstacle and the predicted obstacle are determined to be the same obstacle, and it is used as a candidate ground truth obstacle and predicted obstacle for matching.

[0088] The above technical solution details the steps of initial screening of ground truth obstacles and predicted obstacles to obtain candidate matching ground truth obstacles and predicted obstacles. When evaluating the prediction results, for the same frame, there is one predicted frame and one ground truth frame. If the predicted data is evaluated against all ground truth data, it will lead to a large amount of computation. The above technical solution uses Euclidean distance matching between predicted obstacles and ground truth obstacles. If the Euclidean distance is too far, it is considered that they are definitely not obstacles, thus eliminating a large part of the computation. If the Euclidean distance between the predicted data and the ground truth data is short and meets the standard, subsequent evaluation can be performed, which can reduce a lot of invalid calculations and improve the evaluation speed and computational efficiency.

[0089] S207. If the true value obstacle of the candidate matching is greater than the set obstacle threshold, the crossover ratio of the true value obstacle and the predicted obstacle is calculated based on the three-dimensional evaluation calculation method, and the initial evaluation index is determined based on the crossover ratio.

[0090] The system sets an obstacle threshold to categorize obstacles into two types: those larger than the threshold and those equal to or smaller. For obstacles larger than the threshold, a 3D evaluation method is used to assess whether the prediction algorithm's data is a positive detection. Specifically, the Intersection over Union (IOU) ratio between the ground truth and predicted obstacles is calculated using this method. This IOU is obtained by dividing the intersection area of ​​the predicted and ground truth obstacles by their union area. If the IOU is greater than a set IOU filtering threshold, it is considered a positive detection. If the IOU is less than or equal to the set IOU filtering threshold, it is considered a false detection. For example, larger obstacle types such as cars, large vehicles, and two-wheeled vehicles are calculated using IOU.

[0091] As a specific implementation method, the steps for determining the initial evaluation index based on the intersection-union ratio can be optimized, including:

[0092] a3) If the cross-union ratio is greater than the set cross-union ratio filtering threshold, it is determined to be a positive detection.

[0093] In this implementation, the intersection-over-union (IoU) filtering threshold changes in real-time as the distance between the obstacle and the visual perception system increases. It should be noted that due to the limitations of visual perception algorithms and the decreasing accuracy with increasing distance, this embodiment sets an evaluation filtering rule that changes with distance. This mechanism dynamically adjusts the evaluation parameters based on the distance between the obstacle and the visual perception system. Specifically, the IoU filtering threshold is set more strictly when the distance is short and more leniently when the distance is long. By setting the IoU filtering threshold to change in real-time with increasing distance, the accuracy and robustness of obstacle recognition are improved.

[0094] Specifically, if the cross-union ratio (CUB) between the true obstacle and the predicted obstacle is greater than the set CUB filtering threshold, then it is determined to be a positive detection.

[0095] b3) If the cross-union ratio is less than or equal to the set cross-union ratio filtering threshold, then the target prediction algorithm data is determined to be a false detection.

[0096] Specifically, if the cross-union ratio (CUB) between the true obstacle and the predicted obstacle is less than or equal to the set CUB filtering threshold, it is determined to be a false detection.

[0097] The above technical solution specifies the steps for determining the initial evaluation index based on the intersection-union ratio and sets evaluation filtering rules that change with distance. This mechanism can dynamically adjust the evaluation parameters according to the distance between the obstacle and the visual perception system, thereby improving the accuracy and robustness of obstacle recognition.

[0098] S208. If the true value obstacle of the candidate matching is less than or equal to the set obstacle threshold, the Euclidean distance between the true value obstacle and the predicted obstacle is calculated based on the two-dimensional evaluation calculation method, and the initial evaluation index is determined based on the Euclidean distance.

[0099] In this embodiment, for obstacles smaller than or equal to a set obstacle threshold, a two-dimensional evaluation calculation method is used to assess whether the prediction algorithm data is a positive detection. Specifically, the Euclidean distance between the ground truth obstacle and the predicted obstacle is calculated based on the two-dimensional evaluation calculation method; that is, the distance between corresponding points of the predicted obstacle and the ground truth obstacle. If the Euclidean distance is less than a second set Euclidean distance, it is considered a positive detection. If the Euclidean distance is greater than or equal to the second set Euclidean distance, it is considered a false detection. For example, for smaller obstacles such as traffic cones and water-filled barriers, the Euclidean distance is calculated to filter and determine positive and false detections. It should be noted that the aforementioned determined Euclidean distance can also be used directly here.

[0100] As a specific implementation method, the steps for determining the initial evaluation index based on Euclidean distance can be optimized, including:

[0101] a4) If the Euclidean distance is less than the second set Euclidean distance, then the target prediction algorithm data is determined to be positive.

[0102] In this embodiment, the second set Euclidean distance is smaller than the first set Euclidean distance, and the second set Euclidean value changes in real time as the distance between the obstacle and the visual perception system increases. It should be noted that due to the limitations of visual perception algorithms and the decreasing accuracy with increasing distance, this embodiment sets an evaluation filtering rule that changes with distance. This mechanism can dynamically adjust the evaluation parameters based on the distance between the obstacle and the visual perception system. That is, the first set Euclidean distance is more stringent when the distance is close, and more lenient when the distance is far. By setting the intersection-union ratio (IUGR) filtering threshold to change in real time with increasing distance, the accuracy and robustness of obstacle recognition are improved.

[0103] Specifically, if the Euclidean distance between the true obstacle and the predicted obstacle is less than the second set Euclidean distance, then it is determined to be a positive detection.

[0104] b4) If the Euclidean distance is greater than or equal to the second set Euclidean distance, then the target prediction algorithm data is determined to be a false detection.

[0105] Specifically, if the Euclidean distance between the true obstacle and the predicted obstacle is greater than or equal to the second set Euclidean distance, then it is determined to be a false detection.

[0106] The above technical solution specifies the steps for determining the initial evaluation index based on Euclidean distance and sets evaluation filtering rules that change with distance. This mechanism can dynamically adjust the evaluation parameters according to the distance between the obstacle and the visual perception system, thereby improving the accuracy and robustness of obstacle recognition.

[0107] Furthermore, the method can be further optimized by including:

[0108] a5) For each obstacle, if the target prediction algorithm data exists but the target ground truth data does not exist, then the target prediction algorithm data is determined to be a false detection.

[0109] In this embodiment, for all obstacles, if the target prediction algorithm data exists but the ground truth data does not exist, it is considered a false detection.

[0110] b5) If the target prediction algorithm data does not exist but the target true value data exists, then the target prediction algorithm data is determined to be a missed detection.

[0111] In this embodiment, for all obstacles, if the target prediction algorithm data does not exist but the ground truth data does, it is considered a missed detection.

[0112] The above technical solution increases the possibility of false detection or missed detection.

[0113] S209. Based on the initial evaluation indicators, determine the number of positive detections, false detections, and missed detections for each obstacle.

[0114] Specifically, the initial evaluation indicators include the results of each obstacle detection as positive (TP), false (FP), and missed (FN), and the positive detection data, number of false detections, and number of missed detections for each obstacle are statistically analyzed.

[0115] S210. Determine the accuracy of the visual perception algorithm based on the number of positive detections and the number of false detections.

[0116] Specifically, the accuracy rate for different types of obstacles is obtained by dividing the number of positive detections by the sum of the number of positive detections and the number of false detections.

[0117] S211. Determine the recall rate of the visual perception algorithm based on the number of positive detections and the number of negative detections.

[0118] Specifically, the recall rate for different types of obstacles is obtained by dividing the number of positive detections by the sum of the number of positive detections and the number of missed detections.

[0119] S212. When the detection result is positive, determine the various types of errors in the target prediction algorithm data and the target true value data.

[0120] In this embodiment, under the premise of positive detection, the confidence and performance of the perception algorithm data are measured by using the true value data as a benchmark and calculating the horizontal and vertical errors, velocity errors, heading angle errors, horizontal average relative errors, and vertical average relative errors frame by frame using the prediction algorithm data and the true value data in the same frame. The horizontal and vertical errors, velocity errors, heading angle errors, horizontal average relative errors, and vertical average relative errors are calculated frame by frame. These errors are the various types of errors between the target prediction algorithm data and the true value data.

[0121] S213. Precision, recall, and various types of errors are used as target evaluation metrics for visual perception algorithms.

[0122] Specifically, precision, recall, and various types of errors such as lateral and longitudinal errors, velocity errors, heading angle errors, lateral average relative errors, and longitudinal average relative errors are used as target evaluation indicators for visual perception algorithms.

[0123] The above technical solution specifies the steps for preprocessing ground truth data and prediction algorithm data, as well as the steps for determining the initial evaluation index of the visual perception algorithm based on the target ground truth data and target prediction algorithm data, combined with preset adaptive evaluation rules, and the steps for determining the target evaluation index of the visual perception algorithm based on the initial evaluation index. This achieves a more scientific and reasonable approach by combining three-dimensional and two-dimensional evaluation calculation methods. Furthermore, the filtering thresholds for three-dimensional and two-dimensional methods change in real time with increasing distance. Through efficient data processing methods, this provides assurance for downstream algorithms, meeting the high data requirements and safety requirements of autonomous driving systems.

[0124] Example 3

[0125] Figure 3 is a schematic diagram of an obstacle visual perception algorithm evaluation device provided in Embodiment 3 of this application. This device is applicable to the evaluation of obstacle visual perception algorithms. The obstacle visual perception algorithm evaluation device can be implemented in hardware and / or software and is generally integrated into an electronic device. As shown in Figure 3, the system includes: a data acquisition module 31, a preprocessing module 32, an initial evaluation module 33, and a target evaluation module 34; wherein,

[0126] The data acquisition module 31 is configured to acquire the three-dimensional point cloud data collected by the lidar and the prediction algorithm data obtained by the visual perception algorithm after processing the road data, and to use the three-dimensional point cloud data as the ground truth data.

[0127] Preprocessing module 32 is configured to preprocess the ground truth data and prediction algorithm data to obtain the corresponding target ground truth data and target prediction algorithm data. The preprocessing includes timestamp alignment, spatial alignment and keyframe extraction.

[0128] The initial evaluation module 33 is set to determine the initial evaluation index of the visual perception algorithm based on the target true value data and the target prediction algorithm data, combined with the preset adaptive evaluation rules. The adaptive evaluation rules are related to the size of the obstacle.

[0129] The target evaluation module 34 is set to determine the target evaluation index of the visual perception algorithm based on the initial evaluation index.

[0130] This application provides an obstacle visual perception algorithm evaluation device. Unlike related technologies where the timestamps of the visual perception algorithm and the LiDAR output differ, and where a single evaluation rule affects the evaluation of the prediction algorithm data, leading to inaccurate evaluation results, the above technical solution first aligns the timestamps and spatially of the ground truth data and the prediction algorithm data to ensure they match. Then, it performs keyframe extraction on the aligned ground truth data and prediction algorithm data, effectively reducing the amount of data that needs to be processed and stored, thus lowering computational and storage costs. Simultaneously, it combines three-dimensional and two-dimensional evaluation calculation methods, adaptively adjusting the evaluation rules to improve the accuracy and efficiency of visual perception algorithm evaluation.

[0131] Optionally, the preprocessing module 32 includes:

[0132] The time alignment unit is configured to perform timestamp alignment on the ground truth data and prediction algorithm data based on the time synchronization algorithm to obtain timestamp aligned ground truth data and prediction algorithm data.

[0133] The spatial alignment unit is configured to spatially align the timestamp-aligned ground truth data and prediction algorithm data to obtain spatially aligned ground truth data and prediction algorithm data.

[0134] The frame extraction processing unit is configured to extract keyframes from the timestamp-aligned and spatially aligned ground truth data and prediction algorithm data based on a preset time interval or an event-triggered mechanism, thereby obtaining the corresponding target ground truth data and target prediction algorithm data.

[0135] Optionally, the frame extraction processing unit includes:

[0136] Obtain the first timestamp of the ground value data for each frame and the second timestamp of the prediction algorithm data for each frame;

[0137] Determine if the first timestamp is equal to the second timestamp;

[0138] If the first timestamp and the second timestamp are equal, then align the truth data corresponding to the first timestamp with the prediction algorithm data corresponding to the second timestamp.

[0139] If the first timestamp and the second timestamp are not equal, then determine whether the difference between the first timestamp and the second timestamp is less than a set time threshold.

[0140] If the difference is less than the set time threshold, the true value data corresponding to the first timestamp will be aligned with the prediction algorithm data corresponding to the second timestamp.

[0141] If the difference is greater than or equal to the set time threshold, it is determined that the true value data corresponding to the first timestamp is not aligned with the prediction algorithm data corresponding to the second timestamp.

[0142] Optionally, the initial evaluation module 33 may include:

[0143] The obstacle acquisition unit is configured to acquire the true obstacles contained in the target true data and the predicted obstacles contained in the target prediction algorithm data;

[0144] The initial matching unit is set to perform initial screening of ground truth obstacles and predicted obstacles to obtain candidate ground truth obstacles and predicted obstacles for matching;

[0145] The three-dimensional evaluation unit is set to calculate the cross-union ratio (CUIR) between the true obstacle and the predicted obstacle based on the three-dimensional evaluation calculation method if the true obstacle is larger than the set obstacle threshold, and determine the initial evaluation index based on the CUIR.

[0146] The two-dimensional evaluation unit is set to calculate the Euclidean distance between the real obstacle and the predicted obstacle based on the two-dimensional evaluation calculation method if the candidate matching ground truth obstacle is less than or equal to the set obstacle threshold size, and determine the initial evaluation index based on the Euclidean distance.

[0147] Optionally, the initial matching unit is set to:

[0148] The Euclidean distance between the true obstacle and the predicted obstacle is calculated based on a two-dimensional evaluation and calculation method.

[0149] If the Euclidean distance is greater than the first set Euclidean distance, then the true obstacle and the predicted obstacle are determined to be different obstacles.

[0150] If the Euclidean distance is less than or equal to the first set Euclidean distance, then the ground truth obstacle and the predicted obstacle are determined to be the same obstacle, and are used as candidate ground truth obstacles and predicted obstacles for matching.

[0151] Optionally, the device further includes a false negative / missed detection determination module, configured as follows:

[0152] For each obstacle, if the target prediction algorithm data exists but the target ground truth data does not exist, then the target prediction algorithm data is determined to be a false detection.

[0153] If the target prediction algorithm data does not exist but the target ground truth data does exist, then the target prediction algorithm data is determined to be a missed detection.

[0154] Optionally, the three-dimensional evaluation unit is set as follows:

[0155] If the cross-union ratio is greater than the set cross-union ratio filtering threshold, it is determined to be a positive detection. The set cross-union ratio filtering threshold changes in real time as the distance between the obstacle and the visual perception system increases.

[0156] If the cross-union ratio (CUNR) is less than or equal to the set CUNR filtering threshold, the target prediction algorithm data is determined to be a false detection.

[0157] Optionally, the two-dimensional evaluation unit is set as follows:

[0158] If the Euclidean distance is less than the second set Euclidean distance, then the target prediction algorithm data is determined to be a positive detection.

[0159] If the Euclidean distance is greater than or equal to the second set Euclidean distance, then the target prediction algorithm data is determined to be a false detection;

[0160] The second set Euclidean distance is less than the first set Euclidean distance, and the second set Euclidean value changes in real time as the distance between the obstacle and the visual perception system increases.

[0161] Optionally, the target evaluation module 34 is set as follows:

[0162] Based on the initial evaluation indicators, determine the number of positive detections, false detections, and missed detections for each obstacle;

[0163] The accuracy of the visual perception algorithm is determined based on the number of positive detections and the number of false detections.

[0164] The recall rate of the visual perception algorithm is determined based on the number of positive detections and the number of negative detections.

[0165] When the detection result is a positive detection, determine the various types of errors in the target prediction algorithm data and the target true value data;

[0166] Precision, recall, and various types of errors are used as target evaluation metrics for visual perception algorithms.

[0167] Optionally, the device also includes an indicator display module, configured as follows:

[0168] The target evaluation index, determined based on all obstacles within the perception range, will be displayed on the human-computer interaction interface in a non-segmented manner.

[0169] And / or target evaluation indicators determined based on obstacles at set intervals within the perception range are displayed in segments on the human-computer interaction interface.

[0170] The obstacle visual perception algorithm evaluation device provided in this application embodiment can execute the obstacle visual perception algorithm evaluation method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of the execution method.

[0171] Example 4

[0172] Figure 4 is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of this application. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of this application described and / or claimed herein.

[0173] As shown in Figure 4, the electronic device 40 includes at least one processor 41 and a memory, such as a read-only memory (ROM) 42 or a random access memory (RAM) 43, communicatively connected to the at least one processor 41. The memory stores computer programs executable by the at least one processor. The processor 41 can perform various appropriate actions and processes based on the computer program stored in the ROM 42 or loaded into the RAM 43 from storage unit 48. The RAM 43 can also store various programs and data required for the operation of the electronic device 40. The processor 41, ROM 42, and RAM 43 are interconnected via a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0174] Multiple components in electronic device 40 are connected to I / O interface 45, including: input unit 46, such as keyboard, mouse, etc.; output unit 47, such as various types of monitors, speakers, etc.; storage unit 48, such as disk, optical disk, etc.; and communication unit 49, such as network card, modem, wireless transceiver, etc. Communication unit 49 allows electronic device 40 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0175] Processor 41 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 41 performs the various methods and processes described above, such as obstacle visual perception algorithm evaluation methods.

[0176] In some embodiments, the obstacle visual perception algorithm evaluation method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 40 via ROM 42 and / or communication unit 49. When the computer program is loaded into RAM 43 and executed by processor 41, one or more steps of the obstacle visual perception algorithm evaluation method described above may be performed. Alternatively, in other embodiments, processor 41 may be configured to perform the obstacle visual perception algorithm evaluation method by any other suitable means (e.g., by means of firmware).

[0177] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0178] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0179] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0180] To provide interaction with the user, the systems and technologies described herein can be implemented in a vehicle having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the vehicle. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0181] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0182] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0183] In implementing a computer program product, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0184] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved.

Claims

1. An evaluation method for obstacle visual perception algorithms, comprising: The three-dimensional point cloud data collected by the lidar and the prediction algorithm data obtained by processing the road data by the visual perception algorithm are acquired, and the three-dimensional point cloud data is used as the ground truth data. The ground value data and the prediction algorithm data are preprocessed to obtain the corresponding target ground value data and target prediction algorithm data. The preprocessing includes timestamp alignment, spatial alignment and keyframe extraction. Based on the target true value data and the target prediction algorithm data, and combined with the preset adaptive evaluation rules, the initial evaluation index of the visual perception algorithm is determined. The adaptive evaluation rules include a three-dimensional evaluation calculation method and a two-dimensional evaluation calculation method. Based on the initial evaluation index, the target evaluation index of the visual perception algorithm is determined.

2. The method according to claim 1, wherein, The preprocessing of the ground truth data and the prediction algorithm data to obtain corresponding target ground truth data and target prediction algorithm data includes: The time synchronization algorithm is used to align the timestamps of the truth data and the prediction algorithm data to obtain timestamp-aligned truth data and prediction algorithm data. Spatially align the timestamp-aligned ground truth data and prediction algorithm data to obtain spatially aligned ground truth data and prediction algorithm data; Based on a preset time interval or event triggering mechanism, keyframes are extracted from the timestamp-aligned and spatially aligned ground truth data and prediction algorithm data to obtain the corresponding target ground truth data and target prediction algorithm data.

3. The method according to claim 2, wherein, The step of aligning the ground truth data and the prediction algorithm data with timestamps based on a time synchronization algorithm to obtain timestamp-aligned ground truth data and prediction algorithm data includes: Obtain the first timestamp of the ground value data for each frame and the second timestamp of the prediction algorithm data for each frame; Determine whether the first timestamp is equal to the second timestamp; If the first timestamp is equal to the second timestamp, then the truth data corresponding to the first timestamp is aligned with the prediction algorithm data corresponding to the second timestamp. If the first timestamp and the second timestamp are not equal, then determine whether the difference between the first timestamp and the second timestamp is less than a set time threshold. If the difference is less than the set time threshold, then the true value data corresponding to the first timestamp is aligned with the prediction algorithm data corresponding to the second timestamp. If the difference is greater than or equal to the set time threshold, it is determined that the true value data corresponding to the first timestamp is not aligned with the prediction algorithm data corresponding to the second timestamp.

4. The method according to claim 1, wherein, The step of determining the initial evaluation index of the visual perception algorithm based on the target ground truth data and the target prediction algorithm data, combined with the preset adaptive evaluation rules, includes: Obtain the ground truth obstacles contained in the target ground truth data and the predicted obstacles contained in the target prediction algorithm data; The ground truth obstacles and the predicted obstacles are initially screened to obtain candidate matching ground truth obstacles and predicted obstacles; If the ground truth obstacle in the candidate matching is larger than the set obstacle threshold, the cross-union ratio between the ground truth obstacle and the predicted obstacle is calculated based on the three-dimensional evaluation calculation method, and the initial evaluation index is determined based on the cross-union ratio. If the candidate matching ground truth obstacle is less than or equal to the set obstacle threshold, the Euclidean distance between the ground truth obstacle and the predicted obstacle is calculated based on the two-dimensional evaluation calculation method, and the initial evaluation index is determined based on the Euclidean distance.

5. The method according to claim 4, wherein, The initial screening of the ground truth obstacles and the predicted obstacles to obtain candidate matching ground truth obstacles and predicted obstacles includes: The Euclidean distance between the true obstacle and the predicted obstacle is calculated based on a two-dimensional evaluation calculation method. If the Euclidean distance is greater than the first set Euclidean distance, then it is determined that the true obstacle and the predicted obstacle are not the same obstacle; If the Euclidean distance is less than or equal to the first set Euclidean distance, then the ground truth obstacle and the predicted obstacle are determined to be the same obstacle, and are used as candidate ground truth obstacles and predicted obstacles for matching.

6. The method according to claim 4, further comprising: For each obstacle, if the target prediction algorithm data exists but the target ground truth data does not exist, then the target prediction algorithm data is determined to be a false detection. If the target prediction algorithm data does not exist but the target ground truth data does exist, then the target prediction algorithm data is determined to be a missed detection.

7. The method according to claim 4, wherein, Determining the initial evaluation index based on the intersection-union ratio includes: If the cross-union ratio is greater than the set cross-union ratio filtering threshold, it is determined to be a positive detection. The set cross-union ratio filtering threshold changes in real time as the distance between the obstacle and the visual perception system increases. If the cross-union ratio is less than or equal to the set cross-union ratio filtering threshold, then the target prediction algorithm data is determined to be a false detection.

8. The method according to claim 5, wherein, The determination of the initial evaluation index based on the Euclidean distance includes: If the Euclidean distance is less than the second set Euclidean distance, then the target prediction algorithm data is determined to be a positive detection; If the Euclidean distance is greater than or equal to the second set Euclidean distance, then the target prediction algorithm data is determined to be a false detection; The second set Euclidean distance is less than the first set Euclidean distance, and the second set Euclidean value changes in real time as the distance between the obstacle and the visual perception system increases.

9. The method according to claim 1, wherein, The step of determining the target evaluation index of the visual perception algorithm based on the initial evaluation index includes: Based on the initial evaluation indicators, determine the number of positive detections, false detections, and missed detections for each obstacle; The accuracy of the visual perception algorithm is determined based on the number of positive detections and the number of false detections. The recall rate of the visual perception algorithm is determined based on the number of positive detections and the number of missed detections. When the detection result is positive, determine the various types of errors in the target prediction algorithm data and the target true value data; Precision, recall, and various types of errors are used as the target evaluation metrics for the visual perception algorithm.

10. The method according to claim 1, further comprising: The target evaluation index, determined based on all obstacles within the perception range, will be displayed on the human-computer interaction interface in a non-segmented manner. And / or target evaluation indicators determined based on obstacles at set intervals within the perception range are displayed in segments on the human-computer interaction interface.

11. An obstacle visual perception algorithm evaluation device, comprising: The data acquisition module is configured to acquire the three-dimensional point cloud data collected by the lidar and the prediction algorithm data obtained by the visual perception algorithm processing the road data, and to use the three-dimensional point cloud data as the ground truth data. The preprocessing module is configured to preprocess the ground value data and the prediction algorithm data to obtain the corresponding target ground value data and target prediction algorithm data. The preprocessing includes timestamp alignment, spatial alignment, and keyframe extraction. The initial evaluation module is configured to determine the initial evaluation index of the visual perception algorithm based on the target ground truth data and the target prediction algorithm data, combined with a preset adaptive evaluation rule, wherein the adaptive evaluation rule is related to the size of the obstacle; The target evaluation module is configured to determine the target evaluation index of the visual perception algorithm based on the initial evaluation index.

12. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the obstacle visual perception algorithm evaluation method according to any one of claims 1-10.

13. A computer-readable storage medium storing computer instructions for causing a processor to execute and implement the obstacle visual perception algorithm evaluation method according to any one of claims 1-10.

14. A computer program product comprising a computer program that, when executed by a processor, implements the obstacle visual perception algorithm evaluation method as described in any one of claims 1-10.

Images