Intelligent driving perception evaluation method and device and medium

Abstract

The invention relates to an intelligent driving perception evaluation method and device and a medium, and relates to the field of intelligent driving, and the method comprises the steps: collecting truth value data and to-be-evaluated data in a time space; based on a preset scene extraction rule, constructing the truth value data into a truth value scene library comprising a plurality of sub-scenes; performing spatial association on the truth value data and the to-be-evaluated data to obtain corresponding truth value output and test output in the sub-scene; and based on a preset evaluation index, comparing the truth value output with the test output to obtain an evaluation result corresponding to the evaluation index. The method is used for solving the problems of high test cost, long test period, low pertinence of evaluation indexes and the like when the driving sensing system is evaluated by using a traditional manual labeling mode.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving, and in particular to an intelligent driving perception evaluation method, device and medium. Background Art

[0002] Intelligent driving is an advanced and complex engineering project consisting of multiple sensors and subsystems such as lidar, millimeter-wave radar, camera, global positioning system (GPS), inertial measurement unit (IMU), etc. The key technologies of intelligent driving are perception, planning and control. The environmental perception part of the intelligent driving system usually needs to obtain a large amount of surrounding environment information to ensure that the vehicle has a correct understanding of the environment around the vehicle. Specifically, it includes: vehicle detection, pedestrian detection, lane line detection, traffic light recognition, traffic sign recognition, etc.

[0003] Intelligent driving needs to be verified through long-term and large-scale road tests. During the test and verification process, accurate perception of the surrounding environment is a prerequisite for good planning and control, so the testing of the perception system becomes extremely important. Intelligent driving needs to be verified through long-term and large-scale road tests, and the perception system also needs it. For a large amount of training set data, the traditional manual labeling method is inefficient. One obstacle takes 10 seconds, one frame (an average of 10 obstacles) takes 100 seconds, and 1 second (the average sensor acquisition frequency is 10hz) takes 1000 seconds. The data collected in one day requires about 3 people to process for a year. The manual labeling efficiency is 1:1000. The time and cost of manual labeling are huge, and manual labeling is affected by human factors and interfered by human errors, resulting in inaccurate labeling results. In general, the traditional manual labeling evaluation method has problems such as high testing cost, long testing cycle, and low pertinence of evaluation indicators. Summary of the invention

[0004] In response to the above technical problems, the present invention provides a method, device and medium for evaluating intelligent driving perception, which are used to solve the problems of high testing cost, long testing cycle and low pertinence of evaluation indicators when evaluating driving perception systems using traditional manual labeling methods.

[0005] A first aspect of the present invention provides an intelligent driving perception evaluation method, comprising: Collect true value data and data to be evaluated in the same space and time; Based on a preset scene extraction rule, constructing the true value data into a true value scene library including a plurality of sub-scenes; Spatially associating the true value data with the data to be evaluated to obtain the corresponding true value output and test output in the sub-scenario; And, based on a preset evaluation indicator, the true value output and the test output are compared to obtain an evaluation result corresponding to the evaluation indicator.

[0006] In a possible implementation, the truth data is constructed into a truth scene library including a plurality of sub-scenes based on a preset scene extraction rule, including: Acquire scene information corresponding to the true value data based on the true value data; Determine the target information true value under the scene information, wherein the target information true value includes the target type true value, the target position true value and the target behavior true value; Based on the scene information, the true value data is divided into a plurality of sub-scenes containing scene information and corresponding target information true values, and finally a true value scene library is formed.

[0007] In a possible implementation manner, determining the true value of the target information under the scene information includes: Determine a focus area around the vehicle in the scene information based on a preset range threshold; Determine a target type true value existing in the area of ​​interest and a target position true value corresponding to the target type true value; And, determine the target behavior truth value corresponding to the target type truth value.

[0008] In a possible implementation, determining the target behavior truth value corresponding to the target type truth value includes: Parsing the sensed lane line data in the true value data; When a sudden change occurs in the sensed lane line data, marking a lane change of the vehicle; According to the perceived lane line data and the target position true value, the target behavior true value of the target at the corresponding moment is identified in combination with the lane change mark of the vehicle.

[0009] In a possible implementation manner, spatially associating the true value data with the data to be evaluated to obtain the corresponding true value output and test output in the sub-scenario includes: Obtaining information of the target to be measured and the lane line to be measured in the data to be evaluated; Obtaining the target information true value and the lane line true value in the true value data; The target information to be tested and the true value of the target information, and the lane line to be tested and the true value of the lane line are matched at the same time and in the same space to obtain the true value output and test output in each sub-scenario.

[0010] In a possible implementation manner, the evaluation index at least includes the detection performance of the data to be evaluated, the geometric error of the detection target, and the distance accuracy of the detection target.

[0011] In a possible implementation, the method further includes: Based on the true value output and the test output, data errors in the data to be evaluated are marked.

[0012] In a possible implementation manner, marking the data error in the to-be-tested data based on the true value output and the test output includes: Traversing the truth value outputs, extracting all target information truth values ​​of the first target, and obtaining the start time truth value and the end time truth value of the first target appearing in the sub-scenario according to each target information truth value; Matching a test output corresponding to the true value output between the start time true value and the end time true value to obtain matching information of the first target; Aggregating the matching information to obtain an expected detection result of the first target; The problem type of the test output is marked based on the expected detection result.

[0013] In a possible implementation, marking the problem type of the test output based on the expected detection result includes: If there are multiple target types in the expected detection result of the first target, then mark the test output as having an ID jump; if the detection start time of the expected detection result of the first target is later than the true value of the start time, then mark the test output as having a late detection; If there is a missing segment in the expected detection result of the first target, the test output is marked as having missed detection.

[0014] In a possible implementation, the method further includes: The data to be evaluated in multiple versions are fed back and compared with the true value data to obtain evaluation results corresponding to the evaluation indicators under different versions.

[0015] A second aspect of the present invention provides an intelligent driving perception evaluation device, comprising: A true value sensor, at least for collecting true value data; The perception sensor to be tested is at least used to collect the data to be evaluated in the same time and space as the true value data; A scene extraction module, at least used to construct the true value data into a true value scene library including a plurality of sub-scenes based on a preset scene extraction rule; A target matching module, at least used to spatially associate the true value data with the data to be evaluated, and obtain the corresponding true value output and test output in the sub-scenario; And, an evaluation output module is at least used to compare the true value output and the test output based on a preset evaluation indicator to obtain an evaluation result corresponding to the evaluation indicator.

[0016] In a possible implementation manner, the device further includes: The error marking module is at least used to mark the data error in the data to be evaluated based on the true value output and the test output.

[0017] In a possible implementation manner, the device further includes: The algorithm re-injection module is at least used to re-inject and compare multiple versions of the to-be-evaluated data with the true value data to obtain evaluation results corresponding to the evaluation indicators under different versions.

[0018] According to a third aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a computer, the method according to the first aspect of the embodiment of the present invention is executed.

[0019] The present invention collects true value data and data to be evaluated in the same space and time, builds a true value scene library in combination with scene extraction rules, and performs spatial correlation and evaluation index comparison on the two types of data, thereby realizing automated and refined scene extraction and evaluation, which can significantly improve evaluation efficiency, reduce manual labeling costs, avoid human errors, and support comprehensive verification of the detection accuracy and reliability of the perception system in real scenarios. Compared with the existing technology, it has higher evaluation accuracy and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 It is a flowchart of a specific embodiment of the intelligent driving perception evaluation method of the present invention.

[0022] Figure 2 It is a schematic diagram of the installation of a specific embodiment of the truth value system of the present invention.

[0023] Figure 3 It is a schematic diagram of a specific embodiment of data processing of true value data and data to be tested of the present invention.

[0024] Figure 4 It is a schematic diagram of another specific embodiment of data processing of true value data and data to be tested of the present invention.

[0025] Figure 5 It is a schematic diagram of a specific embodiment of scene extraction of the present invention.

[0026] Figure 6 It is a flowchart of another specific embodiment of the intelligent driving perception evaluation method of the present invention.

[0027] Figure 7 It is a flowchart of another specific embodiment of the intelligent driving perception evaluation method of the present invention.

[0028] Figure 8 It is a flowchart of another specific embodiment of the intelligent driving perception evaluation method of the present invention.

[0029] Fig. 9 It is a structural schematic diagram of a specific embodiment of the intelligent driving perception evaluation device of the present invention.

[0030] Fig.10 It is a flow chart of a specific embodiment of the intelligent driving perception evaluation device of the present invention. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application. In addition, it should be understood that the specific implementation methods described herein are only used to illustrate and explain the present application, and are not used to limit the present application.

[0032] It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments of the present application. In addition, in the following embodiments, the description of each embodiment has its own emphasis, and for parts not described in detail in one embodiment, reference can be made to the relevant description of other embodiments.

[0033] The method steps described in the embodiments of the present invention may be executed in the order described in the specific implementation manner, or the execution order of each step may be adjusted according to actual needs and on the premise that the technical problem can be solved, which will not be listed here one by one.

[0034] Reference Figure 1 An embodiment of the present invention provides an intelligent driving perception evaluation method, which includes the following sub-steps.

[0035] S1, collect the true value data and the data to be evaluated in the same space and time. Specifically, the true value system and the perception system to be evaluated are integrated on the vehicle, and the lidar, camera, millimeter wave radar and other sensors in the true value system are used to collect high-precision dynamic and static environmental data (true value data). At the same time, the perception system to be evaluated collects the detection data in the driving environment (data to be evaluated). The two systems collect data at the same time and space, and synchronize the time through a high-precision timing system (such as RTK) to ensure that the true value data and the data to be evaluated accurately correspond to each other on the same time axis; during the collection process, the data is stored in the ros2bag format to provide basic data support for subsequent scene extraction and evaluation index calculation.

[0036] S2, based on the preset scene extraction rules, construct the true value data into a true value scene library including multiple sub-scenes. Specifically, based on the preset scene extraction rules, the collected true value data (such as the location information of the target object, lane line parameters, etc.) are automatically divided according to the scene characteristics (such as road type, lighting conditions), and a true value scene library is constructed. Specifically, the specific region of interest ROI (region of interest) is identified through the rules, and the key targets and lane information of each sub-scene are screened out, and the target behavior is classified in combination with the label, such as the movement status of vehicles, pedestrians, and cyclists in scenes such as straight roads, ramps, and intersections. Finally, the sorted true value data fragments are marked and stored in the true value scene library to provide high-accuracy comparison scenes for subsequent evaluations.

[0037] S3, spatially associate the true value data with the data to be evaluated, and obtain the corresponding true value output and test output in the sub-scenario. Specifically, first align the collected data of the two systems by timestamp, and then use a matching algorithm (such as the Hungarian algorithm) to associate the true value data and the data to be evaluated one-to-one in the same time frame based on spatial information such as the target position. Through this association process, it can be ensured that the true value output and the test output in each sub-scenario match, so as to obtain perception results with corresponding relationships in different time segments and spatial positions, laying the foundation for the comparative analysis of subsequent evaluation indicators.

[0038] S4, based on the preset evaluation indicators, compare the true value output and the test output to obtain the evaluation results corresponding to the evaluation indicators. Specifically, based on the preset evaluation indicators (such as detection performance, geometric error, distance accuracy, etc.), the true value output and test output associated in step S3 are compared item by item to evaluate the accuracy and reliability of the system to be evaluated. Specifically, the performance of the system to be evaluated in terms of detection rate, false detection and missed detection, position accuracy, target type recognition accuracy, etc. is calculated according to the evaluation indicators. The final output includes the quantitative results of each evaluation indicator, and the performance of the system to be evaluated in different scenarios is displayed with systematic evaluation data.

[0039] Exemplarily, during the implementation of step S2, refer to Figure 2 First, the driving perception system and the truth value system are integrated on a vehicle. The vehicle-mounted end integrates a multi-sensor fusion hardware platform to realize the data collection function of the vehicle-mounted end.

[0040] Next, the true value sensor and the perception sensor to be tested are integrated into a vehicle to collect scenes in the same time and space. The content of the collected data can be in the form of ros2bag. After the hardware platform of the true value system is deployed, the relevant sensors are calibrated and the vehicle-side acquisition software is deployed. When collecting true value data on the vehicle, one-click recording and sub-packaging are performed. The main laser radar is triggered and synchronized by RTK PPS. RTK obtains satellite time and synchronizes the industrial computer system on the acquisition end. The industrial computer uses the system time to synchronize the time of each device and write the bag data to the disk. It uses the NTP protocol or CAN protocol to send timestamps to the outside world to facilitate time synchronization of other devices. Among them, there are six cameras, including two telephoto and four wide-angle, covering the surrounding 360° environment. The cameras are synchronized by hardware triggering.

[0041] Furthermore, the truth value system has built a high-performance offline server to realize the offline data processing of the truth value. The vehicle-mounted end collects multiple .bag files, which are divided by time, carry GPS time, and contain the original data of the truth value sensor. The truth value post-processing algorithm integrates the latest lidar intelligent perception algorithm and post-processing fusion algorithm, which can perform time synchronization fusion, intelligent perception and other processing on the sensor data collected by the reference sensor system, and finally provide reliable and accurate truth value information. The post-processing algorithm needs to install the graphics card driver, Cuda, Cudnn and environmental dependencies, and insert the USB Key on the post-processing server for verification to ensure the success of the reference authorization.

[0042] During the true value data collection process, start running the post-processing algorithm and start the true value data collection command.

[0043] Under the default configuration, each running module will output a progress bar to reflect the running progress of the module. The final generated true value data is stored in the result folder of the data directory. Later, you can get the target true value data GT.csv output in the local coordinate system and the fitting parameters lane.csv of the true value lane line from the result folder with GPS time.

[0044] It should be pointed out that since the two systems collect their own data respectively, both parties use their own GPS time, and the collection frequency of the two systems is not necessarily the same. First, the data of the two systems need to be time-aligned. In view of the difference in data frequency, a linear interpolation alignment method is used to interpolate the true value data based on the data to be tested. For example, the frequency of the true value data is 10hz, the frequency of the data to be tested is 20hz, and the period of the data to be tested is 50ms. Then, the true value data needs to be aligned to each frame of the data to be tested, and the time correspondence is performed according to the reference time and the allowable error. The corresponding data is interpolated and calculated according to the time difference. The position and heading angle are interpolated and calculated, and the speed change can be ignored. After processing, the frame number of the true value data can be re-marked according to the reference time.

[0045] Specifically, refer to Figure 3 In the data processing process, the true value data and the data to be evaluated are interpolated and aligned: Since the acquisition frequency of the true value system and the system to be evaluated is different, the timestamps of the two (true value data time, test data time) are first matched according to the time axis, and the low-frequency true value data is aligned to each frame of the high-frequency test data using the linear interpolation method. Specifically, based on the time point of the test data, the interpolation value of two adjacent frames of true value data is calculated to obtain the true value position, heading and other information at each time point of the test data, thereby forming accurately corresponding true value data and test data, providing highly consistent data support for subsequent spatial association and evaluation analysis.

[0046] Further, see Figure 4 In the data processing flow, the interpolation calculation method is used to align the true value data to the data to be evaluated: for the different sampling frequencies of the two systems, the corresponding true value data is inferred at each time point of the data to be evaluated through linear interpolation in the figure. The specific steps are to calculate the interpolation values ​​of the true value position, heading angle and other parameters at the time point of the data to be evaluated based on the adjacent timestamps before and after the true value data, so that there are accurate corresponding true value data in all the data frames to be evaluated, thereby ensuring the time consistency of the data and providing a reliable basis for subsequent accurate spatial association and indicator evaluation.

[0047] Further, as an embodiment of the present invention, refer to Figure 5 , step S2, based on the preset scene extraction rules, constructing the true value data into a true value scene library including multiple sub-scenes, including the following sub-steps.

[0048] S21, based on the true value data, the scene information corresponding to the true value data is obtained. Specifically, based on the time, location, lighting, weather, road type and other information recorded in the true value data, the scene feature information of each segment of the true value data is identified and extracted. Specifically, by analyzing the GPS location information in the true value data in combination with the map data, it is possible to automatically identify that the current data belongs to different road scenes such as highways, urban areas, ramps or intersections; at the same time, the lighting conditions (such as daytime and nighttime) are inferred in combination with the timestamp, and the weather information at that time (such as sunny days and rainy days) is obtained through an external interface or sensor data. The extraction of these scene information provides basic support for the subsequent division of target information and the construction of a true value scene library.

[0049] S22, determine the target information true value under the scene information, the target information true value includes the target type true value, the target position true value and the target behavior true value. Specifically, the target information true value in the scene is extracted and determined from the true value data, mainly including the target type, target position and target behavior. Specifically, by analyzing the data of sensors such as lidar and cameras in the true value system, the type of each target (such as vehicle, pedestrian, bicycle, etc.) is identified, and its precise spatial position (coordinate data) is obtained, and then the motion trajectory of the target is calculated by combining the data of multiple time frames, and its behavior in the scene (such as going straight, changing lanes, stopping) is identified. After being labeled and sorted, this information forms the precise target information true value in each sub-scene, providing a comparison benchmark for target detection and behavior judgment in subsequent evaluations.

[0050] S23, based on the scene information, the truth data is divided into a plurality of sub-scenes containing scene information and corresponding target information truth values, and finally a truth scene library is formed. Specifically, based on the scene information extracted in step S21 and the target information truth value determined in step S22, the truth data is divided into a plurality of sub-scenes and integrated into a truth scene library. Specifically, the truth data is classified according to the scene information (such as road type, lighting, weather, etc.), and then the corresponding target type, position, behavior and other information truth values ​​are associated with each classified scene to form a complete scene-target comparison relationship. Finally, these sub-scenes containing scene information and target information truth values ​​are structured and stored as a truth scene library, providing real data support for the evaluation of perception systems under different driving environments.

[0051] Furthermore, as an embodiment of the present invention, in step S22, determining the true value of the target information under the scene information includes: determining the focus area around the vehicle in the scene information based on a preset range threshold; determining the target type true value existing in the focus area and the target position true value corresponding to the target type true value; determining the target behavior true value corresponding to the target type true value.

[0052] Specifically, the focus area around the vehicle is delineated by a preset range threshold, so as to focus on the key target information in the area. First, the focus area is defined according to the preset lateral and longitudinal distance limits (for example, 200 meters ahead and lateral adjacent lanes) to filter out targets that have no direct impact on the vehicle. Next, the target type truth value (such as vehicles, pedestrians, etc.) in the focus area is identified, and its precise spatial position truth value is recorded. Finally, by analyzing the position information of these targets in different time frames, the dynamic behavior truth value of the target (such as straight, overtaking, changing lanes) is extracted, so as to comprehensively determine the target information truth value in the focus area, providing basic data for accurate scene construction and evaluation.

[0053] Furthermore, as an embodiment of the present invention, determining the target behavior truth value corresponding to the target type truth value includes: parsing the perceived lane line data in the truth value data; marking the lane change of the self-vehicle when a sudden change occurs in the perceived lane line data; and identifying the target behavior truth value of the target at the corresponding moment based on the perceived lane line data and the target position truth value in combination with the lane change mark of the self-vehicle.

[0054] Specifically, when determining the target's behavioral truth value, the lane change mark of the ego vehicle and the target's position information truth value are combined to specifically analyze the relative position changes between the target and the ego vehicle in the lane line. First, when the ego vehicle has a lane change mark, observe whether the target approaches or moves away from the lane line in the direction of the ego vehicle's lane change; if the target gradually approaches and maintains the position of the adjacent lane, the target is identified as a "following" behavior; if the target overtakes the ego vehicle from the adjacent lane during the ego vehicle's lane change, the target is marked as an "overtaking" behavior. For situations where there is no lane change, the "maintaining" behavior is judged by the stability of the target's relative position; when the target gradually crosses the lane line and deviates from the original lane position, the target is marked as a "lane change" behavior. Through the specific analysis of these position and behavior changes, the target's behavioral truth value in different scenarios can be accurately calibrated.

[0055] For example, refer to Figure 5 Taking the extraction of a simple target scene within the range of three straight roads as an example, the scene extraction rules are constructed as follows: Initial screening of the ROI (region of interest), horizontally considering the left adjacent lane, the own vehicle's driving lane, and the right adjacent lane, and vertically considering 200m in front and 50m in the back; Divide the targets in the ROI area into lanes, with lane_id of the vehicle lane = 0, lane_id of the left neighbor = 1, and lane_id of the right neighbor = -1. Focus on the more critical targets in the three lanes, namely the closest target in front, the closest target in the back, and the second closest targets in front and behind. Use the variable obj_loc to represent the location of the target, and assign values ​​to the variable obj_loc. For example, assign values ​​1 to 12 to the 12 positions closest to the vehicle. According to the motion parameters of the ego vehicle, first combine the lane line changes to determine whether the ego vehicle has lane change behavior and mark it. The judgment method is as follows: 1) parse the data in the perceived lane line; 2) use the variable y_coeffs.0 output by the perceived lane line, and the mutation value of this variable exceeding 3 is recorded as a lane change of the ego vehicle; 3) output the moment before the mutation as the crossing moment of the ego vehicle's lane change; then select the 6 targets closest to the ego vehicle in the three lanes, identify each target, and identify the special actions of the target at certain moments according to the changes in lane_id, obj_loc and motion parameters, and combined with the above-mentioned mark of whether the ego vehicle has changed lanes, such as "the front car changes lanes to the left", traverse all targets, classify and organize them according to the target behavior, and extract the sub-scene results.

[0056] The following is an example of sub-scene extraction results: Scenario Description illumination weather Road Section Target Type Target location Target Behavior high speed daytime sunny Main Road car straight ahead Keep Urban area dusk cloudy day intersection SUV Directly behind Cut left Night rain Ramp truck front left Cut right Snowy Days bus front right Cut in pedestrian Left rear Beyond the car Cyclist Right rear Being surpassed by the car Cone barrel The car cuts out to the left and overtakes the target The car cuts out to the right and overtakes the target ……

[0057] Furthermore, as an implementation mode of the present invention, step S3 spatially associates the true value data and the data to be evaluated to obtain the corresponding true value output and test output under the sub-scenario, including: obtaining the target information to be tested and the lane line to be tested in the data to be evaluated; obtaining the target information true value and the lane line true value in the true value data; matching the target information to be tested with the target information true value, and the lane line to be tested with the lane line true value at the same time and in the same space to obtain the true value output and test output under each sub-scenario.

[0058] Specifically, in order to obtain the true output and test output in each sub-scenario, the target information and lane line data in the data to be evaluated are first extracted as the target information to be tested and the lane line to be tested; at the same time, the target information truth and lane line truth are extracted from the true data. Then, timestamp alignment is used to ensure that the test data and the true data are at the same time point, and then the Hungarian algorithm is used to match the target to be tested with the true target, and the lane line to be tested with the true lane line in the same space to establish the optimal one-to-one association between them. Finally, this process generates the true output and test output in each sub-scenario, providing a comparison basis for subsequent precise evaluation.

[0059] Specifically, the Hungarian algorithm is used to achieve the optimal one-to-one matching between the target to be tested and the true target, and between the lane line to be tested and the true lane line in the same space, including: firstly, calculating the spatial distance or position difference between each target to be tested and the true target, and between each lane line to be tested and the true lane line, and constructing an association cost matrix. Based on the cost matrix, the Hungarian algorithm establishes the optimal matching relationship between the target to be tested and the true target, and between the lane line to be tested and the true lane line by minimizing the overall matching cost, thereby ensuring that each target or lane line obtains a unique and closest corresponding item in the same space, and achieving accurate one-to-one association.

[0060] Exemplarily, an example of output interface data after association matching is performed is shown in the following table, where gt represents true value output and dut represents test output:

[0061] Further, as an implementation mode of the present invention, the evaluation index in step S4 at least includes the detection performance of the data to be evaluated, the geometric error of the detection target, and the distance accuracy of the detection target.

[0062] Specifically, based on the output interface content above, the matched GT and DUT information has been obtained. The next step is to use GT to compare and output the DUT evaluation results, mainly evaluating the detection performance of DUT (such as precision and recall rate, type accuracy), the geometric error of the detection target (such as vehicle length, width, and height errors), and the distance accuracy of the detection target (such as horizontal and vertical distance errors).

[0063] The detection performance indicators are calculated by comparing the true value output with the test output, including key indicators such as detection accuracy, recall rate and false positive rate. First, identify the number of correctly detected targets in the test output (that is, the number of targets that successfully match the true value output and the test output), and calculate its proportion to the total number of true value targets to obtain accuracy and recall rate; for detection items that do not have corresponding true value outputs in the test output, record them as false positives and calculate the false positive rate. In addition, by analyzing the type and quantity matching of the targets, the detection performance is further refined to ensure a comprehensive assessment of the recognition ability and missed and false positives of the system under test.

[0064] The geometric error of the detected target is obtained by comparing the target size (such as length, width, and height) in the true value output and the test output. Specifically, for each successfully matched target, the difference between the geometric size of the test output and the true value output size is calculated, and the errors in the three dimensions of length, width, and height are quantified and recorded separately. Then, the geometric errors of all matched targets are statistically analyzed (such as average error and maximum error) to evaluate the accuracy of the system under test in target size detection, thereby reflecting its geometric accuracy in different scenarios.

[0065] The distance accuracy of the detected target is obtained by calculating the difference between the test output and the true output at the target position. Specifically, for each successfully matched target, the deviation between the test output and the true output in the lateral and longitudinal distances is compared to obtain the lateral and longitudinal error values. Then, these errors are statistically analyzed (such as the mean error, standard deviation, and maximum error) to quantify the accuracy of the system under test in detecting the target position, thereby evaluating its distance measurement accuracy in different sub-scenarios.

[0066] Further, as an embodiment of the present invention, refer to Figure 6 , an embodiment of the present invention provides an intelligent driving perception evaluation method, which also includes the following sub-steps.

[0067] S5: Marking data errors in the data to be evaluated based on the true value output and the test output.

[0068] Specifically, in addition to calculating the precision-recall rate, type accuracy and other information based on the target matching results, the obvious errors in the test output (i.e., the data to be tested) will also be marked, that is, by constructing rules to extract the moments when the dut has obvious errors, such as target over-segmentation, target missed detection, and target position deviation.

[0069] Further, as an embodiment of the present invention, refer to Figure 7 , step S5, marking the data error in the data to be evaluated based on the true value output and the test output, including the following sub-steps.

[0070] S51, traverse the truth value output, extract all the target information truth values ​​of the first target, and obtain the start time truth value and end time truth value of the first target appearing in the sub-scene according to each of the target information truth values. Specifically, traverse each target in the truth value output, extract all the target information truth values ​​of the target (referred to as the first target), including its type, position, size, behavior and other detailed data. Then, based on these target information truth values, locate the start time truth value of the target detected by the truth value system for the first time and the end time truth value of the last time detected in the current sub-scene. By determining these time ranges, the actual appearance interval of the first target in the scene can be clarified, providing an accurate time basis for matching with the test output and error analysis in subsequent steps.

[0071] S52, match the test output corresponding to the true value output between the start time true value and the end time true value, and obtain the matching information of the first target; specifically, between the start time true value and the end time true value of the first target determined in step S41, filter out the detection data corresponding to the time interval from the test output. Specifically, compare the target information of the test output frame by frame within the time period, and find the closest match to the first target of the true value output in terms of position, type and other features, so as to generate the matching information of the first target in the test output. This process ensures that the detection results in the test output are correctly associated with the target of the true value output within the corresponding time interval, laying the foundation for subsequent error analysis and problem marking.

[0072] S53, aggregate the matching information to obtain the expected detection result of the first target; specifically, aggregate all the matching information of the first target obtained in step S42 to form the expected detection result of the target. Specifically, by summarizing the multi-frame matching data of the first target in the test output, calculate the detection performance of the target in the entire time interval, such as the duration of the detection, recognition accuracy, location accuracy, etc. At the same time, the overall performance of the target in the test output is counted, such as whether there is a missed detection, false detection, or ID jump, etc., to obtain a comprehensive expected detection result, which provides a clear comparison standard for the subsequent marking of the problem type in the test output.

[0073] S54, marking the problem type of the test output based on the expected detection result; specifically, according to the expected detection result generated in step S43, the detection items in the test output that do not meet the expectations are marked with problem types. Specifically, the test output is compared item by item against the standards such as target position, size, and behavior in the expected detection result; if the detection item has deviations such as missed detection, false detection, or recognition delay, it is marked as the corresponding problem type (such as missed detection, false detection, or late detection), and the abnormal time period and deviation value are recorded. In addition, for detection items with large ID jumps or positioning errors, problem marking is also performed to systematically reflect the quality of each detection result in the data to be evaluated, and provide data support for subsequent evaluation and improvement.

[0074] In addition, based on the true value output and the corresponding test output, the relevant evaluation indicators of the data to be tested can be further developed, such as the recall rate, distance error, speed error, heading angle error of dynamic targets such as vehicles, pedestrians, and cyclists, the lateral error, type, color, curvature, etc. of lane lines.

[0075] Further, as an embodiment of the present invention, step S54, marking the problem type of the test output based on the expected detection result, includes: if there are multiple target types in the expected detection result of the first target, marking the test output as having an ID jump; if the detection start time of the expected detection result of the first target is later than the true value of the start time, marking the test output as having a late detection; if there is a missing fragment in the expected detection result of the first target, marking the test output as having a missed detection.

[0076] Specifically, all gt targets are screened according to the gt and dut matching timestamp files, and the corresponding start and end times are output for each gt_id according to the target position information. This process requires simple filtering of jump frames; for each fragment with different positions of gt_id, the matching dut_id is obtained. If dut_id has multiple different values ​​in a certain time segment corresponding to the same gt_id, it is marked as an ID jump of the dut target; if there is a part of the continuous time in which the dut target is not matched successfully in a certain time segment corresponding to the same gt_id, it is marked as a missed detection of the dut target, and the missed detection timestamp fragment is recorded; if for the first appearance timestamp of a certain gt_id, the dut target cannot be matched successfully at the start timestamp, it is marked as a late detection of the dut target, and the late detection time is recorded, which is the difference between the timestamp of the first detection frame of the dut target and the timestamp of the first frame of the gt target.

[0077] For example, take the example of late missed detection of a marked dut target.

[0078] First, traverse the first target 14 in the true value output, and the traversal results are as follows: gt_timestamp gt_id …… position 20 14 …… 2 …… 14 …… …… 100 14 …… 2 101 14 …… 1 …… 14 …… …… 200 14 …… 1 …… 14 …… 5 The table takes the first target (gt_id=14) as an example. Its position information changes in different timestamps (20, 100, 101, 200, etc.), indicating that the target moves from position 2 to position 1 in the scene. This table shows the timestamp information (gt_timestamp) of each frame of the first target in the test output, the specific position information (position) and other information.

[0079] Next, get the start time and end time of the first target in the true output and divide it into different time segments: gt_id gt_start_timestamp gt_end_timestamp position 14 20 100 2 14 101 200 1 For gt_id=14, the first segment is from timestamp 20 to 100, position 2, while the second segment is from 101 to 200, position 1.

[0080] Then, match the matching information corresponding to all target fragments of the first target in the test output: In the table, in the first time segment (20 to 100, position 2) of gt_id=14, the test output detected dut_id=1, 2, and 3 respectively, and the time periods corresponding to each dut_id are (40 to 60), (61 to 70), and (71 to 100); for the second time segment (101 to 200, position 1) of gt_id=14, the test output detected dut_id=4 and 5 respectively.

[0081] Then, combine the true value output and the test output into the expected output result: gt_id gt_start_timestamp gt_end_timestamp dut_id duration Target Type 14 20 39 - 19 Small car 14 40 100 1、2、3 60 Small car 14 101 109 - 8 Small car 14 110 190 4、5 80 Small car 14 191 200 - 9 Small car In the table, for the first segment (20 to 100) of gt_id=14, the test output did not detect the target in part of the time (20 to 39), so it was marked as undetected, and the time period (duration) was 19; for the time period 40 to 100, the test output detected dut_id=1, 2, 3. Similarly, the second segment is from 101 to 200, the test output did not detect it in the time period 101 to 109, and detected dut_id=4, 5 from 110 to 190.

[0082] Finally, label the question type based on the expected output: In the table, the test output failed to detect in time at the start time (20) of the true value output, and did not start detection until 40, with a time difference of 19, which is marked as late detection; in the time period from 40 to 100, the test output detected dut_id=1, 2, and 3 successively, indicating that the test output did not identify the same target consistently, which is marked as ID jump; in the time period from 101 to 109, the test output did not detect the first target at all, which is marked as missed detection. The marking method for other time periods is similar to the above, which will not be repeated here.

[0083] Through the above-mentioned marking process, the detection accuracy of the system under test of the test output pair in different time periods can be systematically analyzed, specific detection problems can be identified and recorded, and data support can be provided for performance improvement of the system under test.

[0084] Further, as an embodiment of the present invention, refer to Figure 8 , an embodiment of the present invention provides an intelligent driving perception evaluation method, which also includes the following sub-steps.

[0085] S6, re-injecting and comparing the multiple versions of the to-be-evaluated data with the true value data to obtain evaluation results corresponding to the evaluation indicators under different versions.

[0086] Specifically, by using the refill feature of the autonomous driving system (ROS system), the algorithm KPI indicators of different versions of perception software in the same scenario can be verified. For example, to compare the performance of the perception system for dynamic target ranging in the scenario of "high speed during the day and cutout of the target ahead" between V1.0 and V2.0, first collect the gt and dut data of V1.0 in the same vehicle and time and space, extract the scene fragments with timestamp information based on the gt data for backup, and when refilling the rosbag of V2.0 software, the timestamp of the refilled rosbag should be consistent with the timestamp of the rosbag of V1.0. The rosbag of V1.0 and the refilled V2.0 can be aligned with the data in the true value scene fragment and the ranging error of the perception target in the fragment can be calculated respectively. The target ranging performance of the two versions of software in this scenario can be evaluated through some statistical values ​​(such as mean, 90th percentile, 95th percentile, etc.). If multiple versions of rosbag are refilled later, they can be compared with the true value data multiple times to achieve the reuse of the collected data, drive the collection end to expand the collection of intelligent driving scenes and the rapid iteration of the perception algorithm.

[0087] An embodiment of the present invention also discloses an intelligent driving perception evaluation device.

[0088] Reference Fig. 9 A smart driving perception evaluation device includes a true value sensor 1, a perception sensor to be tested 2, a scene extraction module 3, an automatic matching module 4, an evaluation output module 5, an error marking module 6 and an algorithm re-injection module 7.

[0089] The true value sensor 1 is at least used to collect true value data. Specifically, a variety of high-precision sensor devices are integrated on the vehicle, including laser radar, millimeter wave radar, high-resolution camera, RTK (real-time dynamic differential positioning), etc., to accurately collect dynamic and static information in the environment. The laser radar provides high-precision 3D spatial data, the millimeter wave radar obtains the distance and speed information of obstacles, the high-resolution camera captures scene images to identify roads and target objects, and RTK provides accurate geographic location information through satellite positioning. These sensors work together to generate high-precision true value data, and synchronize the timing to ensure that the data of each sensor is collected under the same time reference, thereby providing a reliable data basis for subsequent scene extraction, target matching and evaluation.

[0090] The perception sensor 2 to be tested is at least used to collect the data to be evaluated in the same time and space as the true value data. The sensors such as the laser radar, millimeter wave radar and camera in the system to be tested are installed on the vehicle, and the environmental data such as the position information, speed and type of the target object are obtained in real time, and the accurate time synchronization mechanism (such as GPS or RTK timing) is used to ensure the consistency of the collection time and space with the true value data, so as to form complete data to be evaluated, and provide standardized basic data for subsequent evaluation and comparison.

[0091] The scene extraction module 3 is at least used to construct the true value data into a true value scene library including multiple sub-scenes based on the preset scene extraction rules. Specifically, the scene extraction module 3 first parses the key parameters such as location information, lighting conditions, weather type, road type, etc. in the true value data, and defines sub-scenes according to these scene characteristics, such as different types of driving environments such as highways, urban roads, and intersections. Then, the target objects (such as vehicles, pedestrians, etc.) in each sub-scene are marked, their behavioral characteristics (such as overtaking, following, etc.) are recorded, and these sub-scene data with scene information and target characteristics are stored in the true value scene library, so as to accurately evaluate the performance of the perception system in different scenarios.

[0092] The target matching module 4 is at least used to spatially associate the true value data with the data to be evaluated, and obtain the corresponding true value output and test output in the sub-scenario. Specifically, the target matching module 4 first aligns the timestamps of the true value data and the data to be evaluated so that the two are compared under the same time reference; then, a matching algorithm (such as the Hungarian algorithm) is used to calculate the spatial distance and position difference between each target to be tested and the true value target, and an association cost matrix is ​​constructed. By minimizing the matching cost of the matrix, the optimal one-to-one matching relationship in each sub-scenario is generated, and the corresponding true value output and test output are obtained, providing basic data for subsequent evaluation index calculation and error analysis.

[0093] The evaluation output module 5 is at least used to compare the true value output and the test output based on preset evaluation indicators to obtain the evaluation results corresponding to the evaluation indicators. Specifically, the evaluation output module 5 compares the true value output and the test output in detail based on preset evaluation indicators (such as detection performance, geometric error and distance accuracy) to calculate and generate evaluation results. Specifically, the evaluation output module first analyzes whether the detection target in the test output is consistent with the true value output, and calculates detection performance indicators such as detection accuracy, recall rate and false detection rate; then, compares the size, position deviation and other information of each matching target, and quantifies the geometric error and distance accuracy. Finally, the evaluation results are sorted and output to provide comprehensive and quantitative data support for the performance of the perception system in different scenarios.

[0094] The error marking module 6 is at least used to mark the data errors in the data to be evaluated based on the true value output and the test output. Specifically, the error marking module 6 automatically marks the errors in the data to be evaluated based on the comparison results of the true value output and the test output. First, the detection performance of each target in the test output is analyzed, and compared with the true value output, the detection results that fail to match are marked as missed detections, and the targets with offset detection results are marked as position errors. If the target ID is inconsistent, it is marked as an ID jump. In addition, the target with delayed detection is marked as late detection, and the timestamp and position difference of the deviation are recorded. Finally, all abnormal marks are attached to the data to be evaluated to provide detailed data basis for error analysis and performance improvement of the perception system.

[0095] The algorithm re-injection module 7 is at least used to re-inject and compare multiple versions of the data to be evaluated with the true value data, and obtain the evaluation results corresponding to the evaluation indicators under different versions. Specifically, the algorithm re-injection module 7 replays different versions of the data to be evaluated into the system, and compares them frame by frame with the true value data under the same scenario to obtain the evaluation indicator results of each version. First, the algorithm re-injection module loads different versions of the perception system data one by one through the re-injection function of ROS to ensure the consistency of time and space with the true value data; then, each version of the data is evaluated item by item according to the preset evaluation indicators (such as detection accuracy, false detection rate, response delay, etc.), and the evaluation results of each version are recorded. Finally, the module outputs a performance comparison of multiple versions to provide reliable quantitative data support for the optimization and iteration of the perception algorithm.

[0096] For example, refer to Fig.10 , demonstrating the overall process of the intelligent driving perception evaluation system, the functions and roles of each module from data collection to algorithm evaluation.

[0097] First, the true value sensor and the perception sensor to be tested are integrated and calibrated through the vehicle to ensure that both collect data in the same time and space. The true value sensor is responsible for collecting high-precision environmental data, while the perception system to be tested collects the data to be evaluated.

[0098] After calibration, the system starts to collect true value data and perception data at the same time, and stores these data in different data pools, namely "true value data automatic labeling" and "perception data algorithm feedback".

[0099] Next, data interpolation is performed: the collected data is aligned in time and space to ensure that the true data and the perceived data are compared under the same benchmark.

[0100] Scenario extraction: According to preset rules, the true value data is constructed into multiple sub-scenarios, and a true value scenario library is built to evaluate the perception system in different driving environments.

[0101] Target matching: Use an algorithm to spatially associate the true value data with the perception data to be tested to obtain the true value output and test output in each sub-scene.

[0102] Evaluation indicator development: Based on preset evaluation indicators (such as detection performance, geometric error, etc.), develop evaluation methods and calculate various evaluation results.

[0103] Algorithm evaluation and iteration: Evaluation results are used to drive algorithm iteration, and combined with data from actual driving scenarios, the perception algorithm is updated and optimized to have higher accuracy and stability in different scenarios.

[0104] Ultimately, the evaluation data and results generated through this process provide systematic and precise support for the performance evaluation and optimization of perception algorithms.

[0105] The embodiment of the present invention also discloses a readable storage medium.

[0106] A readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the intelligent driving perception evaluation method described in any one of the above embodiments. Computer-readable storage media may include: any entity or device capable of carrying a computer program, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), and a software distribution medium, etc. The computer program includes a computer program code. The computer program code may be in source code form, object code form, an executable file, or some intermediate form, etc. The computer-readable storage medium may include: any entity or device capable of carrying a computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), and a software distribution medium, etc.

[0107] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0108] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processing module, or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.

[0109] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for evaluating intelligent driving perception, characterized in that: include: Collect true value data and data to be evaluated in the same space and time; Based on a preset scene extraction rule, constructing the true value data into a true value scene library including a plurality of sub-scenes; Spatially associating the true value data with the data to be evaluated to obtain the corresponding true value output and test output in the sub-scenario; And, based on a preset evaluation index, the true value output and the test output are compared to obtain an evaluation result corresponding to the evaluation index.

2. The intelligent driving perception evaluation method according to claim 1 is characterized in that: The method of constructing the true value data into a true value scene library including a plurality of sub-scenes based on a preset scene extraction rule includes: Acquire scene information corresponding to the true value data based on the true value data; Determine the target information true value under the scene information, wherein the target information true value includes the target type true value, the target position true value and the target behavior true value; Based on the scene information, the true value data is divided into a plurality of sub-scenes containing scene information and corresponding target information true values, and finally a true value scene library is formed.

3. The intelligent driving perception evaluation method according to claim 2 is characterized in that: The determining the true value of the target information under the scene information includes: Determine a focus area around the vehicle in the scene information based on a preset range threshold; Determine a target type true value existing in the area of ​​interest and a target position true value corresponding to the target type true value; And, determine the target behavior truth value corresponding to the target type truth value.

4. The intelligent driving perception evaluation method according to claim 3 is characterized in that: Determining the target behavior truth value corresponding to the target type truth value includes: Parsing the sensed lane line data in the true value data; When a sudden change occurs in the sensed lane line data, marking a lane change of the vehicle; According to the perceived lane line data and the target position true value, the target behavior true value of the target at the corresponding moment is identified in combination with the lane change mark of the vehicle.

5. The intelligent driving perception evaluation method according to claim 1, characterized in that: The spatially associating the true value data with the data to be evaluated to obtain the corresponding true value output and test output in the sub-scenario includes: Obtaining information of the target to be measured and the lane line to be measured in the data to be evaluated; Obtaining the target information true value and the lane line true value in the true value data; The target information to be tested and the true value of the target information, and the lane line to be tested and the true value of the lane line are matched at the same time and in the same space to obtain the true value output and test output in each sub-scenario.

6. The intelligent driving perception evaluation method according to claim 1, characterized in that: The evaluation index at least includes the detection performance of the data to be evaluated, the geometric error of the detection target, and the distance accuracy of the detection target.

7. The intelligent driving perception evaluation method according to claim 1, characterized in that: The method further comprises: Based on the true value output and the test output, data errors in the data to be evaluated are marked.

8. The intelligent driving perception evaluation method according to claim 7, characterized in that: The marking of data errors in the data to be evaluated based on the true value output and the test output includes: Traversing the truth value outputs, extracting all target information truth values ​​of the first target, and obtaining the start time truth value and the end time truth value of the first target appearing in the sub-scenario according to each of the target information truth values; Matching a test output corresponding to the true value output between the start time true value and the end time true value to obtain matching information of the first target; Aggregating the matching information to obtain an expected detection result of the first target; The problem type of the test output is marked based on the expected detection result.

9. The intelligent driving perception evaluation method according to claim 8, characterized in that: The marking of the problem type of the test output based on the expected detection result includes: If there are multiple target types in the expected detection result of the first target, marking the test output as having an ID jump; If the detection start time of the expected detection result of the first target is later than the true value of the start time, then marking the test output as having a late detection; If there is a missing segment in the expected detection result of the first target, the test output is marked as missing.

10. The intelligent driving perception evaluation method according to claim 1, characterized in that: The method further comprises: The data to be evaluated in multiple versions are fed back and compared with the true value data to obtain evaluation results corresponding to the evaluation indicators under different versions.

11. An intelligent driving perception evaluation device, characterized in that: include: A true value sensor, at least for collecting true value data; The perception sensor to be tested is at least used to collect the data to be evaluated in the same time and space as the true value data; A scene extraction module, at least used to construct the true value data into a true value scene library including a plurality of sub-scenes based on a preset scene extraction rule; A target matching module, at least used to spatially associate the true value data with the data to be evaluated, and obtain the corresponding true value output and test output in the sub-scenario; And, an evaluation output module is at least used to compare the true value output and the test output based on a preset evaluation indicator to obtain an evaluation result corresponding to the evaluation indicator.

12. The intelligent driving perception evaluation device according to claim 11, characterized in that: Also includes: The error marking module is at least used to mark the data error in the data to be evaluated based on the true value output and the test output.

13. The intelligent driving perception evaluation device according to claim 11, characterized in that: Also includes: The algorithm re-injection module is at least used to re-inject and compare multiple versions of the to-be-evaluated data with the true value data to obtain evaluation results corresponding to the evaluation indicators under different versions.

14. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is run by a computer, the intelligent driving perception evaluation method as described in any one of claims 1 to 10 is executed.

Images