Method and device for determining evaluation data, storage medium and electronic device

Abstract

The present application discloses a method and device for determining evaluation data, a storage medium, and an electronic device. The method includes: obtaining a road data set collected under multiple data collection conditions; determining a target road object set according to the object matching results between a first road object set and a second road object set corresponding to each of the multiple data collection conditions; performing model evaluation on the perception model to be evaluated according to the multiple target road object sets, and obtaining model evaluation results that match the multiple target road object sets respectively; and determining at least one key data collection condition from multiple data collection conditions according to the multiple model evaluation results. The present application solves the technical problems that the evaluation data determination method provided by the related art has inaccurate data determination and low efficiency.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving technology, and more specifically, to a method and device for determining evaluation data, a storage medium, and an electronic device. Background Art

[0002] In intelligent driving systems, the closed loop of data in autonomous driving has always been the key to improving various algorithms, especially perception algorithms. In the process of improving perception performance, on the one hand, various high-value scene data are needed to generate true values ​​to support model training iterations, and on the other hand, a complete evaluation system is needed to evaluate and analyze the perception algorithms to find the current deficiencies of the perception algorithms, and at the same time support the screening of subsequent high-value scene data based on the evaluation results.

[0003] However, when evaluating perception algorithms, related technologies mainly output the recall rate and precision rate of algorithm models, and rarely output more comprehensive evaluation indicators from multiple dimensions. The core is that it is difficult to obtain specific speed and depth information of the target through manually labeled true values, and many current perception algorithm model evaluation and analysis results are mainly used to evaluate the pros and cons of perception algorithm models, without showing direct support for subsequent data set screening and slicing, nor closed-loop support for iterative optimization of perception algorithm models. In other words, the related technology's method of determining evaluation data has problems such as inaccurate data determination and low efficiency.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a method and device for determining evaluation data, a storage medium, and an electronic device, so as to at least solve the technical problems that the method for determining evaluation data provided by the related art has inaccurate data determination and low efficiency.

[0006] According to one aspect of an embodiment of the present application, a method for determining evaluation data is provided, comprising: obtaining a road data set respectively collected under a plurality of data collection conditions, wherein the road data set includes at least one road image obtained by image collection of a target road scene, and at least one point cloud image obtained by point cloud collection of a target road scene; determining a target road object set according to object matching results between a first road object set and a second road object set respectively corresponding to the plurality of data collection conditions, wherein the first road object set is obtained by object recognition of at least one road image, and the second road object set is obtained by object recognition of at least one point cloud image; performing model evaluation on perception models to be evaluated respectively according to the plurality of target road object sets, and obtaining model evaluation results respectively matching the plurality of target road object sets; determining at least one key data collection condition from the plurality of data collection conditions according to the plurality of model evaluation results, wherein the key data collection condition indicates an acquisition condition of an evaluation road data set for subsequent evaluation of the perception model to be evaluated.

[0007] According to another aspect of an embodiment of the present application, a device for determining evaluation data is also provided, including: an acquisition unit, which acquires a road data set collected respectively under multiple data collection conditions, wherein the road data set includes at least one road image obtained by image collection of a target road scene, and at least one point cloud image obtained by point cloud collection of a target road scene; a first determination unit, which determines a target road object set according to object matching results between a first road object set and a second road object set corresponding to each of the multiple data collection conditions, wherein the first road object set is obtained by object recognition of at least one road image, and the second road object set is obtained by object recognition of at least one point cloud image; an evaluation unit, which performs model evaluation on perception models to be evaluated respectively according to the multiple target road object sets, and obtains model evaluation results that match the multiple target road object sets respectively; a second determination unit, which determines at least one key data collection condition from the multiple data collection conditions according to the multiple model evaluation results, wherein the key data collection condition indicates a collection condition of an evaluation road data set for subsequent evaluation of the perception model to be evaluated.

[0008] Optionally, the above-mentioned acquisition unit includes: a training module, used to update the data collection conditions according to the key data collection conditions to obtain multiple target data collection conditions; obtain the target road data sets collected according to the multiple target data collection conditions; and train the perception model to be evaluated based on the target road data sets.

[0009] Optionally, the above-mentioned training module is also used to determine the target collection description information that matches each key data collection condition; determine the target data collection condition that matches each key data collection condition according to the target collection time interval, target collection road scene label and target collection road environment label indicated by the target collection description information; obtain the target road data sets collected under multiple target data collection conditions; and train the perception model to be evaluated based on the target road data set.

[0010] Optionally, the above-mentioned first determination unit also includes: a data selection module, used to select target road data collected in a single frame from road data containing target scene labels; a single frame evaluation module, used to determine the model perception results and true value data corresponding to the target road data, and match the model perception results and true value data corresponding to the target road data to obtain a single frame evaluation result.

[0011] Optionally, the above-mentioned model evaluation unit further includes: a matching module, which is used to determine the object matching relationship between the first candidate object and the second candidate object when the first annotation box corresponding to the first candidate object in the first road object set and the second annotation box corresponding to the second candidate object in the second road object set meet the target position condition, wherein the first candidate object and the second candidate object with the object matching relationship are used to indicate the same target road object in the target road scene, and the target position condition includes at least one of the following: the area of ​​the overlapping area between the first annotation box and the second annotation box is greater than or equal to the area threshold; the annotation box distance between the first annotation box and the second annotation box is less than or equal to the distance threshold; A labeling box, when the second labeling box corresponding to any candidate object in the second road object set does not meet the target position condition, a second candidate object that meets the historical matching condition is determined from the second road object set according to the historical matching information of the first candidate object, wherein the historical matching information includes a historical object matching result corresponding to at least one historical road scene, and the acquisition timestamp corresponding to the historical road scene is earlier than the acquisition timestamp of the target road scene; wherein the first labeling box is used to indicate a first image area of ​​the first candidate object in the road image, and the second labeling box is used to indicate a second image area of ​​the second candidate object in the road image, and the second image area is a projection result of point cloud information based on the matching of the second candidate object on the road image.

[0012] Optionally, the matching module is further configured to, when the overlapping areas between the first annotation frame and the candidate annotation frames of any candidate object in the second road object set are all smaller than an area threshold, obtain a second reference annotation frame subset matching the first annotation frame from the second reference annotation frame set matching the second road object set, wherein the area of ​​the overlapping areas between the second reference annotation frame and the first annotation frame in the second reference annotation frame subset is greater than 0 and smaller than the area threshold; determine a target annotation frame from the second reference annotation frame subset, wherein the area of ​​the overlapping area between the first annotation frame and the target annotation frame is greater than the area of ​​the overlapping areas corresponding to other annotation frames in the second reference annotation frame subset; and when the annotation frame distances between the target annotation frame and the first annotation frame are all smaller than the annotation frame distances corresponding to other annotation frames in the first reference annotation frame set, determine the annotation frame distance between the target annotation frame and the first annotation frame, meet the target distance condition, and determine that there is an object matching relationship between the third candidate object indicated by the target annotation frame and the first road object, wherein the first reference annotation frame is used to indicate an image area of ​​the candidate object in the first road object set in the target road image matching the target road scene, and the plurality of first reference annotation frames include the first annotation frame.

[0013] Optionally, the above-mentioned data acquisition unit also includes: a data determination module, used to determine the target category and the target distance threshold interval; a data acquisition module, used to obtain target road data containing a target scene label and a target category within the target distance threshold interval, and determine the target road data as training data; a model training module, used to use the training data to train the perception model to be evaluated.

[0014] Optionally, the matching module is also used to obtain historical matching information, wherein the historical matching information includes historical object matching results of the first candidate object in multiple historical road scenes, the multiple historical road scenes and the target road scene are road scenes corresponding to multiple information collection operations respectively, the multiple information collection operations are information collection operations corresponding to multiple continuous timestamps respectively, the information collection operations include image data collection operations and point cloud data collection operations, and the historical object matching results are used to indicate the second reference object that matches the first candidate object in the current road scene; the second reference object with the largest number of matches is determined as the second candidate object that meets the historical matching conditions; or the historical matching information of the second reference object with the largest number of matches is obtained; when the historical matching information of the second reference object indicates that the candidate object with the largest number of matches with the second reference object is not the first candidate object, the object acquisition order is determined according to the number of matches corresponding to at least one second reference object; according to the object acquisition order, a second reference object is sequentially obtained from at least one second reference object as the current reference object; when the historical matching information of the current reference object indicates that the first candidate object is the candidate object with the largest number of matches with the current reference object, the current reference object is determined as the second candidate object that meets the historical matching conditions.

[0015] Optionally, the above-mentioned matching module also includes: a verification module, used to obtain a second annotation box corresponding to the second candidate object that meets the historical matching condition, and a second verification annotation box corresponding to the second verification candidate object matched with the first candidate object determined according to the key frame matching information; when the area of ​​the overlapping area between the second verification annotation box and the second annotation box is greater than or equal to the area threshold, determine that the matching result is correct; when the area of ​​the overlapping area between the second verification annotation box and the second annotation box is less than the area threshold, determine that the second verification annotation box is the second candidate object to be corrected; according to the key frame matching information corresponding to the key frame where the second candidate object to be corrected is located, modify the second candidate object to be corrected that matches the first candidate object to be corrected into the second correction object.

[0016] Optionally, the above-mentioned evaluation unit is also used to: input multiple target road object sets into the perception model to be evaluated to obtain model prediction results that respectively match the multiple target road object sets; perform model evaluation on the perception model to be evaluated according to the model prediction results and true value data that respectively match each target road object set to obtain a category evaluation result, wherein the category evaluation result is used to indicate the evaluation results corresponding to multiple categories of perception targets; determine the category of the perception target to be evaluated based on the category evaluation result, and use road data collected from multiple continuous frame segments to evaluate the perception model to be evaluated on the dimension of the category of the perception target to be evaluated to obtain a model evaluation result.

[0017] Optionally, the above-mentioned evaluation unit also includes: a fourth determination unit, which is used to perform model evaluation on the perception model to be evaluated based on the model prediction results and true value data corresponding to each of the road data sets collected from multiple continuous frame segments, and obtain the model evaluation results corresponding to each of the multiple continuous frame segments; determine the model evaluation results that meet the preset conditions among the model evaluation results corresponding to each of the multiple continuous frame segments as the evaluation results to be determined; determine the repeated scene labels in the scene labels matched by at least one target road object set corresponding to the evaluation results to be determined as the target scene labels; select the target road data collected in a single frame from the target road object set matching the target scene label; determine the model prediction results and true value data corresponding to the target road data, and match the model prediction results and true value data corresponding to the target road data to obtain a single frame evaluation result.

[0018] Optionally, the above-mentioned evaluation unit also includes: a fifth determination unit, used to determine the distance between at least one reference obstacle in at least one target road object set and the vehicle according to the true value data corresponding to at least one target road object set, wherein the reference obstacle is used to indicate the static obstacles and dynamic obstacles around the vehicle; according to the distance between at least one reference obstacle and the vehicle and a preset distance threshold, at least one reference obstacle is divided into at least one reference obstacle set; according to the target road object set corresponding to at least one reference obstacle set, the model evaluation of the perception model to be evaluated is performed in sequence to obtain a model evaluation result, including at least one of the following: calculating the difference between the model prediction result and the distance in the true value data to obtain an absolute distance error; calculating the ratio between the absolute distance error and the distance in the true value data to obtain a relative distance error; calculating the difference between the speed in the model prediction result and the speed in the true value data to obtain an absolute speed error.

[0019] Optionally, the above-mentioned evaluation unit is also used to: determine the category of the perception target to be evaluated and the target distance threshold interval; obtain target road data that belongs to the target scene label and the category of the perception target to be evaluated is within the target distance threshold interval, and determine the target road data as updated training data; use the updated training data to train the perception model to be evaluated.

[0020] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned method for determining evaluation data when running.

[0021] According to another aspect of the embodiment of the present application, a computer program product is provided, the computer program product comprising a computer program / instruction, the computer instruction being stored in a computer-readable storage medium. A processor of a computer device reads the computer program / instruction from the computer-readable storage medium, and the processor executes the computer program / instruction, so that the computer device executes the above method for determining evaluation data.

[0022] According to another aspect of the embodiments of the present application, there is further provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the method for determining the evaluation data through the computer program.

[0023] In an embodiment of the present application, a road data set collected under multiple data collection conditions is obtained; then, a target road object set is determined according to the object matching results between the first road object set and the second road object set corresponding to each of the multiple data collection conditions, wherein the first road object set is obtained by object recognition of at least one road image, and the second road object set is obtained by object recognition of at least one point cloud image; according to the multiple target road object sets, model evaluation is performed on the perception model to be evaluated respectively, and model evaluation results respectively matched with the multiple target road object sets are obtained; thus, according to the multiple model evaluation results, at least one key data collection condition is determined from the multiple data collection conditions, wherein the key data collection condition indicates the collection condition of the evaluation road data set for subsequent evaluation of the perception model to be evaluated. By determining the key data collection conditions, training data for improving the poor results of the perception model output is obtained in a directionally manner. The technical problems of inaccurate data determination and low efficiency in the evaluation data determination method provided by the related technology are solved. The optimization efficiency of the perception model is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0025] Figure 1 is a schematic diagram of a hardware environment of an optional method for determining evaluation data according to an embodiment of the present application;

[0026] Figure 2 is a flowchart of an optional method for determining evaluation data according to an embodiment of the present application;

[0027] Figure 3 is a schematic diagram of an optional method for determining evaluation data according to an embodiment of the present application;

[0028] Figure 4 is a schematic diagram of an optional truth value matching method according to an embodiment of the present application;

[0029] Figure 5 is a schematic diagram of another optional truth value matching method according to an embodiment of the present application;

[0030] Figure 6 is a schematic diagram of another optional truth value matching method according to an embodiment of the present application;

[0031] Figure 7 is a schematic diagram of another optional method for determining evaluation data according to an embodiment of the present application;

[0032] Figure 8 is a flowchart of another optional method for determining evaluation data according to an embodiment of the present application;

[0033] Fig. 9 is a schematic diagram of another optional method for determining evaluation data according to an embodiment of the present application;

[0034] Fig.10 is a schematic diagram of another optional method for determining evaluation data according to an embodiment of the present application;

[0035] Fig.11 is a schematic diagram of another optional method for determining evaluation data according to an embodiment of the present application;

[0036] Fig.12 is a schematic diagram of another optional method for determining evaluation data according to an embodiment of the present application;

[0037] Fig.13 is a schematic diagram of another optional method for determining evaluation data according to an embodiment of the present application;

[0038] Fig.14 is a flowchart of another optional method for determining evaluation data according to an embodiment of the present application;

[0039] Fig.15 is a schematic diagram of an optional device for determining evaluation data according to an embodiment of the present application;

[0040] Fig.16 is a schematic diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0042] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0043] According to one aspect of the embodiment of the present application, a method for determining evaluation data is provided. As an optional implementation, the method for determining evaluation data can be but is not limited to being applied to: Figure 1 The system for determining evaluation data in the hardware environment shown. Optionally, the above evaluation data determination method can be applied to a vehicle terminal. Figure 1 A side view of a vehicle terminal 101 is shown, which is disposed on a travel surface 113 and is capable of traversing the travel surface 113. The vehicle terminal 101 includes an onboard navigation system 103, a computer-readable storage device or medium (memory) 102 including a digitized road map 104, a spatial monitoring system 117, a vehicle controller 109, a GPS (global positioning system) sensor 110, an HMI (human / machine interface) device 111, and also includes an autonomous controller 112 and a telematics controller 114. The vehicle terminal 101 may include, but is not limited to, commercial vehicles, industrial vehicles, agricultural vehicles, passenger vehicles, all-terrain vehicles, personal mobile devices, robots, and similar forms of mobile platforms to achieve the purpose of the present application.

[0044] In one embodiment, the spatial monitoring system 117 includes: one or more spatial sensors and systems arranged to monitor the visible area 105 in front of the vehicle terminal 101; and a spatial monitoring controller 118. The spatial sensors used to monitor the visible area 105 include, for example, a lidar sensor 106, a radar sensor 107, a camera 108, and the like. The placement of the spatial sensors allows the spatial monitoring controller 118 to monitor traffic flow, including approaching vehicles, intersections, lane markings, and other objects around the vehicle terminal 101. The spatial sensors of the spatial monitoring system 117 may include object positioning sensing devices. The lidar sensor 106 uses pulsed and reflected laser beams to measure the range or distance to an object. The radar sensor 107 uses radio waves to determine the range, angle, and / or speed of an object. The camera 108 includes an image sensor, a lens, and a camera controller.

[0045] The camera 108 is advantageously mounted and positioned on the vehicle terminal 101 in a position that allows capturing images of the visible area 105, wherein at least part of the visible area 105 includes a portion of the travel surface 113 in front of the vehicle terminal 101 and including the trajectory of the vehicle terminal 101. The visible area 105 may also include the surrounding environment. Other cameras (not shown) may also be used, for example, including a second camera disposed on the rear portion or side portion of the vehicle terminal 101 to monitor the rear of the vehicle terminal 101 and one of the right or left sides of the vehicle terminal 101. The camera 108 may collect road information to generate a road image sequence during the driving of the vehicle terminal 101.

[0046] The autonomous controller 112 is configured to implement autonomous driving or advanced driver assistance system (ADAS) vehicle functionality. Such functionality may include a vehicle onboard control system capable of providing a certain level of driving automation. Driving automation may include a series of dynamic driving and vehicle operations. Driving automation may include simultaneous automatic control of vehicle driving functions (including steering, acceleration and braking), wherein the driver abandons control of the vehicle for a period of time during the journey. Driving automation may include simultaneous automatic control of vehicle driving functions (including steering, acceleration and braking), wherein the driver abandons control of the vehicle terminal 101 for the entire journey. Driving automation includes hardware and a controller configured to monitor the spatial environment in various driving modes to perform various driving tasks during dynamic vehicle operations. By way of non-limiting example, autonomous vehicle functions include adaptive cruise control (ACC) operations, lane guidance and lane keeping operations, lane change operations, steering assistance operations, object avoidance operations, parking assistance operations, vehicle braking operations, vehicle speed and acceleration operations, vehicle lateral motion operations, for example, as part of lane guidance, lane keeping and lane change operations, etc.

[0047] The autonomous controller can be equipped with an operating system and an automatic driving system. The operating system can be responsible for the management of the hardware resources (including sensors, system buses, networks, etc.) of the vehicle terminal 101 and the scheduling of computing resources. The automatic driving system can implement various algorithms required for automatic driving, including positioning, environmental perception, path planning and control, etc., and can make decisions in situations such as driving on curves, driving in a straight line, driving on complex roads, and changing lanes. The autonomous controller can execute step S1 while the vehicle is driving, and obtain a road data set collected under multiple data collection conditions, wherein the road data set includes at least one road image obtained by image acquisition of the target road scene, and at least one point cloud image obtained by point cloud acquisition of the target road scene.

[0048] The vehicle terminal 101 may include a telematics controller 114 including a wireless telematics communication system capable of performing off-vehicle communications, including communications with a communication network 115 having wireless and wired communication capabilities. Alternatively or additionally, the telematics controller 114 directly performs off-vehicle communications by communicating with an off-board server 116 via the communication network 115. The server side can execute step S2 to determine the target road object set according to the object matching results between the first road object set and the second road object set corresponding to each of the multiple data collection conditions, wherein the first road object set is obtained by performing object recognition on at least one road image, and the second road object set is obtained by performing object recognition on at least one point cloud image; step S3 to perform model evaluation on the perception models to be evaluated respectively according to the multiple target road object sets to obtain model evaluation results that match the multiple target road object sets respectively; step S4 to determine at least one key data collection condition from the multiple data collection conditions according to the multiple model evaluation results, wherein the key data collection condition indicates the collection condition of the evaluation road data set used for subsequent evaluation of the perception model to be evaluated.

[0049] In an optional embodiment, the vehicle terminal 101 can collect road images in real time using sensors such as cameras 108 through the space monitoring system 117. The road images can contain surrounding road condition data, such as movement information of surrounding vehicles, pedestrians and other traffic participants, and location information of static obstacles such as roadblocks and road fences. The collected road image sequence can be stored in the memory 102. Since the storage space of the memory 102 is limited, based on storage efficiency considerations, images with higher priority can be stored in the memory 102, and a large amount of low-priority data can be stored in the non-airborne server 116 via the communication network 115 using the telematics controller 114.

[0050] In an optional embodiment, the method for determining the evaluation data of the present application can be applied to the vehicle side, where the perception model can be tested and evaluated on an actual vehicle, and real-time feedback can be provided in this way. The performance of the perception model can be verified directly in the actual operating environment of the vehicle. The road data of multiple continuous frame fragments collected by the vehicle can be input into the on-board perception model to be evaluated to obtain the model perception result, and the model perception result and the true value data are matched to evaluate the perception model to obtain the evaluation result. The performance of the perception model in the current vehicle driving scenario can be determined based on the evaluation result. When the evaluation result does not meet the preset conditions, it is determined that the perception effect of the perception model in the current vehicle driving scenario is not good, and data with labels for the current vehicle driving scenario can be obtained to train and update the perception model. In this way, the perception model is updated in a targeted manner to improve the efficiency of updating and iterating the perception model.

[0051] In an optional embodiment, the method for determining the evaluation data of the present application can be applied to the server side, and the perception model can be tested and evaluated on the server side or in the cloud to determine the evaluation data, and the evaluation data of multiple scenarios can be determined. Road data with different scene labels can be obtained, and the road data with different scene labels can be input into the perception model. The prediction results of the perception model can be obtained and matched with the true value to evaluate the perception model. The scene label corresponding to the evaluation result that meets the preset conditions is determined as the target scene label, and the road data of the scene corresponding to the target scene label is obtained to update the perception model. In multiple scenarios, scenes with weak performance of the perception model can be quickly identified, and training can be performed to improve the training efficiency of the perception model.

[0052] In an optional embodiment, if Figure 2 As shown, the method for determining the above evaluation data includes the following steps:

[0053] S202, acquiring road data sets respectively collected under a plurality of data collection conditions, wherein the road data sets include at least one road image obtained by image collection of a target road scene, and at least one point cloud image obtained by point cloud collection of the target road scene;

[0054] S204, determining a target road object set according to object matching results between a first road object set and a second road object set corresponding to each of the plurality of data collection conditions, wherein the first road object set is obtained by performing object recognition on at least one road image, and the second road object set is obtained by performing object recognition on at least one point cloud image;

[0055] S206, performing model evaluation on the perception models to be evaluated respectively according to the multiple target road object sets, and obtaining model evaluation results that match the multiple target road object sets respectively;

[0056] S208, determining at least one key data collection condition from multiple data collection conditions based on multiple model evaluation results, wherein the key data collection condition indicates a collection condition of an evaluation road data set for subsequent evaluation of the perception model to be evaluated.

[0057] In the above step S202, a road data set collected respectively under a plurality of data collection conditions is obtained, wherein the road data set includes at least one road image obtained by image collection of a target road scene, and at least one point cloud image obtained by point cloud collection of the target road scene;

[0058] As an optional implementation, the data collection condition may be that the vehicle collects the position information of the preceding vehicle in a cloudy crossroad road scene with low illumination intensity at a time interval of 2 ms, and a road data set collected under multiple data collection conditions may be obtained. The road image may be a road image obtained by collecting 2D information, and the point cloud image may be 3D information obtained by collecting point cloud information.

[0059] In the above step S204, a target road object set is determined according to the object matching results between the first road object set and the second road object set corresponding to each of the plurality of data collection conditions, wherein the first road object set is obtained by performing object recognition on at least one road image, and the second road object set is obtained by performing object recognition on at least one point cloud image;

[0060] Optionally, the first road object set may be a 2D information set that matches the road object, and the second road object set may be a 3D information set that matches the road object. The matching of the 2D information and the 3D information may be determined by, but is not limited to, the following methods:

[0061] Project the truth frame of the 3D truth data onto the image of the 2D truth data, and filter out objects that are not completely blocked; Figure 3 As shown, taking the front view image as an example, the dotted box represents the front view image, the solid line truth box on the left is the 3D truth box corresponding to the vehicle object 302 projected onto the 2D image, and the dotted line truth box on the left is the 3D truth box corresponding to the vehicle object 304 projected onto the 2D image. The solid line truth box on the left surrounds the dotted line truth box on the left, and the vehicle object 304 is farther away from the vehicle 308 longitudinally, so the dotted line truth box on the left can be eliminated because this object cannot be seen under the front view camera.

[0062] By eliminating the 3D truth boxes that are not visible from the camera's perspective and projected onto the 2D image, the 2D truth data and 3D truth data whose intersection over union (IoU) is greater than a preset threshold are associated. The IoU can be the ratio of the intersection area to the union area of ​​two bounding boxes.

[0063] The remaining two-dimensional truth data and three-dimensional truth data can be sorted according to the intersection-and-union ratio, and then the Euclidean distances between the two-dimensional truth data and the three-dimensional truth data can be calculated respectively. If the three-dimensional truth data has the smallest Euclidean distance to a certain two-dimensional truth data and the Euclidean distance between the three-dimensional truth data and the two-dimensional truth data is also the smallest, the two can be matched.

[0064] For the remaining two-dimensional truth data and three-dimensional truth data with an intersection-and-union ratio of 0, the data of the previous n consecutive frames can be used to analyze them, and the two-dimensional truth data and the three-dimensional truth data with the most association times can be combined. After the truth data are combined, the truth data can include target category, two-dimensional truth box, two-dimensional target ID, three-dimensional target ID, two-dimensional target box xmin, ymin, xmax, ymax, target head or tail longitudinal distance, target head or tail lateral distance, target side longitudinal distance, target side lateral distance, target longitudinal speed, target lateral speed target, two-dimensional truth box x, y, z, rz, l, w, h and information, target speed information, target three-dimensional projection box information and 8 corner point coordinate information of the target's three-dimensional rectangle on the two-dimensional image.

[0065] In an optional embodiment, when matching the true value data with the model prediction results, a cost matrix can be constructed, and the rows of the cost matrix can represent the model prediction results, and the columns can represent the true value data, so that each element in the matrix can represent the matching cost between the corresponding prediction results and the true value data. The matching cost can be an intersection ratio, a Euclidean distance, etc. The matrix can be initialized, and the minimum value of each row is subtracted from the minimum value of the row, and the same operation is performed on each column. Use the minimum number of horizontal and vertical lines to cover all zero elements in the matrix. These zero elements represent the possible best match. If the number of lines covering the zero elements is equal to the dimension of the matrix (i.e., the number of rows or columns), it means that the best match has been found. If it is not equal, it means that further adjustment is required. If the best match is not found, the cost matrix needs to be adjusted. Find the smallest element that is not covered, subtract the minimum value from each element in the row where the element is located, and add the minimum value to each element in the column where the element is located, and then repeat the above process. Until the best match is found. The best match will be represented by the index of the row and column, and these indexes point to the zero elements in the cost matrix.

[0066] It should be noted that in the process of matching the true value data and the model prediction results, the large depth error of the distant target may lead to inaccurate matching. However, by matching the combined two-dimensional and three-dimensional data, the matching can be performed within the two-dimensional frame, avoiding the influence of depth information.

[0067] Then, in the above step S206, model evaluation is performed on the perception models to be evaluated respectively according to the multiple target road object sets, and model evaluation results matching the multiple target road object sets are obtained;

[0068] Optionally, the model can be evaluated by comparing the true trajectory of the preceding vehicle with the predicted trajectory output by the perception model. The evaluation results may be recall rate, precision rate, MOTA, MOTP, FN (missed detection), FP (false detection), etc.

[0069] Optionally, in the above step S208, at least one key data collection condition is determined from multiple data collection conditions based on multiple model evaluation results, wherein the key data collection condition indicates a collection condition of an evaluation road data set for subsequent evaluation of the perception model to be evaluated.

[0070] The above-mentioned key data collection conditions may be road scenes with poor evaluation results, or weather, etc., and the data features corresponding to the key data collection conditions may be added to input the perception model to be evaluated for optimization training.

[0071] Through the above implementation, a road data set collected under multiple data collection conditions is obtained; then, according to the object matching results between the first road object set and the second road object set corresponding to each of the multiple data collection conditions, a target road object set is determined, wherein the first road object set is obtained by object recognition of at least one road image, and the second road object set is obtained by object recognition of at least one point cloud image; according to the multiple target road object sets, model evaluation is performed on the perception model to be evaluated, and model evaluation results respectively matched with the multiple target road object sets are obtained; thus, according to the multiple model evaluation results, at least one key data collection condition is determined from the multiple data collection conditions, wherein the key data collection condition indicates the collection condition of the evaluation road data set for subsequent evaluation of the perception model to be evaluated. By determining the key data collection conditions, training data for improving the poor results of the perception model output is obtained in a directionally manner. The technical problems of inaccurate data determination and low efficiency in the evaluation data determination method provided by the related technology are solved. The optimization efficiency of the perception model is improved.

[0072] In an optional implementation, after determining at least one key data collection condition from a plurality of data collection conditions according to a plurality of model evaluation results, the method further comprises:

[0073] S1, updating the data collection conditions according to the key data collection conditions to obtain multiple target data collection conditions;

[0074] S2, obtaining target road data sets collected under multiple target data collection conditions;

[0075] S3, training the perception model to be evaluated based on the target road dataset.

[0076] As an optional implementation, in the above steps S1-S3, the data collection conditions are updated according to the key data collection conditions to obtain multiple target data collection conditions; the target road data sets collected under the multiple target data collection conditions are obtained; and the perception model to be evaluated is trained according to the target road data sets.

[0077] For example, when the traffic flow at the intersection reaches a congested state, the error value between the trajectory prediction result output by the perception model and the real data is greater than a preset threshold, then the key data collection condition may be the road congestion state, and the data collection condition may be updated to obtain multiple target data collection conditions, which may be the vehicle location information under the road scene where the road traffic flow value is greater than 100;

[0078] Furthermore, we can collect target road data sets in congested road scenes under different lighting conditions, input the target road data sets into the model for training, and improve the accuracy of the perception model output results under congested conditions.

[0079] Through the above implementation, the road data corresponding to the labels with poor model output effects are obtained in a targeted manner, thereby realizing closed-loop improvement of slice data collection, improving the accuracy and efficiency of evaluation data selection, and also improving the efficiency of perception model training.

[0080] In an optional implementation, after determining at least one key data collection condition from a plurality of data collection conditions according to a plurality of model evaluation results, the method further comprises:

[0081] S1, determine the target collection description information matching each key data collection condition;

[0082] S2, determining a target data collection condition matching each key data collection condition according to the target collection time interval, the target collection road scene label, and the target collection road environment label indicated by the target collection description information;

[0083] S3, obtaining target road data sets collected under multiple target data collection conditions;

[0084] S4, training the perception model to be evaluated based on the target road dataset.

[0085] In the above steps S1-S4, the target collection description information matching each key data collection condition is determined; the target data collection condition matching each key data collection condition is determined according to the target collection time interval, the target collection road scene label and the target collection road environment label indicated by the target collection description information; the target road data sets collected under each of the multiple target data collection conditions are obtained; and the perception model to be evaluated is trained according to the target road data sets.

[0086] As an optional implementation, for example, the predicted vehicle driving trajectory output by the perception model under the collection conditions of night, rainy days, and crossroads fluctuates greatly and has a large error with the actual trajectory, then the target collection description information that matches the time-critical data collection condition is determined to be night, the target collection description information that matches the weather-critical data collection condition is rainy days, and the target collection description information that matches the scene-critical data collection condition is a crossroad, and then according to the target road data sets collected under multiple target data collection conditions, the perception model to be evaluated is trained according to the target road data sets to optimize the accuracy of the model prediction results in the above scenarios.

[0087] In an optional implementation manner, before determining the target road object set according to the object matching results between the first road object set and the second road object set corresponding to each of the plurality of data collection conditions, the method includes:

[0088] S1, determining an object matching relationship between the first candidate object and the second candidate object when a first annotation box corresponding to a first candidate object in a first road object set and a second annotation box corresponding to a second candidate object in a second road object set meet a target position condition, wherein the first candidate object and the second candidate object having the object matching relationship are used to indicate the same target road object in a target road scene, and the target position condition includes at least one of the following: an area of ​​an overlapping area between the first annotation box and the second annotation box is greater than or equal to an area threshold; a distance between the first annotation box and the second annotation box is less than or equal to a distance threshold;

[0089] S2, when neither the first annotation box corresponding to the first candidate object nor the second annotation box corresponding to any candidate object in the second road object set satisfies the target position condition, determining a second candidate object that satisfies the historical matching condition from the second road object set according to historical matching information of the first candidate object, wherein the historical matching information includes historical object matching results corresponding to at least one historical road scene, and the acquisition timestamp corresponding to the historical road scene is earlier than the acquisition timestamp of the target road scene;

[0090] Among them, the first annotation box is used to indicate the first image area of ​​the first candidate object in the road image, and the second annotation box is used to indicate the second image area of ​​the second candidate object in the road image, and the second image area is the projection result of the point cloud information matched based on the second candidate object on the road image.

[0091] In S1, when a first annotation box corresponding to a first candidate object in a first road object set and a second annotation box corresponding to a second candidate object in a second road object set satisfy a target position condition, an object matching relationship between the first candidate object and the second candidate object is determined, wherein the first candidate object and the second candidate object having the object matching relationship are used to indicate the same target road object in a target road scene, and the target position condition includes at least one of the following: an area of ​​an overlapping area between the first annotation box and the second annotation box is greater than or equal to an area threshold; a annotation box distance between the first annotation box and the second annotation box is less than or equal to a distance threshold.

[0092] Matching can be performed based on IoU, by calculating whether the value of the bounding_box_overlap (area of ​​the overlapping area) of the 2D true value box and the 3D projection box is greater than the set threshold. If the IoU of the 2D true value box and the 3D projection box of the same category is greater than the set threshold, the 2D true value box and the 3D projection box will be associated to determine that the object matching relationship between the first candidate object and the second candidate object is successful.

[0093] The object matching relationship between the first candidate object and the second candidate object may also be determined when the distance between the first annotation box corresponding to the first candidate object and the second annotation box of the second candidate object is less than or equal to a distance threshold.

[0094] The above-mentioned annotation box distance can be but is not limited to the distance between the center point of the first annotation box and the center point of the second annotation box. When the distance between the center point of the 2D truth box and the center point of the 3D projection box is less than a certain threshold, the object matching relationship between the first candidate object and the second candidate object is determined to be a successful match.

[0095] It should be noted that the above matching methods can be combined. For example, the area of ​​the 2D true value box is the smallest. A 2D true value box matching the first candidate object and a 3D projection box matching the second candidate object are obtained. The overlapping area between the 2D true value box and the 3D projection box is greater than 50% of the area of ​​the 2D true value box and less than 90% of the 2D true value box (i.e., partial overlap). It is further determined whether the distance between the center point of the 2D true value box and the center point of the 3D projection box is less than 1 mm. If the distance between the center point of the 2D true value box and the center point of the 3D projection box is less than 1 mm, the object matching relationship between the first candidate object and the second candidate object is determined to be a successful match.

[0096] It should be noted that, to determine whether the distance between the center point of the 2D true value box and the center point of the 3D projection box is less than a certain threshold, it is also possible to obtain the distance between the center point of the 2D true value box matching the first candidate object and the center point of the 3D projection box matching all the second candidate objects, and determine that the distance between the center point of the first candidate object and the center point of the target second candidate object is the smallest of all distances. In this case, it can be determined that the object matching relationship between the first candidate object and the target second candidate object is a successful match, and no specific restrictions are made here.

[0097] In S2, when the first annotation box corresponding to the first candidate object and the second annotation box corresponding to any candidate object in the second road object set do not meet the target position condition, a second candidate object that meets the historical matching condition is determined from the second road object set based on historical matching information of the first candidate object, wherein the historical matching information includes historical object matching results corresponding to at least one historical road scene, and the collection timestamp corresponding to the historical road scene is earlier than the collection timestamp of the target road scene.

[0098] It can be understood that, when the 2D truth box and the 3D projection box do not meet the target position condition (for example, no overlap), a second candidate object that meets the historical matching condition is further determined from the second road object set based on the historical matching information of the first candidate object. For example, the above-mentioned first candidate object is the first car in the left lane, and the historical matching information matching the car is obtained. For example, the historical matching information indicates that the 3D information corresponding to the vehicle ID 3 has the largest number of successful matches with the car, then the 3D information corresponding to the vehicle ID 3 in the second road object set is determined to be the second candidate object that meets the historical matching condition.

[0099] Through the above-mentioned implementation mode of the present application, judging the matching road objects through overlapping conditions, center point distance conditions and historical matching relationships can significantly improve the matching accuracy, efficiency and robustness, while improving the consideration of spatial relationships and optimizing the entire matching process.

[0100] In an optional implementation, when a first annotation box corresponding to a first candidate object in the first road object set and a second annotation box corresponding to a second candidate object in the second road object set meet a target position condition, determining an object matching relationship between the first candidate object and the second candidate object further includes:

[0101] S1, when the overlapping areas between the first annotation box and the candidate annotation boxes of any candidate object in the second road object set are all smaller than the area threshold, a second reference annotation box subset matching the first annotation box is obtained from the second reference annotation box set matching the second road object set, wherein the overlapping area between the second reference annotation box in the second reference annotation box subset and the first annotation box is greater than 0 and smaller than the area threshold;

[0102] S2, determining a target annotation frame from the second reference annotation frame subset, wherein the area of ​​an overlapping region between the first annotation frame and the target annotation frame is greater than the area of ​​an overlapping region corresponding to other annotation frames in the second reference annotation frame subset;

[0103] S3, when the annotation frame distances between the target annotation frame and the first annotation frame are all smaller than the annotation frame distances corresponding to other annotation frames in the first reference annotation frame set, determine the annotation frame distance between the target annotation frame and the first annotation frame, satisfy the target distance condition, and determine that there is an object matching relationship between the third candidate object indicated by the target annotation frame and the first road object, wherein the first reference annotation frame is used to indicate an image area of ​​the candidate object in the first road object set in the target road image matching the target road scene, and the multiple first reference annotation frames include the first annotation frame.

[0104] In the above steps S1-S3, when the overlapping areas between the first annotation frame and the respective candidate annotation frames of any candidate object in the second road object set are all smaller than the area threshold, a second reference annotation frame subset matching the first annotation frame is obtained from the second reference annotation frame set matching the second road object set; a target annotation frame is determined from the second reference annotation frame subset; and when the annotation frame distances between the target annotation frame and the first annotation frame are all smaller than the annotation frame distances corresponding to other annotation frames in the first reference annotation frame set, a annotation frame distance between the target annotation frame and the first annotation frame is determined, the target distance condition is satisfied, and it is determined that there is an object matching relationship between the third candidate object indicated by the target annotation frame and the first road object.

[0105] As an optional implementation, for example, the first annotation box is the 2D true value box corresponding to the first car on the first lane on the left, and the overlapping area between the first annotation box and the candidate annotation box (3D projection box) of any candidate object in the second road object set is smaller than the area threshold (for example, the area threshold is equal to the area size of the 2D box of the car), then a second reference annotation box subset matching the first annotation box is obtained. It should be noted that the area of ​​the overlapping area between the second reference annotation box in the second reference annotation box subset and the first annotation box is greater than 0 and smaller than the area threshold. For example, the overlapping area between the second reference annotation box in the second reference annotation box subset and the first annotation box is 50%-80% of the area of ​​the 2D box of the car.

[0106] Arrange the IoU values ​​with non-zero values ​​of the 2D true value frame from large to small, determine the target annotation frame with the largest overlapping area with the first annotation frame in the second reference annotation frame subset, and calculate the Euclidean distance of the center pixel coordinates of the 2D true value frame (first annotation frame) and all second reference annotation frame subsets (3D projection frames) and sort them, and further match them based on the IoU sorting results and the center point Euclidean distance sorting results;

[0107] For example, the 3D information corresponding to the target annotation box with the largest overlapping area with the first annotation box in the second reference annotation box subset is then determined to determine whether the distance between the center point of the first annotation box and the center point of the reference annotation box with index 3 is less than 1 mm. If the target distance condition is met, it is determined that the first candidate object corresponding to the first annotation box and the second candidate object corresponding to the reference annotation box with index 3 are successfully matched, that is, they belong to the same road object.

[0108] Through the above-mentioned implementation mode of the present application, dual judgment based on the overlapping area and the center point distance is realized, which improves the accuracy of the matching results between 2D information and 3D information, thereby improving the accuracy of the selected model evaluation results and the speed of perception model optimization iteration.

[0109] In an optional implementation, determining a second candidate object that meets the historical matching condition from the second road object set according to the historical matching information of the first candidate object includes:

[0110] S1, obtaining historical matching information, wherein the historical matching information includes historical object matching results of a first candidate object in a plurality of historical road scenes, the plurality of historical road scenes and the target road scene are road scenes corresponding to a plurality of information collection operations respectively, the plurality of information collection operations are information collection operations corresponding to a plurality of continuous timestamps respectively, the information collection operations include image data collection operations and point cloud data collection operations, and the historical object matching results are used to indicate a second reference object that matches the first candidate object in a current road scene;

[0111] S2, determining the second reference object with the largest number of matches as the second candidate object that meets the historical matching condition; or

[0112] S3, obtaining historical matching information of the second reference object with the largest number of matches; when the historical matching information of the second reference object indicates that the candidate object with the largest number of matches with the second reference object is not the first candidate object, determining the object acquisition order according to the number of matches corresponding to at least one second reference object; according to the object acquisition order, sequentially acquiring one second reference object from at least one second reference object as the current reference object; when the historical matching information of the current reference object indicates that the first candidate object is the candidate object with the largest number of matches with the current reference object, determining the current reference object as the second candidate object that meets the historical matching conditions.

[0113] As an optional implementation, for example, the second candidate object that matches the first candidate object the most times is obtained from the historical matching information as the car target with an index of 3, then the information of the second candidate object that matches the first candidate object is determined according to the historical matching information as the 3D information corresponding to the car target with an index of 3.

[0114] By acquiring historical matching information and then determining a second candidate object that meets the historical matching conditions from at least one second reference object based on the object matching results corresponding to each of a plurality of historical road scenes, the optimal matching result between the 2D information and the 3D information is guaranteed to the greatest extent, thereby improving the accuracy of matching the road object information.

[0115] Specifically, for example, the Car target with 2D true value ID 5 has matched with the Car target with 3D projection ID 3 5 times, with the Car target with 3D projection ID 6 4 times, and with the Car target with 3D projection ID 7 once in 10 consecutive historical frames. We sort the historical matching results in descending order. From the descending results, we can see that the Car target with 2D true value ID 5 has matched with the Car target with 3D projection ID 3 the most times in the historical matching stage, so the Car target with 3D projection ID 3 is determined to be the second candidate object that meets the historical matching conditions.

[0116] As another optional implementation, for example, the Car target with a 2D true value ID of 5 matches the Car target with a 3D projection ID of 3 the most times in 10 consecutive historical frames, and further obtains the historical matching information of the Car target with a 3D projection ID of 3 (the second reference object), and searches for the 2D true value ID that matches the Car target with a 3D projection ID of 3 the most times in the historical matching results. If the 2D true value ID is 5, the second reference object is determined to be the second candidate object that meets the historical matching conditions. This is only an example and is not specifically limited.

[0117] The above process S1-S3 is described below in a complete implementation manner:

[0118] First, assume that the Car target with 2D true value ID 5 in the nth frame has no 3D projection frame target matching it after two rounds of matching strategies: overlapping area matching and projection frame center point distance matching. For example, the IoU between the Car target with 2D true value ID 5 and all 3D projection frames is 0. In S1-S4, first count whether the Car target with 2D true value ID 5 has always existed in the past 10 consecutive frames. If it has always existed, it is the most ideal situation. If it has not always existed, only count the continuous scenes where the Car target with 2D true value ID 5 in the past has appeared continuously to the current frame, such as the n-5th frame to the nth frame.

[0119] Assume that the Car target with 2D true value frame ID 5 has always appeared in 10 consecutive historical frames, and count the matching status of the Car target with 2D true value frame ID 5 in 10 consecutive historical frames. Figure 4 As shown, the Car target with 2D truth frame ID 5 has matched the Car target with 3D truth projection frame ID 3 5 times, matched the Car target with 3D truth projection frame ID 6 4 times, and matched the Car target with 3D truth projection frame ID 7 once in 10 consecutive historical frames. The historical matching results are arranged in descending order.

[0120] From the descending results, we can see that in the historical matching stage, the Car target with 2D truth frame ID 5 matches the Car target with 3D truth projection frame ID 3 the most times. This is the process of counting the matching information of the 3D projection frame based on the 2D truth target frame. It is also necessary to count the matching information of the 2D truth target frame based on the 3D projection frame.

[0121] Further based on the result that the Car target with 2D truth box ID 5 matches the Car target with 3D projection ID 3 the most times in the above historical matching stage, the matching information of the Car target with 3D truth projection box ID 3 in the historical matching stage is further counted.

[0122] Assuming that the Car target with the ID of 3 in the 3D true value projection frame is also matched the most times with the Car target with the ID of 5 in the 2D true value frame in the historical matching information, the Car target with the ID of 5 in the 2D true value frame is matched with the Car target with the ID of 3 in the 3D true value projection frame in the nth frame;

[0123] If the Car target with 3D truth projection frame ID 3 is not matched the most times with the Car target with 2D truth frame ID 5 in the historical matching information, further based on the statistical results of the above step, the historical matching information statistics of the Car target with 3D truth projection frame ID 6, which has the second most matches with the Car target with 2D truth ID 5, are performed, and the second candidate object is determined in this way, thereby improving the accuracy of the matching results.

[0124] In an optional implementation manner, after determining the object matching relationship between the first candidate object and the second candidate object, the method includes:

[0125] S1, acquiring a second annotation box corresponding to a second candidate object that meets a historical matching condition, and a second verification annotation box corresponding to a second verification candidate object that matches the first candidate object and is determined according to key frame matching information;

[0126] S2, when the area of ​​the overlapping region between the second verification annotation box and the second annotation box is greater than or equal to the area threshold, determining that the matching result is correct;

[0127] S3, when the area of ​​the overlapping area between the second verification annotation box and the second annotation box is less than the area threshold, determine the second verification annotation box as the second candidate object to be corrected; and modify the second candidate object to be corrected that matches the first candidate object to be corrected into the second correction object according to the key frame matching information corresponding to the key frame where the second candidate object to be corrected is located.

[0128] The above steps S1-S3 are described below in a specific implementation manner:

[0129] Find the nearest key frame based on the current nth frame. The key frame is manually extracted and sent for manual annotation. The number of key frames in a continuous frame segment is fixed. The number of key frames and the key frame extraction rules are set in advance. If the nearest key frame cannot be found, no key frame correction is performed. Here, it is assumed that based on the nth frame, the nearest key frame m is found, and the key frame matching information corresponding to key frame m is obtained.

[0130] For example, the second candidate object that meets the historical matching conditions and matches the first candidate object is the Car target with a 2D truth value ID of 5 in the current nth frame. The 2d truth value information (second verification candidate object) that matches the first candidate object is determined based on the joint annotation matching information of the 2D continuous frame truth value and the 3D continuous frame truth value in the key frame m (for example, the key frame matching information). It is determined whether the overlapping area between the 3d truth value box of the second verification candidate object in the key frame matching information and the 2d truth box corresponding to the Car target with a 2D truth value ID of 5 is greater than the threshold condition. If the threshold condition is met, the matching result can be verified to be correct.

[0131] Through the above implementation, it can be confirmed whether the Car target with 2D true value ID 5 in the nth frame of the current continuous frame segment also exists in the key frame m; if so, Figure 5 As shown, for example, according to the IoU overlap value, it is determined that the Car target with a 2D true value box ID of 5 corresponds to the Car target with a 2D true value box ID of 14 in the key frame joint annotation result;

[0132] Then the matching result is verified, that is, the Car target with 3D true value projection frame ID 15 matched with the Car target with 2D true value frame ID 14 determined according to the key frame matching information (key frame joint annotation result) in key frame m is calculated, and whether the IoU with the Car target with 3D true value projection frame ID 3 in the nth frame in the matching result is greater than the threshold. If it is greater than the threshold condition, it is considered that the Car target with 2D true value frame ID 14 matched by the key frame m joint annotation and the Car target with 3D true value projection frame ID 15 matched therewith are completely corresponding to the Car target with 2D true value frame ID 5 and the Car target with 3D true value projection frame ID 3 matched therewith in the nth frame, thereby confirming that the result based on the third round of matching for the nth frame is correct, that is, the matching result of the nth frame with the Car target with 2D true value frame ID 5 and the Car target with 3D true value projection frame ID 3 is correct.

[0133] like Figure 6 As shown, in the n-10th frame of the left matching result, the Car target with a 2D true value ID of 5 matches the 3D true value projection frame ID of 7, and this match only occurs in one frame, which is the least number of times. Therefore, the second candidate object with a 3D true value projection frame ID of 7 is determined as the second candidate object to be corrected;

[0134] Further optimization is performed with the help of key frames. Assume that the key frame closest to the n-10th frame is key frame m. For example, the Car target with 2D truth frame ID 5 in the nth frame corresponds to the Car target with 2D truth frame ID 14 in the manual joint matching annotation (key frame matching information) of the key frame. The Car target with 3D truth projection frame ID 15 in key frame m that matches the Car target with 2D truth frame ID 14 is calculated based on the Car target with 3D truth projection frame ID 15. The IoU of all 3D projection frames in the n-10th frame is calculated, and then the matching is sorted from large to small. It is found that among all 3D projection frames in the n-10th frame, the Car target with ID 6 has the largest overlap with the Car target with 3D truth projection frame ID 15 in key frame m. Based on this, the 3D truth projection frame ID of the n-10th frame that matches the Car target with 2D truth frame ID 5 is modified to 6, such as Figure 6 The matching results are shown on the right.

[0135] Through the above implementation method, the successfully matched matching results are verified to determine whether they are reasonable, and unreasonable matches are corrected to correct results, thereby improving the accuracy of the selected model evaluation results and the speed of perception model optimization iteration.

[0136] In an optional implementation, model evaluation is performed on the perception models to be evaluated respectively according to the multiple target road object sets, and model evaluation results respectively matching the multiple target road object sets are obtained, including:

[0137] S1, inputting multiple target road object sets into the perception model to be evaluated to obtain model prediction results that match the multiple target road object sets respectively;

[0138] S2, performing model evaluation on the perception model to be evaluated according to the model prediction results and true value data respectively matched with each target road object set, and obtaining a category evaluation result, wherein the category evaluation result is used to indicate the evaluation results corresponding to the perception targets of multiple categories;

[0139] S3, based on the category evaluation results, determine the category of the perception target to be evaluated, and use the road data collected by multiple continuous frame segments to evaluate the perception model to be evaluated in the dimension of the perception target category to obtain the model evaluation results.

[0140] It should be noted that in the above step S1, during the driving process of the vehicle, the camera sensor can be used to collect road information, and the road information can be collected as a road image. As a visual method, the road image can include a large amount of traffic environment information, such as traffic signal signs, road lane line markings, and obstacles ahead, etc. Multiple frames of road images can be determined as road data.

[0141] In an optional embodiment, a monocular camera may be used to collect images of the road on which the vehicle is traveling. When the vehicle is traveling, the scene within the collection range may be photographed at a certain collection frequency, and a road image may be generated based on the camera's internal and external parameters. The road data may be a plurality of road images collected by the vehicle in a series of continuous time frames, or may be road images collected by the vehicle under specific conditions. For example, a sequence of road images collected by the vehicle under specific weather conditions may be obtained, and the road images collected in multiple time frames when the vehicle is traveling under specific weather conditions may be used as road data.

[0142] It should be noted that the target road object set can be data acquired in real time during vehicle driving, or can be road data collected during vehicle driving and stored in a server. The target road object set can be two-dimensional data or three-dimensional data, and its form is not limited to images. Point cloud data can also be used as road data.

[0143] It should be noted that the perception model needs to work reliably in various traffic scenarios. By evaluating the perception model, the effect of the perception model can be quantitatively measured to intuitively determine the effectiveness of the perception model in different scenarios. The perception model can perform different tasks, such as 2D or 3D object detection, semantic segmentation, and lane line detection. Different perception models can be used as perception models to be evaluated using the above method.

[0144] In an optional implementation, inputting the target road object set into the perception model to be evaluated can be a re-injection of the perception model to be evaluated, and the sensor data (including images, radar signals, etc.) collected historically by the vehicle can be re-input into the perception model to be evaluated to simulate the real-world driving scene. The model prediction results output by the perception model to be evaluated can be determined according to the specific tasks performed by the perception model, such as environmental perception results (obstacle detection, traffic sign recognition, lane line detection, etc.), vehicle status information (vehicle speed, position, acceleration, etc.) or driving behavior (steering, braking, acceleration, etc.).

[0145] It should be noted that scene labels can be weather labels (sunny, rainy, snowy, etc.), light labels (night, day, dusk, etc.), road scene labels (city, tunnel, elevated, etc.), etc. How to obtain road data with scene labels will be described in detail below and will not be repeated here.

[0146] In the above steps S2-S3, the perception model to be evaluated is evaluated according to the model prediction results and true value data respectively matched with each target road object set to obtain a category evaluation result, wherein the category evaluation result is used to indicate the evaluation results corresponding to the perception targets of multiple categories; based on the category evaluation result, the category of the perception target to be evaluated is determined, and the perception model to be evaluated is evaluated on the dimension of the category of the perception target to be evaluated using road data collected by multiple continuous frame segments to obtain a model evaluation result.

[0147] The category evaluation results are used to indicate the evaluation results corresponding to multiple categories of perception targets. In an optional implementation, the obtained model prediction results are matched with the true value data, and the evaluation results corresponding to the evaluation set are calculated according to preset evaluation parameters. The evaluation results can be category evaluation results, and the category evaluation results can indicate the categories of all objects and participants around the ego vehicle of specific categories in different scenarios. The categories of objects and participants can be obstacles or traffic participants. Obstacles can be static objects that block the vehicle's driving path, such as trees, buildings, walls, rocks or other vehicles parked in non-driving areas. Traffic participants can be dynamic objects, which can be other vehicles, motorcycles, bicycles, pedestrians, animals, etc., which can move around the ego vehicle and may affect the vehicle's driving path.

[0148] The categories of category evaluation results can be set according to actual needs. For example, large categories such as traffic participants can be identified to obtain category evaluation results of static obstacles or category evaluation results of dynamic traffic participants. Small categories such as pedestrians and vehicles can also be identified to obtain category evaluation results of pedestrians or category evaluation results of vehicles. Through the evaluation results of perception targets of different categories, it can be determined which category of perception targets the perception model to be evaluated is not effective for, and then the evaluation data can be selected in a targeted manner, which effectively improves the efficiency and accuracy of model training.

[0149] In an optional implementation, after obtaining the category evaluation result, the category evaluation result can be compared with a preset threshold, and the category of the perceived target corresponding to the result less than the preset threshold in the category evaluation result is determined as the category of the perceived target to be evaluated. The category of the perceived target to be evaluated can be a perceived target with poor model perception effect under road data collected using multiple continuous frame segments.

[0150] In an optional implementation, after determining the target road object set according to the object matching results between the first road object set and the second road object set corresponding to each of the plurality of data collection conditions, the method includes:

[0151] S1, performing model evaluation on the perception model to be evaluated using model prediction results and true value data corresponding to each of the road data sets collected from multiple continuous frame segments, and obtaining model evaluation results corresponding to each of the multiple continuous frame segments;

[0152] S2, determining, among the model evaluation results corresponding to each of the plurality of continuous frame segments, a model evaluation result that satisfies a preset condition as an evaluation result to be determined;

[0153] S3, determining a repeated scene label in the scene labels matched by at least one target road object set corresponding to the evaluation result to be determined as a target scene label;

[0154] S4, selecting target road data collected in a single frame from a target road object set matching the target scene label;

[0155] S5, determining the model prediction result and the true value data corresponding to the target road data, and matching the model prediction result and the true value data corresponding to the target road data to obtain a single-frame evaluation result.

[0156] In the above steps S1-S5, the model prediction results and true value data corresponding to each of the road data sets collected by multiple continuous frame segments are used to perform model evaluation on the perception model to be evaluated, so as to obtain the model evaluation results corresponding to each of the multiple continuous frame segments; among the model evaluation results corresponding to each of the multiple continuous frame segments, the model evaluation results that meet the preset conditions are determined as the evaluation results to be determined; the repeated scene labels in the scene labels matched by at least one target road object set corresponding to the evaluation result to be determined are determined as the target scene labels; the target road data collected by a single frame are selected from the target road object set matching the target scene label; the model prediction results and true value data corresponding to the target road data are determined, and the model prediction results and true value data corresponding to the target road data are matched to obtain the single frame evaluation results.

[0157] In an optional embodiment, the perception model to be evaluated is evaluated with model prediction results and true value data corresponding to each of the road data collected from multiple continuous frame segments to obtain evaluation results corresponding to each of the multiple continuous frame segments; among the evaluation results corresponding to each of the multiple continuous frame segments, the evaluation results that meet preset conditions are determined as the evaluation results to be determined; and repeated scene labels in the scene labels carried by the road data corresponding to the evaluation results to be determined are determined as target scene labels, for example, repeated labels are labels for which the number of scene labels with inaccurate prediction results is greater than a preset threshold.

[0158] In an optional implementation, after the above-mentioned evaluation based on multiple data segments, if the evaluation results indicate that the overall indicators of the pedestrian category are poor, the pedestrian category can be determined as the perception target category to be evaluated, and a single data segment can be evaluated, and the evaluation results of the pedestrian category on each continuous frame segment can be screened. The road data corresponding to the results that meet the preset conditions (such as less than the preset threshold) after the single data segment evaluation is extracted, and the scene labels contained in the road data are determined to determine which scene label has the poor effect of the perception model on the pedestrian category.

[0159] In an optional implementation, repeated scene tags among the screened scene tags may be determined as target scene tags, or all screened scene tags may be determined as target scene tags. For example, after evaluating the pedestrian category, if the evaluation result indicates that the evaluation result meets the preset conditions in the night and rainy scenes, then the night and rainy days may be determined as target scene tags.

[0160] Through the above implementation of the present application, after the category of the perceived target to be evaluated is determined by the evaluation of multiple data segments, a single data segment can be used to determine in which scenarios the evaluation effect of the perceived target category to be evaluated is poor, so as to more accurately determine the perceived weaknesses of the perceived model to be evaluated. This improves the accuracy of the model evaluation and makes the training data obtained subsequently more targeted, thereby improving the efficiency of the iterative update of the model.

[0161] The road data collected in a single frame can be target road data collected in a single frame selected from road data containing target scene labels after single data segment evaluation, or it can be target road data collected in a single frame selected from segments containing perception target categories to be evaluated after multiple data segment evaluation.

[0162] In an optional implementation, evaluation items can be set as needed when using single-frame data for evaluation. For example, the ability of the perception model to be evaluated to detect small targets can be specifically evaluated, and the data between the vehicle and the specified target can be determined in the single-frame data and detected.

[0163] Fig. 9 is a schematic diagram of an optional perceptual model evaluation method according to an embodiment of the present application; Fig. 9 As shown, after obtaining road data of multiple continuous frame segments, a multi-data segment evaluation can be performed, and the perception targets to be evaluated, such as pedestrians, vehicles, etc., can be determined based on the multi-data segment evaluation. After determining the perception target to be evaluated, the road data of the continuous frame segments containing the perception target to be evaluated can be extracted, and each continuous frame segment can be input into the perception model to be evaluated to obtain the evaluation results corresponding to each continuous frame segment. The ones that meet the preset conditions are determined as the evaluation results to be determined, and the corresponding target scene labels are determined. After that, a single frame of road data containing the perception target to be evaluated and the target scene label can be found and input into the perception model to be evaluated to perform a single-frame evaluation on the perception model to determine whether the perception model has poor detection effect on the perception target to be evaluated in the scene corresponding to the target scene label, so as to achieve multiple verifications.

[0164] Through the above implementation of the present application, the target scene label and the perception target to be evaluated that matches the evaluation result to be determined can be determined after the multi-data segment evaluation and the single data segment evaluation, and the evaluation results that do not meet the preset conditions can be screened. The perception model can be evaluated again through a single frame of road data containing the perception target to be evaluated and the target scene label to detect the effect of the perception model. A complete evaluation result can be formed with the multi-data segment evaluation and the single data segment evaluation, and it can be verified to obtain an accurate perception model evaluation result. The accuracy of the perception model evaluation is improved.

[0165] It should be noted that the above process is the overall process of evaluating the perception model to be evaluated. The following will explain in detail how to evaluate the road data of each continuous frame segment in the specific evaluation process.

[0166] In an optional implementation, based on multiple target road object sets, model evaluation is performed on the perception models to be evaluated respectively to obtain model evaluation results that match the multiple target road object sets respectively, further comprising:

[0167] S1, determining a distance between at least one reference obstacle in at least one target road object set and the ego vehicle according to true value data corresponding to at least one target road object set, wherein the reference obstacle is used to indicate static obstacles and dynamic obstacles around the ego vehicle;

[0168] S2, dividing the at least one reference obstacle into at least one reference obstacle set according to a distance between the at least one reference obstacle and the vehicle and a preset distance threshold;

[0169] S3, performing model evaluation on the perception model to be evaluated in sequence according to the target road object set corresponding to at least one reference obstacle set, and obtaining a model evaluation result, including at least one of the following:

[0170] Method 1: Calculate the difference between the model prediction result and the distance in the true value data to obtain the absolute distance error;

[0171] Method 2: Calculate the ratio between the absolute distance error and the distance in the true value data to obtain the relative distance error;

[0172] Method 3: Calculate the difference between the speed in the model prediction result and the true value data to obtain the absolute speed error.

[0173] In the above steps S1-S2, the distance between at least one reference obstacle in at least one target road object set and the vehicle is determined based on the true value data corresponding to at least one target road object set, wherein the reference obstacle is used to indicate static obstacles and dynamic obstacles around the vehicle; and the at least one reference obstacle is divided into at least one reference obstacle set based on the distance between the at least one reference obstacle and the vehicle and a preset distance threshold.

[0174] As mentioned above, the perception model can detect and identify static and dynamic obstacles around the ego vehicle. The true value data can be a combination of two-dimensional data and three-dimensional data. Therefore, the true value data contains depth information, which can indicate the distance between the reference obstacle and the ego vehicle.

[0175] In an optional implementation, the reference obstacle can be segmented according to the distance between the reference obstacle and the vehicle to obtain different segmented intervals. For example, it can be divided into five intervals of 0~10m, 10~30m, 30~60m, 60~100m, and 100~150m, and the perception model is evaluated within the five intervals. For example, when evaluating vehicle category targets, the distance between the reference obstacle (surrounding vehicle) and the vehicle of each vehicle category in the continuous frame road data can be obtained, and it can be divided into five distance intervals according to the distance between the vehicle and the surrounding vehicles. The general evaluation results can be calculated for the perception targets of the vehicle category first.

[0176] In an optional implementation, after determining the category to be evaluated, the recall rate, precision rate, multiple object tracking accuracy (Multiple Object Tracking Accuracy, MOTA), multiple object tracking precision (Multiple Object Tracking Precision, MOTP), etc. can be calculated. For example, the following formula can be used to calculate the recall rate:

[0177]

[0178] To calculate the accuracy, you can use the following formula:

[0179]

[0180] To calculate MOTA, you can use the following formula:

[0181]

[0182] To calculate MOTP, you can use the following formula:

[0183]

[0184] TP stands for true positive, i.e., the target correctly detected and tracked by the perception model. In a given frame, if the intersection / intersection ratio of the target box predicted by the perception model and the true value data exceeds a certain threshold, it can be considered correct.

[0185] FP stands for false positive, i.e., the perception model mistakenly detects background or other non-target objects as targets.

[0186] FN represents false negative examples, i.e., actual targets that the perception model fails to detect. They are targets that actually exist in the ground-truth data, but are not detected by the perception model in a given frame.

[0187] ID represents ID switching error, i.e. the perception model mistakenly changes the ID of the target during tracking.

[0188] GT stands for true value, that is, true value data.

[0189] d ti represents the positioning error of the i-th matching pair in the t-th frame, c t represents the number of matching pairs in the t-th frame.

[0190] This solution can also match the model prediction results corresponding to the reference obstacle with the true value data corresponding to the reference obstacle. In the above step S3, multiple combinations are included to obtain the evaluation results.

[0191] In an optional implementation, after a category of targets has been evaluated using the above-mentioned general indicators, they can be evaluated by distance segment. The set of errors between the perceived target (reference obstacle) and the matching target in the true data can be calculated, and the measurement can be performed in segments according to the distance. By evaluating the distance in segments, the performance changes of the perception model in different distance ranges can be captured more finely, thereby more accurately evaluating the error performance of the perception model. It can also distinguish between distant targets and close targets, so the depth error of distant targets is larger. By evaluating the targets in segments, the influence of the long-distance error on the overall performance can be avoided.

[0192] In an optional embodiment, the model can be evaluated using an absolute distance error, a relative distance error, and an absolute speed error. The absolute distance error can be a set of absolute lateral and longitudinal distance errors, including a lateral error indicating the lateral distance between the center of mass of the vehicle and the true position, and a longitudinal error indicating the deviation of the true position of the vehicle along the desired trajectory. The relative distance error can be a set of relative lateral and longitudinal distance errors, including a relative lateral error indicating the lateral distance of the vehicle relative to the center line of the lane or the desired trajectory, and a relative longitudinal error indicating the deviation of the front and rear positions of the vehicle on the desired trajectory. The absolute speed error can be a set of absolute lateral and longitudinal speed errors, including a lateral speed error indicating the difference between the actual lateral speed of the vehicle and the desired lateral speed, and a longitudinal speed error indicating the difference between the actual longitudinal speed of the vehicle and the desired longitudinal speed.

[0193] The error can be calculated using the following formula:

[0194]

[0195] Among them, d i represents the true value data, represents the predicted value of the perception model, and M can represent the number of perceived targets in the current distance segment.

[0196] In an optional implementation, after obtaining the error set of the target in the segmented interval, indicators such as mean and variance may also be calculated, such as the root mean square error:

[0197]

[0198] In an optional implementation, when performing segmented evaluation, scene tags may also be combined for evaluation. Fig.10 FIG. 1 is a schematic diagram of another optional perceptual model evaluation method according to an embodiment of the present application; as mentioned above, the scene label may include multiple types. Fig.10 As shown, it is an example that multiple continuous frame segments can be divided according to preset scene labels. For a continuous frame segment, it can contain multiple scene labels. For example, for a segment, it can be sunny, night, and city. It can be divided into sunny, night, and city according to the labels, and the segments with the same labels are evaluated. For example, a sunny day may contain 8 segments, and the 8 segments can be evaluated in segments. And mixed evaluation of multiple scene labels can be performed, such as forming an evaluation group with the segments corresponding to the rainy day, night, and tunnel labels, and performing segmented evaluation.

[0199] Through the above implementation of the present application, the pre-detected data set can be freely set according to the scene label, and can be divided according to the distance during the evaluation. After determining the general indicators, the error and other indicators can be calculated for the targets of different segmented distances to evaluate the perception model. The performance of the perception model at different distances can be more accurately indicated, so as to optimize the model in a targeted manner.

[0200] In an optional implementation, based on multiple target road object sets, model evaluation is performed on the perception models to be evaluated respectively, and after obtaining model evaluation results that match the multiple target road object sets respectively, the method includes:

[0201] S1, determine the category of the perceived target to be evaluated and the target distance threshold interval;

[0202] S2, obtaining target road data that belongs to the target scene label and the perceived target category to be evaluated is within the target distance threshold interval, and determining the target road data as updated training data;

[0203] S3, using the updated training data to train the perception model to be evaluated.

[0204] In an optional implementation, multiple rounds of evaluation can be used to determine in which categories, distance intervals, and scene labels the perception model does not work well. For example, if the current perception model does not work well for pedestrians in a dark scene within the range of 10 to 30 meters, the number of target road data in the training set of the current perception model that contains target scene labels and target categories within the target distance threshold interval can be detected. When it is determined that the number of qualified road data in the acquired training set is less than the quantity threshold, the target road data that contains the target scene label and the target category within the target distance threshold interval can be acquired in the storage and used as training data to update the perception model.

[0205] Through the above implementation of the present application, when it is determined that the perception model has poor prediction effect on the data of those dimensions, the number of data that meet the requirements in the data contained in the current training set can be determined, and when the number is insufficient, it can be supplemented in a targeted manner. In this way, with the same amount of data, the perception model can be iteratively updated in a targeted manner to improve the performance of the perception model.

[0206] It should be noted that before evaluating the perception model, the present application will also cut the road data in the reference road data that meets the preset data conditions to obtain road data in multiple continuous frame segments that are consistent in time; perform scene recognition on the road data to obtain corresponding scene labels, and add the scene labels to the road data to obtain updated road data.

[0207] In an optional implementation, a true value vehicle can be used for road sampling, and reference road data can be collected through a variety of high-precision sensors carried by the true value vehicle. Sensors include laser radar, cameras, millimeter wave radar, GPS, etc. The reference road data can be stored in the form of rosbag data packets, which can carry accurate timestamps and ensure the integrity of the reference road data. After obtaining the reference road data, data screening can be performed on it, and high-value data can be selected from it. Reference indicators can be set in advance for high-value data, such as vehicle interaction, pedestrian crossing, lane change and other conditions can be set as reference indicators. The reference road data can be divided into continuous frame segments and sliced.

[0208] In an optional implementation, after the data is expanded, scene tags can be added to the slice segments, and scene recognition can be performed on the slice segments to determine their scene tags and mark them. After the road data is acquired, it can be stored in the cloud or a storage device.

[0209] Through the above implementation of the present application, reference road data can be obtained through true value vehicle road sampling, and the reference road data can be processed to obtain marked road data, which is convenient for evaluating the perception model, and when the perception model is updated, training data can be obtained more quickly. By ensuring the accuracy of the data, the accuracy of subsequent perception model evaluation is improved.

[0210] Alternatively, if Fig.11 As shown in the figure, the collected road data can be screened and processed to obtain true value data. When evaluating the model, the road data can be obtained and fed back into the perception model to obtain the model prediction results. The model evaluation results can be obtained based on the model prediction results and the true value data. According to the scene labels with poor results in the model evaluation results, data is screened again to obtain the training set of the perception model, and the perception model is updated and iterated.

[0211] Through the above implementation of the present application, true value data of two-dimensional and three-dimensional information can be obtained, which can make the perception model evaluation avoid the problem that the three-dimensional space cannot be associated with the true value due to the large error in the distance measurement of long-distance targets, and can also provide more information on the basis of two-dimensional data. And it can be divided into distance measurement and speed measurement evaluation, and classification based on scene labels can be performed to obtain more detailed evaluation results and provide clearer guidance for subsequent model updates. The update of the perception model can achieve significant results based on less data.

[0212] The following three modules are used to explain this solution:

[0213] First is the data slicing module:

[0214] like Fig.12 As shown, in the data slicing module, the data mainly comes from various sensor data collected by real-value vehicles and roads, including lidar data 1202, camera data 1204, GNSS data 1206, IMU data 1208, and other sensor data 1210.

[0215] The various sensor data collected by the road will be stored in the form of RosBag data packets 1212. After data screening, high-value scene data will be selected for data expansion. The high-value scene data slices 1214 are continuous frame segments with consistent segment lengths.

[0216] In the data expansion stage, scene labels are first added to the slice segments. The scene labeling is mainly done by marking the road scene, weather, lighting conditions and other scenes on the selected data slices through the scene recognition model 1216. Each data slice segment corresponds to a set of scene labels. After that, the expanded image data and point cloud data are aligned 1218 (timestamp alignment) and stored in the cloud.

[0217] The expanded data of the data slices stored in the cloud are subjected to frame extraction, and the extracted frames are regarded as key frames, which are manually labeled, i.e., data frame extraction and labeling 1220. The returned manual labeling results can be mapped back to the original slice data segments and will be used for the subsequent training of the cloud-based true value large model.

[0218] All data expanded from the data slices and the key frame data returned from frame extraction and labeling will be stored in the object storage 1222, which is convenient for the subsequent acquisition of cloud-based truth model training data and scene label data.

[0219] Then comes the joint annotation module:

[0220] Different from the current method of obtaining true values ​​for perception model algorithm evaluation, the cloud-based truth value large model trained in this application supports the output of 2D continuous frame truth value results and 3D continuous frame truth value results. In order to solve the problem of serious offset in the position and size of the 3D truth value box of the 3D continuous frame truth value, and the possible inability to output the truth value due to the sparse point cloud at distant targets, after the 2D and 3D continuous frame truth values ​​are generated, three rounds of matching and one round of key frame manual annotation auxiliary optimization algorithms are designed to perform joint annotation of 2D and 3D continuous frame truth values.

[0221] Joint matching annotation can also effectively solve the problem that the vehicle-side algorithm model has large errors in measuring the distance to distant targets in pure 3D space, resulting in the inability to associate with the true value during evaluation. This is because through joint matching annotation, this step can be associated in a 2D frame.

[0222] The unified perspective of joint truth matching is the 2D image perspective. First, based on the camera intrinsic parameters and the joint calibration parameters of the camera and lidar, the 3D target truth frame is projected to the 2D image. This step projects the 3D frame to the 2D image. Due to the maximum circumscribed rectangle, the projected 2D frame will expand, which is normal.

[0223] Secondly, the 2D frame projected onto the 2D image from the 3D target truth frame needs to be screened. Due to the characteristics of point clouds, the cloud-based truth model can output targets that are completely invisible in 2D images based on point cloud data. In this step, the truth targets that are completely invisible in 2D images need to be eliminated, because the framework of the entire joint matching annotation process is set within the 2D image range. Three discrimination conditions are set, based on the target truth value being larger when near and smaller when far, and the fact that the depth information of the 3D truth frame is still retained despite the projection operation on the 3D truth frame. The size of the 2D projection frame, the distance, and the relative relationship of the center pixel coordinates of the frame can be used to eliminate the 3D projection frames of the truth targets that are completely invisible from the 2D image perspective.

[0224] like Figure 3 As shown, taking the front view image as an example, the dotted box represents the front view image, the solid line truth box on the left is the 3D truth box corresponding to the vehicle object 302 projected onto the 2D image, and the dotted line truth box on the left is the 3D truth box corresponding to the vehicle object 304 projected onto the 2D image. The solid line truth box on the left surrounds the dotted line truth box on the left, and the vehicle object 304 is farther away from the vehicle 308 longitudinally, so the dotted line truth box on the left can be eliminated because this object cannot be seen under the front view camera.

[0225] By eliminating the 3D true value boxes that are not visible from the camera's perspective and projecting them onto the 2D image, we enter the first round of matching.

[0226] In the first round of matching, the target category is first divided into 2D truth boxes and 3D projection boxes, and the matching process is framed in the same category of 2D truth boxes and 3D projection boxes. The first round of matching is mainly based on IoU matching, and direct matching is performed by calculating whether the numerical value of the bounding_box_overlap of the 2D truth box and the 3D projection box is greater than the set threshold. If the IoU of the 2D truth box and the 3D projection box of the same category is greater than the set threshold, the 2D truth box and the 3D projection box will be associated. Because we retain the information of the original 3D truth box of the 3D projection box, we can give the 2D truth box 3D information, such as horizontal and vertical depth and horizontal and vertical speed information, and because it is a continuous frame truth, the target tracking ID of the 2D truth box and the 3D projection frame can also be associated. In this round of matching, targets that are close to the vehicle and not too coupled with other targets will be jointly matched. However, because this round of matching is completely based on the IoU value, only when the IoU result is greater than the set threshold will it be associated, which will cause many missed matches. Therefore, the first round of matching is a coarse match.

[0227] like Figure 7 As shown, box 704 represents the 3D true value projection box, and box 702 represents the 2D true value box. Since the first round of matching will only associate 2D and 3D true value targets whose IoU values ​​are greater than the selected threshold, there will be many missed matching targets in one round of matching results, because the 3D true value projection box of the farther target is not only expanded but also offset, and there will be no IoU information, so it cannot be matched.

[0228] In order to solve the problem of complex scenes with many true targets or targets occluding or overlapping each other, a two-round matching method is proposed.

[0229] The second round of matching is based on the matching process after eliminating the results of the first round of matching. This round of matching will only associate the targets whose IoU values ​​of the 2D true value box and the 3D projection box are not 0. This step will sort the IoU values ​​of the 2D true value box from large to small, calculate the Euclidean distance of the pixel coordinates of the center point of the 2D true value box and the 3D projection box, and sort them. Matching is performed based on the IoU sorting results and the center point Euclidean distance sorting results.

[0230] Double judgment is performed on the two types of sorting results. For example, the 2D truth box with index 1 has the largest IoU value with its 3D truth projection box with index 3, and the 2D truth box with the closest Euclidean distance to the center pixel coordinates of the 3D truth projection box with index 3 has index 1. That is, double judgment is performed, and the 2D truth box with index 1 and the 3D truth projection box with index 3 are associated and matched.

[0231] The second round of matching is still to solve the matching of targets whose IoU values ​​between the 2D true value box and the 3D projection box are not 0, and the third round of matching is mainly to solve the matching of targets whose IoU values ​​are 0.

[0232] First, based on the elimination of the first and second round matching results, the 2D true value boxes and 3D target boxes with an IoU value of 0 are screened out, and the matching results of each 2D true value box in the historical 10 consecutive frames (if less than 10 frames, from the start frame to the current frame) are calculated for analysis, and the joint annotation information of the key frames is combined to assist in optimization. The specific matching ideas are as follows:

[0233] First, assume that the Car target with 2D true value ID 5 in the nth frame has no 3D projection frame target matching it after the first and second rounds of matching. That is, the IoU between the Car target with 2D true value ID 5 and all 3D projection frames is 0. At the beginning of the third round of matching, we first count whether the Car target with 2D true value ID 5 has always existed in the past 10 consecutive frames. If it has always existed, it is the most ideal situation. If it has not always existed, we only count the continuous scenes where the Car target with 2D true value ID 5 in the past has appeared continuously to the current frame, such as the n-5th frame to the nth frame.

[0234] Now assume that in the 10 consecutive historical frames, the Car target with 2D true value ID 5 has always appeared, and count the matching of the Car target with 2D true value ID 5 in the 10 consecutive historical frames. Figure 4As shown in the figure, the Car target with 2D truth value ID 5 has matched with the Car target with 3D projection ID 3 5 times, 4 times, and 1 time in 10 consecutive historical frames. We sort the historical matching results in descending order. From the descending results, we can see that the Car target with 2D truth value ID 5 matches the Car target with 3D projection ID 3 the most times in the historical matching stage. This is the process of counting the matching information of the 3D projection frame based on the 2D truth value target frame. We also need to count the matching information of the 2D truth value target frame based on the 3D projection frame.

[0235] For example, based on the result that the Car target with 2D true value ID 5 matches the Car target with 3D projection ID 3 the most times in the above historical matching stage, we will count the matching information of the Car target with 3D projection ID 3 in the historical matching stage. Assuming that the Car target with 3D projection ID 3 also matches the Car target with 2D true value ID 5 the most times in the historical matching information, we will match the Car target with 2D true value ID 5 with the Car target with 3D projection ID 3 in the nth frame; if the Car target with 3D projection ID 3 does not match the Car target with 2D true value ID 5 the most times in the historical matching information, then we will count the historical matching information of the Car target with 3D projection ID 6, which has the second most matches with the Car target with 2D true value ID 5, based on the statistical results of the above step, and so on.

[0236] After the above two steps, assuming that the Car target with 2D true value ID 5 has been matched with the Car target with 3D projection ID 3, the matching sequence is corrected and optimized based on the manual joint annotation results of the key frames. The specific ideas are as follows:

[0237] First, we look for the nearest key frame based on the current nth frame. The key frame is manually extracted and sent for manual annotation. The number of key frames in a continuous frame segment is fixed. The number of key frames and the key frame extraction rules are set in advance. If the nearest key frame cannot be found, no key frame correction is performed. Suppose that the nearest key frame m is found based on the nth frame. Based on the joint annotation information of the key frame, we can know the joint annotation matching information of the 2D continuous frame truth value and the 3D continuous frame truth value of the key frame (such as Figure 5 As shown in the joint annotation result of the key frames on the right, based on the joint annotation information of the key frames, the IoU is calculated for the Car target with the 2D truth value ID of 5 in the current nth frame and the 2D truth box annotated by the joint matching of the key frames. The purpose is: first, to confirm whether the Car target with the 2D truth value ID of 5 in the nth frame of the current continuous frame segment also exists in the key frame m; second, if it exists, Figure 5, the Car target with 2D truth value ID 5 corresponds to the Car target with 2D truth value ID 14 marked by the manual joint matching of the key frame, and then calculates whether the IoU of the Car target with 3D truth value ID 15 matched with the Car target with 2D truth value ID 14 in the key frame m and the Car target with 3D projection ID 3 in the nth frame is greater than the threshold. If it is greater than the threshold condition, it is considered that the Car target with 2D truth value ID 14 and the Car target with 3D truth value ID 15 matched by the joint annotation of the key frame m are completely corresponding to the Car target with 2D truth value ID 5 and the Car target with 3D projection ID 3 matched by the nth frame, thereby confirming that the result based on the third round of matching is correct for the nth frame.

[0238] Then, yes Figure 6 In the n-10th frame, the Car target with 2D truth ID 5 matches the 3D projection truth ID 7, and this match only appears in one frame, so the key frame is used for optimization. Assume that the key frame closest to the n-10th frame is also key frame m. This assumption is valid in most cases because there are few key frames and the span is large.

[0239] Based on the previous step, it is known that the Car target with 2D truth value ID 5 corresponds to the Car target with 2D truth value ID 14 marked by the manual joint matching of the key frame. The Car target with 3D truth value ID 15 in the key frame m matches the Car target with 2D truth value ID 14. Based on the Car target with 3D truth value ID 15, the IoU of all 3D projection frames in the n-10th frame and it is calculated, and then sorted from large to small for matching. It is concluded that in all 3D projection frames of the n-10th frame, the Car target with ID 6 corresponds to the Car target with 3D truth value ID 15 of the key frame m. Based on this, the 3D projection truth value ID of the Car target matching the 2D truth value ID 5 in the n-10th frame is modified to 6. The modified result is as follows Figure 6 Shown on the right.

[0240] Flowchart Figure 8 The above process is explained as follows:

[0241] On the one hand, S802 is executed to obtain the area of ​​the 2D truth box; S804, the 3D truth box point cloud xyz determines the distance before and after; S806, the 2D truth box border coordinates determine the relative position;

[0242] Execute S808 according to S802-S806, and comprehensively judge the inclusion relationship to determine the occlusion;

[0243] On the other hand, S810 is executed to obtain the true value of the 2D continuous frame;

[0244] S812, obtaining 3D continuous frame true values;

[0245] S814, projecting the 3D target truth frame onto the 2D image; executing S816 in conjunction with S808 to filter out objects that are not completely blocked;

[0246] On the other hand, S818, manual labeling of key frames; combining S810 and S816 to perform S820, three rounds of matching;

[0247] S822, detection result of the perception algorithm model to be evaluated; and executing S824 in combination with step S820 to associate the detection result.

[0248] In the evaluation and analysis module: through the joint matching annotation of the true values ​​of 2D and 3D continuous frames and the true values ​​of continuous frame fragments, the 2D true value box and the 3D true value box are associated. Each 2D continuous frame true value target box will contain the target category, 2D target ID, 3D target ID, 2D target box xmin, ymin, xmax, ymax, target head or tail longitudinal distance, target head or tail lateral distance, target side longitudinal distance, target side lateral distance, target longitudinal speed, target lateral speed, 3D true value box x, y, z, rz, l, w, h and the 8 corner point coordinates of the target's three-dimensional rectangle on the 2D image. The specific joint matching annotation result example is as follows:

[0249] Car 1 6 2187 1198 2333 1314 28.89 -4.35 31.08 -3.44 21.13 -0.77 31.06-4.41 0.74 -0.025 4.34 1.93 1.63 2177 1202 2297 1203 2342 1197 2204 1196 21761304 2296 1304 2341 1315 2203 1315.

[0250] Based on the joint annotation truth value, the detection results of the vehicle-side algorithm model to be evaluated under the joint annotation truth value continuous frame fragments are evaluated. Based on the characteristics of the above-mentioned joint matching truth value annotation generation module, the vehicle-side algorithm evaluation has the advantages of multi-angle and multi-dimensional evaluation, and the entire set of algorithm evaluation is not limited to specific 2D continuous frame or 3D continuous frame evaluation, but is more suitable for the evaluation of the current mainstream BEV algorithm and pure vision algorithm.

[0251] For the feedback results of the vehicle-side perception algorithm, single-frame evaluation, single continuous frame segment evaluation, and multiple continuous frame segment evaluation can be performed in combination with the joint matching true value.

[0252] First, the joint matching annotations and the feedback results of the algorithm to be evaluated are associated and output in the format required for cloud evaluation. When the true value and algorithm results are associated in the 3D space in a pure 3D evaluation, there is no associated information and no association can be performed due to the large depth error of the vehicle-side perception algorithm for distant targets. However, this method is not performed in 3D space, but the 2D frame of the joint matching annotation is associated, which can effectively solve the problem of no associated information in pure 3D space. After association, single frame evaluation, single data segment evaluation, and multiple data segment evaluation can be performed:

[0253] Multi-data fragment evaluation: The evaluation set of this evaluation system is generated by combining multiple data fragments. Multi-fragment evaluation can intuitively reflect the comprehensive performance of the vehicle-side perception algorithm. The fragments that make up the evaluation set, based on the scene recognition labels of the data slicing module, can better balance the diversity and richness of the fragment labels contained in the evaluation set, and show the performance indicators of the vehicle-side perception algorithm under a variety of scene labels as much as possible, so as to make the evaluation results more convincing and representative. Through the evaluation index results, we can also conduct targeted performance analysis. For example, the evaluation indicators show that the overall indicators of the pedestrian category are poor. Based on this result, further single-fragment evaluation can be carried out.

[0254] Single data segment evaluation: Single segment evaluation can help us further analyze the performance of the algorithm for specific categories under different scene labels, and can also help us further analyze the performance of the algorithm for specific scene labels under different target categories. For example, based on the evaluation results of multiple data segments, the overall evaluation indicators of the pedestrian category are poor. Through the single segment evaluation results, the evaluation results of the pedestrian category in each segment are filtered and grouped, and the possible reasons for the poor evaluation results of the pedestrian category can be analyzed, such as poor pedestrian detection effect on a rainy night.

[0255] Single-frame evaluation: Single-frame evaluation can be customized by us. For example, to evaluate the detection ability of the vehicle-side perception algorithm for small targets, we can filter all single frames containing small targets for evaluation through object storage. And because single frames can also be mapped back to the fragment information of the single frame and detailed fragment scene label information through object storage, single-frame evaluation can be combined with single-fragment evaluation and multi-fragment evaluation to form a complete evaluation system.

[0256] In addition, the evaluation module outputs more refined evaluation indicators. First, based on the data screening slicing and joint matching annotation modules, it is assumed that the joint matching annotation truth values ​​of 20 consecutive frame segments are generated. In the data screening slicing module, scene labels are generated for the 20 consecutive frame segments. Based on the scene labels, the 20 consecutive frame segments are divided.

[0257] like Fig.10As shown in the figure, the data is divided based on weather labels, light labels, road scene labels, etc. In each divided label, 20 consecutive frame segments can be evaluated, for example, under the weather label, there are 10 segments on sunny days, 4 segments on rainy days, 3 segments on snowy days, and 3 other segments. Different types of segment data can be selected for model evaluation.

[0258] Because the joint annotation truth contains depth information, the targets can be divided into different distance segments according to the longitudinal distance from the target to the vehicle. In each distance segment, each type of target is evaluated. The evaluation indicators include the absolute error, relative error, and quantile of the depth information, the absolute error and quantile of the speed information, and the total number of each type of target in the distance segment.

[0259] Then, for the evaluation under each scene label segmentation, the evaluation indicators include recall rate, precision rate, MOTA, MOTP, FN (missed detection), FP (false detection), etc. Assuming that based on the 2D and 3D joint annotation truth results of the above 20 consecutive frame segments, the longitudinal and lateral distance information of each target from the vehicle can be obtained. Based on the truth information, each type of target is first divided into distance segments. For example, for the Car category targets, based on the joint annotation truth results, they are divided into 5 intervals of 0~10m, 10~30m, 30~60m, 60~100m, and 100~150m from the vehicle. The target set of the Car category based on the truth value distance division of these 20 consecutive frame segments is obtained.

[0260] Based on the re-injection results of the vehicle-side algorithm model in these 20 consecutive frame segments, we can first calculate the recall rate, precision rate, MOTA, MOTP and other general target tracking indicators of the Car category. Secondly, for example, if we need to calculate the statistical indicators of the Car category within the range of 0~10m from the vehicle, we first need to obtain the above-mentioned target set with a distance of 0~10m from the vehicle based on the joint true value annotation. We can calculate the absolute horizontal and longitudinal distance error set of each re-injected target that matches the true value target. The matching of the re-injected target and the joint annotation target is mainly based on the Hungarian algorithm. Then, based on the error set, we can calculate the absolute error standard deviation, mean, and variance. At the same time, we can also calculate the relative horizontal and longitudinal distance error set of each re-injected target that matches the true value target. Then, based on the error set, we can calculate the relative error standard deviation, mean, and variance. For speed, we can calculate the absolute horizontal and longitudinal speed error set of each re-injected target that matches the true value target. Then, based on the error set, we can calculate the absolute error standard deviation, mean, and variance. In order to calculate the quantiles of the distance error and the speed error, we can arrange the distance error set and the speed error set from large to small, and then calculate the 0.95 quantile (i.e., 2 times sigma).

[0261] Based on the cloud-based scene recognition model, we can label each continuous frame segment with a scene label. We can customize the combination of scene labels for evaluation, such as the evaluation of the scene segment of exiting a tunnel on a rainy day, the evaluation of the scene segment of an elevated road on a rainy day at night, etc. The evaluation is multidimensional and the indicators are enriched. Based on the analysis of evaluation indicators, we can provide a priori for data slicing and screening.

[0262] like Fig.13 As shown in the figure, based on the comprehensive performance indicators of multi-segment evaluation, the analysis shows that the current vehicle-side perception model has a low recall rate in the pedestrian category. We can filter out the segments containing pedestrians in the current evaluation segments for single-segment evaluation. Based on the single-segment evaluation results, such as the evaluation results of segments ①, ②…n, it can be seen that the overall recall rate of the pedestrian category is low at night, because the intersection of labels shows that the recall rate of the pedestrian category is lower than that of the day in the night scene. Based on this conclusion, all single frames containing pedestrians and at night can be filtered out for evaluation. At the same time, the number of training data samples containing pedestrian categories and at night in the current vehicle-side algorithm model training set can be counted to provide information on whether the subsequent data collection and screening steps need to provide more training data containing pedestrians and at night.

[0263] The following is a complete flow chart to illustrate this solution:

[0264] 1402, the data slicing process includes slicing data, extracting frames of data, and manually annotating data;

[0265] 1404 is executed on the data after data slicing processing to perform joint matching, including inputting the data after data slicing processing into the cloud-based truth value model to obtain 2D continuous frame truth values ​​and 3D continuous frame truth values, and then performing one round of matching, two rounds of refined matching and three rounds of historical information matching, combined with manual annotation of key frames to obtain joint matching annotations, and then associating the detection data of the perception model to be evaluated.

[0266] S1406, evaluation and analysis, conduct multi-dimensional index evaluation on the associated results, perform index result analysis, and close the loop to the step of adjusting data slices.

[0267] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0268] According to another aspect of the embodiment of the present application, a device for determining evaluation data for implementing the above evaluation data determination method is also provided. Fig.15 As shown, the device comprises:

[0269] An acquisition unit 1502 acquires road data sets respectively collected under a plurality of data collection conditions, wherein the road data sets include at least one road image obtained by performing image collection on a target road scene, and at least one point cloud image obtained by performing point cloud collection on the target road scene;

[0270] A first determining unit 1504 determines a target road object set according to object matching results between a first road object set and a second road object set corresponding to each of the plurality of data collection conditions, wherein the first road object set is obtained by performing object recognition on at least one road image, and the second road object set is obtained by performing object recognition on at least one point cloud image;

[0271] An evaluation unit 1506 performs model evaluation on the perception models to be evaluated respectively according to the multiple target road object sets, and obtains model evaluation results that match the multiple target road object sets respectively;

[0272] The second determination unit 1508 determines at least one key data collection condition from multiple data collection conditions based on multiple model evaluation results, wherein the key data collection condition indicates a collection condition of an evaluation road data set for subsequent evaluation of the perception model to be evaluated.

[0273] Optionally, the above-mentioned acquisition unit includes: a training module, used to update the data collection conditions according to the key data collection conditions to obtain multiple target data collection conditions; obtain the target road data sets collected according to the multiple target data collection conditions; and train the perception model to be evaluated based on the target road data sets.

[0274] Optionally, the above-mentioned training module is also used to determine the target collection description information that matches each key data collection condition; determine the target data collection condition that matches each key data collection condition according to the target collection time interval, target collection road scene label and target collection road environment label indicated by the target collection description information; obtain the target road data sets collected under multiple target data collection conditions; and train the perception model to be evaluated based on the target road data set.

[0275] Optionally, the above-mentioned first determination unit also includes: a data selection module, used to select target road data collected in a single frame from road data containing target scene labels; a single frame evaluation module, used to determine the model perception results and true value data corresponding to the target road data, and match the model perception results and true value data corresponding to the target road data to obtain a single frame evaluation result.

[0276] Optionally, the above-mentioned model evaluation unit further includes: a matching module, which is used to determine the object matching relationship between the first candidate object and the second candidate object when the first annotation box corresponding to the first candidate object in the first road object set and the second annotation box corresponding to the second candidate object in the second road object set meet the target position condition, wherein the first candidate object and the second candidate object with the object matching relationship are used to indicate the same target road object in the target road scene, and the target position condition includes at least one of the following: the area of ​​the overlapping area between the first annotation box and the second annotation box is greater than or equal to the area threshold; the annotation box distance between the first annotation box and the second annotation box is less than or equal to the distance threshold; A labeling box, when the second labeling box corresponding to any candidate object in the second road object set does not meet the target position condition, a second candidate object that meets the historical matching condition is determined from the second road object set according to the historical matching information of the first candidate object, wherein the historical matching information includes a historical object matching result corresponding to at least one historical road scene, and the acquisition timestamp corresponding to the historical road scene is earlier than the acquisition timestamp of the target road scene; wherein the first labeling box is used to indicate a first image area of ​​the first candidate object in the road image, and the second labeling box is used to indicate a second image area of ​​the second candidate object in the road image, and the second image area is a projection result of point cloud information based on the matching of the second candidate object on the road image.

[0277] Optionally, the matching module is further configured to, when the overlapping areas between the first annotation frame and the candidate annotation frames of any candidate object in the second road object set are all smaller than an area threshold, obtain a second reference annotation frame subset matching the first annotation frame from the second reference annotation frame set matching the second road object set, wherein the area of ​​the overlapping areas between the second reference annotation frame and the first annotation frame in the second reference annotation frame subset is greater than 0 and smaller than the area threshold; determine a target annotation frame from the second reference annotation frame subset, wherein the area of ​​the overlapping area between the first annotation frame and the target annotation frame is greater than the area of ​​the overlapping areas corresponding to other annotation frames in the second reference annotation frame subset; and when the annotation frame distances between the target annotation frame and the first annotation frame are all smaller than the annotation frame distances corresponding to other annotation frames in the first reference annotation frame set, determine the annotation frame distance between the target annotation frame and the first annotation frame, meet the target distance condition, and determine that there is an object matching relationship between the third candidate object indicated by the target annotation frame and the first road object, wherein the first reference annotation frame is used to indicate an image area of ​​the candidate object in the first road object set in the target road image matching the target road scene, and the plurality of first reference annotation frames include the first annotation frame.

[0278] Optionally, the above-mentioned data acquisition unit also includes: a data determination module, used to determine the target category and the target distance threshold interval; a data acquisition module, used to obtain target road data containing a target scene label and a target category within the target distance threshold interval, and determine the target road data as training data; a model training module, used to use the training data to train the perception model to be evaluated.

[0279] Optionally, the matching module is also used to obtain historical matching information, wherein the historical matching information includes historical object matching results of the first candidate object in multiple historical road scenes, the multiple historical road scenes and the target road scene are road scenes corresponding to multiple information collection operations respectively, the multiple information collection operations are information collection operations corresponding to multiple continuous timestamps respectively, the information collection operations include image data collection operations and point cloud data collection operations, and the historical object matching results are used to indicate the second reference object that matches the first candidate object in the current road scene; the second reference object with the largest number of matches is determined as the second candidate object that meets the historical matching conditions; or the historical matching information of the second reference object with the largest number of matches is obtained; when the historical matching information of the second reference object indicates that the candidate object with the largest number of matches with the second reference object is not the first candidate object, the object acquisition order is determined according to the number of matches corresponding to at least one second reference object; according to the object acquisition order, a second reference object is sequentially obtained from at least one second reference object as the current reference object; when the historical matching information of the current reference object indicates that the first candidate object is the candidate object with the largest number of matches with the current reference object, the current reference object is determined as the second candidate object that meets the historical matching conditions.

[0280] Optionally, the above-mentioned matching module also includes: a verification module, used to obtain a second annotation box corresponding to the second candidate object that meets the historical matching condition, and a second verification annotation box corresponding to the second verification candidate object matched with the first candidate object determined according to the key frame matching information; when the area of ​​the overlapping area between the second verification annotation box and the second annotation box is greater than or equal to the area threshold, determine that the matching result is correct; when the area of ​​the overlapping area between the second verification annotation box and the second annotation box is less than the area threshold, determine that the second verification annotation box is the second candidate object to be corrected; according to the key frame matching information corresponding to the key frame where the second candidate object to be corrected is located, modify the second candidate object to be corrected that matches the first candidate object to be corrected into the second correction object.

[0281] Optionally, the above-mentioned evaluation unit is also used to: input multiple target road object sets into the perception model to be evaluated to obtain model prediction results that respectively match the multiple target road object sets; perform model evaluation on the perception model to be evaluated according to the model prediction results and true value data that respectively match each target road object set to obtain a category evaluation result, wherein the category evaluation result is used to indicate the evaluation results corresponding to multiple categories of perception targets; determine the category of the perception target to be evaluated based on the category evaluation result, and use road data collected from multiple continuous frame segments to evaluate the perception model to be evaluated on the dimension of the category of the perception target to be evaluated to obtain a model evaluation result.

[0282] Optionally, the above-mentioned evaluation unit also includes: a fourth determination unit, which is used to perform model evaluation on the perception model to be evaluated based on the model prediction results and true value data corresponding to each of the road data sets collected from multiple continuous frame segments, and obtain the model evaluation results corresponding to each of the multiple continuous frame segments; determine the model evaluation results that meet the preset conditions among the model evaluation results corresponding to each of the multiple continuous frame segments as the evaluation results to be determined; determine the repeated scene labels in the scene labels matched by at least one target road object set corresponding to the evaluation results to be determined as the target scene labels; select the target road data collected in a single frame from the target road object set matching the target scene label; determine the model prediction results and true value data corresponding to the target road data, and match the model prediction results and true value data corresponding to the target road data to obtain a single frame evaluation result.

[0283] Optionally, the above-mentioned evaluation unit also includes: a fifth determination unit, used to determine the distance between at least one reference obstacle in at least one target road object set and the vehicle according to the true value data corresponding to at least one target road object set, wherein the reference obstacle is used to indicate the static obstacles and dynamic obstacles around the vehicle; according to the distance between at least one reference obstacle and the vehicle and a preset distance threshold, at least one reference obstacle is divided into at least one reference obstacle set; according to the target road object set corresponding to at least one reference obstacle set, the model evaluation of the perception model to be evaluated is performed in sequence to obtain a model evaluation result, including at least one of the following: calculating the difference between the model prediction result and the distance in the true value data to obtain an absolute distance error; calculating the ratio between the absolute distance error and the distance in the true value data to obtain a relative distance error; calculating the difference between the speed in the model prediction result and the speed in the true value data to obtain an absolute speed error.

[0284] Optionally, the above-mentioned evaluation unit is also used to: determine the category of the perception target to be evaluated and the target distance threshold interval; obtain target road data that belongs to the target scene label and the category of the perception target to be evaluated is within the target distance threshold interval, and determine the target road data as updated training data; use the updated training data to train the perception model to be evaluated.

[0285] Optionally, in this embodiment, the embodiments to be implemented by the above-mentioned various unit modules can refer to the above-mentioned various method embodiments, which will not be repeated here.

[0286] According to another aspect of the embodiment of the present application, an electronic device for implementing the above-mentioned method for determining evaluation data is also provided. The electronic device may be Fig.16 The terminal device or server shown in the figure. This embodiment is described by taking the electronic device as a terminal device as an example. Fig.16 As shown, the electronic device includes a memory 1602 and a processor 1604. The memory 1602 stores a computer program, and the processor 1604 is configured to execute the steps in any of the above method embodiments through the computer program.

[0287] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0288] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0289] S1, obtaining road data sets respectively collected under multiple data collection conditions, wherein the road data sets include at least one road image obtained by image collection of a target road scene, and at least one point cloud image obtained by point cloud collection of the target road scene;

[0290] S2, determining a target road object set according to object matching results between a first road object set and a second road object set corresponding to each of the plurality of data collection conditions, wherein the first road object set is obtained by performing object recognition on at least one road image, and the second road object set is obtained by performing object recognition on at least one point cloud image;

[0291] S3, performing model evaluation on the perception models to be evaluated respectively according to the multiple target road object sets, and obtaining model evaluation results that match the multiple target road object sets respectively;

[0292] S4. Determine at least one key data collection condition from multiple data collection conditions based on multiple model evaluation results, wherein the key data collection condition indicates a collection condition of an evaluation road data set for subsequent evaluation of the perception model to be evaluated.

[0293] Alternatively, a person skilled in the art may understand that: Fig.16The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, and other terminal devices. Fig.16 The electronic device and the electronic equipment described above are not limited in structure. Fig.16 More or fewer components (such as network interfaces, etc.) as shown in, or with Fig.16 Different configurations are shown.

[0294] Among them, the memory 1602 can be used to store software programs and modules, such as the program instructions / modules corresponding to the method and device for determining the evaluation data in the embodiment of the present application. The processor 1604 executes various functional applications and data processing by running the software programs and modules stored in the memory 1602, that is, realizing the above-mentioned method for determining the evaluation data. The memory 1602 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1602 may further include a memory remotely located relative to the processor 1604, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 1602 can be specifically, but not limited to, used to store information such as road images. As an example, such as Fig.16 As shown, the memory 1602 may include, but is not limited to, the acquisition unit 1502, the first determination unit 1504, the evaluation unit 1506, and the second determination unit 1508 in the evaluation data determination device. In addition, other module units in the evaluation data determination device may also be included but are not limited to, which will not be repeated in this example.

[0295] Optionally, the transmission device 1606 is used to receive or send data via a network. Specific examples of the network may include a wired network and a wireless network. In one example, the transmission device 1606 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 1606 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0296] In addition, the electronic device mentioned above further includes: a display 1608 for displaying a simulated road scene; and a connection bus 1610 for connecting various module components in the electronic device mentioned above.

[0297] In other embodiments, the terminal device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes in the form of network communication. Among them, the nodes may form a peer-to-peer (P2P) network, and any form of computing device, such as a server, terminal or other electronic device, may become a node in the blockchain system by joining the peer-to-peer network.

[0298] According to one aspect of the present application, a computer program product is provided, the computer program product comprising a computer program / instruction, the computer program / instruction comprising a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit, various functions provided by the embodiments of the present application are executed.

[0299] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0300] According to one aspect of the present application, a computer-readable storage medium is provided, and a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned method for determining evaluation data.

[0301] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0302] S1, obtaining road data sets respectively collected under multiple data collection conditions, wherein the road data sets include at least one road image obtained by image collection of a target road scene, and at least one point cloud image obtained by point cloud collection of the target road scene;

[0303] S2, determining a target road object set according to object matching results between a first road object set and a second road object set corresponding to each of the plurality of data collection conditions, wherein the first road object set is obtained by performing object recognition on at least one road image, and the second road object set is obtained by performing object recognition on at least one point cloud image;

[0304] S3, performing model evaluation on the perception models to be evaluated respectively according to the multiple target road object sets, and obtaining model evaluation results that match the multiple target road object sets respectively;

[0305] S4. Determine at least one key data collection condition from multiple data collection conditions based on multiple model evaluation results, wherein the key data collection condition indicates a collection condition of an evaluation road data set for subsequent evaluation of the perception model to be evaluated.

[0306] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0307] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which may be personal computers, servers or network devices, etc.) to execute all or part of the steps of the above methods in each embodiment of the present application.

[0308] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0309] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0310] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0311] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0312] The above are only preferred implementations of the present application. It should be pointed out that ordinary technicians in this technical field can make several improvements and modifications without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for determining evaluation data, characterized in that: include: Acquire a road data set collected respectively under a plurality of data collection conditions, wherein the road data set includes at least one road image obtained by performing image collection on a target road scene, and at least one point cloud image obtained by performing point cloud collection on the target road scene; In the case where a first annotation box corresponding to a first candidate object in a first road object set and a second annotation box corresponding to a second candidate object in a second road object set meet a target position condition, determining an object matching relationship between the first candidate object and the second candidate object, comprising: in the case where an overlapping area between the first annotation box and a candidate annotation box of any candidate object in the second road object set is smaller than an area threshold, acquiring a second reference annotation box subset matching the first annotation box from a second reference annotation box set matching the second road object set, wherein an area of ​​an overlapping area between a second reference annotation box in the second reference annotation box subset and the first annotation box is greater than 0 and smaller than the area threshold; determining a target annotation box from the second reference annotation box subset, wherein an area of ​​an overlapping area between the first annotation box and the target annotation box is greater than an area of ​​an overlapping area corresponding to other annotation boxes in the second reference annotation box subset; When the annotation frame distances between the target annotation frame and the first annotation frame are smaller than the annotation frame distances corresponding to other annotation frames in the first reference annotation frame set, the annotation frame distance between the target annotation frame and the first annotation frame is determined to meet the target distance condition, and it is determined that the third candidate object indicated by the target annotation frame has the object matching relationship with the first road object, wherein the first reference annotation frame is used to indicate the image area of ​​the candidate object in the first road object set in the target road image matching the target road scene, the first reference annotation frames include the first annotation frame, the first candidate object and the second candidate object having the object matching relationship are used to indicate the same target road object in the target road scene, and the target position condition includes at least one of the following: the area of ​​the overlapping area between the first annotation frame and the second annotation frame is greater than or equal to the area threshold; the annotation frame distance between the first annotation frame and the second annotation frame is less than or equal to the distance threshold; In a case where the first annotation box corresponding to the first candidate object and the second annotation box corresponding to any one of the candidate objects in the second road object set do not satisfy the target position condition, determining the second candidate object satisfying the historical matching condition from the second road object set according to historical matching information of the first candidate object, wherein the historical matching information includes historical object matching results corresponding to at least one historical road scene, and the acquisition timestamp corresponding to the historical road scene is earlier than the acquisition timestamp of the target road scene; The first annotation box is used to indicate a first image region of the first candidate object in the road image, and the second annotation box is used to indicate a second image region of the second candidate object in the road image, and the second image region is a projection result of point cloud information matched based on the second candidate object in the road image; Determining a target road object set according to object matching results between a first road object set and a second road object set corresponding to each of the plurality of data collection conditions, wherein the first road object set is obtained by performing object recognition on at least one of the road images, and the second road object set is obtained by performing object recognition on at least one of the point cloud images; According to the plurality of target road object sets, model evaluation is performed on the perception models to be evaluated respectively, and model evaluation results that match the plurality of target road object sets are obtained respectively; According to the multiple model evaluation results, at least one key data collection condition is determined from the multiple data collection conditions, wherein the key data collection condition indicates the collection condition of the evaluation road data set for subsequent evaluation of the perception model to be evaluated, and the key data collection condition is the data collection condition with poor evaluation results, thereby adding data features corresponding to the key data collection condition to the perception model to be evaluated for optimization training.

2. The method according to claim 1, characterized in that After determining at least one key data collection condition from the plurality of data collection conditions according to the plurality of model evaluation results, the method further comprises: updating the data collection conditions according to the key data collection conditions to obtain a plurality of target data collection conditions; Acquire target road data sets collected respectively under a plurality of target data collection conditions; The perception model to be evaluated is trained according to the target road data set.

3. The method according to claim 2, characterized in that After determining at least one key data collection condition from the plurality of data collection conditions according to the plurality of model evaluation results, the method further comprises: Determine target acquisition description information matching each of the key data acquisition conditions; Determine the target data collection condition that matches each of the key data collection conditions according to the target collection time interval, the target collection road scene label, and the target collection road environment label indicated by the target collection description information; Acquire target road data sets collected respectively under a plurality of target data collection conditions; The perception model to be evaluated is trained according to the target road data set.

4. The method according to claim 1, characterized in that The determining, based on the historical matching information of the first candidate object, the second candidate object that meets the historical matching condition from the second road object set includes: Acquire the historical matching information, wherein the historical matching information includes historical object matching results of the first candidate object in a plurality of historical road scenes, the plurality of historical road scenes and the target road scene are road scenes corresponding to a plurality of information collection operations respectively, the plurality of information collection operations are information collection operations corresponding to a plurality of continuous timestamps respectively, the information collection operations include image data collection operations and point cloud data collection operations, and the historical object matching results are used to indicate a second reference object that matches the first candidate object in the current road scene; Determine the second reference object having the largest number of matches as the second candidate object satisfying the historical matching condition; or Obtain the historical matching information of the second reference object with the largest number of matches; when the historical matching information of the second reference object indicates that the candidate object with the largest number of matches with the second reference object is not the first candidate object, determine the object acquisition order according to the number of matches corresponding to at least one of the second reference objects; according to the object acquisition order, sequentially acquire one second reference object from at least one of the second reference objects as the current reference object; when the historical matching information of the current reference object indicates that the first candidate object is the candidate object with the largest number of matches with the current reference object, determine the current reference object as the second candidate object that meets the historical matching condition.

5. The method according to claim 1, characterized in that After determining the object matching relationship between the first candidate object and the second candidate object, the method further includes: Acquire a second annotation box corresponding to the second candidate object that meets the historical matching condition, and a second verification annotation box corresponding to the second verification candidate object that matches the first candidate object and is determined according to the key frame matching information; When the area of ​​the overlapping region between the second verification annotation box and the second annotation box is greater than or equal to the area threshold, determining that the matching result is correct; When the area of ​​the overlapping area between the second verification annotation box and the second annotation box is less than the area threshold, the second verification annotation box is determined to be the second candidate object to be corrected; and according to the key frame matching information corresponding to the key frame where the second candidate object to be corrected is located, the second candidate object to be corrected that matches the first candidate object is modified to a second correction object.

6. The method according to claim 1, characterized in that The step of performing model evaluation on the perception models to be evaluated respectively according to the plurality of target road object sets to obtain model evaluation results that match the plurality of target road object sets respectively includes: Inputting the plurality of target road object sets into the perception model to be evaluated to obtain model prediction results respectively matching the plurality of target road object sets; Performing the model evaluation on the perception model to be evaluated according to the model prediction results and true value data respectively matched with each of the target road object sets to obtain a category evaluation result, wherein the category evaluation result is used to indicate the evaluation results corresponding to the perception targets of multiple categories; The category of the perception target to be evaluated is determined based on the category evaluation result, and the perception model to be evaluated is evaluated on the dimension of the perception target category to be evaluated using road data collected from multiple continuous frame segments to obtain the model evaluation result.

7. The method according to claim 6, characterized in that After determining the target road object set according to the object matching results between the first road object set and the second road object set corresponding to the plurality of data collection conditions, the method includes: Performing the model evaluation on the perception model to be evaluated based on the model prediction results and the true value data corresponding to each of the road data sets collected from a plurality of continuous frame segments, to obtain the model evaluation results corresponding to each of the plurality of continuous frame segments; Determining, among the model evaluation results corresponding to each of the plurality of continuous frame segments, the model evaluation results that meet a preset condition as the evaluation results to be determined; Determine a repeated scene label in the scene labels matched by at least one target road object set corresponding to the to-be-determined evaluation result as a target scene label; Selecting target road data collected in a single frame from the target road object set matching the target scene label; The model prediction result and the true value data corresponding to the target road data are determined, and the model prediction result and the true value data corresponding to the target road data are matched to obtain a single-frame evaluation result.

8. The method according to claim 6, characterized in that The step of performing model evaluation on the perception models to be evaluated respectively according to the plurality of target road object sets to obtain model evaluation results that match the plurality of target road object sets respectively further includes: Determining a distance between at least one reference obstacle in at least one of the target road object sets and the vehicle according to true value data corresponding to at least one of the target road object sets, wherein the reference obstacle is used to indicate static obstacles and dynamic obstacles around the vehicle; Differentiate the at least one reference obstacle into at least one reference obstacle set according to the distance between the at least one reference obstacle and the vehicle and a preset distance threshold; The model evaluation is performed on the perception model to be evaluated in sequence according to the target road object set corresponding to the at least one reference obstacle set, to obtain the model evaluation result, including at least one of the following: Calculate the difference between the model prediction result and the distance in the true value data to obtain the absolute distance error; Calculate the ratio between the absolute distance error and the distance in the true value data to obtain a relative distance error; The absolute speed error is obtained by calculating the difference between the speed in the model prediction result and the speed in the true value data.

9. The method according to any one of claims 6 to 8, characterized in that After respectively performing model evaluation on the perception models to be evaluated according to the plurality of target road object sets and obtaining model evaluation results respectively matching the plurality of target road object sets, the method comprises: Determining the category of the perceived target to be evaluated and the target distance threshold interval; Acquire target road data that belongs to the target scene label and the perceived target category to be evaluated is within the target distance threshold interval, and determine the target road data as updated training data; The updated training data is used to train the perception model to be evaluated.

10. A device for determining evaluation data, characterized in that: include: An acquisition unit is configured to acquire road data sets respectively collected under a plurality of data collection conditions, wherein the road data sets include at least one road image obtained by performing image collection on a target road scene, and at least one point cloud image obtained by performing point cloud collection on the target road scene; The first determining unit determines the object matching relationship between the first candidate object and the second candidate object when a first annotation box corresponding to the first candidate object in the first road object set and a second annotation box corresponding to the second candidate object in the second road object set meet the target position condition, including: when the overlapping areas between the first annotation box and the candidate annotation boxes of any one of the candidate objects in the second road object set are both smaller than an area threshold, acquiring a second reference annotation box subset matching the first annotation box from a second reference annotation box set matching the second road object set, wherein the area of ​​the overlapping area between the second reference annotation box in the second reference annotation box subset and the first annotation box is greater than 0 and smaller than the area threshold; determining a target annotation box from the second reference annotation box subset, wherein the area of ​​the overlapping area between the first annotation box and the target annotation box is greater than the area of ​​the overlapping areas corresponding to other annotation boxes in the second reference annotation box subset; and when the annotation box distances between the target annotation box and the first annotation box are both smaller than the annotation box distances corresponding to other annotation boxes in the first reference annotation box set, determining The annotation box distance between the target annotation box and the first annotation box satisfies the target distance condition, and it is determined that the third candidate object indicated by the target annotation box has the object matching relationship with the first road object, wherein the first reference annotation box is used to indicate the image area of ​​the candidate object in the first road object set in the target road image matching the target road scene, and the first reference annotation boxes include the first annotation box, and the first candidate object and the second candidate object having the object matching relationship are used to indicate the same target road object in the target road scene, and the target position condition includes at least one of the following: the area of ​​the overlapping area between the first annotation box and the second annotation box is greater than or equal to the area threshold; the annotation box distance between the first annotation box and the second annotation box is less than or equal to the distance threshold; the target road object set is determined according to the object matching results between the first road object set and the second road object set corresponding to each of the plurality of data acquisition conditions, wherein the first road object set is obtained by performing object recognition on at least one of the road images, and the second road object set is obtained by performing object recognition on at least one of the point cloud images;In a case where the first annotation box corresponding to the first candidate object and the second annotation box corresponding to any one of the candidate objects in the second road object set do not satisfy the target position condition, determining the second candidate object satisfying the historical matching condition from the second road object set according to historical matching information of the first candidate object, wherein the historical matching information includes historical object matching results corresponding to at least one historical road scene, and the acquisition timestamp corresponding to the historical road scene is earlier than the acquisition timestamp of the target road scene; The first annotation box is used to indicate a first image region of the first candidate object in the road image, and the second annotation box is used to indicate a second image region of the second candidate object in the road image, and the second image region is a projection result of point cloud information matched based on the second candidate object in the road image; An evaluation unit, performing model evaluation on the perception models to be evaluated respectively according to the plurality of target road object sets, and obtaining model evaluation results respectively matching the plurality of target road object sets; A second determination unit determines at least one key data collection condition from the multiple data collection conditions based on the multiple model evaluation results, wherein the key data collection condition indicates the collection condition of the evaluation road data set for subsequent evaluation of the perception model to be evaluated, and the key data collection condition is the data collection condition with poor evaluation results, thereby adding data features corresponding to the key data collection condition to the perception model to be evaluated for optimization training.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method described in any one of claims 1 to 9 when executed.

12. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

13. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 9 through the computer program.

Images