A method and device for determining annotation data for monocular 3D target detection

Abstract

The present invention discloses a method and device for determining annotation data for monocular 3D target detection, and relates to the field of computer technology. A specific implementation of the method includes: receiving multiple 3D point cloud data and multiple image data synchronously collected by a laser radar and a camera; using a 3D point cloud detection model to process the multiple 3D point cloud data to obtain multiple 3D detection results, and sequentially converting and mapping the multiple 3D detection results to obtain multiple first 2D frame information; wherein the first 2D frame information indicates the two-dimensional information of the annotated object; using a 2D target detection model to process the multiple image data to obtain multiple second 2D frame information; calculating the intersection-and-union ratio of the multiple first 2D frame information and the multiple second 2D frame information, and determining the annotation data according to the intersection-and-union ratio and a first threshold. This implementation reduces the cost of determining the annotation data, and can quickly and efficiently determine large quantities of annotation data.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for determining labeled data for monocular 3D target detection. Background Art

[0002] Monocular 3D Object Detection is one of the most widely used computer vision technologies, which can be applied to vehicle autonomous driving systems, intelligent robots, intelligent transportation and other fields. Especially in the field of autonomous driving system perception, 3D object detection has the huge advantages of dense information and low cost compared to multi-line laser radar.

[0003] Monocular 3D target detection requires a large amount of annotation data in different scenarios to display the attribute information (such as length, width, height, position, etc.) of the annotated objects. Based on 2D images, only two-dimensional information of the annotated objects can be obtained, such as length, width, and position in the image coordinate system; however, it is difficult to determine the three-dimensional information of the annotated objects in the real-world coordinate system, such as the actual length, width, height, deflection angle, and distance of the object.

[0004] There are at least the following problems in the related art:

[0005] In the related art, the annotation data determined only by 2D images only includes two-dimensional information of the annotated object, resulting in low accuracy of the annotation data and difficulty in obtaining large quantities of annotation data. Summary of the invention

[0006] In view of this, an embodiment of the present invention provides a method and device for determining annotation data for monocular 3D target detection, which can combine the three-dimensional information and two-dimensional information of the annotated object according to the point cloud data collected by the lidar and the image data collected by the camera, thereby reducing the cost of determining the annotation data and being able to quickly and efficiently determine large quantities of annotation data.

[0007] To achieve the above object, according to one aspect of an embodiment of the present invention, a method for determining labeled data for monocular 3D target detection is provided, comprising:

[0008] Receive multiple 3D point cloud data and multiple image data collected synchronously by the laser radar and camera;

[0009] Using a 3D point cloud detection model to process multiple 3D point cloud data to obtain multiple 3D detection results, and sequentially performing conversion processing and mapping processing on the multiple 3D detection results to obtain multiple first 2D frame information; wherein the first 2D frame information indicates two-dimensional information of the labeled object;

[0010] Processing the multiple image data using a 2D object detection model to obtain multiple second 2D frame information;

[0011] The intersection-and-union ratios of the plurality of first 2D frame information and the plurality of second 2D frame information are calculated, and the labeled data is determined according to the intersection-and-union ratios and a first threshold.

[0012] Furthermore, the multiple 3D detection results are sequentially converted and mapped to obtain multiple first 2D frame information, which may include:

[0013] The multiple 3D detection results are converted respectively to obtain multiple 3D frame information in the camera coordinate system; wherein the 3D frame information indicates the three-dimensional information of the labeled object;

[0014] Mapping processing is performed on the multiple 3D frame information respectively to obtain multiple first 2D frame information corresponding to the multiple 3D frame information.

[0015] Furthermore, determining the labeled data according to the intersection-over-union ratio and the first threshold value may also include:

[0016] Determine whether the intersection-to-union ratio is less than a first threshold;

[0017] If yes, delete the 3D frame information corresponding to the first 2D frame information, retain the second 2D frame information, and use the second 2D frame information as the annotation data;

[0018] If not, the matching result of the first 2D frame information and the second 2D frame information is determined according to the intersection-and-union ratio, the 3D frame information corresponding to the matched first 2D frame information and the unmatched second 2D frame information are retained, and the retained information is used as the annotation data.

[0019] Furthermore, if the intersection-over-union ratio is greater than or equal to a first threshold, the method further includes:

[0020] Determine a 3D projection center point corresponding to the 3D frame information in the retained information, and a 2D frame center point corresponding to the second 2D frame information in the retained information;

[0021] Calculate the Euclidean distance between the center point of the 3D projection and the center point of the 2D frame, and calculate the frame diagonal distance of the second 2D frame information in the retained information;

[0022] The annotation error of the annotated object is calculated based on the Euclidean distance and the box diagonal distance;

[0023] The labeled data is updated according to the labeling error and the second threshold.

[0024] Furthermore, the retained information includes one or more annotated objects; if the number of the annotated objects is one, the step of calculating the annotation error of the image according to the Euclidean distance and the box diagonal distance includes:

[0025] Calculate the normalized distance between the 3D projection center point and the 2D box center point based on the Euclidean distance and the box diagonal distance;

[0026] The annotation error is calculated based on the normalized distance.

[0027] Furthermore, if there are multiple labeled objects, the step of calculating the labeling error of the image according to the Euclidean distance and the box diagonal distance includes:

[0028] Calculate the average of the normalized distances between the 3D projection center point and the 2D box center point based on the number of annotations, Euclidean distance, and box diagonal distance;

[0029] The annotation error is calculated based on the average of the normalized distance and the number of annotations.

[0030] According to another aspect of an embodiment of the present invention, there is provided a device for determining labeling data for monocular 3D object detection, comprising:

[0031] The data receiving module is used to receive multiple 3D point cloud data and multiple image data collected synchronously by the laser radar and the camera;

[0032] The point cloud data processing module is used to: use the 3D point cloud detection model to process multiple 3D point cloud data to obtain multiple 3D detection results, and sequentially perform conversion processing and mapping processing on the multiple 3D detection results to obtain multiple first 2D frame information; wherein the first 2D frame information indicates the two-dimensional information of the labeled object;

[0033] An image data processing module is used to: process a plurality of image data using a 2D object detection model to obtain a plurality of second 2D frame information;

[0034] The labeling data determination module is used to calculate the intersection-and-union ratios of the multiple first 2D frame information and the multiple second 2D frame information, and determine the labeling data according to the intersection-and-union ratios and a first threshold.

[0035] Furthermore, the labeling data determination module is also used to:

[0036] Determine whether the intersection-to-union ratio is less than a first threshold;

[0037] If yes, delete the 3D frame information corresponding to the first 2D frame information, retain the second 2D frame information, and use the second 2D frame information as the annotation data;

[0038] If not, the matching result of the first 2D frame information and the second 2D frame information is determined according to the intersection-and-union ratio, the 3D frame information corresponding to the matched first 2D frame information and the unmatched second 2D frame information are retained, and the retained information is used as the annotation data.

[0039] According to another aspect of an embodiment of the present invention, there is provided an electronic device for determining annotation data for monocular 3D target detection, characterized in that it includes:

[0040] one or more processors;

[0041] a storage device for storing one or more programs,

[0042] When one or more programs are executed by one or more processors, the one or more processors implement any of the above-mentioned methods for determining labeled data for monocular 3D target detection.

[0043] According to another aspect of an embodiment of the present invention, there is provided a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, any of the above-mentioned methods for determining labeled data for monocular 3D target detection is implemented.

[0044] An embodiment of the above invention has the following advantages or beneficial effects: because it adopts receiving multiple 3D point cloud data and multiple image data synchronously collected by laser radar and camera; using 3D point cloud detection model to process multiple 3D point cloud data to obtain multiple 3D detection results, and converting and mapping the multiple 3D detection results in turn to obtain multiple first 2D frame information; wherein the 2D frame information indicates the two-dimensional information of the labeled object; using 2D target detection model to process multiple image data to obtain multiple second 2D frame information; calculating the intersection and union ratio of multiple first 2D frame information and multiple second 2D frame information respectively, and determining the labeled data according to the intersection and union ratio and the first threshold. The technical means overcomes the technical problem that the labeled data determined by only relying on 2D images in the related art only includes the two-dimensional information of the labeled object, resulting in low accuracy of the labeled data and difficulty in obtaining a large amount of labeled data, thereby achieving the technical effect of combining the three-dimensional information and the two-dimensional information of the labeled object according to the point cloud data collected by the laser radar and the image data collected by the camera, reducing the cost of determining the labeled data, and quickly and efficiently determining a large amount of labeled data.

[0045] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0047] Figure 1 is a schematic diagram of the main process of a method for determining labeled data for monocular 3D target detection provided according to an embodiment of the present invention;

[0048] Figure 2is a schematic diagram of the main process of a method for determining labeled data for monocular 3D target detection provided according to yet another embodiment of the present invention;

[0049] Figure 3 is a schematic diagram of main modules of a device for determining annotation data for monocular 3D target detection provided according to another embodiment of the present invention;

[0050] Figure 4 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0051] Figure 5 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0053] Figure 1 is a schematic diagram of the main process of a method for determining labeled data for monocular 3D target detection provided according to an embodiment of the present invention; Figure 1 As shown, the method for determining the labeled data for monocular 3D target detection provided by the embodiment of the present invention mainly includes:

[0054] Step S101, receiving a plurality of 3D point cloud data and a plurality of image data synchronously collected by a laser radar and a camera.

[0055] Specifically, the calibration parameters and conversion matrix parameters between the vehicle-mounted sensors can be pre-set, and the laser radar and camera can be used to synchronously collect (synchronize the collection timestamp) the point cloud data and image data of the surrounding environment during the vehicle's driving. According to an embodiment of the present invention, a laser radar with fewer beams, such as a 16-beam laser radar, can be used to reduce the cost of the laser radar, thereby reducing the cost of determining the labeled data. Although the point cloud collected by the laser radar with fewer beams is relatively sparse, and the labeled objects at a distance are very likely to have no scanned point cloud, combined with the image data collected by the camera, the accuracy of the determined labeled data can be improved while ensuring low cost.

[0056] Step S102, using a 3D point cloud detection model to process multiple 3D point cloud data to obtain multiple 3D detection results, and converting and mapping the multiple 3D detection results in turn to obtain multiple first 2D frame information; wherein the first 2D frame information indicates the two-dimensional information of the labeled object, and further, the two-dimensional information of the labeled object is determined based on the 3D point cloud data.

[0057] Specifically, according to an embodiment of the present invention, an existing 3D point cloud detection model with high detection accuracy and relatively mature detection (such as PointPillars, etc.) can be used to process all 3D point cloud data to obtain a 3D detection result corresponding to each point cloud data, wherein the 3D detection result indicates the three-dimensional information of the annotated object in the point cloud coordinate system. Then, by sequentially converting and mapping the 3D detection results, the first 2D frame information can be obtained, indicating the two-dimensional information of the annotated object in the camera coordinate system, so as to combine it with the two-dimensional information of the annotated object obtained after image data processing (the second 2D frame information) to determine the annotated data with higher accuracy.

[0058] Further, according to an embodiment of the present invention, the multiple 3D detection results are sequentially converted and mapped to obtain multiple first 2D frame information, including:

[0059] The multiple 3D detection results are converted respectively to obtain multiple 3D frame information in the camera coordinate system; wherein the 3D frame information indicates the three-dimensional information of the labeled object;

[0060] Mapping processing is performed on the multiple 3D frame information respectively to obtain multiple first 2D frame information corresponding to the multiple 3D frame information.

[0061] Through the above settings, the 3D detection results in multiple point cloud coordinate systems are converted and processed respectively to obtain 3D frame information in the camera coordinate system, which is used to indicate the three-dimensional information of the labeled object. Then, the 3D frame information is mapped to obtain the two-dimensional information of the labeled object in the camera coordinate system.

[0062] Step S103: Process the multiple image data using the 2D object detection model to obtain multiple second 2D frame information.

[0063] Specifically, according to an embodiment of the present invention, an existing 2D target detection model (such as SSD, YOLO, etc.) can be used to process multiple image data collected by the camera to obtain the second 2D frame information. The intersection-and-union ratio of the labeled object in different 2D frame information is determined by processing the second 2D frame information obtained by image data processing and the first 2D frame information obtained by processing the 3D point cloud data, and then the intersection-and-union ratio is compared with the first threshold value, so that the labeled data with higher accuracy can be determined quickly and in large quantities for monocular 3D target detection. Among them, the second 2D frame information indicates the two-dimensional information of the labeled object determined based on the image data.

[0064] Step S104 , calculating intersection-over-union ratios of the plurality of first 2D frame information and the plurality of second 2D frame information, and determining the labeled data according to the intersection-over-union ratios and a first threshold.

[0065] Specifically, according to an embodiment of the present invention, determining the labeled data according to the intersection-over-union ratio and the first threshold value may further include:

[0066] Determine whether the intersection-to-union ratio is less than a first threshold;

[0067] If yes, delete the 3D frame information corresponding to the first 2D frame information, retain the second 2D frame information, and use the second 2D frame information as the annotation data;

[0068] If not, the matching result of the first 2D frame information and the second 2D frame information is determined according to the intersection-and-union ratio, the 3D frame information corresponding to the matched first 2D frame information and the unmatched second 2D frame information are retained, and the retained information is used as the annotation data.

[0069] Through the above settings, the intersection-and-union ratio is compared with the first threshold to determine the two types of annotation data. According to a specific implementation of an embodiment of the present invention, the first threshold can be set to 0.5 (the value range of the first threshold is between 0 and 1), that is, if the two-dimensional information displayed in the first 2D frame information of the annotated object and the two-dimensional information displayed in the second 2D frame information are half completely matched, that is, the 3D frame information corresponding to the first 2D frame information of the successfully matched part is retained, and the second 2D frame information that is not successfully matched is retained, and the retained information is used as the annotation data; if the two-dimensional information displayed in the first 2D frame information of the annotated object and the two-dimensional information displayed in the second 2D frame information are less than half of the successfully matched part, only the second 2D frame information is retained as the annotation data. It should be noted that the above setting of the first threshold is only an example, and different first thresholds can be set according to the actual accuracy to be achieved and the amount of annotation data to be determined. If the accuracy to be achieved is higher, a higher first threshold can be set, and if more annotation data is to be obtained, a lower first threshold can be set.

[0070] Further, according to an embodiment of the present invention, if the intersection-over-union ratio is greater than or equal to the first threshold, the method may further include:

[0071] Determine a 3D projection center point corresponding to the 3D frame information in the retained information, and a 2D frame center point corresponding to the second 2D frame information in the retained information;

[0072] Calculate the Euclidean distance between the center point of the 3D projection and the center point of the 2D frame, and calculate the frame diagonal distance of the second 2D frame information in the retained information;

[0073] The annotation error of the annotated object is calculated based on the Euclidean distance and the box diagonal distance;

[0074] The labeled data is updated according to the labeling error and the second threshold.

[0075] Through the above settings, in the case where both the first 2D frame information and the second 2D frame information are retained, in order to improve the accuracy of the determined labeled data, it is necessary to calculate the calibration error, and use the calibration error and the second threshold to clean the data. That is, determine the 3D projection center point of the 3D frame information corresponding to the first 2D frame information in the retained information, and the 2D frame center point corresponding to the second 2D frame information in the retained information, and then calculate the Euclidean distance between the two center points, and the frame diagonal distance of the second 2D frame information, and calculate the labeling error of the labeled object based on the Euclidean distance and the frame diagonal distance; then update the labeled data based on the labeling error and the second threshold.

[0076] Exemplarily, according to an embodiment of the present invention, the retained information includes one or more annotated objects; if the number of the annotated objects is one, the step of calculating the annotation error of the image according to the Euclidean distance and the box diagonal distance includes:

[0077] According to the Euclidean distance and the diagonal distance of the box, the normalized distance between the center point of the 3D projection and the center point of the 2D box is calculated; and the annotation error is calculated according to the normalized distance.

[0078] Preferably, according to an embodiment of the present invention, if there are multiple labeled objects, the step of calculating the labeling error of the image according to the Euclidean distance and the box diagonal distance may include:

[0079] Calculate the average of the normalized distances between the 3D projection center point and the 2D box center point based on the number of annotations, Euclidean distance, and box diagonal distance;

[0080] The annotation error is calculated based on the average of the normalized distance and the number of annotations.

[0081] It can be understood that, the more the number of annotated objects included in the information is, the more accurate the annotation error ultimately determined by calculating the average value of the normalized distance is, thereby making the accuracy of the determined annotation data higher.

[0082] According to the technical solution of the embodiment of the present invention, because multiple 3D point cloud data and multiple image data synchronously collected by the laser radar and the camera are received; the multiple 3D point cloud data are processed by the 3D point cloud detection model to obtain multiple 3D detection results, and the multiple 3D detection results are sequentially converted and mapped to obtain multiple first 2D frame information; wherein the first 2D frame information indicates the two-dimensional information of the labeled object; the multiple image data are processed by the 2D target detection model to obtain multiple second 2D frame information; the intersection and union ratios of the multiple first 2D frame information and the multiple second 2D frame information are calculated, and the technical means for determining the labeled data according to the intersection and union ratio and the first threshold value, overcomes the technical problem that the labeled data determined by only relying on 2D images in the related art, because only the two-dimensional information of the labeled object is included, the accuracy of the labeled data is low, and it is difficult to obtain a large amount of labeled data, thereby achieving the ability to combine the three-dimensional information and the two-dimensional information of the labeled object according to the point cloud data collected by the laser radar and the image data collected by the camera, reducing the cost of determining the labeled data, and being able to quickly and efficiently determine a large amount of labeled data.

[0083] Figure 2 is a schematic diagram of the main process of a method for determining labeled data for monocular 3D target detection provided by yet another embodiment of the present invention; Figure 2 As shown, the method for determining the labeled data for monocular 3D target detection provided by the embodiment of the present invention mainly includes:

[0084] Step S201, receiving multiple 3D point cloud data and multiple image data synchronously collected by the laser radar and the camera.

[0085] Specifically, the calibration parameters and conversion matrix parameters between the on-board sensors can be set in advance, and the laser radar and camera can be used to synchronously collect (synchronize the collection timestamp) the point cloud data and image data of the surrounding environment during the vehicle's driving process.

[0086] Step S202: Process multiple 3D point cloud data using a 3D point cloud detection model to obtain multiple 3D detection results.

[0087] Specifically, according to an embodiment of the present invention, an existing 3D point cloud detection model with high detection accuracy and relatively mature detection (such as PointPillars, etc.) can be used to process all 3D point cloud data to obtain a 3D detection result corresponding to each point cloud data, wherein the 3D detection result indicates the three-dimensional information of the marked object in the point cloud coordinate system.

[0088] Step S203, converting the multiple 3D detection results respectively to obtain multiple 3D frame information in the camera coordinate system; wherein the 3D frame information indicates the three-dimensional information of the labeled object; mapping the multiple 3D frame information respectively to obtain multiple first 2D frame information corresponding to the multiple 3D frame information.

[0089] By converting and mapping the 3D detection results in sequence, the first 2D frame information can be obtained, indicating the two-dimensional information of the labeled object in the camera coordinate system, so as to be combined with the two-dimensional information of the labeled object obtained after image data processing (second 2D frame information) to determine the labeled data with higher accuracy.

[0090] Step S204: Process the multiple image data using the 2D object detection model to obtain multiple second 2D frame information.

[0091] Specifically, according to an embodiment of the present invention, an existing 2D target detection model (such as SSD, YOLO, etc.) can be used to process multiple image data collected by a camera to obtain second 2D frame information. The intersection-and-union ratio of the labeled object in different 2D frame information is determined by processing the second 2D frame information obtained by image data processing and the first 2D frame information obtained by processing the 3D point cloud data, and then the intersection-and-union ratio is compared with the first threshold, so that the labeled data with high accuracy can be determined quickly and in large quantities for monocular 3D target detection.

[0092] Step S205 , calculating intersection-and-union ratios of the plurality of first 2D frame information and the plurality of second 2D frame information respectively.

[0093] Through the above settings, the intersection-and-union ratio is used to reflect the matching degree between the two-dimensional information displayed in the first 2D frame information and the two-dimensional information displayed in the second 2D frame information of the annotated object. Then, the two types of annotated data are determined by comparing the intersection-and-union ratio with the first threshold, which helps to further improve the accuracy of the determined annotated data.

[0094] Step S206, determining whether the intersection-to-union ratio is less than a first threshold.

[0095] Specifically, if yes, that is, the intersection-over-union ratio is less than the first threshold, step S207 is executed; if no, that is, the intersection-over-union ratio is greater than or equal to the first threshold, step S208 is executed.

[0096] Step S207 , deleting the 3D frame information corresponding to the first 2D frame information, retaining the second 2D frame information, and using the second 2D frame information as the annotation data.

[0097] According to an embodiment of the present invention, if the intersection-to-union ratio is less than the first threshold, it means that the two-dimensional information of the labeled object displayed in the current first 2D frame information has a low matching degree with the two-dimensional information reflected in the image data, indicating that the point cloud data may not be able to well reflect the three-dimensional information of the labeled object due to sparse point clouds and other reasons. Therefore, the 3D frame information corresponding to the current first 2D frame information is deleted, and only the second 2D frame information is retained, and the two-dimensional information of the labeled object indicated by the second 2D frame information is used as the labeled data.

[0098] Step S208, determining the matching result of the first 2D frame information and the second 2D frame information according to the intersection-to-union ratio, retaining the 3D frame information corresponding to the matched first 2D frame information and the unmatched second 2D frame information, and using the retained information as the annotation data.

[0099] According to an embodiment of the present invention, if the intersection-to-union ratio is greater than or equal to the first threshold value, and the two-dimensional information of the annotated object displayed in the current first 2D frame information matches the two-dimensional information reflected in the image data to a high degree, the 3D frame information corresponding to the first 2D frame information that has been matched and the second 2D frame information that has not been matched are retained. This is because the 3D frame information reflects the three-dimensional information of the annotated object, which is more comprehensive and accurate than the two-dimensional information reflected in the image data, so the 3D frame information corresponding to the first 2D frame information that has been matched is retained. For the first 2D frame information that has not been matched, similar to step S207, a deletion process is performed, and only the corresponding (i.e., the unmatched) second 2D frame information is retained.

[0100] Step S209, determine the 3D projection center point corresponding to the 3D frame information in the retained information, and the 2D frame center point corresponding to the second 2D frame information in the retained information; calculate the Euclidean distance between the 3D projection center point and the 2D frame center point, and calculate the frame diagonal distance of the second 2D frame information in the retained information; calculate the annotation error of the annotated object according to the Euclidean distance and the frame diagonal distance; update the annotation data according to the annotation error and the second threshold.

[0101] Specifically, according to an embodiment of the present invention, the calculation formula of the Euclidean distance d between the 3D projection center point and the 2D frame center point is as follows:

[0102]

[0103] Among them, (c x , c y ) represents the center point of the 2D box, (c′ x , c′ y ) represents the 3D projection center point.

[0104] The calculation formula of the frame diagonal distance l of the second 2D frame information is as follows:

[0105]

[0106] Among them, (x1, y1) and (x2, y2) represent the diagonal coordinates of the 2D box respectively.

[0107] Further, according to an embodiment of the present invention, the above-mentioned retained information includes one or more annotated objects; if the number of annotated objects is one, the step of calculating the annotation error of the image according to the Euclidean distance and the box diagonal distance includes: calculating the normalized distance between the 3D projection center point and the 2D box center point according to the Euclidean distance and the box diagonal distance; and calculating the annotation error according to the normalized distance.

[0108] Preferably, according to an embodiment of the present invention, if the number of labeled objects is multiple, the step of calculating the labeling error of the image according to the Euclidean distance and the box diagonal distance includes:

[0109] Calculate the average of the normalized distances between the 3D projection center point and the 2D box center point based on the number of annotations, Euclidean distance, and box diagonal distance;

[0110] The annotation error is calculated based on the average of the normalized distance and the number of annotations.

[0111] It can be understood that, the more the number of annotated objects included in the information is, the more accurate the annotation error ultimately determined by calculating the average value of the normalized distance is, thereby making the accuracy of the determined annotation data higher.

[0112] Specifically, according to an embodiment of the present invention, the calculation formula of the annotation error (metric) is as follows:

[0113]

[0114] Where N represents the number of labeled objects.

[0115] According to the technical solution of the embodiment of the present invention, because multiple 3D point cloud data and multiple image data synchronously collected by the laser radar and the camera are received; the multiple 3D point cloud data are processed by the 3D point cloud detection model to obtain multiple 3D detection results, and the multiple 3D detection results are sequentially converted and mapped to obtain multiple first 2D frame information; wherein the first 2D frame information indicates the two-dimensional information of the labeled object; the multiple image data are processed by the 2D target detection model to obtain multiple second 2D frame information; the intersection and union ratios of the multiple first 2D frame information and the multiple second 2D frame information are calculated, and the technical means for determining the labeled data according to the intersection and union ratio and the first threshold value, overcomes the technical problem that the labeled data determined by only relying on 2D images in the related art, because only the two-dimensional information of the labeled object is included, the accuracy of the labeled data is low, and it is difficult to obtain a large amount of labeled data, thereby achieving the ability to combine the three-dimensional information and the two-dimensional information of the labeled object according to the point cloud data collected by the laser radar and the image data collected by the camera, reducing the cost of determining the labeled data, and being able to quickly and efficiently determine a large amount of labeled data.

[0116] Figure 3 is a schematic diagram of main modules of a device for determining labeled data for monocular 3D target detection provided by another embodiment of the present invention; Figure 3 As shown, the labeling data determination device 300 for monocular 3D target detection provided by the embodiment of the present invention mainly includes:

[0117] The data receiving module 301 is used to receive multiple 3D point cloud data and multiple image data collected synchronously by the laser radar and the camera.

[0118] Specifically, the calibration parameters and conversion matrix parameters between the vehicle-mounted sensors can be pre-set, and the laser radar and camera can be used to synchronously collect the point cloud data and image data of the surrounding environment during the vehicle's driving process. According to an embodiment of the present invention, a laser radar with fewer beams, such as a 16-beam laser radar, can be used to reduce the cost of the laser radar, thereby reducing the cost of determining the labeled data. Although the point cloud collected by the laser radar with fewer beams is relatively sparse, and the labeled objects at a long distance are very likely to have no scanned point cloud, combined with the image data collected by the camera, the accuracy of the determined labeled data can be improved while ensuring low cost.

[0119] The point cloud data processing module 302 is used to: use a 3D point cloud detection model to process multiple 3D point cloud data to obtain multiple 3D detection results, and perform conversion processing and mapping processing on the multiple 3D detection results in turn to obtain multiple first 2D frame information; wherein the first 2D frame information indicates the two-dimensional information of the labeled object, and further, the two-dimensional information of the labeled object is determined based on the 3D point cloud data.

[0120] Specifically, according to an embodiment of the present invention, an existing 3D point cloud detection model with high detection accuracy and relatively mature detection (such as PointPillars, etc.) can be used to process all 3D point cloud data to obtain a 3D detection result corresponding to each 3D point cloud data, wherein the 3D detection result indicates the three-dimensional information of the annotated object in the point cloud coordinate system. Then, by sequentially converting and mapping the 3D detection results, the first 2D frame information can be obtained, indicating the two-dimensional information of the annotated object in the camera coordinate system, so as to combine it with the two-dimensional information of the annotated object obtained after image data processing (the second 2D frame information) to determine the annotated data with higher accuracy.

[0121] Furthermore, according to an embodiment of the present invention, the point cloud data processing module 302 is also used to: respectively convert multiple 3D detection results to obtain multiple 3D frame information in the camera coordinate system; wherein the 3D frame information indicates the three-dimensional information of the labeled object; and respectively map the multiple 3D frame information to obtain multiple first 2D frame information corresponding to the multiple 3D frame information.

[0122] Through the above settings, the 3D detection results in multiple point cloud coordinate systems are converted and processed respectively to obtain 3D frame information in the camera coordinate system, which is used to indicate the three-dimensional information of the labeled object. Then, the 3D frame information is mapped to obtain the two-dimensional information of the labeled object in the camera coordinate system.

[0123] The image data processing module 303 is used to process the multiple image data using the 2D object detection model to obtain multiple second 2D frame information.

[0124] Specifically, according to an embodiment of the present invention, an existing 2D target detection model (such as SSD, YOLO, etc.) can be used to process multiple image data collected by the camera to obtain the second 2D frame information. The intersection-and-union ratio of the labeled object in different 2D frame information is determined by processing the second 2D frame information obtained by image data processing and the first 2D frame information obtained by processing the 3D point cloud data, and then the intersection-and-union ratio is compared with the first threshold value, so that the labeled data with higher accuracy can be determined quickly and in large quantities for monocular 3D target detection. Among them, the second 2D frame information indicates the two-dimensional information of the labeled object determined based on the image data.

[0125] The labeling data determining module 304 is used to calculate the intersection-and-union ratios of the plurality of first 2D frame information and the plurality of second 2D frame information, and determine the labeling data according to the intersection-and-union ratios and a first threshold.

[0126] Specifically, according to an embodiment of the present invention, the above-mentioned annotation data determination module 304 is also used for:

[0127] Determine whether the intersection-to-union ratio is less than a first threshold;

[0128] If yes, delete the 3D frame information corresponding to the first 2D frame information, retain the second 2D frame information, and use the second 2D frame information as the annotation data;

[0129] If not, the matching result of the first 2D frame information and the second 2D frame information is determined according to the intersection-and-union ratio, the 3D frame information corresponding to the matched first 2D frame information and the unmatched second 2D frame information are retained, and the retained information is used as the annotation data.

[0130] Through the above settings, the intersection-and-union ratio is compared with the first threshold to determine the two types of annotation data. According to a specific implementation of an embodiment of the present invention, the first threshold can be set to 0.5 (the value range of the first threshold is between 0 and 1), that is, if the two-dimensional information displayed in the first 2D frame information of the annotated object and the two-dimensional information displayed in the second 2D frame information are half completely matched, that is, the 3D frame information corresponding to the first 2D frame information of the successfully matched part is retained, and the second 2D frame information that is not successfully matched is retained, and the retained information is used as the annotation data; if the two-dimensional information displayed in the first 2D frame information of the annotated object and the two-dimensional information displayed in the second 2D frame information are less than half of the successfully matched part, only the second 2D frame information is retained as the annotation data. It should be noted that the above setting of the first threshold is only an example, and different first thresholds can be set according to the actual accuracy to be achieved and the amount of annotation data to be determined. If the accuracy to be achieved is higher, a higher first threshold can be set, and if more annotation data is to be obtained, a lower first threshold can be set.

[0131] Further, according to an embodiment of the present invention, if the intersection-over-union ratio is greater than or equal to the first threshold, the labeling data determination module 304 is further configured to:

[0132] Determine a 3D projection center point corresponding to the 3D frame information in the retained information, and a 2D frame center point corresponding to the second 2D frame information in the retained information;

[0133] Calculate the Euclidean distance between the center point of the 3D projection and the center point of the 2D frame, and calculate the frame diagonal distance of the second 2D frame information in the retained information;

[0134] The annotation error of the annotated object is calculated based on the Euclidean distance and the box diagonal distance;

[0135] The labeled data is updated according to the labeling error and the second threshold.

[0136] Through the above settings, in the case where both the first 2D frame information and the second 2D frame information are retained, in order to improve the accuracy of the determined labeled data, it is necessary to calculate the calibration error, and use the calibration error and the second threshold to clean the data. That is, determine the 3D projection center point of the 3D frame information corresponding to the first 2D frame information in the retained information, and the 2D frame center point corresponding to the second 2D frame information in the retained information, and then calculate the Euclidean distance between the two center points, and the frame diagonal distance of the second 2D frame information, and calculate the labeling error of the labeled object based on the Euclidean distance and the frame diagonal distance; then update the labeled data based on the labeling error and the second threshold.

[0137] Exemplarily, according to an embodiment of the present invention, the retained information includes one or more annotated objects; if the number of the annotated objects is one, the annotated data determination module 304 is further configured to:

[0138] According to the Euclidean distance and the diagonal distance of the box, the normalized distance between the center point of the 3D projection and the center point of the 2D box is calculated; and the annotation error is calculated according to the normalized distance.

[0139] Preferably, according to an embodiment of the present invention, if there are multiple labeled objects, the above-mentioned labeling data determination module 304 is further used to:

[0140] Calculate the average of the normalized distances between the 3D projection center point and the 2D box center point based on the number of annotations, Euclidean distance, and box diagonal distance;

[0141] The annotation error is calculated based on the average of the normalized distance and the number of annotations.

[0142] It can be understood that, the more the number of annotated objects included in the information is, the more accurate the annotation error ultimately determined by calculating the average value of the normalized distance is, thereby making the accuracy of the determined annotation data higher.

[0143] According to the technical solution of the embodiment of the present invention, because multiple 3D point cloud data and multiple image data synchronously collected by the laser radar and the camera are received; the multiple 3D point cloud data are processed by the 3D point cloud detection model to obtain multiple 3D detection results, and the multiple 3D detection results are sequentially converted and mapped to obtain multiple first 2D frame information; wherein the first 2D frame information indicates the two-dimensional information of the labeled object; the multiple image data are processed by the 2D target detection model to obtain multiple second 2D frame information; the intersection and union ratios of the multiple first 2D frame information and the multiple second 2D frame information are calculated, and the technical means for determining the labeled data according to the intersection and union ratio and the first threshold value, overcomes the technical problem that the labeled data determined by only relying on 2D images in the related art, because only the two-dimensional information of the labeled object is included, the accuracy of the labeled data is low, and it is difficult to obtain a large amount of labeled data, thereby achieving the ability to combine the three-dimensional information and the two-dimensional information of the labeled object according to the point cloud data collected by the laser radar and the image data collected by the camera, reducing the cost of determining the labeled data, and being able to quickly and efficiently determine a large amount of labeled data.

[0144] Figure 4 An exemplary system architecture 400 is shown to which the method for determining labeled data for monocular 3D object detection or the apparatus for determining labeled data for monocular 3D object detection according to the embodiments of the present invention can be applied.

[0145] like Figure 4 As shown, the system architecture 400 may include terminal devices 401, 402, 403, a network 404 and a server 405 (this architecture is only an example, and the components included in the specific architecture may be adjusted according to the specific application). The network 404 is used to provide a medium for a communication link between the terminal devices 401, 402, 403 and the server 405. The network 404 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0146] Users can use terminal devices 401, 402, 403 to interact with server 405 through network 404 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 401, 402, 403, such as data processing applications, annotated data determination applications, search applications, instant messaging tools, social platform software, etc. (only as examples).

[0147] The terminal devices 401 , 402 , and 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0148] The server 405 may be a server that provides various services, such as a server that processes data (only as an example) received by the user using the terminal devices 401, 402, and 403. The server may analyze and process the received 3D point cloud data, image data, and other data, and feed back the processing results (such as annotation data - only as an example) to the terminal device.

[0149] It should be noted that the method for determining the labeled data for monocular 3D target detection provided in the embodiment of the present invention is generally executed by the server 405 , and accordingly, the device for determining the labeled data for monocular 3D target detection is generally arranged in the server 405 .

[0150] It should be understood that Figure 4 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0151] Reference below Figure 5 , which shows a schematic diagram of the structure of a computer system 500 of a terminal device or a server suitable for implementing an embodiment of the present invention. Figure 5 The terminal device or server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0152] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the system 500 are also stored. The CPU 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0153] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.

[0154] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-mentioned functions defined in the system of the present invention are executed.

[0155] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0156] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0157] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided in a processor. For example, they may be described as follows: a processor includes a data receiving module, a point cloud data processing module, an image data processing module, and a labeling data determination module. The names of these modules do not, in some cases, constitute limitations on the modules themselves. For example, the data receiving module may also be described as a "module for receiving multiple 3D point cloud data and multiple image data synchronously collected by a laser radar and a camera."

[0158] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by a device, the device includes: receiving multiple 3D point cloud data and multiple image data synchronously collected by a laser radar and a camera; using a 3D point cloud detection model to process multiple 3D point cloud data to obtain multiple 3D detection results, and sequentially converting and mapping the multiple 3D detection results to obtain multiple first 2D frame information; wherein the first 2D frame information indicates the two-dimensional information of the labeled object; using a 2D target detection model to process multiple image data to obtain multiple second 2D frame information; calculating the intersection and union ratio of multiple first 2D frame information and multiple second 2D frame information, and determining the labeled data according to the intersection and union ratio and the first threshold.

[0159] According to the technical solution of the embodiment of the present invention, because multiple 3D point cloud data and multiple image data synchronously collected by the laser radar and the camera are received; the multiple 3D point cloud data are processed by the 3D point cloud detection model to obtain multiple 3D detection results, and the multiple 3D detection results are sequentially converted and mapped to obtain multiple first 2D frame information; wherein the first 2D frame information indicates the two-dimensional information of the labeled object; the multiple image data are processed by the 2D target detection model to obtain multiple second 2D frame information; the intersection and union ratios of the multiple first 2D frame information and the multiple second 2D frame information are calculated, and the technical means for determining the labeled data according to the intersection and union ratio and the first threshold value, overcomes the technical problem that the labeled data determined by only relying on 2D images in the related art, because only the two-dimensional information of the labeled object is included, the accuracy of the labeled data is low, and it is difficult to obtain a large amount of labeled data, thereby achieving the ability to combine the three-dimensional information and the two-dimensional information of the labeled object according to the point cloud data collected by the laser radar and the image data collected by the camera, reducing the cost of determining the labeled data, and being able to quickly and efficiently determine a large amount of labeled data.

[0160] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for determining labeled data for monocular 3D target detection, characterized in that: include: Receive multiple 3D point cloud data and multiple image data collected synchronously by the laser radar and camera; Using a 3D point cloud detection model, the plurality of 3D point cloud data are processed to obtain a plurality of 3D detection results, and the plurality of 3D detection results are sequentially converted and mapped to obtain a plurality of first 2D frame information; wherein the first 2D frame information indicates two-dimensional information of the labeled object; Processing the plurality of image data using a 2D object detection model to obtain a plurality of second 2D frame information; Calculate the intersection-and-union ratios of multiple first 2D frame information and multiple second 2D frame information respectively, and determine whether the intersection-and-union ratio is less than a first threshold; if so, delete the 3D frame information corresponding to the first 2D frame information, retain the second 2D frame information, and use the second 2D frame information as the annotation data; if not, determine the matching result of the first 2D frame information and the second 2D frame information according to the intersection-and-union ratio, retain the 3D frame information corresponding to the matched first 2D frame information and the unmatched second 2D frame information, and use the retained information as the annotation data.

2. The method for determining annotation data for monocular 3D target detection according to claim 1, characterized in that: The converting and mapping the multiple 3D detection results in sequence to obtain multiple first 2D frame information includes: The multiple 3D detection results are respectively converted to obtain multiple 3D frame information in the camera coordinate system; wherein the 3D frame information indicates the three-dimensional information of the labeled object; Mapping processing is performed on the multiple 3D frame information respectively to obtain multiple first 2D frame information corresponding to the multiple 3D frame information.

3. The method for determining annotation data for monocular 3D target detection according to claim 1, characterized in that: If the intersection-over-combination ratio is greater than or equal to the first threshold, the method further includes: Determine a 3D projection center point corresponding to the 3D frame information in the retained information, and a 2D frame center point corresponding to the second 2D frame information in the retained information; Calculating the Euclidean distance between the 3D projection center point and the 2D frame center point, and calculating the frame diagonal distance of the second 2D frame information in the retained information; Calculating a labeling error of the labeled object according to the Euclidean distance and the frame diagonal distance; The labeled data is updated according to the labeling error and a second threshold.

4. The method for determining annotation data for monocular 3D target detection according to claim 3, characterized in that: The reserved information includes one or more marked objects; If the number of the labeled object is one, the step of calculating the labeling error of the image according to the Euclidean distance and the frame diagonal distance includes: Calculating a normalized distance between the 3D projection center point and the 2D frame center point according to the Euclidean distance and the frame diagonal distance; The annotation error is calculated according to the normalized distance.

5. The method for determining annotation data for monocular 3D target detection according to claim 4, characterized in that: If the number of the labeled objects is multiple, the step of calculating the labeling error of the image according to the Euclidean distance and the frame diagonal distance includes: Calculate the average value of the normalized distance between the 3D projection center point and the 2D box center point according to the number of annotations, the Euclidean distance and the box diagonal distance; The annotation error is calculated according to an average value of the normalized distance and the number of annotations.

6. A device for determining label data for monocular 3D target detection, characterized in that: include: The data receiving module is used to receive multiple 3D point cloud data and multiple image data collected synchronously by the laser radar and the camera; A point cloud data processing module is used to: use a 3D point cloud detection model to process the multiple 3D point cloud data to obtain multiple 3D detection results, and sequentially perform conversion processing and mapping processing on the multiple 3D detection results to obtain multiple first 2D frame information; wherein the first 2D frame information indicates two-dimensional information of the labeled object; An image data processing module is used to: process the multiple image data using a 2D object detection model to obtain multiple second 2D frame information; The labeling data determination module is used to: calculate the intersection-and-union ratios of the multiple first 2D frame information and the multiple second 2D frame information respectively, and determine whether the intersection-and-union ratios are less than a first threshold; if so, delete the 3D frame information corresponding to the first 2D frame information, retain the second 2D frame information, and use the second 2D frame information as the labeling data; if not, determine the matching result of the first 2D frame information and the second 2D frame information according to the intersection-and-union ratio, retain the 3D frame information corresponding to the matched first 2D frame information and the unmatched second 2D frame information, and use the retained information as the labeling data.

7. An electronic device for determining annotation data for monocular 3D target detection, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images