Unsupervised monocular three-dimensional target detection training processing and label generation method and system
Through instance segmentation and point cloud density clustering technology, a three-dimensional enclosure box is generated as a depth label, which solves the problem of large workload and uncertainty of manual labeling in monocular 3D object detection, and improves the training efficiency and deep recognition effect of the model.
Patent Information
- Application Number
- CN202311039215.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-16
- Publication Date
- 2025-02-21
AI Technical Summary
Due to the lack of depth information, monocular 3D object detection leads to high uncertainty when mapping two-dimensional space to three-dimensional space. The existing methods require a large number of and relatively accurate depth tags for training, and manual labeling is large and difficult to be accurate.
The image data is mapped to point cloud data through the instance segmentation module, the point cloud density clustering module is used to divide the clusters, and the image data is combined to generate a three-dimensional enclosure box as a depth tag to reduce the workload and error of manual annotation.
It effectively reduces the workload and error of manual labeling of depth tags, and improves the training efficiency and depth recognition effect of monocular object detection model.
Smart Images
Figure CN119495091A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to an unsupervised monocular three-dimensional target detection training processing, label generation method and system, and belong to the field of computer technology. Background Art
[0002] At present, most 3D object detection methods are based on LiDAR devices, which improve detection accuracy by collecting accurate 3D point cloud depth information. Another detection method is to detect through a monocular camera. Monocular cameras have low equipment prices, robust external parameter changes, and a wide range of application scenarios. Therefore, although monocular 3D detection has the disadvantage of lacking depth information and 3D position, it is also a popular research direction.
[0003] However, monocular 3D object detection is inherently inappropriate because the lack of depth information in monocular images will cause uncertainty when mapping two-dimensional space to three-dimensional space. Currently, the general approach is to regress the size and position of the corresponding target through a deep learning network, which requires a large number of relatively accurate depth labels to train the monocular 3D object detection model. However, manually labeling depth labels on 2D images undoubtedly requires a huge amount of work and is difficult to label accurately. Summary of the invention
[0004] The embodiments of the present invention provide an unsupervised monocular 3D target detection training processing, label generation method and system to reduce the workload and error of manually annotating depth labels, improve the training efficiency of the monocular target detection model, and enhance the depth recognition effect of the monocular target detection model.
[0005] An embodiment of the present invention provides a deep label generation system, including:
[0006] An instance segmentation module, used to perform instance segmentation on the image data, generate an instance mask, map the instance mask to the point cloud data calibrated with the image data, and generate point cloud instance segmentation data;
[0007] The point cloud density clustering module is used to divide the point cloud instance segmentation data into different clusters according to the point cloud density, and determine the point cloud cluster data of the target instance according to the preset screening strategy;
[0008] A three-dimensional depth label generation module is used to project the point cloud cluster data of the target instance onto the bird's-eye view to generate a two-dimensional bounding box of the bird's-eye view, determine the instance height according to the point cloud cluster data in the vertical direction corresponding to the image data, and generate a three-dimensional bounding box according to the two-dimensional bounding box and the instance height as a depth label for training a monocular target detection model.
[0009] An embodiment of the present invention provides a training processing system for unsupervised monocular three-dimensional object detection, comprising:
[0010] The deep label generation processing module is used to:
[0011] Acquire a training data set, wherein the training data set includes registered point cloud data and image data;
[0012] Using the aforementioned deep label generation system, generate a deep label;
[0013] The training processing module of the monocular 3D object detection model is used to:
[0014] The depth label and the image data corresponding thereto are used as monocular three-dimensional object detection training data to train a monocular three-dimensional object detection model.
[0015] An embodiment of the present invention provides a method for generating a deep label, including:
[0016] Performing instance segmentation on the image data to generate an instance mask, and mapping the instance mask to point cloud data calibrated with the image data to generate point cloud instance segmentation data;
[0017] According to the point cloud density, the point cloud instance segmentation data is divided into different clusters, and the point cloud cluster data of the target instance is determined according to the preset screening strategy;
[0018] The point cloud cluster data of the target instance is projected onto the bird's-eye view to generate a two-dimensional bounding box of the bird's-eye view, the instance height is determined according to the point cloud cluster data in the vertical direction corresponding to the image data, and a three-dimensional bounding box is generated according to the two-dimensional bounding box and the instance height as a depth label for training a monocular target detection model.
[0019] The embodiment of the present invention provides a training processing method for unsupervised monocular three-dimensional object detection, comprising:
[0020] Acquire a training data set, wherein the training data set includes registered point cloud data and image data;
[0021] Using the aforementioned deep label generation system, generate a deep label;
[0022] The depth label and the image data corresponding thereto are used as monocular three-dimensional object detection training data to train a monocular three-dimensional object detection model.
[0023] The unsupervised monocular three-dimensional target detection training processing, label generation method and system of the embodiments of the present invention utilize the three-dimensional depth characteristics of point cloud data and the existing aligned point cloud and image data sets to generate depth labels that can be used to train monocular target detection models, thereby reducing the workload and errors of manual labeling, more efficiently training the monocular target detection model, and improving the depth recognition effect of the monocular target detection model.
[0024] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 A schematic diagram of the structure of a deep label generation system according to an embodiment of the present invention;
[0026] Figure 2 A schematic diagram of the structure of a training and processing system for unsupervised monocular three-dimensional object detection according to an embodiment of the present invention;
[0027] Figure 3 A schematic diagram of a flow chart of a method for generating a deep tag according to an embodiment of the present invention;
[0028] Figure 4 The present invention is a flowchart of a training method for unsupervised monocular three-dimensional object detection according to an embodiment of the present invention. DETAILED DESCRIPTION
[0029] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.
[0030] In order to provide better training data for the monocular target detection model, an embodiment of the present invention proposes a network architecture as a deep label generation system, and efficiently trains the three-dimensional monocular target detection model based on the image training data with deep labels generated by this system.
[0031] In an embodiment of the present invention, the KITTI dataset can be used as training data for a monocular target detection model, and a system and method for generating a deep label according to an embodiment of the present invention can be used to generate a deep label according to unlabeled point cloud data in the KITTI dataset, and combined with the corresponding image data to form training data for a three-dimensional monocular target detection model. It can be understood by those skilled in the art that other datasets that calibrate point cloud data and image data can also be used as training datasets.
[0032] like Figure 1 As shown, it is a schematic diagram of the structure of a deep label generation system according to an embodiment of the present invention, and the system includes:
[0033] The instance segmentation module 11 is used to perform instance segmentation on the image data, generate instance masks, map the instance masks to the point cloud data calibrated with the image data, and generate point cloud instance segmentation data.
[0034] As shown in the figure, the point cloud data and image data are data that have been calibrated. The part of the instance segmentation process that performs image instance segmentation can be implemented using a general instance segmentation model for image processing, such as the MaskRCNN (Mask Region Convolutional Neural Network) model and other models with instance segmentation functions. Specifically, the image data can be convolved to extract the ROI (Region of Interest) area of the image data, and the ROI area can be processed by a fully convolutional network connection (FCC) to generate an instance mask.
[0035] Specifically, a conventional convolution operation is performed on the image data to obtain the region of interest (ROI). Subsequently, a full convolutional connection (FCC) is used to predict the mask for each ROI. The specific formula is as follows:
[0036] M=f out (f conv (f deconv (f conv (RoI)))) Formula (1)
[0037] where f deconv represents the deconvolution network, which can ensure that the up-sampling of the convolution network output features is aligned with the original feature map. out It represents the output network, which converts the upsampled feature map into the corresponding instance mask. Then, the instance mask is mapped to the three-dimensional point cloud space to obtain the corresponding lidar ROI point set, that is, the point cloud instance segmentation data.
[0038] Since the point cloud data and image data have been calibrated, the mask generated after instance segmentation in the image can be mapped to the point cloud data, and the corresponding part of the point cloud data can be obtained as the above-mentioned point cloud instance segmentation data. There may be multiple instances segmented here, so there will also be multiple point cloud instance segmentation data.
[0039] The point cloud density clustering module 12 is used to divide the point cloud instance segmentation data into different clusters according to the point cloud density, and determine the point cloud cluster data of the target instance according to the preset screening strategy. There may be many noise points in the obtained point cloud instance segmentation data, such as occlusion points, background points, etc. Here, the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm can be used to denoise the point cloud instance segmentation data and divide it into different clusters.
[0040] DBSCAN is used to find clusters of data points with similar density from a data set. It clusters data points by dividing them into core points, boundary points, and noise points. A core point is a data point that has a sufficient number of neighbor points within a specified radius, a boundary point is a data point that does not have a sufficient number of neighbor points within the radius but has core points among its neighbors, and a noise point is a data point that is neither a core point nor a boundary point. In the clustering process of DBSCAN, a cluster is formed by expanding from the core point through the neighbor relationship of the point until no more neighbor points can be added. In this way, DBSCAN can find clusters of any shape and size, and can identify isolated data points as noise points.
[0041] The screening strategy mentioned in the embodiment of the present invention can be to use clusters greater than a certain density threshold as point cloud cluster data of the target instance, or to select the cluster with the largest density as the point cloud data of the target instance. In addition, the family containing a large number of point clouds can also be used as the point cloud data of the target instance. The screening strategy depends on the purpose of the deep label to be generated and the training purpose of the three-dimensional monocular target detection model. In an application scenario, it is more desirable to train a monocular vision model for the recognition of vehicles within a certain distance range, so the cluster with the largest density can be used as the point cloud cluster data of the target instance.
[0042] The three-dimensional depth label generation module 13 is used to project the point cloud cluster data of the target instance onto the bird's-eye view, generate a two-dimensional bounding box of the bird's-eye view, determine the instance height according to the point cloud cluster data in the vertical direction corresponding to the image data, and generate a three-dimensional bounding box based on the two-dimensional bounding box and the instance height as a depth label for training the monocular target detection model.
[0043] Specifically, after projecting the point cloud cluster onto the bird's-eye view, the rotating caliper algorithm can be used to determine the two-dimensional bounding box. As shown in the figure, the convex hull of the point cloud cluster is first used to form an irregular figure that encloses the point cloud at the minimum, and then the rotating caliper method is combined to form a rectangular box that encloses the point cloud at the minimum.
[0044] Since the image data and point cloud data have been calibrated, the height of the 3D cuboid bounding box can be determined directly based on the height information on the image data, or the height of the 3D cuboid bounding box can be determined by finding the difference between the farthest points in the point cloud cluster data of the target instance based on the direction determined in the image data. In addition, the center point can be calculated based on the point cloud cluster data and output as part of the depth label.
[0045] It should be noted that the models contained in each module in the above-mentioned deep label generation system can be pre-trained using training data of other point clouds or images, and are not limited to training data sets for monocular three-dimensional target detection.
[0046] like Figure 2 The figure is a schematic diagram of the structure of the unsupervised monocular 3D target detection training processing system according to an embodiment of the present invention, which includes a depth label generation processing module 21 and a monocular 3D target detection model training processing module 22.
[0047] The depth label generation processing module 21 is used to: obtain a training data set, the training data set includes the registered point cloud data and image data; and generate a depth label using the depth label generation system of the aforementioned embodiment.
[0048] The training processing module 22 of the monocular 3D object detection model is used to train the monocular 3D object detection model using the depth label and the image data corresponding thereto as the monocular 3D object detection training data.
[0049] like Figure 3 As shown, it is a schematic diagram of a flow chart of a method for generating a deep label according to an embodiment of the present invention, the method comprising:
[0050] S101: performing instance segmentation on the image data, generating an instance mask, and mapping the instance mask to the point cloud data calibrated with the image data to generate point cloud instance segmentation data.
[0051] Among them, performing instance segmentation on image data and generating instance masks may include: performing a convolution operation on the image data, extracting a region of interest from the image data, performing a full convolutional network connection process on the region of interest, and generating an instance mask.
[0052] Partitioning the point cloud instance segmentation data into different clusters according to the point cloud density may include: performing denoising processing on the point cloud instance segmentation data and dividing the data into different clusters by using a density-based spatial clustering algorithm.
[0053] S102: According to the point cloud density, the point cloud instance segmentation data is divided into different clusters, and the point cloud cluster data of the target instance is determined according to the preset screening strategy. Among them, the screening strategy mentioned in the embodiment of the present invention can be to use the cluster greater than a certain density threshold as the point cloud cluster data of the target instance, or to select the cluster with the largest density as the point cloud data of the target instance. In addition, the family containing a large number of point clouds can also be used as the point cloud data of the target instance. The screening strategy depends on the purpose of the deep label to be generated and the training purpose of the three-dimensional monocular target detection model.
[0054] S103: Project the point cloud cluster data of the target instance onto the bird's-eye view to generate a two-dimensional bounding box of the bird's-eye view, determine the instance height according to the point cloud cluster data in the vertical direction corresponding to the image data, and generate a three-dimensional bounding box based on the two-dimensional bounding box and the instance height as a depth label for training a monocular target detection model.
[0055] The step of projecting the point cloud cluster data of the target instance onto the bird's-eye view and generating a two-dimensional bounding box of the bird's-eye view may include: determining the two-dimensional bounding box using a rotating caliper algorithm according to the convex hull points of the point cloud data projected onto the bird's-eye view.
[0056] In addition, since the image data and point cloud data have been calibrated, the height of the 3D cuboid bounding box can be determined directly based on the height information on the image data, or the height of the 3D cuboid bounding box can be determined by finding the difference between the farthest points in the point cloud cluster data of the target instance based on the direction determined in the image data. In addition, the center point can be calculated based on the point cloud cluster data and output as part of the depth label.
[0057] like Figure 4 As shown, it is a flow chart of a training processing method for unsupervised monocular three-dimensional target detection according to an embodiment of the present invention, and the method includes:
[0058] S201: Acquire a training data set, where the training data set includes registered point cloud data and image data;
[0059] S202: Generate a depth label using the depth label generation system of the previous embodiment;
[0060] S203: Use the depth labels and the image data corresponding thereto as monocular 3D object detection training data to train a monocular 3D object detection model.
[0061] The technical principles and technical effects of the method embodiment of the present invention are essentially consistent with those of the aforementioned system embodiment. In view of the fact that they have been described in detail above, they will not be repeated here.
[0062] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.
[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A deep label generation system, comprising: An instance segmentation module, used to perform instance segmentation on the image data, generate an instance mask, map the instance mask to the point cloud data calibrated with the image data, and generate point cloud instance segmentation data; The point cloud density clustering module is used to divide the point cloud instance segmentation data into different clusters according to the point cloud density, and determine the point cloud cluster data of the target instance according to the preset screening strategy; A three-dimensional depth label generation module is used to project the point cloud cluster data of the target instance onto the bird's-eye view to generate a two-dimensional bounding box of the bird's-eye view, determine the instance height according to the point cloud cluster data in the vertical direction corresponding to the image data, and generate a three-dimensional bounding box according to the two-dimensional bounding box and the instance height as a depth label for training a monocular target detection model.
2. The system according to claim 1, wherein: The method for performing instance segmentation on the image data and generating the instance mask includes: performing a convolution operation on the image data, extracting a region of interest from the image data, performing a full convolutional network connection process on the region of interest, and generating the instance mask.
3. The system according to claim 1, wherein: According to the point cloud density, the point cloud instance segmentation data is divided into different clusters, including: denoising the point cloud instance segmentation data through a density-based spatial clustering algorithm, and dividing it into different clusters.
4. The system according to claim 1, wherein: According to a preset density screening strategy, determining the point cloud cluster data of the target instance includes: acquiring a cluster with the largest density as the point cloud cluster data of the target instance.
5. The system according to claim 1, wherein: Projecting the point cloud cluster data of the target instance onto the bird's-eye view, generating a two-dimensional bounding box of the bird's-eye view includes: The two-dimensional bounding box is determined using a rotating caliper algorithm according to the convex hull points of the point cloud data projected onto the bird's-eye view.
6. A training and processing system for unsupervised monocular three-dimensional object detection, comprising: The deep label generation processing module is used to: Acquire a training data set, wherein the training data set includes registered point cloud data and image data; Using the deep label generation system according to any one of claims 1 to 5 to generate a deep label; The training processing module of the monocular 3D object detection model is used to: The depth label and the image data corresponding thereto are used as monocular three-dimensional object detection training data to train a monocular three-dimensional object detection model.
7. A method for generating a deep label, comprising: Performing instance segmentation on the image data to generate an instance mask, and mapping the instance mask to point cloud data calibrated with the image data to generate point cloud instance segmentation data; According to the point cloud density, the point cloud instance segmentation data is divided into different clusters, and the point cloud cluster data of the target instance is determined according to the preset screening strategy; The point cloud cluster data of the target instance is projected onto the bird's-eye view to generate a two-dimensional bounding box of the bird's-eye view, the instance height is determined according to the point cloud cluster data in the vertical direction corresponding to the image data, and a three-dimensional bounding box is generated according to the two-dimensional bounding box and the instance height as a depth label for training a monocular target detection model.
8. The method according to claim 7, wherein: The method for performing instance segmentation on the image data and generating the instance mask includes: performing a convolution operation on the image data, extracting a region of interest from the image data, performing a full convolutional network connection process on the region of interest, and generating the instance mask.
9. The method according to claim 7, wherein: According to the point cloud density, the point cloud instance segmentation data is divided into different clusters, including: denoising the point cloud instance segmentation data through a density-based spatial clustering algorithm, and dividing it into different clusters.
10. A training and processing method for unsupervised monocular three-dimensional object detection, comprising: Acquire a training data set, wherein the training data set includes registered point cloud data and image data; Using the deep label generation system according to any one of claims 1 to 5 to generate a deep label; The depth label and the image data corresponding thereto are used as monocular three-dimensional object detection training data to train a monocular three-dimensional object detection model.