Weak supervision monocular three-dimensional target detection training processing and label generation method and system

Abstract

The embodiment of the invention provides a weak supervision monocular three-dimensional target detection training processing and label generation method and system, and the method comprises the steps: generating global feature data through three-dimensional sparse convolution of different scales; key points are extracted, and key point feature vectors are generated; extracting feature data related to the key points, and encoding the feature data and the feature vectors of the key points to generate global feature data of the key points; splicing the global feature data and the key point global feature data; performing up-sampling to generate point cloud proposal data; and generating a depth label according to the point cloud proposal data. According to the embodiment of the invention, the depth label which can be used for training the monocular target detection model is generated by using the three-dimensional depth characteristic of the point cloud data and the data set of the existing registered point cloud and image, so that the workload and error of manual labeling are reduced, the monocular model is trained more efficiently, and the detection accuracy of the monocular target detection model is improved. And the depth identification effect of the monocular target detection model is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to a weakly supervised monocular three-dimensional target detection training processing, label generation method and system, and belong to the field of computer technology. Background Art

[0002] At present, most 3D object detection methods are based on LiDAR devices, which improve detection accuracy by collecting accurate 3D point cloud depth information. Another detection method is to detect through a monocular camera. Monocular cameras have low equipment prices, robust external parameter changes, and a wide range of application scenarios. Therefore, although monocular 3D detection has the disadvantage of lacking depth information and 3D position, it is also a popular research direction.

[0003] However, monocular 3D object detection is inherently inappropriate because the lack of depth information in monocular images will cause uncertainty when mapping two-dimensional space to three-dimensional space. Currently, the general approach is to regress the size and position of the corresponding target through a deep learning network, which requires a large number of relatively accurate depth labels to train the monocular 3D object detection model. However, manually labeling depth labels on 2D images undoubtedly requires a huge amount of work and is difficult to label accurately. Summary of the invention

[0004] The embodiments of the present invention provide a weakly supervised monocular 3D target detection training processing, label generation method and system to reduce the workload and error of manually annotating depth labels, improve the training efficiency of the monocular target detection model, and enhance the depth recognition effect of the monocular target detection model.

[0005] An embodiment of the present invention provides a deep label generation system, including:

[0006] The 3D multi-scale global feature extraction module is used to divide the point cloud data into voxels and extract the voxel feature data of each voxel, and then process the voxel feature data with 3D sparse convolution of different scales to generate global feature data;

[0007] A key point extraction module, used to extract multiple key points from the point cloud data and generate a key point feature vector;

[0008] A key point global feature encoding module is used to extract feature data related to each key point from the global feature data at each scale, and encode these feature data with the key point feature vector at each scale to generate key point global feature data corresponding to each key point;

[0009] A feature data merging module, used for splicing the global feature data and the key point global feature data to generate merged feature data;

[0010] An upsampling processing module, used to upsample the merged feature data to generate point cloud proposal data;

[0011] A three-dimensional depth label generation module is used to generate a three-dimensional bounding box surrounding the target according to the point cloud proposal data as a depth label for training a monocular target detection model.

[0012] An embodiment of the present invention provides a training processing system for weakly supervised monocular three-dimensional object detection, including:

[0013] The depth label generation processing module is used to:

[0014] Acquire a training data set, the training data set comprising registered point cloud data and image data, wherein the point cloud data comprises first point cloud data with a first depth label and second point cloud data without a depth label, and the image data has a category label;

[0015] Using the first point cloud data with the first depth label to train the aforementioned depth label generation system;

[0016] Using the trained depth label generation system, processing the second point cloud data without the depth label to generate a second depth label;

[0017] The training processing module of the monocular 3D target detection model is used to:

[0018] Using the second depth label and the image data corresponding to the second point cloud data as monocular three-dimensional object detection training data to train a monocular three-dimensional object detection model;

[0019] or,

[0020] The first depth label and the image data corresponding to the first point cloud data, and the second depth label and the image data corresponding to the second point cloud data are used as monocular three-dimensional object detection training data to train a monocular three-dimensional object detection model.

[0021] An embodiment of the present invention provides a method for generating a deep label, including:

[0022] Divide the point cloud data into voxels and extract the voxel feature data of each voxel. Then, process the voxel feature data with three-dimensional sparse convolution of different scales to generate global feature data.

[0023] Extracting multiple key points from the point cloud data and generating key point feature vectors;

[0024] Extracting feature data related to each key point from the global feature data at each scale, and encoding the feature data with the key point feature vector at each scale to generate key point global feature data corresponding to each key point;

[0025] Splicing the global feature data and the key point global feature data to generate merged feature data;

[0026] Upsampling the merged feature data to generate point cloud proposal data;

[0027] A three-dimensional bounding box surrounding the target is generated according to the point cloud proposal data as a depth label for training a monocular target detection model.

[0028] The embodiment of the present invention provides a training processing method for weakly supervised monocular three-dimensional object detection, comprising:

[0029] Acquire a training data set, the training data set comprising registered point cloud data and image data, wherein the point cloud data comprises first point cloud data with a first depth label and second point cloud data without a depth label, and the image data has a category label;

[0030] Using the first point cloud data with the first depth label to train the aforementioned depth label generation system;

[0031] Using the trained depth label generation system, processing the second point cloud data without the depth label to generate a second depth label;

[0032] Use the second depth label and the image data corresponding to the second point cloud data as monocular three-dimensional target detection training data to train a monocular three-dimensional target detection model; or use the first depth label and the image data corresponding to the first point cloud data, and the second depth label and the image data corresponding to the second point cloud data as monocular three-dimensional target detection training data to train a monocular three-dimensional target detection model.

[0033] The weakly supervised monocular three-dimensional target detection training processing, label generation method and system of the embodiments of the present invention utilize the three-dimensional depth characteristics of point cloud data and the existing aligned point cloud and image data sets to generate depth labels that can be used to train monocular target detection models, thereby reducing the workload and errors of manual labeling, more efficiently training the monocular target detection model, and improving the depth recognition effect of the monocular target detection model.

[0034] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of the structure of a deep label generation system according to an embodiment of the present invention;

[0036] Figure 2 A schematic diagram of the structure of a training and processing system for weakly supervised monocular three-dimensional object detection according to an embodiment of the present invention;

[0037] Figure 3 A schematic diagram of a flow chart of a method for generating a deep tag according to an embodiment of the present invention;

[0038] Figure 4 The present invention is a flowchart of a training method for weakly supervised monocular 3D object detection according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0040] In order to provide better training data for the monocular target detection model, an embodiment of the present invention proposes a network architecture as a deep label generation system, and efficiently trains the monocular target detection model based on the image training data with deep labels generated by this system.

[0041] In an embodiment of the present invention, a weak supervision model is adopted to construct a training system. By using a small amount of existing labels in the data set and adopting a neural network learning method, high-quality deep labels are generated, thereby greatly improving the monocular detection accuracy.

[0042] like Figure 1 As shown, it is a schematic diagram of the structure of a deep label generation system according to an embodiment of the present invention, and the system includes:

[0043] The three-dimensional multi-scale global feature extraction module 11 is used to divide the point cloud data into voxels and extract the voxel feature data of each voxel, and then process the voxel feature data using three-dimensional sparse convolutions of different scales to generate global feature data.

[0044] There are two mainstream ways to infer the three-dimensional bounding box of a point cloud. One is a point-based method, which has a larger point set receptive field, but has the characteristics of high computational cost. The other is a voxel-based method, which has high computational efficiency, but low local detail of the point cloud. The method for inferring the three-dimensional bounding box in the weakly supervised mode proposed in the embodiment of the present invention integrates the advantages of both methods. The key is to combine and encode the voxel-based multi-scale point cloud features with the key points of the point cloud data. The main function of the three-dimensional multi-scale global feature extraction module 11 is to extract global feature data of different scales based on a voxel-based method. Specifically, the three-dimensional multi-scale global feature extraction module 11 may include:

[0045] The voxel feature extraction module is used to perform voxel division on the point cloud data and extract the voxel feature data of each voxel. This module is not shown in the figure. It is divided into voxel space through voxels. The spatial resolution of a unit voxel is L×W×H (length, width and height), that is, the entire point cloud space is divided into multiple small cubic grids. The voxel feature data of a unit voxel can be the average value after all the features of the voxel are extracted point by point. Specifically, for each point, the position, normal vector, color, curvature, etc. of the point are calculated. These features are then merged point by point and feature by feature into the features of the voxel by various methods, such as local neighborhood statistics, normal estimation, color histogram, etc., merged into voxel features. The voxel features of each voxel form the voxel feature data for the entire point cloud, so that further global feature extraction can be performed.

[0046] The multi-scale voxel feature extraction module includes a plurality of three-dimensional sparse convolutional layers and a 2D bird's-eye view feature processing layer connected in sequence, and is used to extract features of different scales from the voxel feature data layer by layer through a plurality of three-dimensional sparse convolutional layers, and finally generate the first bird's-eye view feature data as global feature data through the bird's-eye view feature processing layer. As shown in the figure, it uses three sparse convolutional layers for voxels (Voxel) (represented as Voxel Layer1 to Voxel Layer3 in the figure) and a bird's-eye view feature processing layer (represented as BEV Layer in the figure). As an example, the figure also marks the number of input channels and output channels of each layer.

[0047] As shown in the figure, after the voxel feature data passes through a series of 3D sparse convolution operations, the original point cloud feature F is converted into feature volumes F1, F2, and F3 sampled at different scales. In order to generate high-quality 3D global features, the global features after 3D convolution are converted into 2D bird's-eye view feature maps, and the BEV-3Dbox method is used to extract high-quality 3D proposals. The specific operation is to stack the 3D feature volumes in the height dimension to obtain a bird's-eye view feature map of size L×W. At the same time, each detection category can have 3 3D anchor boxes, and the sizes of these anchor boxes can be determined according to the average value of the 3D object. In addition, the completely overlapping anchor boxes are associated with the vertical position of each pixel point in the bird's-eye view feature map to improve the recall performance.

[0048] The key point extraction module 12 is used to extract multiple key points from the point cloud data and generate a key point feature vector. Specifically, the key point extraction can use the farthest point sampling (FPS) algorithm to sample n key points from the point cloud P, denoted as K = {p1, ..., pn}, where for the KITTI dataset, n = 2,048, and for the Waymo dataset, n = 4,096. The key points sampled by this strategy can be evenly distributed around non-empty voxels, so that the key point features and can represent the entire scene.

[0049] The key point global feature encoding module 13 is used to extract feature data related to each key point from the global feature data at each scale, and encode these feature data with the key point feature vector at each scale to generate key point global feature data corresponding to each key point. The main function of this module is to encode multi-scale global features onto key points.

[0050] Specifically, first, the points around the key point are voxels with multiple semantic features. In this embodiment, As the 3D voxel vector set in the kth level (the vector includes the global features of the voxel and the coordinates of the voxel), the kth level can also be understood as the voxel features obtained through the k-level convolution layer, so l is used in the formula k The non-empty voxels in the set are denoted by g. k is the number of non-empty voxels in level k. Key point P i The feature generation is as follows:

[0051] in,

[0052] Among them, E represents the operation of encoding voxel features by the multi-layer perception layer. Max represents the maximum pooling operation along the channel, thereby aggregating different numbers of adjacent voxel features to the key point P iThe eigenvector of Then, 3D global features at different k scales are cascaded to generate key points P i The multi-scale semantic features of . The specific formula can be expressed as:

[0053]

[0054] Furthermore, the original point cloud features associated with the key points can be and 2D Bird's Eye View Features To further enrich the key point features, thereby capturing richer multi-scale contextual information. Specifically, the original point cloud features and bird's-eye view features can be selected within a specified range around the key point as the original point cloud features related to the key point. and 2D Bird's Eye View Features

[0055]

[0056] Among them, for the bird's-eye view feature In the embodiment of the present invention, the key point p i Map it to the 2D bird's-eye view map and use interpolation to obtain the bird's-eye view feature vector corresponding to the key point This strategy makes the key point features generated It combines the global feature information of the entire scene integrated by 3D voxels, and at the same time, it also has the local precise position feature information based on points. The key point features finally obtained can be used as follows:

[0057]

[0058] Through the processing of the above-mentioned key point global map feature encoding module, feature information including the key point features themselves and feature information of various scales and bird's-eye view perspectives is finally generated, and feature information of the original point cloud is also included.

[0059] The feature data merging module 14 is used to splice the global feature data and the key point global feature data to generate merged feature data. As shown in the figure, the module merges the output bird's-eye view features after multi-layer convolution (i.e., the most abstract global feature data) with the key point global feature data encoded based on each key point and containing features of each scale, and then performs subsequent deep label recognition processing based on the merged features.

[0060] The upsampling processing module 15 is used to upsample the merged feature data to generate point cloud proposal data. The feature data output by the feature data merging module 14 is relatively abstract data. After upsampling processing, it forms relatively specific feature data, thereby meeting the data requirements for three-dimensional depth label recognition.

[0061] The three-dimensional depth label generation module 16 is used to generate a three-dimensional bounding box surrounding the target according to the point cloud proposal data as a depth label for training a monocular target detection model. The three-dimensional depth label generation module 16 here can be processed by a target recognition model such as PointNet, PointNet++, etc., wherein the feature data output by the upsampling processing module 15 is used as the data of the feature input part of the target recognition model, and then the output 3D candidate boxes in these target recognition models are obtained as the depth labels of the embodiments of the present invention.

[0062] It should be noted that the above-mentioned deep label generation system can be trained using a small amount of point cloud data with deep labels in the data set, and then use the trained deep label generation system to process a large amount of point cloud data without deep labels, thereby automatically generating deep labels. Then, in conjunction with the image data collected and calibrated at the same time as the point cloud data, training data with deep labels for training a monocular three-dimensional target detection model is formed, which is used for training a monocular three-dimensional target recognition model. Therefore, in an embodiment of the present invention, this method of using data for training is called a weak supervision mode.

[0063] like Figure 2 The figure is a schematic diagram of the structure of the training processing system of weakly supervised monocular 3D object detection according to an embodiment of the present invention, which includes a depth label generation processing module 21 and a monocular 3D object detection model training processing module 22.

[0064] The depth label generation processing module 21 is used for:

[0065] Acquire a training data set, the training data set includes registered point cloud data and image data, wherein the point cloud data includes first point cloud data with a first depth label and second point cloud data without a depth label, and the image data has a category label. Specifically, the training data set may be a KITTI data set;

[0066] Training the aforementioned depth label generation system using the first point cloud data with the first depth label;

[0067] Using the trained depth label generation system, processing the second point cloud data without the depth label to generate a second depth label;

[0068] The training processing module 22 of the monocular 3D target detection model is used to:

[0069] Using the second depth label and the image data corresponding to the second point cloud data as monocular three-dimensional object detection training data to train the monocular three-dimensional object detection model;

[0070] or,

[0071] The monocular 3D object detection model is trained using the first depth label and the image data corresponding to the first point cloud data, and the second depth label and the image data corresponding to the second point cloud data as monocular 3D object detection training data.

[0072] like Figure 3 As shown, it is a schematic diagram of a flow chart of a method for generating a deep label according to an embodiment of the present invention, the method comprising:

[0073] S101: Divide the point cloud data into voxels, extract voxel feature data of each voxel, and then process the voxel feature data using three-dimensional sparse convolutions of different scales to generate global feature data. Specifically, this step may include:

[0074] Divide the point cloud data into voxels and extract the voxel feature data of each voxel;

[0075] The voxel feature data is subjected to three-dimensional sparse convolution layer by layer through multiple sequentially connected three-dimensional sparse convolution layers, so as to extract features of different scales, and finally the first bird's-eye view feature data is generated as the global feature data through the bird's-eye view feature processing layer.

[0076] S102: extract multiple key points from the point cloud data and generate key point feature vectors.

[0077] S103: extracting feature data related to each key point from the global feature data at each scale, and encoding the feature data with the key point feature vector at each scale to generate key point global feature data corresponding to each key point. Specifically, this step may include:

[0078] Respectively obtain the first feature extraction data output by each three-dimensional sparse convolutional layer, extract the second feature extraction data corresponding to the adjacent voxels within the specified range of the key point, and obtain the second bird's-eye view feature data corresponding to the adjacent voxels within the specified range of the key point from the first bird's-eye view feature data;

[0079] The extracted multiple second feature extraction data and second bird's-eye view feature data are aggregated with feature vectors corresponding to the key points to generate key point global feature data.

[0080] S104: Joining the global feature data and the key point global feature data to generate merged feature data.

[0081] S105: Upsample the merged feature data to generate point cloud proposal data.

[0082] S106: Generate a three-dimensional bounding box surrounding the target according to the point cloud proposal data as a depth label for training a monocular target detection model.

[0083] like Figure 4 As shown, it is a flow chart of a training processing method for weakly supervised monocular 3D target detection according to an embodiment of the present invention, the method comprising:

[0084] S201: Acquire a training data set, the training data set comprising registered point cloud data and image data, wherein the point cloud data comprises first point cloud data with a first depth label and second point cloud data without a depth label, and the image data has a category label;

[0085] S202: training the aforementioned depth label generation system using first point cloud data having a first depth label;

[0086] S203: Using the trained depth label generation system, processing the second point cloud data without the depth label to generate a second depth label;

[0087] S204: Use the second depth label and the image data corresponding to the second point cloud data as monocular three-dimensional object detection training data to train the monocular three-dimensional object detection model; or, use the first depth label and the image data corresponding to the first point cloud data, and the second depth label and the image data corresponding to the second point cloud data as monocular three-dimensional object detection training data to train the monocular three-dimensional object detection model.

[0088] The technical principles and technical effects of the method embodiment of the present invention are essentially consistent with those of the aforementioned system embodiment. In view of the fact that they have been described in detail above, they will not be repeated here.

[0089] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A deep label generation system, comprising: The 3D multi-scale global feature extraction module is used to divide the point cloud data into voxels and extract the voxel feature data of each voxel, and then process the voxel feature data with 3D sparse convolution of different scales to generate global feature data; A key point extraction module, used to extract multiple key points from the point cloud data and generate a key point feature vector; A key point global feature encoding module is used to extract feature data related to each key point from the global feature data at each scale, and encode these feature data with the key point feature vector at each scale to generate key point global feature data corresponding to each key point; A feature data merging module, used for splicing the global feature data and the key point global feature data to generate merged feature data; An upsampling processing module, used to upsample the merged feature data to generate point cloud proposal data; A three-dimensional depth label generation module is used to generate a three-dimensional bounding box surrounding the target according to the point cloud proposal data as a depth label for training a monocular target detection model.

2. The generation system according to claim 1, wherein: The three-dimensional multi-scale global feature extraction module includes: A voxel feature extraction module is used to perform voxel division on the point cloud data and extract voxel feature data of each voxel; The multi-scale voxel feature extraction module includes a plurality of three-dimensional sparse convolutional layers and a 2D bird's-eye view feature processing layer connected in sequence, and is used to extract features of different scales on the voxel feature data layer by layer through the plurality of three-dimensional sparse convolutional layers, and finally generate the first bird's-eye view feature data as the global feature data through the bird's-eye view feature processing layer.

3. The system according to claim 2, wherein: The extracting feature data related to each key point from the global feature data at each scale, and encoding the feature data with the key point feature vector at each scale to generate key point global feature data corresponding to each key point includes: Respectively obtain the first feature extraction data output by each three-dimensional sparse convolutional layer, extract the second feature extraction data corresponding to the adjacent voxels within the specified range of the key point, and obtain the second bird's-eye view feature data corresponding to the adjacent voxels within the specified range of the key point from the first bird's-eye view feature data; The extracted plurality of second feature extraction data and the second bird's-eye view feature data are aggregated with the feature vector corresponding to the key point to generate the key point global feature data.

4. A training and processing system for weakly supervised monocular 3D object detection, comprising: The deep label generation processing module is used to: Acquire a training data set, the training data set comprising registered point cloud data and image data, wherein the point cloud data comprises first point cloud data with a first depth label and second point cloud data without a depth label, and the image data has a category label; Using first point cloud data with a first depth label to train the depth label generation system according to any one of claims 1 to 3; Using the trained depth label generation system, processing the second point cloud data without the depth label to generate a second depth label; The training processing module of the monocular 3D object detection model is used to: Using the image data corresponding to the second depth label and the second point cloud data as monocular three-dimensional object detection training data to train a monocular three-dimensional object detection model; or, The monocular 3D object detection model is trained using the image data corresponding to the first depth label and the first point cloud data, and the image data corresponding to the second depth label and the second point cloud data as monocular 3D object detection training data.

5. A method for generating a deep label, comprising: Divide the point cloud data into voxels and extract the voxel feature data of each voxel. Then, process the voxel feature data with three-dimensional sparse convolution of different scales to generate global feature data. Extracting multiple key points from the point cloud data and generating key point feature vectors; Extracting feature data related to each key point from the global feature data at each scale, and encoding the feature data with the key point feature vector at each scale to generate key point global feature data corresponding to each key point; Splicing the global feature data and the key point global feature data to generate merged feature data; Upsampling the merged feature data to generate point cloud proposal data; A three-dimensional bounding box surrounding the target is generated according to the point cloud proposal data as a depth label for training a monocular target detection model.

6. The method according to claim 5, wherein the step of dividing the point cloud data into voxels, extracting voxel feature data of each voxel, and then processing the voxel feature data using three-dimensional sparse convolutions of different scales to generate global feature data comprises: Divide the point cloud data into voxels and extract the voxel feature data of each voxel; The voxel feature data is subjected to three-dimensional sparse convolution layer by layer through a plurality of sequentially connected three-dimensional sparse convolution layers, so as to extract features of different scales, and finally, the first bird's-eye view feature data is generated as the global feature data through a bird's-eye view feature processing layer.

7. According to the method of claim 6, extracting feature data related to each key point from the global feature data at each scale, encoding the feature data with the key point feature vector at each scale, and generating key point global feature data corresponding to each key point comprises: Respectively obtain the first feature extraction data output by each three-dimensional sparse convolutional layer, extract the second feature extraction data corresponding to the adjacent voxels within the specified range of the key point, and obtain the second bird's-eye view feature data corresponding to the adjacent voxels within the specified range of the key point from the first bird's-eye view feature data; The extracted plurality of second feature extraction data and the second bird's-eye view feature data are aggregated with the feature vector corresponding to the key point to generate the key point global feature data.

8. A training and processing method for weakly supervised monocular three-dimensional object detection, comprising: Acquire a training data set, the training data set comprising registered point cloud data and image data, wherein the point cloud data comprises first point cloud data with a first depth label and second point cloud data without a depth label, and the image data has a category label; Using first point cloud data with a first depth label to train the depth label generation system according to any one of claims 1 to 3; Using the trained depth label generation system, processing the second point cloud data without the depth label to generate a second depth label; Use the second depth label and the image data corresponding to the second point cloud data as monocular three-dimensional target detection training data to train a monocular three-dimensional target detection model; or use the first depth label and the image data corresponding to the first point cloud data, and the second depth label and the image data corresponding to the second point cloud data as monocular three-dimensional target detection training data to train a monocular three-dimensional target detection model.

Images