A multimodal weakly supervised three-dimensional target detection method, system and device

Abstract

The present invention relates to a multimodal weakly supervised three-dimensional target detection method, system and device. The method comprises: shooting multiple 2D RGB images, generating a three-dimensional visual cone based on the two-dimensional box annotation on the 2D RGB image; in the three-dimensional visual cone, filtering out the ground points in the three-dimensional point cloud collected by the vehicle laser radar sensor, and determining the area with the largest number of three-dimensional points by relying on the seed point region growth method; generating a three-dimensional pseudo-annotated box of the object according to the area with the largest number of three-dimensional points based on the density-based three-dimensional parameter estimation method; training a multimodal superpixel dual-branch network according to the 2D RGB image and the three-dimensional point cloud with the three-dimensional pseudo-annotated box as the target; inputting the 2D RGB image of the current frame and the three-dimensional point cloud of the current scene into the trained multimodal superpixel dual-branch network to generate an overall three-dimensional point cloud. The present invention can alleviate the problem of inconsistent resolution between two-dimensional and three-dimensional modalities, and make full use of multimodal information to identify and locate three-dimensional objects.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and in particular to a multimodal weakly supervised three-dimensional target detection method, system and device. Background Art

[0002] Deep learning has developed rapidly in recent years, but the network currently requires strong supervision information. Three-dimensional object detection, as an important task in the field of autonomous driving and robotics, has also attracted the attention of researchers. The accuracy of fully supervised three-dimensional object detection depends on manually labeled three-dimensional bounding boxes, which requires cumbersome manual operations.

[0003] At present, weakly supervised 3D object detection methods mainly target vehicle categories, but in actual roads, there are also categories such as pedestrians and cyclists in addition to vehicles. At present, weakly supervised 3D object detection methods still lack research on objects with unclear outer edges such as pedestrians and cyclists. The current difficulties in detecting pedestrians and cyclists are:

[0004] 1) Difficulty in estimating 3D parameters: Vehicle objects have obvious 3D outer edges, are relatively larger in size, and are numerous in the dataset, so it is relatively easy to infer their 3D parameters (3D position, 3D size, and orientation). The 3D points of pedestrians and cyclists are concentrated in a pattern with high density in the center and low density on both sides, so it is impossible to find their top-view bounding boxes by directly inferring corner points and deleting them in order, as in Frustum-Aware Geometric Reasoning (FGR).

[0005] 2) Since their number is smaller than that of vehicles, they are prone to overfitting under a single mode, resulting in insufficient generalization ability.

[0006] 3) Use multimodal features to locate three-dimensional objects. Simply splicing color information or two-dimensional high-dimensional features of three-dimensional projection points cannot fully utilize multimodal features. At present, multimodal fusion methods often use cascade or point feature fusion methods; among them, the cascade method will have the problem of error accumulation. Objects that cannot be detected in the previous modality cannot be detected in the next modality. Although the point feature fusion method fuses features at the network level and avoids the error accumulation problem of the cascade method, the resolution of two-dimensional and three-dimensional imaging is not the same, and multimodal information cannot be fully utilized. Summary of the invention

[0007] The purpose of the present invention is to provide a multimodal weakly supervised three-dimensional target detection method, system and device to solve the problem that the information density of two-dimensional and three-dimensional modalities cannot be aligned, and the multimodal information cannot be fully utilized to identify and locate three-dimensional objects.

[0008] To achieve the above object, the present invention provides the following solutions:

[0009] A multimodal weakly supervised three-dimensional object detection method, comprising:

[0010] Using a camera to capture multiple 2D RGB images, and generating a three-dimensional view cone based on a two-dimensional box annotation on each of the 2D RGB images;

[0011] In the three-dimensional view cone, ground points in the three-dimensional point cloud collected by the vehicle laser radar sensor are filtered out, and a region with the largest number of three-dimensional points in the filtered three-dimensional view cone is determined by using a seed point region growing method;

[0012] A density-based three-dimensional parameter estimation method uses a main control chip to generate a three-dimensional pseudo-annotation frame of an object according to the area with the largest number of three-dimensional points in each filtered three-dimensional cone; the object is a pedestrian or a bicycle;

[0013] Utilizing the main control chip and taking the three-dimensional pseudo-annotation box as the target, a multimodal superpixel two-branch network is trained according to the 2D RGB image and the three-dimensional point cloud to generate a trained multimodal superpixel two-branch network; the network finally inputs the 2D RGB image and the three-dimensional point cloud, and outputs a three-dimensional prediction box of the object, which is used to locate objects around the vehicle and estimate the size and orientation of the object.

[0014] The 2D RGB image of the current frame taken by the camera and the three-dimensional point cloud of the current scene collected by the vehicle lidar sensor are obtained, and the 2D RGB image of the current frame and the three-dimensional point cloud of the current scene are input into the trained multimodal superpixel dual-branch network to generate a three-dimensional object prediction frame of the current scene; the three-dimensional object prediction frame is used to locate objects around the vehicle and predict the size and orientation of objects around the current vehicle.

[0015] Optionally, the density-based 3D parameter estimation method uses a main control chip to generate a 3D pseudo-annotation frame of the object according to the area with the largest number of 3D points in each filtered 3D cone, specifically including:

[0016] Filtering out the adhesion noise point cloud in the area with the largest number of three-dimensional points to generate a filtered three-dimensional point cloud;

[0017] Iteratively search the top view rotation box of the filtered three-dimensional point cloud, gradually delete the lowest point cloud density in the filtered three-dimensional point cloud, until the key point position remains unchanged, exit the iteration, and generate the top view rotation box of the filtered three-dimensional point cloud; the key point is the corner point of the four corner points of the rotation box that contains the most three-dimensional points;

[0018] The key points are expanded to both sides of the filtered three-dimensional view cone, and the statistical information of the object is used to constrain the expansion length, and the vector direction of the long side is used as the actual orientation of the object to generate a three-dimensional pseudo-annotation box.

[0019] Optionally, the using of the main control chip, taking the three-dimensional pseudo-annotated frame as a target, training a multimodal superpixel dual-branch network according to the 2D RGB image and the three-dimensional point cloud, and generating a trained multimodal superpixel dual-branch network specifically includes:

[0020] Using the local consistency prior, a local pixel block in two dimensions is spatially corresponded to a point in three dimensions to generate a projection relationship;

[0021] ResNet50 is used to extract an image feature map of the 2D RGB image, and the image feature map is upsampled to a size consistent with an input image; the input image is any of the 2D RGB images;

[0022] Performing superpixel segmentation on the input image, and obtaining each superpixel feature on the image feature map according to the superpixel segmentation result;

[0023] Assigning the superpixel feature to each three-dimensional point according to the projection relationship;

[0024] Using three-dimensional convolution to extract initial features of the point cloud and point cloud features of three-dimensional points with superpixel features, and projecting the point cloud features into a top view to generate a top view feature map;

[0025] Fusion of the point cloud features in the top-view feature map and the initial point cloud features to generate fused features;

[0026] The fusion features are used as features for target classification, a multimodal superpixel two-branch network is trained, and a trained multimodal superpixel two-branch network is generated.

[0027] Optionally, also include:

[0028] During the training process of the multimodal superpixel dual-branch network, a training method of randomly discarding superpixel features is adopted to train the multimodal superpixel dual-branch network, and the discarded superpixel features are filled with 0.

[0029] A multimodal weakly supervised three-dimensional object detection system, comprising:

[0030] A three-dimensional view cone generation module, used to capture multiple 2D RGB images using a camera, and generate a three-dimensional view cone based on a two-dimensional box annotation on each of the 2D RGB images;

[0031] A filtering module is used to filter out ground points in the three-dimensional point cloud collected by the vehicle laser radar sensor in the three-dimensional view cone, and determine the area with the largest number of three-dimensional points in the filtered three-dimensional view cone by relying on a seed point region growing method;

[0032] A 3D pseudo-annotation frame generation module is used for a density-based 3D parameter estimation method, using a main control chip to generate a 3D pseudo-annotation frame of an object according to the area with the largest number of 3D points in each filtered 3D cone; the object is a pedestrian or a bicycle;

[0033] A training module, used to use the main control chip, take the three-dimensional pseudo-annotated frame as a target, train a multimodal superpixel two-branch network according to the 2D RGB image and the three-dimensional point cloud, and generate a trained multimodal superpixel two-branch network;

[0034] A detection module is used to obtain the 2D RGB image of the current frame taken by the camera and the three-dimensional point cloud of the current scene collected by the vehicle lidar sensor, and input the 2D RGB image of the current frame and the three-dimensional point cloud of the current scene into the trained multimodal superpixel dual-branch network to generate a three-dimensional object prediction frame of the current scene; the three-dimensional object prediction frame is used to locate objects around the vehicle and predict the size and orientation of objects around the current vehicle.

[0035] Optionally, the three-dimensional pseudo-annotation box generation module specifically includes:

[0036] A filtering unit, used to filter out the adhesion noise point cloud in the area with the largest number of three-dimensional points, and generate a filtered three-dimensional point cloud;

[0037] A top view rotation frame generating unit is used to iteratively search the top view rotation frame of the filtered three-dimensional point cloud, gradually delete the lowest point cloud density in the filtered three-dimensional point cloud, until the key point position remains unchanged, exit iteration, and generate a top view rotation frame of the filtered three-dimensional point cloud; the key point is the corner point of the four corner points of the rotation frame that contains the most three-dimensional points;

[0038] The three-dimensional pseudo-annotation frame generating unit is used to expand the key points to both sides of the filtered three-dimensional visual cone, and at the same time use the statistical information of the object to constrain the expansion length, and use the vector direction of the long side as the actual orientation of the object to generate a three-dimensional pseudo-annotation frame.

[0039] Optionally, the training module specifically includes:

[0040] A projection relationship generating unit, used to generate a projection relationship by spatially corresponding a local pixel block in two dimensions to a point in three dimensions using a local consistency prior;

[0041] An upsampling unit, configured to extract an image feature map of the 2D RGB image by using ResNet50, and upsample the image feature map to a size consistent with that of an input image; the input image is any of the 2D RGB images;

[0042] A superpixel feature obtaining unit, configured to perform superpixel segmentation on the input image, and obtain each superpixel feature on the image feature map according to the superpixel segmentation result;

[0043] An assignment unit, configured to assign the superpixel feature to each three-dimensional point according to the projection relationship;

[0044] A projection unit, configured to extract initial features of the point cloud and point cloud features of three-dimensional points with superpixel features by using three-dimensional convolution, and project the point cloud features into a top view to generate a top view feature map;

[0045] A fusion feature generating unit, used for fusing the point cloud features in the top view feature map with the initial point cloud features to generate fusion features;

[0046] The trained multimodal superpixel two-branch network generation unit is used to use the fusion feature as the target classification feature, train the multimodal superpixel two-branch network, and generate the trained multimodal superpixel two-branch network.

[0047] Optionally, also include:

[0048] The superpixel feature random discarding module is used to train the multimodal superpixel two-branch network by randomly discarding superpixel features during the training process of the multimodal superpixel two-branch network, and fill the discarded superpixel features with 0.

[0049] An electronic device comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program so that the electronic device executes the multimodal weakly supervised three-dimensional target detection method as described above.

[0050] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the multimodal weakly supervised three-dimensional target detection method as described above.

[0051] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects: the present invention provides a multimodal weakly supervised three-dimensional target detection method, system and device, because the three-dimensional points of pedestrians and cyclists are concentrated and present a form with high central density and low density on both sides. The density-based three-dimensional parameter estimation method adopted by the present invention effectively filters out the noise points of the point cloud, and accurately predicts the three-dimensional parameters of the object to form a three-dimensional pseudo-annotation; the pseudo-annotation generated by the present invention does not depend on the network structure, and since the accuracy is improved from the pseudo-annotation itself, it can be connected to any network for training; in addition, since the number of training samples of pedestrians and cyclists is small, it is easy to overfit in a single mode. In order to avoid the network overfitting to a single mode, resulting in insufficient generalization performance, the present invention adopts a superpixel dual-branch network with underlying color and texture similarity, and under the premise of local consistency assumption, each three-dimensional point is given a 2D superpixel feature with similar semantics, so that the information density of the two-dimensional and three-dimensional modes can be aligned, and at the same time, multimodal information is fully utilized to identify and locate three-dimensional objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 A flow chart of the multimodal weakly supervised three-dimensional target detection method provided by the present invention;

[0054] Figure 2 This is a structural diagram of the multimodal weakly supervised three-dimensional target detection system provided by the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] The purpose of the present invention is to provide a multimodal weakly supervised three-dimensional target detection method, system and device, which can alleviate the resolution mismatch problem between two-dimensional and three-dimensional modalities, and make full use of multimodal information to identify and locate three-dimensional objects.

[0057] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0058] There are still two problems with pedestrians and cyclists. First, due to the small size of pedestrians and cyclists, background points tend to stick together after rough segmentation. Second, pedestrians and cyclists do not have obvious outer bounding shapes. Their point clouds are mainly distributed in the center of the object, with a small number of points distributed at the edge of the top view rotation box. Therefore, the assumption that the point cloud density of corner points is high is no longer valid, and the accuracy of directly obtaining the minimum outer bounding box is poor.

[0059] Embodiment 1

[0060] The present invention provides a multimodal weakly supervised three-dimensional target detection method. Figure 1 The flowchart of the multimodal weakly supervised three-dimensional target detection method provided by the present invention is as follows: Figure 1 As shown, a multimodal weakly supervised three-dimensional object detection method includes:

[0061] Step 101: Use a camera to capture multiple 2D RGB images, and generate a three-dimensional viewing cone based on a two-dimensional box annotation on each of the 2D RGB images.

[0062] Step 102: In the three-dimensional view cone, ground points in the three-dimensional point cloud collected by the vehicle laser radar sensor are filtered out, and a region with the largest number of three-dimensional points in the filtered three-dimensional view cone is determined by using a seed point region growing method.

[0063] Step 103: Based on the density-based 3D parameter estimation method, using the main control chip, a 3D pseudo-annotation frame of the object is generated according to the area with the largest number of 3D points in each filtered 3D visual cone; the object is a pedestrian or a bicycle.

[0064] In practical applications, the step 103 specifically includes: filtering out the adhesion noise point cloud in the area with the largest number of three-dimensional points to generate a filtered three-dimensional point cloud; using an iterative search for the top view rotation box of the filtered three-dimensional point cloud, gradually deleting the lowest point cloud density in the filtered three-dimensional point cloud until the key point position remains unchanged, exiting the iteration, and generating a top view rotation box of the filtered three-dimensional point cloud; the key point is the corner point among the four corner points of the rotation box that contains the most three-dimensional points; expanding the key point to both sides of the filtered three-dimensional view cone, and at the same time using the statistical information of the object to constrain the expansion length, using the vector direction of the long side as the actual orientation of the object, to generate a three-dimensional pseudo-annotation box.

[0065] In practical applications, the generation process of the 3D pseudo-annotation box is as follows:

[0066] Filter out the sticky noise point cloud in the generated seed area. The point cloud density is defined as:

[0067]

[0068] In formula (1), d p is the density of point p, p i is a point in the K nearest neighbors of point p, K is the number of neighbors in the K nearest neighbor algorithm, i is the index number, and N is the total number of points in the region. p The distance between points is used to represent the point density. The higher the density, the smaller the distance between points and the larger its reciprocal.

[0069] The center of density is defined as:

[0070]

[0071] Since step 101 provides an accurate two-dimensional frame annotation of the object, the points of the main part of the object are denser in the view cone generated by the two-dimensional frame, while the background points are relatively low-density. The background points here have been ground estimated and ground point filtered out.

[0072] In formula (2), p c Based on the point cloud density center, d i For point p i Point cloud density calculated according to formula (1).

[0073] The filtering formula is defined as:

[0074]

[0075] In formula (3), M is the mask matrix of the point set P, where M i Indicates whether to keep point p i , 0 represents removal, 1 represents retention. c To calculate the object center based on the point cloud density by formula (2), T c is the object center distance threshold, d i Point p i The density of Point density average, σ d is the point density standard deviation. In point cloud imaging, there are some points that belong to objects, but they are far away from the remaining points. Simply using density-based filtering methods may delete them incorrectly, so formula (3) adds a distance limit for the object density center.

[0076] According to the filtering formula (3), the noise part in the point cloud is filtered out. When the value is 1, the point is retained, and when the value is 0, the point is filtered out.

[0077] On the basis of the first filtering mentioned above, the point cloud top view rotation box is searched. For the 3D parameter estimation of pedestrians and bicycles, a new parameter estimation process is designed based on the morphological characteristics of pedestrians and cyclists. The morphology of pedestrians and cyclists is distributed in an elliptical shape in the top view. Therefore, step 102 performs ellipse parameter fitting on the filtered point cloud and calculates its ellipse eccentricity θ, that is, the intersection angle between the major axis of the ellipse and the x-axis in the coordinate system xOz. In the interval [θ+T θ ,θ-T θ ] The objective function of formula (4) is used for angle search. The initial angle obtained by the ellipse eccentricity limits the search interval and accelerates the search to a certain extent. On the other hand, the restriction of the search interval prevents the search from falling into the local optimal solution to a certain extent.

[0078]

[0079] Q0={p|p∈Q,||p,l1||>T rect ,||p,l2||>T rect} (4)

[0080] In formula (4), Q0 is the subset of Q that satisfies the distance constraint, l is the edge, and T rect is the distance threshold, area is the area of ​​the rotation box to be searched, and α is the balance factor. The item is used to limit the target to approach the corners of the box, and the area item is used to keep the search rotation box close to the target.

[0081] In the process of searching the rotating box, an iterative search is used to gradually remove noise points in the point set. For pedestrians and bicycles, the parts with low point cloud density are considered as noise points and deleted. The iteration is exited when the position of the key point remains basically unchanged. The key point is the corner point with the most points among the four corner points of the rotating box.

[0082] Find the part with the highest density of corner points as the key point, and expand it to both sides of the cone along this key point, and use the statistical information of the object to constrain the expansion length. Finally, the vector direction of the long side is regarded as the actual orientation of the object to generate a 3D pseudo-annotation box.

[0083] The main reason for using point cloud density as the basis for noise point removal is that the rotating box can be close to the center of the object. Since the actual distribution of the corner points of the object is difficult to estimate, it is weakened to expand from the central rotating box to the viewing cone.

[0084] The three-dimensional points of pedestrians and cyclists are concentrated in a pattern with high density in the center and low density on both sides.

[0085] The density-based three-dimensional parameter estimation method adopted by the present invention can effectively filter out noise points in the point cloud and accurately predict the three-dimensional parameters of the object to form a three-dimensional pseudo-annotation frame; the three-dimensional pseudo-annotation frame generated by the present invention does not depend on the network structure. Since the accuracy is improved from the pseudo-annotation itself, it can be connected to any network for training.

[0086] In the following steps, the multimodal superpixel dual-branch network is trained based on the three-dimensional pseudo-annotated frame generated in step 103. The three-dimensional pseudo-annotated frame is the training target of step 104, and the required input of step 104 is the 2D RGB image of the current frame and the overall three-dimensional point cloud acquired by LiDAR.

[0087] Step 104: using the main control chip, taking the three-dimensional pseudo-annotation box as a target, training a multimodal superpixel dual-branch network according to the 2D RGB image and the three-dimensional point cloud, and generating a trained multimodal superpixel dual-branch network.

[0088] The step 104 specifically includes: using local consistency prior to spatially correspond a local pixel block in two dimensions to a point in three dimensions to generate a projection relationship; using ResNet50 to extract an image feature map of the 2D RGB image, and upsampling the image feature map to a size consistent with that of the input image; the input image is any of the 2D RGB images; performing superpixel segmentation on the input image, and obtaining each superpixel feature on the image feature map according to the superpixel segmentation result; assigning the superpixel feature to each three-dimensional point according to the projection relationship; using three-dimensional convolution to extract initial point cloud features and point cloud features of three-dimensional points with superpixel features, and projecting the point cloud features to a top view to generate a top view feature map; fusing the point cloud features in the top view feature map with the initial point cloud features to generate a fused feature; using the fused feature as a feature for target classification, training a multimodal superpixel dual-branch network, and generating a trained multimodal superpixel dual-branch network.

[0089] In practical applications, there are two aspects that need to be considered in the way of modal fusion.

[0090] First, the number of points in the same scene is not consistent between 2D and 3D data, so how to align 2D and 3D data needs to be considered.

[0091] Second, the choice of network structure for modal fusion is also a part that needs to be carefully designed. For the two-dimensional and three-dimensional data alignment methods, methods represented by PointPainting choose to directly splice 2D image attributes on 3D projection points, but for accuracy considerations, this method needs to introduce time-consuming segmentation and annotation to improve accuracy. Methods represented by MVX-net (MultimodalVoxelNet) choose to use point alignment to extract 3D projection point features on the feature map, but this method does not align the amount of 2D and 3D information.

[0092] The present invention proposes semantic alignment of two-dimensional and three-dimensional images, and uses local consistency priors to spatially correspond a local pixel block in two dimensions to a point in three dimensions. Superpixels can use the underlying color and texture information to achieve the effect of aggregating 2D points. In this way, a three-dimensional point and a set of points composed of superpixels in two dimensions can be matched based on the projection relationship. In addition, this method avoids the introduction of new annotations.

[0093] Specifically, we first use ResNet50 to extract image features and upsample the image feature map to the same size as the input image. Then we perform superpixel segmentation on the input image, and use the superpixel segmentation results to obtain the features of each superpixel on the image feature map. Here, we select average pooling as the superpixel feature extraction method. After obtaining the superpixel features, we assign them to each 3D point based on the projection relationship (the 3D points here are actually the point clouds of the entire scene scanned by the vehicle LiDAR sensor). If a 3D point falls within a certain superpixel range, its superpixel features are assigned to this 3D point.

[0094] Since separating target category features and size features can effectively improve network accuracy, the present invention adopts a dual-branch structure to separate features for classification and regression.

[0095] The initial features of the point cloud are extracted by 3D convolution. For image features, after assigning superpixel features to each 3D point cloud, 3D convolution is used to extract the point cloud features of the 3D points with superpixel features. Then, the point cloud features of the 3D points with superpixel features are projected to the top view and fused with the point cloud features (the point cloud features are high-dimensional vectors extracted by 3D convolution after voxelization) to generate fused features. Since the classification of the target depends not only on its appearance in the 2D image, but also on its shape in 3D, the fused features are used as the features for target classification.

[0096] First, the point cloud has clearer positioning information; second, three-dimensional data enhancement will be performed during the network training process. In order for the network to simultaneously learn two-dimensional image features for size regression, it is necessary to maintain two-dimensional and three-dimensional consistency, but this is very difficult, so only point cloud features are used for target three-dimensional parameter regression.

[0097] The multimodal superpixel multi-branch network is used to locate pedestrians or bicycles. It can effectively improve the current problem of poor performance of a single modality and can be used in the field of unmanned driving to form a perception of the scene.

[0098] When adopting a multimodal network structure, its training method still needs to be considered. In order to enhance the generalization performance of the network, the 3D object detector needs to add data enhancement methods such as random rotation, random translation, random scaling, and random replication during the training process. However, these methods will destroy the consistency between the 2D image and the 3D point cloud, and may cause inconsistency in the semantics corresponding to the 3D point cloud and the 2D image. In order to resist the problem of multimodal data consistency, the network randomly discards superpixel features with a certain probability during the training process and fills them with 0. During the target replication process, the superpixel features of the newly replicated object are filled with 0.

[0099] Step 105: Obtain the 2D RGB image of the current frame taken by the camera and the three-dimensional point cloud of the current scene collected by the vehicle lidar sensor, and input the 2D RGB image of the current frame and the three-dimensional point cloud of the current scene into the trained multimodal superpixel dual-branch network to generate a three-dimensional object prediction frame of the current scene; the three-dimensional object prediction frame is used to locate objects around the vehicle and predict the size and orientation of objects around the current vehicle.

[0100] Since there are fewer training samples of pedestrians and cyclists, they are prone to overfitting under a single mode. In order to avoid overfitting of the network to a single mode, resulting in insufficient generalization performance, the present invention uses multimodal fusion to train a weakly supervised three-dimensional target detector. However, the method of simply splicing color information or two-dimensional high-dimensional features of three-dimensional projection points cannot fully utilize multimodal features. The superpixel dual-branch network adopted by the present invention, which relies on the similarity of underlying color and texture, gives each 3D point a 2D superpixel feature with similar semantics under the premise of local consistency assumption, so that the information density of two-dimensional and three-dimensional modes can be aligned, while making full use of multimodal information to identify and locate three-dimensional objects.

[0101] The superpixel dual-branch network structure proposed in the present invention does not rely on a specific three-dimensional network structure, and can be combined with most three-dimensional network structures to form a new multimodal fusion network. The random discard superpixel feature training method proposed in the present invention is simpler and easier to implement than the method that needs to maintain consistency between two-dimensional and three-dimensional images while ensuring a certain accuracy.

[0102] Embodiment 2

[0103] In order to execute the method corresponding to the above-mentioned embodiment 1 to achieve the corresponding functions and technical effects, a multimodal weakly supervised three-dimensional object detection system is provided below.

[0104] Figure 2 The structure diagram of the multimodal weakly supervised three-dimensional target detection system provided by the present invention is as follows: Figure 2 As shown, a multimodal weakly supervised three-dimensional object detection system includes:

[0105] The three-dimensional view cone generation module 201 is used to capture multiple 2D RGB images using a camera, and generate a three-dimensional view cone based on a two-dimensional box annotation on each of the 2D RGB images.

[0106] The filtering module 202 is used to filter out ground points in the three-dimensional point cloud collected by the vehicle laser radar sensor in the three-dimensional view cone, and determine the area with the largest number of three-dimensional points in the filtered three-dimensional view cone by relying on the seed point region growing method.

[0107] The 3D pseudo-annotation frame generation module 203 is used for the density-based 3D parameter estimation method, using the main control chip to generate a 3D pseudo-annotation frame of an object according to the area with the largest number of 3D points in each filtered 3D visual cone; the object is a pedestrian or a bicycle.

[0108] The three-dimensional pseudo-annotation frame generation module 203 specifically includes: a filtering unit, which is used to filter out the adhesion noise point cloud in the area with the largest number of three-dimensional points, and generate a filtered three-dimensional point cloud; a top-view rotation frame generation unit, which is used to use an iterative search for the top-view rotation frame of the filtered three-dimensional point cloud, and gradually delete the lowest point cloud density in the filtered three-dimensional point cloud until the key point position remains unchanged, exit the iteration, and generate a top-view rotation frame of the filtered three-dimensional point cloud; the key point is the corner point with the most three-dimensional points among the four corner points of the rotation frame; the three-dimensional pseudo-annotation frame generation unit is used to expand the key point to both sides of the filtered three-dimensional visual cone, and at the same time use the statistical information of the object to constrain the expansion length, and use the vector direction of the long side as the actual orientation of the object to generate a three-dimensional pseudo-annotation frame.

[0109] The training module 204 is used to use the main control chip, take the three-dimensional pseudo-annotation box as the target, train a multimodal superpixel two-branch network according to the 2DRGB image and the three-dimensional point cloud, and generate a trained multimodal superpixel two-branch network.

[0110] The training module 204 specifically includes: a projection relationship generation unit, which is used to use the local consistency prior to spatially correspond a local pixel block in two dimensions to a point in three dimensions to generate a projection relationship; an upsampling unit, which is used to extract the image feature map of the 2D RGB image using ResNet50, and upsample the image feature map to the same size as the input image; the input image is any of the 2D RGB image; a superpixel feature obtaining unit, used to perform superpixel segmentation on the input image, and obtain each superpixel feature on the image feature map according to the superpixel segmentation result; an assignment unit, used to assign the superpixel feature to each three-dimensional point according to the projection relationship; a projection unit, used to extract the initial features of the point cloud and the point cloud features of the three-dimensional points with superpixel features by using three-dimensional convolution, and project the point cloud features into a top view to generate a top-view feature map; a fusion feature generation unit, used to fuse the point cloud features in the top-view feature map with the initial features of the point cloud to generate a fusion feature; a trained multimodal superpixel dual-branch network generation unit, used to use the fusion feature as the feature for target classification, train a multimodal superpixel dual-branch network, and generate a trained multimodal superpixel dual-branch network.

[0111] The detection module 205 is used to obtain the 2D RGB image of the current frame taken by the camera and the three-dimensional point cloud of the current scene collected by the vehicle lidar sensor, and input the 2D RGB image of the current frame and the three-dimensional point cloud of the current scene into the trained multimodal superpixel dual-branch network to generate a three-dimensional object prediction frame of the current scene; the three-dimensional object prediction frame is used to locate objects around the vehicle and predict the size and orientation of objects around the current vehicle.

[0112] The present invention also includes: a superpixel feature random discarding module, which is used to train the multimodal superpixel two-branch network by using a training method of randomly discarding superpixel features during the training process of the multimodal superpixel two-branch network, and fill the discarded superpixel features with 0.

[0113] In order to balance the data characteristics of the two-dimensional and three-dimensional modalities, the present invention aligns the two-dimensional and three-dimensional at the semantic and spatial levels, thereby making full use of the rich semantic information of the two-dimensional image and the precise positioning accuracy of the three-dimensional point cloud. The present invention further proposes a superpixel dual-branch network based on the underlying color and texture similarity to align the information density of the two-dimensional and three-dimensional modalities, while making full use of multimodal information to identify and locate three-dimensional objects.

[0114] Embodiment 3

[0115] An embodiment of the present invention provides an electronic device including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the multimodal weakly supervised three-dimensional target detection method of embodiment one.

[0116] In practical applications, the above electronic device may be a server.

[0117] In practical applications, an electronic device includes: at least one processor, a memory, a bus and a communications interface.

[0118] Wherein: the processor, the communication interface, and the memory communicate with each other via a communication bus.

[0119] Communication interface, used to communicate with other devices.

[0120] The processor is used to execute the program, and specifically can execute the method described in the above embodiment.

[0121] Specifically, the program may include program codes including computer operation instructions.

[0122] The processor may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the electronic device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0123] The memory is used to store programs. The memory may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory.

[0124] Based on the description of the above embodiments, the embodiments of the present application provide a storage medium on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the method described in any embodiment.

[0125] The multimodal weakly supervised three-dimensional object detection system provided in the embodiments of the present application exists in various forms, including but not limited to:

[0126] (1) Mobile communication devices: These devices are characterized by their mobile communication functions and their main purpose is to provide voice and data communications. These terminals include: smart phones (such as iPhone), multimedia phones, functional phones, and low-end phones.

[0127] (2) Ultra-mobile personal computer devices: These devices fall into the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access capabilities. These terminals include: PDA, MID and UMPC devices, such as iPad.

[0128] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players (such as iPods), handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0129] (4) Other electronic devices with data interaction functions.

[0130] Thus far, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired results. Additionally, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing may be advantageous.

[0131] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0132] For the convenience of description, the above device is described by being divided into various units according to their functions and described separately. Of course, when implementing the present application, the functions of each unit can be implemented in one or more software and / or hardware. It should be understood by those skilled in the art that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0133] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0134] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0135] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0136] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0137] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0138] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM),

[0139] Digital Versatile Disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device

[0140] Or any other non-transmission medium that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0141] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0142] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0143] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific transactions or implement specific abstract data types. The present application may also be practiced in distributed computing environments where transactions are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0144] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0145] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A multimodal weakly supervised three-dimensional object detection method, characterized in that: include: Using a camera to capture multiple 2D RGB images, and generating a three-dimensional view cone based on a two-dimensional box annotation on each of the 2D RGB images; In the three-dimensional view cone, ground points in the three-dimensional point cloud collected by the vehicle laser radar sensor are filtered out, and a region with the largest number of three-dimensional points in the filtered three-dimensional view cone is determined by using a seed point region growing method; The density-based 3D parameter estimation method uses the main control chip to generate a 3D pseudo-annotation box of the object according to the area with the largest number of 3D points in each filtered 3D cone; The object is a pedestrian or a bicycle; Using the main control chip, taking the three-dimensional pseudo-annotated frame as the target, training a multimodal superpixel dual-branch network according to the 2D RGB image and the three-dimensional point cloud, and generating a trained multimodal superpixel dual-branch network, specifically includes: Using the local consistency prior, a local pixel block in two dimensions is spatially corresponded to a point in three dimensions to generate a projection relationship; ResNet50 is used to extract an image feature map of the 2D RGB image, and the image feature map is upsampled to a size consistent with an input image; the input image is any of the 2D RGB images; Performing superpixel segmentation on the input image, and obtaining each superpixel feature on the image feature map according to the superpixel segmentation result; Assigning the superpixel feature to each three-dimensional point according to the projection relationship; Using three-dimensional convolution to extract initial features of the point cloud and point cloud features of three-dimensional points with superpixel features, and projecting the point cloud features into a top view to generate a top view feature map; Fusion of the point cloud features in the top-view feature map and the initial point cloud features to generate fused features; Using the fusion feature as a feature for target classification, training a multimodal superpixel two-branch network, and generating a trained multimodal superpixel two-branch network; The 2D RGB image of the current frame taken by the camera and the three-dimensional point cloud of the current scene collected by the vehicle lidar sensor are obtained, and the 2D RGB image of the current frame and the three-dimensional point cloud of the current scene are input into the trained multimodal superpixel dual-branch network to generate a three-dimensional object prediction frame of the current scene; the three-dimensional object prediction frame is used to locate objects around the vehicle and predict the size and orientation of objects around the current vehicle.

2. The multimodal weakly supervised three-dimensional object detection method according to claim 1, characterized in that: The density-based 3D parameter estimation method uses a main control chip to generate a 3D pseudo-annotation frame of an object according to the area with the largest number of 3D points in each filtered 3D view cone, specifically including: Filtering out the adhesion noise point cloud in the area with the largest number of three-dimensional points to generate a filtered three-dimensional point cloud; Iteratively search the top view rotation box of the filtered three-dimensional point cloud, gradually delete the lowest point cloud density in the filtered three-dimensional point cloud, until the key point position remains unchanged, exit the iteration, and generate the top view rotation box of the filtered three-dimensional point cloud; the key point is the corner point of the four corner points of the rotation box that contains the most three-dimensional points; The key points are expanded to both sides of the filtered three-dimensional view cone, and the statistical information of the object is used to constrain the expansion length, and the vector direction of the long side is used as the actual orientation of the object to generate a three-dimensional pseudo-annotation box.

3. The multimodal weakly supervised three-dimensional target detection method according to claim 2, characterized in that: Also includes: During the training process of the multimodal superpixel dual-branch network, a training method of randomly discarding superpixel features is adopted to train the multimodal superpixel dual-branch network, and the discarded superpixel features are filled with 0.

4. A multimodal weakly supervised three-dimensional object detection system, characterized in that: include: A three-dimensional view cone generation module, used to capture multiple 2D RGB images using a camera, and generate a three-dimensional view cone based on a two-dimensional box annotation on each of the 2D RGB images; A filtering module is used to filter out ground points in the three-dimensional point cloud collected by the vehicle laser radar sensor in the three-dimensional view cone, and determine the area with the largest number of three-dimensional points in the filtered three-dimensional view cone by relying on a seed point region growing method; A 3D pseudo-annotation box generation module is used for a density-based 3D parameter estimation method, which uses a main control chip to generate a 3D pseudo-annotation box of an object according to the area with the largest number of 3D points in each filtered 3D view cone; The object is a pedestrian or a bicycle; A training module, used to use the main control chip, take the three-dimensional pseudo-annotated frame as a target, train a multimodal superpixel two-branch network according to the 2D RGB image and the three-dimensional point cloud, and generate a trained multimodal superpixel two-branch network; The training module specifically includes: A projection relationship generating unit, used to generate a projection relationship by spatially corresponding a local pixel block in two dimensions to a point in three dimensions using a local consistency prior; An upsampling unit, configured to extract an image feature map of the 2D RGB image by using ResNet50, and upsample the image feature map to a size consistent with that of an input image; the input image is any of the 2D RGB images; A superpixel feature obtaining unit, configured to perform superpixel segmentation on the input image, and obtain each superpixel feature on the image feature map according to the superpixel segmentation result; An assignment unit, configured to assign the superpixel feature to each three-dimensional point according to the projection relationship; A projection unit, configured to extract initial features of the point cloud and point cloud features of three-dimensional points with superpixel features by using three-dimensional convolution, and project the point cloud features into a top view to generate a top view feature map; A fusion feature generating unit, used for fusing the point cloud features in the top view feature map with the initial point cloud features to generate fusion features; A trained multimodal superpixel two-branch network generating unit, used to train a multimodal superpixel two-branch network using the fusion feature as a feature for target classification, and generate a trained multimodal superpixel two-branch network; A detection module is used to obtain the 2D RGB image of the current frame taken by the camera and the three-dimensional point cloud of the current scene collected by the vehicle lidar sensor, and input the 2D RGB image of the current frame and the three-dimensional point cloud of the current scene into the trained multimodal superpixel dual-branch network to generate a three-dimensional object prediction frame of the current scene; the three-dimensional object prediction frame is used to locate objects around the vehicle and predict the size and orientation of objects around the current vehicle.

5. The multimodal weakly supervised three-dimensional object detection system according to claim 4, characterized in that: The three-dimensional pseudo-annotation frame generation module specifically includes: A filtering unit, used to filter out the adhesion noise point cloud in the area with the largest number of three-dimensional points, and generate a filtered three-dimensional point cloud; A top view rotation frame generating unit is used to iteratively search the top view rotation frame of the filtered three-dimensional point cloud, gradually delete the lowest point cloud density in the filtered three-dimensional point cloud, until the key point position remains unchanged, exit iteration, and generate a top view rotation frame of the filtered three-dimensional point cloud; the key point is the corner point of the four corner points of the rotation frame that contains the most three-dimensional points; The three-dimensional pseudo-annotation frame generating unit is used to expand the key points to both sides of the filtered three-dimensional visual cone, and at the same time use the statistical information of the object to constrain the expansion length, and use the vector direction of the long side as the actual orientation of the object to generate a three-dimensional pseudo-annotation frame.

6. The multimodal weakly supervised three-dimensional object detection system according to claim 5, characterized in that: Also includes: The superpixel feature random discarding module is used to train the multimodal superpixel two-branch network by randomly discarding superpixel features during the training process of the multimodal superpixel two-branch network, and fill the discarded superpixel features with 0.

7. An electronic device, characterized in that: It includes a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the multimodal weakly supervised three-dimensional target detection method as described in any one of claims 1 to 3.

8. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the multimodal weakly supervised three-dimensional target detection method as described in any one of claims 1 to 3.

Images