3D reconstruction method based on target detection

Abstract

The invention discloses a 3D reconstruction method based on target detection. A 3D target detection model provides target point cloud information for a 3D target reconstruction model. According to the 3D target reconstruction model, firstly, inter-frame feature matching is carried out on input continuous frames, rapid layered optimization is carried out on the pose of each detection object, then color, depth and weight information of voxel blocks is fused and stored as the 3D target reconstruction model, and high-quality object surface 3D modeling is realized according to newly calculated pose estimation. The inter-frame matching method, the layered optimization method and the surface reconstruction method work cooperatively, so that the 3D target reconstruction method can keep high local precision and global consistency. According to the 3D reconstruction method based on target detection, high-quality surface modeling can be achieved, high local precision can be kept, and a clear topological structure can also be reserved.

Description

Technical Field

[0001] The present invention relates to the technical field of 3D reconstruction, and in particular to a 3D reconstruction method based on target detection. Background Art

[0002] As electronic information technology becomes increasingly mature and demand continues to grow, 3D reconstruction technology can be applied to many industrial scenarios. For example, workers can remotely perform accurate scale measurements of targets without using professional tools such as rangefinders and rulers. This requires 3D reconstruction algorithms to be able to identify and reconstruct individual targets. However, traditional 3D reconstruction technology focuses on positioning, and due to the limitation of computational complexity, it can only achieve dense mapping of small scenes, and the error is large.

[0003] 3D target reconstruction methods include: 3D target detection and 3D target reconstruction. In the 3D target detection method, the dimension reduction principle is followed, and the advantages of mature 2D detection algorithms are used to extract features to reduce the search space, and then the semantic class of each point in the point cloud is predicted by 3D instance segmentation to obtain target information. The focus of existing methods is on how to obtain the depth information of the target scene or object with high accuracy. The distance sensing technology for obtaining depth information can be roughly divided into three types, namely ToF (Time-of-Flight), structured light and stereo vision sensors. ToF-based sensors, such as LIDAR, measure the reflection time of the emitted light between the illuminator and the object to generate a depth map. L i DAR has accurate distance perception and is not easily affected by factors such as light. It is also more suitable for detecting the 3D structural information of the scene, but L i DAR is expensive and greatly affected by weather. Structured light sensors estimate depth information by calculating the parallax of the distorted perspective of infrared light projected onto the object. Stereo vision uses multiple cameras to capture the reflection of natural light, and uses trigonometric functions to calculate the disparity between two viewpoints when perceiving depth information, establish the relative relationship between the object and the camera, and thus obtain 3D information of the object surface. Depth information is usually recorded in the data format of raw points, RGB images, or intermediate derived 3D data representations such as voxels and projections. Therefore, fusing and calibrating camera and LiDAR data is an attractive challenge. Summary of the invention

[0004] The purpose of the present invention is to provide a 3D reconstruction method based on target detection, which can achieve high-quality surface modeling, maintain high local accuracy, and retain a clear topological structure.

[0005] The technical solution adopted by a 3D reconstruction method based on target detection disclosed in the present invention is:

[0006] A 3D reconstruction method based on target detection, comprising:

[0007] 3D object detection: The 3D object detection module uses a 2D convolutional neural network to identify the 2D object area and its category in a frame image, then maps the detected 2D boundary area into a frustum point cloud space in the 3D space, and aligns the point cloud in the input space according to the point cloud centroid correction, and finally estimates the 3D bounding box of the target object;

[0008] 3D object reconstruction, first use feature detection, feature matching and correspondence filtering steps to achieve sparse matching between frames. In order to obtain global alignment, a set of sparse features that provide loop closure detection and re-positioning are used. The 3D object reconstruction module calculates SI FT key points and descriptors at a speed of 4-5 milliseconds per frame and matches a pair of frames within 0.05 milliseconds. Therefore, 20K frames can be found for matching in real time. After completing the basic matching, the calibration is further refined by photometric and geometric consistency, potential matches between each pair of frames are filtered to remove false positives, and valid paired frames are generated as input to the hierarchical optimization module. In order to efficiently process up to tens of thousands of RGB input frames, the algorithm uses the matched frame correspondence to perform a hierarchical optimization strategy - local to global posture optimization;

[0009] Size measurement, first build a 3D stereo box to completely surround the object to be reconstructed, then divide the 3D stereo box into n voxels, and calculate the TSDF value and weight of each voxel on the current depth map, and then use the new voxel value and the previously stored voxel value to weighted average cumulative update TSDF value. TSDF is the weighted signed distance from the voxel point to the entire object surface, SDF is the signed distance from the voxel point to the tangent plane of the nearest sample point, trunc represents the truncation distance,

[0010]

[0011] As a preferred solution, the DSSD branch algorithm improved based on the SSD algorithm realizes 2D target detection. Fast R-CNN takes the detection area extracted by the DSSD branch algorithm as input, and first processes the entire image with several convolution and maximum pooling layers to produce a convolution feature map. For each region of interest Ro I, the Ro I pooling layer extracts a fixed-length feature vector from the feature map. Each feature vector is fed into a series of fully connected layers, which eventually branch to two output layers of the same level: one layer produces a softmax probability estimate for the object class plus a background class, and the other layer outputs four values ​​for each detected object. These four values ​​further encode the position of the refined bounding box;

[0012] Using the known camera intrinsic parameters, a frustum sequence is obtained by sliding a pair of parallel planes along the frustum axis with the same step, where the plane pair is also perpendicular to the frustum axis. The region proposal module generates such a frustum sequence for each 2D region proposal. The point cloud in the obtained frustum constitutes the view cone point cloud for the detected object. For different target detection areas, the frustum may face many different directions. The camera coordinate system is initially adjusted based on the T-net network, and the frustum is normalized through a series of geometric transformations - by translating and rotating the points in the frustum, the central axis of the frustum is made orthogonal to the image plane, and the frustum is rotated to a direction orthogonal to the camera, so that the camera optical axis is perpendicular to this 2D area. This normalization helps to improve the rotational invariance of the algorithm and stabilizes the distribution of the point cloud inside the frustum.

[0013] As a preferred solution, a set of sparse features that provide loop closure detection and relocalization are used for feature matching in inter-frame matching. SI FT features are used to address the main changes encountered during RGB-D camera scanning. The detected SI FT keypoints are matched with all previous frames and carefully filtered to eliminate mismatches and avoid false loop closures.

[0014] In local pose optimization, the input continuous frames are divided into a short block, and the adjacent short blocks overlap by one frame, and the pose optimization of local alignment is performed on the frames in the short block. In order to ensure that the local pose optimization results are accurate enough, dense verification test is also applied here. According to the pose after local optimization, a representative block keyframe feature set is calculated, that is, all feature points in the short block are projected into the same coordinate system to generate a set of consistent feature positions and depths, and the least squares method is used to fit an optimal 3D position. Finally, the data within the block, including the relationship within the block, the features within the block, etc., are discarded, the sparse feature relationship within the fixed block, and no local pose optimization is performed later;

[0015] The global pose optimization uses sparse and dense constraints to calculate the best global alignment for a set of global keyframes. First, the pose of the global keyframe is initialized using the globally optimized pose, then the globally consistent transformation of all input frames is obtained, and finally the transformation of the corresponding short block from the local optimization is applied to each frame image in the short block to obtain the camera pose of each frame image in the short block.

[0016] As a preferred solution, the specific calculation method of size measurement is as follows: first, a 3D stereo frame is established to completely surround the target to be reconstructed, and then the 3D stereo frame is divided into n voxels, and the TSDF value and weight of each voxel on the current depth map are calculated, and then the TSDF value is updated by weighted average of the new voxel value and the previously stored voxel value. The formula is as follows:

[0017] TSDF( p ) =[W ( p ) TSDF ( p ) +w ( p ) tsdf ( p ) ] / [W ( p ) +w ( p ) ], (2)

[0018] W(p) is the updated weight value, TSDF(p) is the TSDF value of the updated frame of voxel p, tsdf(p) is the TSDF value of the current frame of voxel p, w(p) is the weight value of the current frame, and the weight calculation formula for each voxel is as follows:

[0019] W ( p ) =W ( p ) +w(p), (3)

[0020] Segmentation is performed by building a triangle mesh along the target sampling surface, discarding triangles with edge lengths exceeding a threshold to avoid subdivision step discontinuities, and finally calculating a weight for each vertex. After the range image is converted into a triangle mesh with a weight for each vertex, the voxel grid can be updated.

[0021] The beneficial effects of a 3D reconstruction method based on target detection disclosed in the present invention are: determining the overall technical route of 3D reconstruction based on target detection, providing knowledge reserves for subsequent further research and development, adding visualization algorithm design, firstly obtaining the 2D coordinate projection of the vertices of the prediction box of the target detection object in the camera coordinate system, and then connecting the point projections according to the connection order of the cube, so as to output a very accurate 3D target detection visualization result, and also performing ablation experiments and comparing with the current mainstream algorithms, proving that the 3D target detection model can achieve good detection effects in various detection tasks, and the 3D reconstruction model can reconstruct a high-quality 3D model according to the input RGB-D data stream, thereby realizing high-precision ranging of the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a structural schematic diagram of a 3D reconstruction method based on target detection of the present invention.

[0023] Figure 2 It is a schematic diagram of a segmentation method in an example of a 3D reconstruction method based on target detection in the present invention.

[0024] Figure 3It is a flow chart of a 3D target reconstruction module in a 3D reconstruction method based on target detection of the present invention. DETAILED DESCRIPTION

[0025] The present invention will be further described and illustrated below in conjunction with specific embodiments and accompanying drawings:

[0026] Please refer to Figure 1 , a 3D reconstruction method based on target detection, comprising:

[0027] 3D object detection,The 3D object detection module uses a 2D convolutional neural network to identify the 2D object area and its category in a frame of image, and then maps the detected 2D boundary area into a frustum point cloud space in 3D space, and aligns the point cloud in the input space according to the point cloud centroid correction, and finally estimates the 3D bounding box of the target object.

[0028] The feature extraction module chooses to use an encoder-decoder structure - the DSSD branch algorithm improved based on the SSD algorithm to achieve 2D target detection. Fast R-CNN takes the detection area extracted by the DSSD branch algorithm as input, and first processes the entire image with several convolution and maximum pooling layers to produce a convolution feature map. For each region of interest RoI, the RoI pooling layer extracts a fixed-length feature vector from the feature map, and each feature vector is fed into a series of fully connected layers, which eventually branch to two output layers of the same level: one layer produces a softmax probability estimate for the object class plus a background class, and the other layer outputs four values ​​for each detected object, which further encodes the location of the refined bounding box.

[0029] Using known camera intrinsic parameters, a (possibly overlapping) frustum sequence is obtained by sliding a pair of parallel planes along the frustum axis with the same step, where the plane pair is also perpendicular to the frustum axis. The region proposal module generates such a frustum sequence for each 2D region proposal, and the point cloud in the obtained frustum constitutes the view cone point cloud for the detected object. For different target detection areas, the frustum may be oriented in many different directions. In order to solve the problem of unstable point cloud distribution, the algorithm also makes an initial adjustment to the camera coordinate system based on the T-net network, and normalizes the frustum through a series of geometric transformations - by translating and rotating the points in the frustum, the central axis of the frustum is orthogonal to the image plane, and the frustum is rotated to a direction orthogonal to the camera, so that the camera optical axis is perpendicular to this 2D area. This normalization helps to improve the rotational invariance of the algorithm and stabilize the point cloud distribution inside the frustum.

[0030] The instance segmentation module needs to further process the point cloud information saved by the region proposal module to obtain the 3D coordinates of the detected object. Although the pixels of occluders, objects, and backgrounds are very close and difficult to distinguish in 2D images, they are often naturally separated in 3D physical space. Therefore, instance segmentation in 3D point clouds is more natural and accurate than segmentation in 2D images. The instance segmentation module changes the strategy of traditional algorithms for instance segmentation in 2D or depth images, and uses a PointNet-based network to group points on the point cloud of a frustum, that is, points falling in the same cone plane are grouped together, thereby achieving 3D instance segmentation.

[0031] The principle flow of the instance segmentation module method is as follows Figure 2 As shown in Figure 1, the network first groups the point clouds in the frustum, and extracts features from the frustum sequence through PointNet, and the features are reassembled to form a 2D feature map. Figure 2 The cube in represents the 2D feature map of the arranged frustum feature vector. The 2D feature map is input to two parallel fully convolutional networks (FCN) for processing, one of which is a rectangular box classification layer (CLS) and the other is a rectangular box regression layer (REG). After each convolution layer, batch normalization and ReLU16 nonlinearity are performed.

[0032] 3D object reconstruction first uses feature detection, feature matching and correspondence filtering steps to achieve sparse matching between frames. In order to obtain global alignment, a set of sparse features that provide loop closure detection and repositioning are used. The 3D object reconstruction module calculates SI FT key points and descriptors at a speed of 4-5 milliseconds per frame and matches a pair of frames within 0.05 milliseconds. Therefore, 20K frames can be found for matching in real time. After completing the basic matching, the calibration is further refined through photometric and geometric consistency, potential matches between each pair of frames are filtered to remove false positives, and valid paired frames are generated as input to the hierarchical optimization module. In order to efficiently process up to tens of thousands of RGB input frames, the algorithm uses the matched frame correspondences to perform a hierarchical optimization strategy - local to global posture optimization.

[0033] The flow chart of 3D reconstruction model module is as follows Figure 3 As shown. In inter-frame matching, feature matching is performed using a set of sparse features that provide loop closure detection and relocalization. The present invention uses SI FT features because it can account for the main changes encountered during RGB-D camera scanning, namely: image translation, scaling and rotation. The detected SI FT keypoints are matched to all previous frames and carefully filtered to eliminate mismatches, thereby avoiding false loop closures.

[0034] In local pose optimization, the 11 consecutive input frames are divided into a short block, and the adjacent short blocks overlap by one frame, and the pose optimization of local alignment is performed on the frames in the short block. In order to ensure that the local pose optimization results are accurate enough, dense validation tests are also applied here. According to the pose after local optimization, a representative block keyframe feature set is also calculated, that is, all feature points in the short block are projected into the same coordinate system (the pose of the block keyframe) to generate a consistent set of feature positions and depths, and the least squares method is used to fit an optimal 3D position. Finally, the data within the block is discarded, including the intra-block relationship, intra-block features, etc., and the sparse feature relationship within the fixed block is fixed, and no local pose optimization is performed subsequently.

[0035] Similar to local pose optimization, global pose optimization uses sparse and dense constraints to calculate the best global alignment for a set of global keyframes. First, the pose of the global keyframe is initialized using the globally optimized pose, then the global consistency transformation of all input frames is obtained, and finally the transformation of the corresponding short block from the local optimization is applied to each frame image in the short block to obtain the camera pose of each frame image in the short block.

[0036] Size measurement, first build a 3D stereo box to completely surround the object to be reconstructed, then divide the 3D stereo box into n voxels, and calculate the TSDF value and weight of each voxel on the current depth map, and then use the new voxel value and the previously stored voxel value to weighted average cumulative update TSDF value. TSDF is the weighted signed distance from the voxel point to the entire object surface, SDF is the signed distance from the voxel point to the tangent plane of the nearest sample point, trunc represents the truncation distance,

[0037]

[0038] The specific calculation method for size measurement is as follows: first, a 3D stereo frame is established to completely surround the target to be reconstructed, and then the 3D stereo frame is divided into n voxels, and the TSDF value and weight of each voxel on the current depth map are calculated, and then the TSDF value is updated by weighted average of the new voxel value and the previously stored voxel value. The formula is shown in (2).

[0039] TSDF ( p ) =[W ( p ) TSDF ( p ) +w ( p ) tsdf ( p ) ] / [W ( p ) +w ( p ) ], (2)

[0040] W(p) is the updated weight value, TSDF(p) is the TSDF value of voxel p in the updated frame, tsdf(p) is the TSDF value of voxel p in the current frame, w(p) is the weight value of the current frame, and the weight calculation formula for each voxel is shown in (3).

[0041] W ( p ) =W ( p ) +w(p), (3)

[0042] Segmentation is performed by building a triangle mesh along the target sampling surface, discarding triangles with edge lengths exceeding a threshold to avoid subdivision step discontinuities, and finally calculating a weight for each vertex. After the range image is converted into a triangle mesh with a weight for each vertex, the voxel grid can be updated.

[0043] In this paper, the 3D target detection model follows the principle of dimensionality reduction, takes advantage of the mature 2D detection algorithm to reduce the search space, and then uses 3D instance segmentation to obtain the point cloud information of the target. A 3D target detection model that integrates 2D target detection and 3D instance segmentation is proposed. The task of this model is to realize 3D target detection by inputting RGB-D image stream.

[0044] The 3D target detection model provides the target point cloud information for the 3D target reconstruction model. The 3D target reconstruction model first performs inter-frame feature matching on the input continuous frames, and performs fast hierarchical optimization on the pose of each detected object. Then, the color, depth and weight information of the voxel block are fused and stored as a 3D target reconstruction model, and high-quality 3D modeling of the object surface is achieved based on the newly calculated pose estimation. Inter-frame matching, hierarchical optimization and surface reconstruction methods work together, and such a 3D target reconstruction method can maintain high local accuracy and global consistency.

[0045] The present invention has found through investigation that there is a lack of research on the reconstruction and size measurement of specific targets in the field of 3D reconstruction. In view of the important demand for accurate reconstruction of targets, the present invention analyzes the key and difficult points of this task, and can determine the overall technical route of 3D reconstruction based on target detection, providing knowledge reserves for subsequent further research and development, and adding visualization algorithm design. First, in the camera coordinate system, the 2D coordinate projection of the vertices of the prediction box of the target detection object is obtained, and then the point projections are connected in the connection order of the cube, so that very accurate 3D target detection visualization results can be output. Ablation experiments and comparisons with current mainstream algorithms are also carried out, which proves that the 3D target detection model can achieve good detection effects in various detection tasks, and the 3D reconstruction model can reconstruct a high-quality 3D model based on the input RGB-D data stream, thereby achieving high-precision ranging of the target object.

[0046] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention, rather than to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the essence and scope of the technical solution of the present invention.

Claims

1. A 3D reconstruction method based on target detection, characterized in that: include: 3D object detection: The 3D object detection module uses a 2D convolutional neural network to identify the 2D object area and its category in a frame image, then maps the detected 2D boundary area into a frustum point cloud space in the 3D space, and aligns the point cloud in the input space according to the point cloud centroid correction, and finally estimates the 3D bounding box of the target object; 3D object reconstruction, first use feature detection, feature matching and correspondence filtering steps to achieve sparse matching between frames. In order to obtain global alignment, a set of sparse features that provide loop closure detection and re-positioning are used. The 3D object reconstruction module calculates SIFT key points and descriptors at a speed of 4-5 milliseconds per frame and matches a pair of frames within 0.05 milliseconds. Therefore, 20K frames can be found for matching in real time. After completing the basic matching, the calibration is further refined by photometric and geometric consistency, potential matches between each pair of frames are filtered to remove false positives, and valid paired frames are generated as input to the hierarchical optimization module. In order to efficiently process up to tens of thousands of RGB input frames, the algorithm uses the matched frame correspondence to perform a hierarchical optimization strategy - local to global posture optimization; Size measurement, first build a 3D stereo box to completely surround the object to be reconstructed, then divide the 3D stereo box into n voxels, and calculate the TSDF value and weight of each voxel on the current depth map, and then use the new voxel value and the previously stored voxel value to weighted average cumulative update TSDF value. TSDF is the weighted signed distance from the voxel point to the entire object surface, SDF is the signed distance from the voxel point to the tangent plane of the nearest sample point, trunc represents the truncation distance, 2. A 3D reconstruction method based on target detection as claimed in claim 1, characterized in that: The DSSD branch algorithm improved based on the SSD algorithm realizes 2D target detection. Fast R-CNN takes the detection area extracted by the DSSD branch algorithm as input. First, it processes the entire image with several convolution and maximum pooling layers to produce a convolution feature map. For each region of interest RoI, the RoI pooling layer extracts a fixed-length feature vector from the feature map. Each feature vector is fed into a series of fully connected layers, which eventually branch to two output layers of the same level: one layer generates a softmax probability estimate for the object class plus a background class, and the other layer outputs four values ​​for each detected object. These four values ​​further encode the location of the refined bounding box; Using the known camera intrinsic parameters, a frustum sequence is obtained by sliding a pair of parallel planes along the frustum axis with the same step, where the plane pair is also perpendicular to the frustum axis. The region proposal module generates such a frustum sequence for each 2D region proposal. The point cloud in the obtained frustum constitutes the view cone point cloud for the detected object. For different target detection areas, the frustum may face many different directions. The camera coordinate system is initially adjusted based on the T-net network, and the frustum is normalized through a series of geometric transformations - by translating and rotating the points in the frustum, the central axis of the frustum is made orthogonal to the image plane, and the frustum is rotated to a direction orthogonal to the camera, so that the camera optical axis is perpendicular to this 2D area. This normalization helps to improve the rotational invariance of the algorithm and stabilizes the distribution of the point cloud inside the frustum.

3. The 3D reconstruction method based on target detection according to claim 1, characterized in that: In inter-frame matching, feature matching is performed using a set of sparse features that provide loop closure detection and relocalization. SIFT features are used to address the main changes encountered during RGB-D camera scanning. The detected SIFT keypoints are matched to all previous frames and carefully filtered to eliminate mismatches and avoid false loop closures. In local pose optimization, the input continuous frames are divided into a short block, and the adjacent short blocks overlap by one frame, and the pose optimization of local alignment is performed on the frames in the short block. In order to ensure that the local pose optimization results are accurate enough, dense verification test is also applied here. According to the pose after local optimization, a representative block keyframe feature set is calculated, that is, all feature points in the short block are projected into the same coordinate system to generate a set of consistent feature positions and depths, and the least squares method is used to fit an optimal 3D position. Finally, the data within the block, including the relationship within the block, the features within the block, etc., are discarded, the sparse feature relationship within the fixed block, and no local pose optimization is performed later; The global pose optimization uses sparse and dense constraints to calculate the best global alignment for a set of global keyframes. First, the pose of the global keyframe is initialized using the globally optimized pose, then the globally consistent transformation of all input frames is obtained, and finally the transformation of the corresponding short block from the local optimization is applied to each frame image in the short block to obtain the camera pose of each frame image in the short block.

4. The 3D reconstruction method based on target detection according to claim 1, characterized in that: The specific calculation method for size measurement is as follows: first, a 3D stereo frame is established to completely surround the object to be reconstructed, then the 3D stereo frame is divided into n voxels, and the TSDF value and weight of each voxel on the current depth map are calculated, and then the TSDF value is updated by weighted average of the new voxel value and the previously stored voxel value. The formula is as follows: TSDF(p)=[W(p)TSDF(p)+w(p)tsdf(p)] / [W(p)+w(p)], (2) W(p) is the updated weight value, TSDF(p) is the TSDF value of the updated frame of voxel p, tsdf(p) is the TSDF value of the current frame of voxel p, w(p) is the weight value of the current frame, and the weight calculation formula for each voxel is as follows: W(p)=W(p)+w(p), (3) Segmentation is performed by building a triangle mesh along the target sampling surface, discarding triangles with edge lengths exceeding a threshold to avoid subdivision step discontinuities, and finally calculating a weight for each vertex. After the range image is converted into a triangle mesh with a weight for each vertex, the voxel grid can be updated.

Images