A three-dimensional reconstruction method and device based on surface element

Through superpixel segmentation and regularization fusion technology, the problem of low accuracy and 'hole' phenomenon based on surface element 3D reconstruction at the edge is solved, which improves the robustness and stability of 3D reconstruction and achieves higher reconstruction accuracy and smoothness.

CN114170288BActive Publication Date: 2024-12-03BEIJING INST OF TECH
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202111439939.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-30
Publication Date
2024-12-03
Estimated Expiration
2041-11-30

AI Technical Summary

Technical Problem

Three-dimensional reconstruction based on surface elements is prone to low accuracy and ‘empty’ at the edges, resulting in insufficient robustness and stability.

Method used

Using superpixel segmentation and regularization fusion technology, the edge point detection of binocular color images can be obtained, superpixels and their spatial coordinates are initialized and updated, and the surface element fusion is used for local maps, and combined with filtering processing is reduced. hollow'generated.

Benefits of technology

It significantly improves the robustness of three-dimensional reconstruction and the reconstruction accuracy and stability at the edges, reduces the dependence of face elements on pixels, and enhances the smoothness and accuracy of reconstruction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114170288B_ABST
    Figure CN114170288B_ABST
Patent Text Reader

Abstract

The present invention discloses a three-dimensional reconstruction method and device based on facets, the method comprising: combining a depth map of a scene to be reconstructed and a binocular color image, performing edge point detection and super pixel extraction based on a facet method, and introducing regularization constraints to obtain a dense three-dimensional map. The present invention can significantly reduce the generation of edge "holes" in the reconstructed image, improve the smoothness of plane reconstruction, the robustness of three-dimensional reconstruction, and greatly improve the reconstruction accuracy and stability of facet-based three-dimensional reconstruction at the edge.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the technical field of a three-dimensional reconstruction method based on surface elements. Background Art

[0002] At present, 3D reconstruction can be divided into surface-based, voxel-based and point cloud-based 3D reconstruction. Among them, point cloud-based 3D reconstruction is not suitable for large scene environment reconstruction, and in order to increase its observability, it usually needs to be used in conjunction with image mapping, and the application requirements are high; voxel-based 3D reconstruction has a high advantage in reconstructing large scenes, but voxels are not easy to deform, and it is difficult to correct the modeling accuracy in real time, so it cannot make good use of the advantages of closed-loop correction; surface-based 3D reconstruction has the advantages of variability and small storage space, and can be applied to large scenes and small environments. 3D reconstruction, but the accuracy of surface-based 3D reconstruction depends on the extraction accuracy of pixels and depth maps, as well as the positioning accuracy of the positioning system, and it is easy to have low reconstruction accuracy at the edges and produce "holes". Summary of the invention

[0003] The purpose of the present invention is to provide a new facet-based 3D reconstruction method, which can reduce the generation of "holes", significantly improve the robustness of 3D reconstruction, and greatly improve the reconstruction accuracy and stability of facet-based 3D reconstruction at the edges, in response to the problem that facet-based 3D reconstruction relies on the extraction accuracy of pixels and depth maps, as well as the positioning accuracy of the positioning system, and is prone to holes at the edges.

[0004] The technical solution of the present invention is as follows:

[0005] A facet-based 3D reconstruction method comprises: performing super pixel segmentation including edge point detection on any binocular color image of a scene to be reconstructed;

[0006] Obtaining the segmented super pixels and the spatial coordinates of the pixels contained therein in a fused image of the binocular color image and its corresponding depth map;

[0007] Using the super pixel as a bin, initializing the bin and updating the bin based on the pixels contained therein to obtain bins of different frames;

[0008] The facets of different frames or their facets with the local map are fused to obtain a reconstructed three-dimensional atlas.

[0009] According to some preferred embodiments of the present invention, the surface element fusion includes: fusing the different frame surfaces or the different frame surfaces and the local map based on regularization constraints.

[0010] According to some preferred embodiments of the present invention, the local map is extracted from an atlas database based on a current frame pose graph corresponding to the fused image.

[0011] According to some preferred embodiments of the present invention, the reconstruction method further includes filtering the depth map, and the filtering model is as follows:

[0012] When the depth value of a pixel in the depth map exceeds a certain threshold, or the depth value difference of pixels in a neighborhood pixel set where the pixel is located exceeds a certain threshold, the original depth of the pixel is replaced by the average depth value of the pixel set; wherein the neighborhood pixel set is a set of pixels within a circular range with the pixel as the center and a certain number of pixels as the radius.

[0013] According to some preferred embodiments of the present invention, obtaining the super pixel includes:

[0014] (1) selecting random seeds at equal distances in the binocular color image;

[0015] (2) performing edge detection on the binocular color image to extract edge points;

[0016] (3) using the random seeds and the edge points as cluster centers, initializing the intensity, depth, and normal vector direction of each pixel in the cluster area, and then performing cluster analysis;

[0017] (4) Perform super pixel segmentation based on the similarity of pixel intensity and depth obtained by the cluster analysis.

[0018] According to some preferred embodiments of the present invention, the cluster analysis comprises:

[0019] When the cluster center C i =[x i ,y i , d i , c i , r i ] T When the edge point is near the edge line formed by the edge point and the number of its clustered pixels does not exceed 6, the radius of the corresponding super pixel is set to be equal to the plane distance from the cluster center to the edge point, as follows:

[0020] D=(x i -u x ) 2 +(y i -u y ) 2 (1)

[0021] When a new pixel is added, all the parameters of the superpixel obtained after adding the new pixel are used to update all the parameters of the superpixel before adding it, and the average depth of the superpixel center is updated using the Huber parameter, as follows:

[0022]

[0023] Among them, [x i ,y i ] T is the average coordinate of clustered pixels, d i is the average depth of the pixels at the cluster center, c i is the average intensity value of the pixel at the cluster center, r i is the radius of the superpixel, which is equal to 1.2 times the plane distance between the cluster center and the farthest point in the image. x ,u y ] T Indicates the coordinates of the pixels to be clustered; u is the new pixel added, u d is the depth of the newly added pixel, d i is the average depth of the superpixel center before updating, L δ is the Huber coefficient.

[0024] According to some preferred embodiments of the present invention, the initialization of the facet includes:

[0025] Set the initialization pixel S = [S p , S n , S c , S w , S r , S t , S i ] T In the figure, the position of the face element S p ∈R 3 is the spatial point coordinate corresponding to the cluster center of the super pixel, and the pixel intensity S c ∈R is the average pixel intensity of the pixels contained in the super pixel, and the surface element radius S r ∈R + The update time S is the distance between the farthest pixel in the superpixel and the cluster center of the superpixel multiplied by the distance weight. t is the index value of the key frame; the index of the key frame S i is 0, and:

[0026] Normal vector S of the surface element n ∈R 3 as follows:

[0027]

[0028] Weight Sw ∈R + as follows:

[0029]

[0030] Among them, p u is the spatial point coordinate of the pixel contained in the super pixel, p is the spatial point coordinate corresponding to the cluster center of the super pixel, b represents the deviation, E s represents the fitting error, f represents the focal length, S p (z) represents the depth of the surface element, σ represents the binocular disparity;

[0031] The updating of the face element based on the contained pixels includes:

[0032] When a new pixel is added, all parameters of the superpixel obtained after adding the new pixel are used to update all parameters of the initialized pixel.

[0033] According to some preferred embodiments of the present invention, the reconstruction method further comprises:

[0034] Obtain the current frame pose graph based on the odometer, and determine whether the system pose graph needs to be updated based on loop detection;

[0035] When the system pose graph does not need to be updated, a local map extracted from a map database according to the current frame pose graph is subjected to face fusion with the initialization face to obtain a fused surface; the fused surface is added to the map database to obtain an updated atlas;

[0036] When the system pose graph needs to be updated, the current frame pose graph obtained by the odometer is used to deform the corresponding map in the system database to obtain a deformed map, and the deformed map is added to the map database to obtain an updated atlas.

[0037] More preferably, the deformation process comprises: using The position and normal vector of the surface element S attached to the key frame F are transformed, where and T w,F They are the poses of the key frame F before and after the loop closure. After deformation, the pose after the loop closure is used to replace the pose before the loop closure for the next local map deformation.

[0038] According to some preferred embodiments of the present invention, the facet fusion is performed based on the following regularized constraint model:

[0039]

[0040] Among them, E regrepresents the regularization loss, v represents the vth face element to be fused, i represents all faces in the face element set except face element v, S represents the face element set, S pi represents the positions of all the extracted facets except facet v, represents the positions of all facets except facet v in the local map, S pv represents the position of the facet v in the extracted facet, represents the position of the surfel v in the local map, S ni Represents the normal vectors of all the extracted face elements except face element v, represents the normal vectors of all facets in the local map except facet v, S nv Represents the normal vector of the face element v in the extracted face element, Represents the normal vector of the surface element v in the local map.

[0041] The present invention further provides a device that can implement the above-mentioned reconstruction method, which includes one or more of the following modules: a super pixel extraction module for implementing super pixel extraction, an edge detection module for implementing edge detection, a facet extraction module for implementing facet extraction, a facet fusion module for implementing regularized facet fusion, a filtering module for implementing depth map filtering, a facet update module for updating facets based on loop detection of odometer, and an image processing module.

[0042] The present invention has the following beneficial effects:

[0043] The present invention is based on super pixels, introduces edge point detection, and further, can introduce super pixel regularization fusion, which can effectively increase edge straight line constraints, improve the robustness of three-dimensional reconstruction, and greatly improve the reconstruction accuracy and stability of three-dimensional reconstruction based on surface elements at the edges, and can further improve the smoothness of surface element fusion and reduce the generation of "holes".

[0044] The present invention uses super pixels instead of pixels to reduce the dependence of facets on pixels. At the same time, by adding edge constraints, the problem of low reconstruction accuracy at edge joints that occurs when using super pixels is effectively eliminated. The two work together to significantly enhance the robustness and accuracy of reconstruction.

[0045] In some specific implementations, the present invention denoises the depth map directly generated by, for example, an RGBD camera through filtering processing, which can achieve better reconstruction effect.

[0046] Reconstructing a large-scale environment may generate millions of facets, but in the specific implementation of the present invention, when performing facet fusion, only a facet subset needs to be extracted based on the posture graph, that is, the local map is fused with the initialized facets.

[0047] If local map fusion is performed directly, an update time will be generated that is independent of the scale of the reconstructed scene. However, due to the accumulated tracking error of the positioning system, the fusion of facets with larger drift will significantly reduce the scene reconstruction accuracy and cannot achieve global consistency. However, in the specific implementation of the present invention, the pose graph in the positioning system is used to identify the local map, and the facets generated by the current key frame are fused with the local map composed of connected key frames, which can effectively improve the scene reconstruction accuracy.

[0048] In a specific implementation of the present invention, only the extracted facets may be deformed so that the relative posture between each facet and its attached keyframe remains unchanged. Therefore, although the surface attached to the same keyframe is rigidly deformed, the overall deformation of the map is non-rigid. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 The figure is a flowchart of a specific facet-based 3D reconstruction method.

[0050] Figure 2 Initialize a partition map for a superpixel in a specific embodiment.

[0051] Figure 3 It is a schematic diagram of the regularization constraint process during face element fusion in a specific embodiment. DETAILED DESCRIPTION

[0052] The present invention is described in detail below in conjunction with the embodiments and drawings, but it should be understood that the embodiments and drawings are only used to exemplify the present invention and do not constitute any limitation on the protection scope of the present invention. All reasonable changes and combinations within the scope of the inventive concept of the present invention fall within the protection scope of the present invention.

[0053] Reference Figure 1 According to the technical solution of the present invention, a specific method for reconstructing a three-dimensional scene based on facets includes:

[0054] S1 aligns and calibrates the depth map of the scene to be reconstructed and its binocular image to generate a fused image;

[0055] Preferably, in a specific implementation, considering that the depth map directly generated by an RGBD camera or the like has large noise, which will affect the reconstruction accuracy, the depth map can be further filtered to improve the reconstruction accuracy. A preferred filtering processing model is as follows:

[0056] When the depth value of a pixel in the depth map exceeds a certain threshold, or when the depth value difference of a pixel in a circular range with a certain pixel as the center and a certain number of pixels as the radius in the depth map exceeds a certain threshold, the original depth value is replaced by the depth average of all pixels in the pixel set as the new depth value of the pixel.

[0057] S2 extracts super pixels including edge point detection from the fused image to obtain a super pixel point segmentation map including edge line constraints;

[0058] Among them, more specifically,

[0059] The edge point detection may use a variety of edge extraction algorithms, such as an Edge Drawing algorithm.

[0060] The extraction of the superpixel can be carried out with reference to the prior art, such as the superpixel extraction process proposed by Kaixuan Wang (Wang K, Gao F, Shen S. Real-time Scalable Dense Surfel Mapping [C] / / 2019 International Conference on Robotics and Automation (ICRA). 2019.).

[0061] In this process, the introduction of edge point detection can reduce the scale of super pixels at the edge points and generate straight line constraints on the edges of the segmented areas.

[0062] Preferably, the present invention can specifically select random seeds for super pixel extraction and allocation as follows:

[0063] (1) Divide the image into k super-pixel regions of equal size, referring to Figure 2 (Original image is in color) as shown;

[0064] (2) Using random seeds and edge points as cluster centers, the intensity, depth, and spatial position of each pixel are initialized;

[0065] (3) Reassign superpixels based on the similarity of pixel intensity and depth after initialization;

[0066] (4) Combined with the edge point detection results, the radius of the super pixel at the edge point shall not be greater than a certain threshold pixel, and a super pixel point set with edge straight line constraints is obtained.

[0067] A more specific method of obtaining a super pixel point set with edge straight line constraints includes:

[0068] Obtain edge point coordinates through edge detection;

[0069] When the cluster center C of the superpixel i =[x i ,y i , d i , c i , r i ] T When the edge line is near the edge line, and the number of pixels in the cluster cannot exceed 6, the radius of the super pixel after redistribution is set equal to the plane distance from the cluster center to the edge point, as follows:

[0070] D=(x i -u x ) 2 +(y i -u y ) 2 (1)

[0071] Among them, [x i ,y i ] T is the average coordinate of clustered pixels, d i is the average depth of the pixels at the cluster center, c i is the average intensity value of the pixel at the cluster center, r i is the radius of the superpixel, which is equal to 1.2 times the plane distance between the cluster center and the farthest point in the image. x ,u y ] T Indicates the coordinates of the pixels to be clustered;

[0072] Every time a new pixel is added, the original super pixel center is updated using the average value of the new pixel, and the average depth of the super pixel center is updated using the Huber parameter, as shown in formula (2):

[0073]

[0074] Where u is the new pixel added, u d is the depth of the newly added pixel, d i is the average depth of the superpixel center before updating, L δ is the Huber coefficient.

[0075] S3 performs facet extraction on the superpixel to obtain facets;

[0076] Specifically, the facet extraction may further include:

[0077] Get the initialization pixel S = [S p , S n , S c , S w , Sr , S t , S i ] T ;

[0078] Among them, S p ∈R 3 is the location of the face element, and the spatial point coordinates corresponding to the cluster center of the superpixel are used as its initialization value; S c ∈R is the intensity of the image pixel, and the cluster average pixel intensity is used as its initialization value; S r ∈R + is the facet radius, which can be determined based on the distance between the farthest pixel in the superpixel and the cluster center. For example, the radius is chosen to be equal to the distance weight multiplied by the distance between the farthest pixel cluster center; S t is the update time, using the index of the keyframe as its initialization value; S i is the index of the key frame where the face is located, and uses 0 as its initialization value;

[0079] S n ∈R 3 is the normal vector of the face element, and its initialization value is obtained by minimizing the normal vector cross product error, as shown in the following formula (3):

[0080]

[0081] Among them, p u is the spatial point coordinate of the pixel contained in the super pixel, p is the spatial point coordinate corresponding to the center coordinate of the super pixel, b represents the deviation, E s represents the fitting error;

[0082] S w ∈R + is the weight, and its initialization value is the inverse depth of the bin, as shown in the following formula (4):

[0083]

[0084] Where f is the focal length, S p (z) represents the depth of the surface element, and σ represents the binocular disparity.

[0085] S4 obtains the current frame pose graph based on the odometer, and determines whether the system pose graph needs to be updated based on loop detection.

[0086] S5: when the system pose graph does not need to be updated, performing facet fusion on the local map extracted from the map database according to the pose graph of the current frame and the facet graph to obtain a fused surface; adding the fused surface to the map database to obtain an updated atlas;

[0087] Regularization constraints are used to fuse the local map and the extracted face elements so that each face element has pose constraints on each other when fused, as shown in the attached figure. Figure 3 As shown, its regularization constraints are as follows:

[0088]

[0089] Where v represents the vth face element, and i is all the faces in the face element set except v. It represents the i-th facet in the local map, the superscript 1 represents the local map, E reg represents the regularization loss.

[0090] After the fusion is completed, the current key frame pose T of the positioning system is used wc , transform the updated local map to the global coordinate system.

[0091] S6 When the system pose graph needs to be updated, the corresponding map in the system database is deformed through the current frame pose graph to ensure global consistency, and an updated atlas containing the deformed surface is obtained.

[0092] Specifically, the deformation process includes:

[0093] use The position and normal vector of the surface element S attached to the key frame F are transformed, where and T w,F They are the poses of the key frame F before and after the loop closure. After deformation, the pose after the loop closure is used to replace the pose before the loop closure for the next local map deformation.

[0094] The above embodiments are only preferred implementations of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A three-dimensional reconstruction method based on surface element, characterized in that: It includes: Perform super pixel segmentation including edge point detection on any binocular color image of the scene to be reconstructed; Obtaining the segmented super pixels and the spatial coordinates of the pixels contained therein in a fused image of the binocular color image and its corresponding depth map; Using the super pixel as a bin, initializing the bin and updating the bin based on the pixels contained therein to obtain bins of different frames; Performing fusion based on regularization constraints between different frame surfaces or between different frame surfaces and local maps, i.e., surface fusion, to obtain a reconstructed three-dimensional atlas; The facet fusion is performed based on the following regularized constraint model: Among them, E reg represents the regularization loss, v represents the vth face element to be fused, i represents all faces in the face element set except face element v, S represents the face element set, S pi represents the positions of all the extracted facets except facet v, represents the positions of all facets except facet v in the local map, S pv represents the position of the facet v in the extracted facet, represents the position of the surfel v in the local map, S ni Represents the normal vectors of all the extracted face elements except face element v, represents the normal vectors of all facets in the local map except facet v, S nv Represents the normal vector of the face element v in the extracted face element, Represents the normal vector of the surface element v in the local map.

2. The reconstruction method according to claim 1, characterized in that: The facet fusion includes: fusing the facets of different frames or fusing the facets of different frames with the local map based on regularization constraints.

3. The reconstruction method according to claim 1, characterized in that: The local map is extracted from an atlas database according to a current frame pose graph corresponding to the fused image.

4. The reconstruction method according to claim 1, characterized in that: It also includes filtering the depth map first, and the filtering model is as follows: When the depth value of a pixel in the depth map exceeds a certain threshold, or the depth value difference of pixels in a neighborhood pixel set where the pixel is located exceeds a certain threshold, the original depth of the pixel is replaced by the average depth value of the pixel set; wherein the neighborhood pixel set is a set of pixels within a circular range with the pixel as the center and a certain number of pixels as the radius.

5. The reconstruction method according to claim 1, characterized in that: The obtaining of the super pixel comprises: (1) selecting random seeds at equal distances in the binocular color image; (2) performing edge detection on the binocular color image to extract edge points; (3) using the random seeds and the edge points as cluster centers, initializing the intensity, depth, and normal vector direction of each pixel in the cluster area, and then performing cluster analysis; (4) Perform super pixel segmentation based on the similarity of pixel intensity and depth obtained by the cluster analysis.

6. The reconstruction method according to claim 5, characterized in that: The cluster analysis includes: When the cluster center C i =[x i ,y i ,d i ,c i ,r i ] T When the edge point is near the edge line formed by the edge point and the number of its clustered pixels does not exceed 6, the radius of the corresponding super pixel is set to be equal to the plane distance from the cluster center to the edge point, as follows: D=(x i -u x ) 2 +(y i -u y ) 2 (1), When a new pixel is added, all the parameters of the superpixel obtained after adding the new pixel are used to update all the parameters of the superpixel before adding it, and the average depth of the superpixel center is updated using the Huber parameter, as follows: Among them, [x i ,y i ] T is the average coordinate of clustered pixels, d i is the average depth of the pixels at the cluster center, c i is the average intensity value of the pixel at the cluster center, r i is the radius of the superpixel, which is equal to 1.2 times the plane distance between the cluster center and the farthest point in the image. x ,u y ] T Indicates the coordinates of the pixels to be clustered; u is the new pixel added, u d is the depth of the newly added pixel, d i is the average depth of the superpixel center before updating, L δ is the Huber coefficient.

7. The reconstruction method according to claim 6, characterized in that: The initialization of the facet includes: Set the initialization pixel S = [S p ,S n ,S c ,S w ,S r ,S t ,S i ] T In the figure, the position of the face element S p ∈R 3 is the spatial point coordinate corresponding to the cluster center of the super pixel, and the pixel intensity S c ∈R is the average pixel intensity of the pixels contained in the super pixel, and the surface element radius S r ∈R + The update time S is the distance between the farthest pixel in the superpixel and the cluster center of the superpixel multiplied by the distance weight. t is the index value of the key frame; the index of the key frame S i is 0, and: Normal vector S of the surface element n ∈R 3 as follows: Weight S w ∈R + as follows: Among them, p u is the spatial point coordinate of the pixel contained in the face element, p is the spatial point coordinate corresponding to the super pixel, b represents the deviation, E s represents the fitting error, f represents the focal length, S p (z) represents the depth of the surface element, σ represents the binocular disparity; The updating of the face element based on the contained pixels includes: When a new pixel is added, all parameters of the superpixel obtained after adding the new pixel are used to update all parameters of the initialized pixel.

8. The reconstruction method according to claim 1, characterized in that: It also includes, Obtain the current frame pose graph based on the odometer, and determine whether the system pose graph needs to be updated based on loop detection; When the system pose graph does not need to be updated, a local map extracted from a map database according to the current frame pose graph is subjected to face fusion with the initialization face to obtain a fused surface; the fused surface is added to the map database to obtain an updated atlas; When the system pose graph needs to be updated, the current frame pose graph obtained by the odometer is used to deform the corresponding map in the system database to obtain a deformed map, and the deformed map is added to the map database to obtain an updated atlas.

9. The reconstruction method according to claim 8, characterized in that: The deformation process includes: using The position and normal vector of the surface element S attached to the key frame F are transformed, where and T w,F They are the poses of the key frame F before and after the loop closure. After deformation, the pose after the loop closure is used to replace the pose before the loop closure for the next local map deformation.

Citation Information

Patent Citations

  • A real-time indoor 3D reconstruction method based on low-cost RGB-D sensor

    CN109087394A

  • Three-dimensional reconstruction method and device of space plane relation constraint

    CN113034681A

  • Dense visual slam with probabilistic surfel map

    US20190026943A1