A 3D point cloud fusion method based on manifold embedding

Abstract

The present application provides a three-dimensional point cloud fusion method based on manifold embedding, including: obtaining multiple frames of point cloud data of a target to be reconstructed, and voxelizing each frame of point cloud data; for each frame of voxelized point cloud data: establishing a neighbor relationship between voxels and corresponding points in an accumulation model through a hash index, wherein, in an initial state, the first frame of point cloud data is used as an accumulation model; adding voxels that have not established a neighbor relationship in the voxelized point cloud data of the current frame to the accumulation model, verifying the accumulation model based on each voxel that has established a neighbor relationship in the voxelized point cloud data of the current frame, and using the voxels that have established a neighbor relationship to perform a point-to-surface fusion update on the verified accumulation model; after completing the fusion update of the accumulation model using all voxelized point cloud data, outputting the accumulation model as the reconstruction result of the target to be reconstructed. The present method can achieve high-precision reconstruction while maintaining low computational complexity.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional reconstruction, and more specifically, to a three-dimensional point cloud fusion method based on manifold embedding. Background Art

[0002] The process of fusing 3D data requires accurate and dense correspondences, which can be achieved through point-to-point or point-to-pixel correspondences, as seen in the work of Keller et al. (2013), Whelan et al. (2016), Schops et al. (2020), Chen et al. (2021), and Chen et al. (2022). However, this is a huge challenge in the field of computer vision, especially for mobile devices with limited battery capacity. Therefore, recent research attempts have explored non-projective models inspired by optimal transport theory to improve the accuracy of reconstruction. Although these methods have higher accuracy, they require a lot of computational resources due to the large number of correspondence candidates.

[0003] The non-projection model based on optimal transport theory is constructed by minimizing L 2 -Wasserstein distance, which introduces dense correspondences between points. Although this model provides higher reconstruction accuracy for noisy input data, it is computationally expensive. To overcome this, recent studies have proposed methods such as GaussianFusion and CyclicalFusion, which exploit the network simplex algorithm and cyclic monotonicity to speed up the optimization process.

[0004] Learning methods have received considerable attention in the field of 3D reconstruction. However, these learning-based methods often lack robustness when dealing with sensor noise, especially for imaging systems different from those used in the training set. There are several reasons why training networks for 3D reconstruction may not be feasible. First, consumer-grade sensors used in conjunction with devices with limited battery capacity often have limited computational power and are not suitable for resource-intensive tasks like training deep learning models. For example, DI-Fusion only processes at 10Hz per second on modern GPU platforms, and the training process itself can take days or even weeks. Other end-to-end networks, such as NICE-SLAM, update 3D representations in a back-propagation style and require a modern GPU (RTX3090 TDP350W) with 24GB of GPU memory to reconstruct an indoor scene. Second, learning methods heavily rely on specific datasets, which are often virtual and may not fully represent real-world scanned scenes. For example, IMLSnet, which was trained on the ShapeNet dataset, has difficulties when processing real scan inputs. Importantly, different imaging systems introduce various types of noise. For example, lidar sensors are affected by ambient light and reflectivity, while stereo sensors suffer from matching errors. Even within the lidar category, solid-state lidars are mechanistically different. Training a unified network to handle these diverse imaging systems remains a challenge. Therefore, we chose a classic approach to design an interpretable algorithm using mathematical tools to address this challenge.

[0005] Reconstruction with high geometric quality is a prominent task in computer vision. This process is mainly based on projective and non-projective models with different geometric representations, namely, surface / point-based methods and voxel-based methods. Voxel-based methods, such as those proposed by Newcombe et al., Niessner et al., Zhou et al., and Dai et al., process volumetric data and represent the surface through the truncated signed distance function (TSDF) in voxels. The most common voxel correspondence lookup procedure for projected TSDF measurement is to perform nearest neighbor queries on the depth map. Then, the surface is updated by applying a weighted average function. Recently, learning-based methods such as NICE-SLAM, iMap, DI-Fusion, NeuralFusion, IMLSNet, etc. have been proposed to improve the reconstruction quality. Among these methods, NICE-SLAM introduces a hierarchical scene representation that combines multi-level local information. The representation is optimized using pre-trained geometric priors to achieve detailed reconstruction of large indoor scenes. iMap uses a keyframe structure and multi-processing computational flow combined with dynamic information-guided pixel sampling to improve computational speed. IMLSNet uses an octree structure to provide a framework for generating moving least squares (MLS) points where needed. In addition, it describes the geometric features of the shape through local learned priors. However, these end-to-end networks require huge GPU memory (typically 24GB) to reconstruct indoor scenes at low mapping / fusion rates (2-10Hz) and are very sensitive to sensor noise, especially noise that is different from the training set.

[0006] Surf and point representations are widely used in computer vision, graphics, and multimedia as they encode additional information such as confidence, radius, and timestamp in addition to spatial coordinates and normals. During the fusion process, surfels and points are projected onto the upcoming depth map using the known sensor pose. Schops et al. addressed the uncertainty issue by defining depth intervals to handle small uncertainties. The projected correspondences are then classified as supported, conflicting, or occluded, depending on the depth interval. The verified correspondences are fused into new surfels or points using a weighted average function, such as Gaussian weights. Finally, the fused model is temporally refined by removing conflicting fused points / surfels subject to confidence and temporal constraints. Recently, non-projective methods, Gaussian fusion and recurrent fusion, have been proposed based on optimal transfer theory to achieve fine reconstruction of indoor scenes. Although both methods use the same metric, i.e., L 2 -Wasserstein distance, but the core optimization idea is different. Gaussian fusion uses displacement interpolation as L 2-Wasserstein space to derive point-to-point correspondences from the optimal transmission plan. In contrast, loop fusion exploits loop monotonicity to efficiently construct point-to-point correspondences. A significant advantage of loop fusion is that its time complexity can reach the quadratic level, while the time complexity of Gaussian fusion is the cubic level.

[0007] In summary, in the existing technology of 3D reconstruction, there is usually a trade-off problem between complexity and reconstruction accuracy, and it is difficult to achieve high-precision reconstruction in a low-complexity solution. Summary of the invention

[0008] The purpose of the embodiments of the present application is to provide a three-dimensional point cloud fusion method based on manifold embedding, which achieves high-precision reconstruction while maintaining low computational complexity.

[0009] In order to achieve the above purpose, the embodiments of the present application are implemented in the following manner:

[0010] In a first aspect, an embodiment of the present application provides a three-dimensional point cloud fusion method based on manifold embedding, comprising: obtaining multi-frame point cloud data of a target to be reconstructed, and voxelizing each frame of point cloud data; for each frame of voxelized point cloud data: establishing a proximity relationship between voxels and corresponding points in an accumulated model through a hash index, wherein, in an initial state, the first frame of point cloud data is used as the accumulated model; adding voxels for which a proximity relationship has not been established in the voxelized point cloud data of the current frame to the accumulated model, verifying the accumulated model based on each voxel for which a proximity relationship has been established in the voxelized point cloud data of the current frame, and performing a point-to-surface fusion update on the verified accumulated model using the voxels for which a proximity relationship has been established; after completing the fusion update of the accumulated model using all voxelized point cloud data, outputting the accumulated model as a reconstruction result of the target to be reconstructed.

[0011] In combination with the first aspect, in a first possible implementation of the first aspect, the cumulative model is verified based on each voxel with a neighboring relationship established in the point cloud data after voxelization of the current frame, and the verified cumulative model is updated from point to surface using the voxels with a neighboring relationship, including: for each voxel with a neighboring relationship established in the point cloud data after voxelization of the current frame: determining a set of three-dimensional sampling points determined by a hash index in the cumulative model Among them, the three-dimensional sampling point set Each 3D sampling point in the cumulative model is a voxel V in the voxelized point cloud data of the current frame. k Neighboring 3D sampling points; using the voxel V in the point cloud data after voxelization of the current frame k For 3D sampling points Perform initial verification; if the three-dimensional sampling points The initial verification of the three-dimensional sampling points Perform secondary verification; if the three-dimensional sampling point The initial verification failed, and the three-dimensional sampling points were determined. Do not participate in the update process of the accumulated model by the point cloud data after voxelization of the current frame; if the 3D sampling point The secondary verification is passed to determine the three-dimensional sampling points Participate in the update process of the accumulated model by the point cloud data after voxelization of the current frame; if the three-dimensional sampling point The secondary verification fails, and the three-dimensional sampling points are determined The point cloud data after voxelization of the current frame does not participate in the updating process of the accumulated model; for the three-dimensional sampling points participating in the updating process of the accumulated model, the three-dimensional sampling points are updated from point to surface based on the point cloud data after voxelization of the current frame.

[0012] In combination with the first possible implementation manner of the first aspect, in a second possible implementation manner of the first aspect, a voxel V in the voxelized point cloud data of the current frame is used. k For 3D sampling points Perform initial verification, including: determining the three-dimensional sampling points With voxel V k Is it the sampling of the same scanning surface, and the three-dimensional sampling points V k Support point, voxel V k is the 3D sampling point The supporting voxels of Satisfy the judgment condition and determine the three-dimensional sampling point The initial verification passed.

[0013] In combination with the second possible implementation of the first aspect, in a third possible implementation of the first aspect, the three-dimensional sampling point is determined With voxel V k Is it the sampling of the same scanning surface, and the three-dimensional sampling points V k Support point, voxel V k is the 3D sampling point Support voxels, including: Determine the three-dimensional sampling points The corresponding unit normal vector And, determine the voxel V k The average unit normal vector set up Inner Product Representation and The cosine between vectors, the target distribution is calculated by the following first target distribution function:

[0014]

[0015] Among them, the first target distribution function is a normal distribution with a mean of 0 and a standard deviation of σ1; the three-dimensional sampling point is determined The unit normal vector Whether it meets:

[0016]

[0017] Among them, μ1 is the value sampled from the uniform distribution U(0,1); if it satisfies Determine 3D sampling points Satisfy the conditions: 3D sampling points With voxel V k is the sampling of the same scanning surface, and the three-dimensional sampling points V k Support point, voxel V k is the 3D sampling point otherwise, determine the 3D sampling points Does not meet the judgment condition: 3D sampling point With voxel V k is the sampling of the same scanning surface, and the three-dimensional sampling points V k Support point, voxel V k is the 3D sampling point of supported voxels.

[0018] In combination with the third possible implementation manner of the first aspect, in a fourth possible implementation manner of the first aspect, the three-dimensional sampling point Perform secondary verification, including:

[0019] set up express Project to The directed projection distance on , the target distribution is calculated by the following second target distribution function:

[0020]

[0021] Among them, the second target distribution function is a normal distribution with a mean of 0 and a standard deviation of σ2; judging the three-dimensional sampling point The unit normal vector Whether it meets:

[0022]

[0023] Where μ2 is the value sampled from the uniform distribution U(0,1); if Determine 3D sampling points Pass the secondary verification, otherwise, determine the three-dimensional sampling points Failed secondary verification.

[0024] In combination with the fourth possible implementation manner of the first aspect, in a fifth possible implementation manner of the first aspect, the standard deviation σ1=0.5, and the standard deviation σ2=0.5.

[0025] In combination with the fourth possible implementation method of the first aspect, in a sixth possible implementation method of the first aspect, for the three-dimensional sampling points participating in the update process of the cumulative model, the three-dimensional sampling points are updated from point to surface based on the point cloud data after voxelization of the current frame, including: for each three-dimensional sampling point participating in the update process of the cumulative model: using an implicit moving least squares surface function for fusion guidance, embedding the three-dimensional sampling points participating in the update process of the cumulative model into a smooth manifold established by the point cloud data after voxelization of the current frame, to achieve point-to-surface fusion update.

[0026] In combination with the sixth possible implementation of the first aspect, in a seventh possible implementation of the first aspect, an implicit moving least squares surface function is used for fusion guidance, and the three-dimensional sampling points participating in the update process of the cumulative model are embedded in a smooth manifold established by the point cloud data after voxelization of the current frame, so as to realize the fusion update from point to surface, including: for satisfying The three-dimensional sampling points Determine the 3D sampling points Update increment in, For vector Projection to unit normal vector The directed projection distance on V k The center point of the update increment Combine the following formula to convert the three-dimensional sampling points Embedded into voxel V k In the corresponding smooth manifold, recursive optimization update is performed:

[0027]

[0028] Where F(x) is the implicit function of the moving least squares surface, whose null space Null(F) approximates the input surface S, which is the voxel V k The corresponding smooth manifold, p i ∈V k , V k ={p i ,i=1,2,…,n}, V k The center point, r k V k The radius of θ(|xp i |,r k ) is the weight function, n i For p i The unit normal vector of .

[0029] In conjunction with the seventh possible implementation manner of the first aspect, in an eighth possible implementation manner of the first aspect, the weight function θ satisfies:

[0030] θ(|xp i |,r k )=exp(-|xp i | 2 / r k 2 ).

[0031] Beneficial effects:

[0032] 1. This scheme designs a manifold embedding framework, which is a non-projective technique. Unlike the existing techniques, the manifold embedding framework uses dense point-to-surface correspondences and is guided by implicitly moving least squares surface functions rather than optimal transfer. These unique designs enable this scheme to achieve high reconstruction accuracy while maintaining low computational complexity. Moreover, these limitations are overcome by introducing a verification-update sampling method for manifold embedding without relying on any prior information. This enables robust 3D reconstruction even in the presence of sensor noise or changes in the imaging system.

[0033] 2. This scheme uses an RGB-D prototype system as the input for reconstruction, and includes other RGB-D systems in the experiments of this scheme. The output of the front end is a multi-combination data frame, which consists of color image frames, depth maps, IMU data, and posture data. This scheme uses the hardware clock and visual calibration of the sensor to align the multi-combination data frames. This scheme uses these combination data frames to reconstruct indoor environments in scanning and reconstruction applications. The algorithm of this scheme mainly fuses color image frames and depth maps, while the posture data is obtained through RealSense TMTracking camera T265 or algorithms such as RoseFusion, or other RGB-D systems are provided by visual odometry and reprojection error optimization. The main goal of this scheme is to simulate 3D fusion, which involves searching and updating dense correspondences. In order to improve the fusion efficiency, this scheme uses a smooth representation of the local surface as a guide to achieve fast point-to-surface fusion. The fusion method given in this scheme is different from previous methods. After the neighbor relationship between points and voxels is established, an implicit moving least squares surface function with a verification update sampling process (i.e., it needs to pass the initial verification and the second verification) is used for fusion guidance. In the fusion stage, this scheme abstracts the sampled data into a mathematical model and considers the 3D point set V generated by multiple combined data frameworks. k The fusion correspondence between its neighboring point sets aims to achieve point-to-surface fusion under the guidance of the implicit moving least squares surface function given in this scheme. This method is different from determining the point-to-point fusion correspondence based on geometric constraints. Under the guidance of the implicit moving least squares surface function given in this scheme, this scheme completes smooth fusion and forms a smooth manifold after multiple fusions. Despite the dense matching, this surface fusion method is efficient and improves computational efficiency. Therefore, this scheme can overcome the technical bottleneck of dense correspondence and achieve faster 3D fusion.

[0034] 3. In addition, this scheme decomposes the fusion process into a two-step filtering process to improve the reconstruction accuracy. The implicit moving least squares surface function method given in this scheme effectively solves the uncertainty interference, improves the point cloud quality, smoothes the noise points, and solves the multi-layer surface effect due to the uncertainty of the pose. Since the implicit moving least squares surface function relies on updating along the normal vector, the normal vector is crucial for fusion. Before starting the fusion, it is necessary to verify the three-dimensional sampling points in the accumulated model according to the normal vector. The normal vector direction of the corresponding nearby three-dimensional sampling points in the accumulated model should be consistent with the normal vector direction of the local surface to be fused. Verifying the normal vector can effectively avoid the degradation of reconstruction quality caused by incorrect fusion. Although multiple samplings of the same surface should have similar normal vector directions during sampling, a few samples that are severely affected by uncertainty can be effectively excluded by normal vector verification. Therefore, this scheme also proposes a two-step process of verification-update sampling to achieve fast and accurate 3D fusion through this two-step process of verification-update sampling (performing initial verification and secondary verification).

[0035] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 Schematic diagram of the optimization update of three-dimensional sampling points under the guidance of the implicit moving least squares function.

[0038] Figure 2 A flowchart of a three-dimensional point cloud fusion method based on manifold embedding is provided in an embodiment of the present application.

[0039] Figure 3 The figure is a comparison chart of the experimental results of the comparison algorithm and the reconstruction algorithm of this scheme in scene one.

[0040] Figure 4 The figure is a comparison chart of the experimental results of the comparison algorithm and the reconstruction algorithm of this scheme in scene 2.

[0041] Figure 5 The experimental results of the reconstruction algorithm of this scheme in scene one are compared with the true value map.

[0042] Figure 6 The experimental results of the reconstruction algorithm of this scheme in scene 2 are compared with the true value map. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0044] This paper proposes a surface fusion framework, focusing on the geometric representation of the surface. This scheme is different from the previous method. This scheme adopts a point-to-surface fusion method instead of a point-to-point method. Secondly, after the voxels and their neighboring points are determined, an implicit moving least squares surface function is used as a guide, combined with a verification-update sampling filter (primary verification and secondary verification are performed, and the fusion process is performed only after the secondary verification passes) to fuse the adjacent points.

[0045] In order to achieve fast neighboring search from point to voxel, this scheme uses hash index of spatial coordinates. Specifically, the point cloud of the input frame is first voxelized (the voxelization process in the prior art can be referred to, which will not be repeated here), and then the neighboring points of the voxels in the cumulative model are searched by hash index. The fusion method of this scheme is different from the previous methods. After the neighboring relationship between the point and the voxel is established, the implicit moving least squares surface function with a verification update sampling filter is used for fusion guidance. This method is different from determining the point-to-point fusion correspondence based on geometric constraints. Therefore, this scheme can overcome the technical bottleneck of dense correspondence and achieve faster 3D fusion.

[0046] The three-dimensional sampling point p in the neighborhood Ω i Represents the local surface. We use the three-dimensional sampling points in the voxel to represent the local surface. The moving least squares surface has many advantages and can handle the interference of depth and posture uncertainty well. This scheme uses the moving least squares surface as the geometric representation of the local surface. Since the projected moving least squares surface requires frequent calculation of the projected reference surface, the calculation time increases significantly with the increase in the number of three-dimensional sampling points. Therefore, the projected moving least squares surface is not suitable for dense sampling. Therefore, this scheme uses the implicit moving least squares surface function F(x) proposed by Kolluri to represent the local surface.

[0047] Assume that the input point cloud (i.e. point cloud data) obtained by the sensor at time t is expressed as Each 3D sampling point p i There is a unit normal vector and a control radius Unknown coordinates Define from x to p inclusive i The signed distance function of the tangent plane is <x-p i ,n i >. Here, is the set of sample points obtained by the sensor, i∈{1,2,...,N} and p i is the 3D coordinate of the 3D sampling point, and the corresponding normal vector is expressed as Take the input point cloud K at time t t The weighted average of all the distance functions in the implicit function F(x) is obtained. The null space Null(F) of F(x) is an effective approximation of the input surface S at time t. Based on this, this scheme defines the implicit moving least squares surface function of the input surface as follows:

[0048]

[0049] The weight function θ satisfies:

[0050] θ(|xpi |,r k )=exp(-|xp i | 2 / r k 2 ), (2)

[0051] Kolluri has proved that if the input point cloud K t is uniformly sampled, under this condition, the implicit moving least squares surface function F(x) accurately represents the original sampling surface S in terms of its geometric and topological properties. In addition, F(x) is an effective approximation of the signed distance function of the original surface. i |)=exp(-|xp i | 2 / r k 2 ) is the average Gaussian weight, r k V k The radius of .

[0052] Since when x is away from p i When the point cloud K is far away, the weight function decays rapidly. Therefore, in order to make good use of this effect, this scheme only considers the three-dimensional sampling points within the same voxel when designing the implicit moving least squares surface function, rather than the entire point cloud K. t , which greatly improves the computational efficiency of F(x). Then, equation (1) can be rewritten as:

[0053]

[0054] In formula (3), V k It is the point cloud K t A voxel in , and the local neighborhood relationship can be represented by voxels. That is, V k ={p i ,i=1,2,…,n}(voxel V k n 3D sampling points are stored in its neighborhood). V k The center point, r k V k The radius is the maximum distance from a 3D sampling point to the voxel center.

[0055] The null space Null(F) of F(x) shown in formula (3) exists in a narrowband region. Minimize F(x) to achieve smooth reconstruction of the point cloud. The implicit moving least squares function F(x) is effective in dealing with uncertainty interference. The goal of this scheme, guided by formula (3), is to achieve smooth point cloud fusion. Specifically, in the input point cloud K tUnder the guidance of the implicit surface function F(x), the three-dimensional sampling points of the cumulative model are smoothly fused. In the cumulative model, the three-dimensional sampling points And its unit normal vector When the cumulative model is combined with the input 3D sampling point cloud K t When the pose is aligned at time t, the input voxel V in the cumulative model is determined by the hash index k The set of nearby points at time t The superscript k indicates is the value corresponding to the kth voxel V k Nearby points of V k p i Since F(x) approximates the original surface signed distance function, the optimization process involves converting the points Embedded into voxel V k In the corresponding smooth manifold. Then, we have:

[0056]

[0057] The fusion update optimization process is expressed as formula (4), where yes Along the unit normal vector The update increment of Is a vector Projection to unit normal vector Here, Describes Update direction, where α∈[0,1]. In particular, when α=1, Along the unit normal vector Voxel V k Center Update, when α=0, remains at the initial position. Therefore, equation (4) indicates that under the guidance of the implicit moving least squares function F (i.e., transformed by F(x), or F(x)), Update in a given direction, and when F(x) approaches 0, the position of the embedded point approaches the sampling surface, such as Figure 1 Therefore, the optimization in a given direction in equation (4) can be achieved quickly through iteration.

[0058] Therefore, the implicit moving least squares guided fusion follows a series of organized steps: First, the function F(x) accurately approximates the original surface signed distance function and smoothly represents the input samples K through the null space Null(F). tThe local surface of . Under the guidance of F(x), the corresponding adjacent point sets in the cumulative model are smoothed through the manifold embedding framework. Smooth fusion is achieved by iteratively minimizing equation (4), as Figure 1 Depicted. Figure 1 In the figure, the black dots represent voxels V k The three-dimensional sampling points in Represents voxel V k The center of represents the corresponding average unit normal vector. The black curve is the null space Null(F) of the implicit function F(x), which is the t-th input point cloud K t The purple dots represent the voxels V in the cumulative model. k The black arrows represent The unit normal vector of Is a vector Projection to unit normal vector The directed projection distance on . Therefore, Representatives along The unit normal vector By updating the point set in a given direction, that is, (where α∈[0,1]) to complete the surface fusion. When α=1, the update moves towards the voxel V k Center Along the unit normal vector (purple dashed line). On the contrary, when α=0, remains in its initial position. Therefore, under the guidance of the implicit function F(x), we complete the smooth fusion, and a smooth manifold is formed after multiple fusions. Despite the dense matching, this surface fusion method is efficient and improves the computational efficiency. In addition, this scheme can effectively solve the uncertainty interference, improve the point cloud quality, smooth the noise points, and solve the multi-layer surface effect due to the uncertainty of the pose. This method relies on updating along the normal vector, which is crucial for fusion. Before starting the fusion, it is necessary to verify the 3D sampling points in the accumulated model according to the normal vector. The normal vector direction of the corresponding nearby 3D sampling points in the accumulated model should be consistent with the normal vector direction of the local surface to be fused. Verifying the normal vector can effectively avoid the degradation of reconstruction quality caused by incorrect fusion. Although multiple samplings of the same surface should have similar normal vector directions during sampling, a few samples that are severely affected by uncertainty can be effectively excluded by normal vector verification. Therefore, we also propose a two-step process of verification-update sampling to achieve fast and accurate 3D fusion through this two-step process of verification-update sampling.

[0059] The above is the theoretical derivation process of this scheme. Below, the three-dimensional point cloud fusion method based on manifold embedding designed based on the idea of ​​this scheme will be introduced.

[0060] See also Figure 2 , Figure 2 A flowchart of a 3D point cloud fusion method based on manifold embedding is provided in an embodiment of the present application. The 3D point cloud fusion method based on manifold embedding can be executed by an electronic device (server or smart terminal), and the method includes step S10, step S20, step S30 and step S40.

[0061] First, the electronic device may execute step S10.

[0062] Step S10: Acquire multiple frames of point cloud data of the target to be reconstructed, and voxelize each frame of point cloud data.

[0063] In this embodiment, the electronic device can obtain multiple frames of point cloud data of the target to be reconstructed.

[0064] This scheme uses an RGB-D prototype system as the input for reconstruction, and includes other RGB-D systems in subsequent experiments of this scheme. The output of the front end is a multi-combined data frame, which consists of color image frames, depth maps, IMU data, and posture data. This scheme uses the hardware clock and visual calibration of the sensor to align the multi-combined data frames so that these combined data frames can be used to reconstruct indoor environments in scanning and reconstruction applications. The algorithm of this scheme mainly fuses color image frames and depth maps, while the posture data is obtained through RealSense TM Tracking camera T265 or algorithms such as RoseFusion, or other RGB-D systems provide visual odometers and reprojection error optimization to form point cloud data. The data scanned by the sensor at a position and angle can be processed to form a frame of point cloud data (corresponding to the input point cloud mentioned above).

[0065] After obtaining multiple frames of point cloud data of the target to be reconstructed, each frame of point cloud data is voxelized. The specific process of voxelization is not an improvement point of this solution. This solution also uses mature means in the existing technology for voxelization, so it will not be described here.

[0066] After voxelizing each frame of point cloud data, the electronic device may execute step S20.

[0067] Step S20: for each frame of voxelized point cloud data: establish a proximity relationship between the voxel and the corresponding point in the accumulation model through a hash index, wherein, in the initial state, the first frame of point cloud data is used as the accumulation model.

[0068] For each frame of voxelized point cloud data, the electronic device can perform a hash index search on the neighboring points of the voxels in the cumulative model based on the voxelized point cloud data of the current frame to obtain the index result. That is, the proximity relationship between the voxel and the corresponding point in the cumulative model is established through the hash index. Here, the cumulative model is a model that is continuously updated and improved by the voxelized point cloud data. The initial cumulative model, that is, the initial state, can use the point cloud data of the first frame input (no voxelization is required) as the cumulative model.

[0069] Afterwards, the electronic device may execute step S30.

[0070] Step S30: Add the voxels that have not established a neighboring relationship in the voxelized point cloud data of the current frame to the cumulative model, verify the cumulative model based on each voxel that has established a neighboring relationship in the voxelized point cloud data of the current frame, and use the voxels that have established a neighboring relationship to perform point-to-surface fusion update on the verified cumulative model.

[0071] For the voxels that have not established neighboring relationships in the voxelized point cloud data of the current frame, these voxels that have not established neighboring relationships can be added to the cumulative model so that they can be verified in the verification process of the cumulative model with the point cloud data of subsequent frames. For these newly added voxels that have not established neighboring relationships in the cumulative model (which can be called three-dimensional sampling points in the cumulative model), they are regarded as the part to be verified of the cumulative model, and there is a life cycle of a certain frame time (for example, 10 consecutive frames, 20 frames, etc.). Each point in the part to be verified, if verified within the life cycle, can be retained in the cumulative model. As part of the cumulative model, it can be continuously verified and updated. If it is still not verified until the end of the life cycle, it will be removed from the cumulative model (if it has not been verified within the life cycle, it means that there are no voxels in the subsequent frames that have established neighboring relationships with these points, and it is very likely to be noise points).

[0072] For each voxel with a neighboring relationship in the voxelized point cloud data of the current frame, the accumulated model can be verified based on each voxel with a neighboring relationship in the voxelized point cloud data of the current frame, and the verified accumulated model can be updated from point to surface using the voxels with neighboring relationships.

[0073] In this embodiment, for each voxel in the voxelized point cloud data of the current frame, in which a neighbor relationship is established:

[0074] The electronic device can determine the three-dimensional sampling point set obtained by hash index in the cumulative model Among them, the three-dimensional sampling point set Each 3D sampling point in the cumulative model is a voxel V in the voxelized point cloud data of the current frame. k Neighboring 3D sampling points.

[0075] Then, the voxel V in the voxelized point cloud data of the current frame is used k For 3D sampling points The initial verification method is mainly to determine the three-dimensional sampling points With voxel V k Is it the sampling of the same scanning surface, and the three-dimensional sampling points V k Support point, voxel V k is the 3D sampling point of supported voxels.

[0076] Specifically, the electronic device can determine the three-dimensional sampling point The corresponding unit normal vector And, determine the voxel V k The average unit normal vector

[0077] set up Inner Product Representation and The cosine between vectors, the target distribution is calculated by the following first target distribution function:

[0078]

[0079] The first target distribution function is a normal distribution with a mean of 0 and a standard deviation of σ1. In this embodiment, the standard deviation σ1=0.5, but is not limited thereto.

[0080] Then determine the three-dimensional sampling point The unit normal vector Whether it meets:

[0081]

[0082] Where μ1 is a value sampled from the uniform distribution U(0,1).

[0083] If satisfied Determine 3D sampling points Satisfy the conditions: 3D sampling points With voxel V k is the sampling of the same scanning surface, and the three-dimensional sampling points V k Support point, voxel V k is the 3D sampling point Otherwise, determine the 3D sampling points Does not meet the judgment condition: 3D sampling point With voxel V k is the sampling of the same scanning surface, and the three-dimensional sampling points V k Support point, voxel V k is the 3D sampling point of supported voxels.

[0084] The three-dimensional sampling points Satisfy the judgment condition (three-dimensional sampling point With voxel V k is the sampling of the same scanning surface, and the three-dimensional sampling points V k Support point, voxel V k is the 3D sampling point When the supporting voxels are The initial verification passed.

[0085] If the three-dimensional sampling point The judgment condition is not met (three-dimensional sampling point With voxel V k is the sampling of the same scanning surface, and the three-dimensional sampling points V k Support point, voxel V k is the 3D sampling point The supporting voxels of ), there may be several cases:

[0086] (1) Three-dimensional sampling points Affected by the uncertainty of posture and depth, it becomes a disturbance point; (2) The corresponding voxel V on the input frame (i.e., the point cloud data of the current frame) k Disturbed by the uncertainty of posture and depth, it becomes a disturbed voxel; (3) The corresponding voxel V on the input frame (i.e., the point cloud data of the current frame) k and the three-dimensional sampling points on the cumulative model Both are affected by the uncertainty of pose and depth, and become disturbance points and disturbance voxels at the same time, and cannot be used for fusion.

[0087] A single validation and update process cannot distinguish between 3D sampling points When the judgment condition is not met, it belongs to one of the situations (1), (2), and (3). However, multiple verification and update processes can judge the disturbed voxels or disturbed points by the number of fusions and the confidence level, thereby distinguishing between situations (1), (2), and (3), and processing them differently (for example, losing the disturbed points, disturbed voxels, etc.), which is conducive to achieving high-quality reconstruction.

[0088] For 3D sampling points After the initial verification, the electronic device can be based on the three-dimensional sampling point Verification results of the three-dimensional sampling points Whether to perform secondary verification. The initial verification of the three-dimensional sampling points Perform secondary verification; if the three-dimensional sampling point The initial verification failed, and the three-dimensional sampling points were determined. It does not participate in the update process of the accumulated model by the voxelized point cloud data of the current frame.

[0089] For 3D sampling points For secondary verification, electronic devices can be verified in the following ways:

[0090] set up express Project to The directed projection distance on , the target distribution is calculated by the following second target distribution function:

[0091]

[0092] The second target distribution function is a normal distribution with a mean of 0 and a standard deviation of σ2. In this embodiment, the standard deviation σ2=0.5, but is not limited thereto.

[0093] Then determine the three-dimensional sampling point The unit normal vector Whether it meets:

[0094]

[0095] Where μ2 is a value sampled from the uniform distribution U(0,1).

[0096] If satisfied Electronics can determine the 3D sampling point Determine the three-dimensional sampling points through secondary verification Participate in the update process of the accumulated model by the voxelized point cloud data of the current frame. Otherwise, the 3D sampling point Failed the secondary verification, 3D sampling points It does not participate in the update process of the accumulated model by the voxelized point cloud data of the current frame.

[0097] For the three-dimensional sampling points participating in the updating process of the accumulation model, the electronic device can perform point-to-surface fusion update on the three-dimensional sampling points based on the voxelized point cloud data of the current frame.

[0098] Exemplarily, for each three-dimensional sampling point participating in the update process of the cumulative model: the electronic device can use an implicit moving least squares surface function for fusion guidance, embedding the three-dimensional sampling points participating in the update process of the cumulative model into a smooth manifold established by the voxelized point cloud data of the current frame, to achieve point-to-surface fusion update.

[0099] Specifically, to meet The three-dimensional sampling points Electronic devices can determine the three-dimensional sampling point Update increment in, For vector Projection to unit normal vector The directed projection distance on V k Then based on the update increment Combine the following formula to convert the three-dimensional sampling points Embedded into voxel V k In the corresponding smooth manifold, recursive optimization update is performed:

[0100]

[0101] Where F(x) is the implicit function of the moving least squares surface, whose null space Null(F) approximates the input surface S, which is the voxel V k The corresponding smooth manifold, p i ∈V k , V k ={p i ,i=1,2,…,n}, V k The center point, r k V k The radius of n i For p i The unit normal vector, θ(|xp i |,r k ) is the weight function, satisfying:

[0102] θ(|xp i |,r k )=exp(-|xp i | 2 / r k 2 ), (11)

[0103] Through the above method, the cumulative model can be verified and updated for each frame of voxelized point cloud data, so that the cumulative model becomes more refined in the continuous verification and update process. For the reconstruction of general scenes, each 3D sampling point in the accumulated model should generally be verified more than three times to ensure the reconstruction accuracy.

[0104] After completing the updating of the accumulated model using all the voxelized point cloud data, the electronic device may execute step S40.

[0105] Step S40: After completing the fusion update of the accumulated model using all voxelized point cloud data, the accumulated model is output as the reconstruction result of the target to be reconstructed.

[0106] In this embodiment, after completing the fusion update of the cumulative model using all voxelized point cloud data, the electronic device can output the cumulative model (i.e., the fusion updated cumulative model) as the reconstruction result of the target to be reconstructed. It should be noted that the cumulative model only outputs the verified part. For the points in the part to be verified that are still within the life cycle but have not been verified, the cumulative model will not output these points because they have not been verified and may be noise points.

[0107] The following will provide an experimental comparison of this scheme with other reconstruction schemes:

[0108] Here, we provide a benchmark of our approach including metrics and full results on the Replica dataset. We tested our reconstruction system and others on a laptop equipped with an 8-core CPU R9 7940HS (TDP 35W), 32GB RAM, and an NVIDIA RTX 4050GPU (optional visual odometry, TDP 115W). The code is implemented in C++ and compiled with GCC 11.4.0. For the learning baseline NICE-SLAM (comparison algorithm 3), we ran the reconstruction on a 24-core laptop. We ran the code on servers with GPUs Gold 5318Y (165W) and NVIDIA A40 (300W). In our experiments, even on the server, NICE-SLAM (compared with Algorithm 3) failed to complete the reconstruction of all replicated scenes and got stuck due to computational performance issues. In this case, we took the mapping time and reconstruction results from the original paper. We remind readers that the original NICE-SLAM was implemented on a desktop computer with a 3.80GHz Intel i7-10700K CPU (TDP 125W) and an NVIDIA RTX3090GPU (TDP 350W).

[0109] In all experiments, we set the minimum fusion time of ManifoldEmbed (the reconstruction algorithm of this scheme) to 3, which means that a 3D sample point needs to be verified at least three times, and the voxel resolution is scaled according to the average point radius. We set the voxel search radius to 2. This means that the point will search for nearby voxels within a step size of 2 voxels. At the same time, we set the target normal distribution of the filter to N(0,0.25). The optimization of α is performed for 10 iterations with a step size of 0.1.

[0110] For real-world scanning, Intel RealSense T265 tracking camera and L515 depth camera were used. The T265 camera was calibrated to align with the depth map. The RGB frames were aligned with the depth frames using the RealSense software development kit. The input to the backend fusion framework includes the refined pose, RGB frames, and depth map.

[0111] For other datasets, we use advanced odometry in FastCaMo-Real to obtain the camera pose (Microsoft Azure Kinect DK). The input to our backend fusion framework includes refined pose, RGB frames and depth maps. For the Replica dataset, we use visual odometry similar to superpoint to optimize the pose with minimum reprojection error. A GPU (RTX4050) can be optionally used for feature extraction and matching. We did not compare tracking times as this is not the focus of this paper. In real-world scans, tracking cameras can provide tracking with visual odometry even without a GPU. For learning-based feature extraction, such as superpoint, it can also be provided by the latest sensors such as OpenCV AIKit-OAK-D series.

[0112] The metrics on different datasets vary slightly. For targets on the real-world scans and replica datasets, we use metrics similar to the original paper. Accuracy (mm / cm) is defined as the average distance between a 3D sample point on the reconstructed mesh and the nearest ground truth point. Completeness is defined as the average distance between a 3D sample point on the ground truth mesh and the nearest reconstructed point. The completeness ratio represents the percentage of points in the reconstructed mesh that are below a threshold for completeness.

[0113] For FastCaMo-Real, we adopted the metrics described in the original paper. Specifically, we evaluated the reconstruction quality, including completeness and accuracy relative to the ground truth surface. Regarding accuracy, we only evaluated the RMS (root mean square) error in the overlapping (inliers) area between the reconstructed surface and the ground truth surface. To determine the threshold for inliers, we set it to 15cm.

[0114] In Table 1, we only give the average results of all scenes (iMAP is the comparison algorithm 1, and no reconstruction result comparison is provided; DI-Fushion is the comparison algorithm 2, and no reconstruction result comparison is provided; NICE-SLAM is the comparison algorithm 3, and reconstruction result comparison is provided, please refer to Figure 3-Figure 4 , including the reconstruction results comparison of 2 scenes under different frame point cloud data, namely 500 frames, 1000 frames, 1500 frames, and 2000 frames; Manifold Embed is the reconstruction algorithm of this solution). For the comparison of the true value experimental results, please refer to Figure 5-Figure 6In this section, we provide detailed results for each scenario (including the comparison results of the two comparison algorithms for which comparison graphs are not provided in this embodiment). This is not an additional experiment and will not affect our conclusions. The detailed results for each scenario are shown in Table 1. It can be seen that this solution has improved reconstruction accuracy. In addition, since the computational complexity of this solution is relatively low compared to algorithms that achieve similar reconstruction accuracy, it can complete reconstruction more efficiently.

[0115] Table 1

[0116]

[0117] In summary, the embodiment of the present application provides a three-dimensional point cloud fusion method based on manifold embedding, and designs a manifold embedding framework, which is a non-projection technology. Different from the prior art, the manifold embedding framework uses the correspondence relationship between dense points and surfaces, and is guided by implicit moving least squares surface functions rather than optimal transmission. These unique designs enable the present solution to achieve high reconstruction accuracy while maintaining low computational complexity. Moreover, these limitations are overcome by introducing a verification-update sampling method for manifold embedding without relying on any prior information. This allows robust 3D reconstruction even in the presence of sensor noise or changes in the imaging system.

[0118] We use an RGB-D prototype system as the input for reconstruction, and include other RGB-D systems in our experiments. The output of the front end is a multi-combined data frame, which consists of color image frames, depth maps, IMU data, and pose data. We use the hardware clock and visual calibration of the sensors to align the multi-combined data frames. We use these combined data frames to reconstruct indoor environments in scanning and reconstruction applications. Our algorithm mainly fuses color image frames and depth maps, while pose data is obtained through RealSense TM The tracking camera T265 or algorithms such as RoseFusion, or other RGB-D systems are provided by visual odometry and reprojection error optimization. The main goal of this scheme is to simulate 3D fusion, which involves searching and updating dense correspondences. In order to improve the fusion efficiency, this scheme uses a smooth representation of the local surface as a guide to achieve fast point-to-surface fusion. The fusion method given in this scheme is different from previous methods. After the neighbor relationship between points and voxels is established, an implicit moving least squares surface function with a verified updated sampling filter (that is, it needs to pass the initial verification and the second verification) is used for fusion guidance. In the fusion stage, this scheme abstracts the sampled data into a mathematical model and considers the 3D point set V generated by multiple combined data frameworks. kThe fusion correspondence between its neighboring point sets aims to achieve point-to-surface fusion under the guidance of the implicit moving least squares surface function given in this scheme. This method is different from determining the point-to-point fusion correspondence based on geometric constraints. Under the guidance of the implicit moving least squares surface function given in this scheme, this scheme completes smooth fusion and forms a smooth manifold after multiple fusions. Despite the dense matching, this surface fusion method is efficient and improves computational efficiency. Therefore, this scheme can overcome the technical bottleneck of dense correspondence and achieve faster 3D fusion.

[0119] In addition, this scheme decomposes the fusion process into a two-step filtering process to improve the reconstruction accuracy. The implicit moving least squares surface function method given in this scheme effectively solves the uncertainty interference, improves the point cloud quality, smoothes the noise points, and solves the multi-layer surface effect due to the uncertainty of the pose. Since the implicit moving least squares surface function relies on updating along the normal vector, the normal vector is crucial for fusion. Before starting the fusion, it is necessary to verify the 3D sampling points in the accumulated model according to the normal vector. The normal vector direction of the corresponding nearby 3D sampling points in the accumulated model should be consistent with the normal vector direction of the local surface to be fused. Verifying the normal vector can effectively avoid the degradation of reconstruction quality caused by incorrect fusion. Although multiple samplings of the same surface should have similar normal vector directions during sampling, a few samples that are severely affected by uncertainty can be effectively excluded by normal vector verification. Therefore, this scheme also proposes a two-step process of verification-update sampling to achieve fast and accurate 3D fusion through this two-step process of verification-update sampling (performing initial verification and secondary verification).

[0120] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0121] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A 3D point cloud fusion method based on manifold embedding, characterized in that: include: Obtain multi-frame point cloud data of the target to be reconstructed, and voxelize each frame of point cloud data; For each frame of voxelized point cloud data: establish the proximity relationship between the voxel and the corresponding point in the accumulation model through hash index, where, in the initial state, the first frame of point cloud data is used as the accumulation model; The voxels that have not established a neighboring relationship in the voxelized point cloud data of the current frame are added to the cumulative model, and the cumulative model is verified based on each voxel that has established a neighboring relationship in the voxelized point cloud data of the current frame, and the verified cumulative model is updated from point to surface using the voxels that have established a neighboring relationship; After completing the fusion update of the accumulated model using all voxelized point cloud data, the accumulated model is output as a reconstruction result of the target to be reconstructed; The accumulated model is verified based on each voxel with a neighboring relationship in the voxelized point cloud data of the current frame, and the verified accumulated model is updated from point to surface using the voxels with a neighboring relationship, including: For each voxel that establishes a neighboring relationship in the voxelized point cloud data of the current frame: Determine the three-dimensional sampling point set determined by the hash index in the cumulative model Among them, the three-dimensional sampling point set Each 3D sampling point in the cumulative model is a voxel V in the voxelized point cloud data of the current frame. k Neighboring three-dimensional sampling points; Using the voxel V in the voxelized point cloud data of the current frame k For 3D sampling points Conduct initial verification; If the three-dimensional sampling point The initial verification of the three-dimensional sampling points Conduct secondary verification; If the three-dimensional sampling point The initial verification failed, and the three-dimensional sampling points were determined. It does not participate in the update process of the accumulated model by the voxelized point cloud data of the current frame; If the three-dimensional sampling point The secondary verification is passed to determine the three-dimensional sampling points Participate in the process of updating the accumulated model with the voxelized point cloud data of the current frame; If the three-dimensional sampling point The secondary verification fails, and the three-dimensional sampling point is determined It does not participate in the update process of the accumulated model by the voxelized point cloud data of the current frame; For the three-dimensional sampling points participating in the updating process of the accumulation model, the three-dimensional sampling points are updated from point to surface based on the voxelized point cloud data of the current frame.

2. The three-dimensional point cloud fusion method based on manifold embedding according to claim 1, characterized in that: Using the voxel V in the voxelized point cloud data of the current frame k For 3D sampling points Perform initial verification, including: Determine the 3D sampling points Whether the conditions are met: 3D sampling points With voxel V k is the sampling of the same scanning surface, and the three-dimensional sampling points V k Support point, voxel V k is the 3D sampling point The supporting voxels of If the three-dimensional sampling point Satisfy this condition and determine the three-dimensional sampling points The initial verification passed.

3. The three-dimensional point cloud fusion method based on manifold embedding according to claim 2 is characterized in that: Determine the 3D sampling points Whether the conditions are met: 3D sampling points With voxel V k is the sampling of the same scanning surface, and the three-dimensional sampling points V k Support point, voxel V k is the 3D sampling point Supported voxels include: Determine the 3D sampling points The corresponding unit normal vector And, determine the voxel V k The mean unit normal vector set up Inner Product Representation and The cosine between vectors, the target distribution is calculated by the following first target distribution function: Among them, the first target distribution function is a normal distribution with a mean of 0 and a standard deviation of σ1; Determine the 3D sampling points The unit normal vector Whether it meets: Among them, μ1 is the value sampled from the uniform distribution U(0,1); If satisfied Determine 3D sampling points Satisfy the conditions: 3D sampling points With voxel V k is the sampling of the same scanning surface, and the three-dimensional sampling points V k Support point, voxel V k is the 3D sampling point The supporting voxels of Otherwise, determine the 3D sampling points Condition not met: 3D sampling point With voxel V k is the sampling of the same scanning surface, and the three-dimensional sampling points V k Support point, voxel V k is the 3D sampling point of supported voxels.

4. The three-dimensional point cloud fusion method based on manifold embedding according to claim 3 is characterized in that: For 3D sampling points Perform secondary verification, including: set up express Project to The directed projection distance on V k The target distribution is calculated by the following second target distribution function: Among them, the second target distribution function is a normal distribution with a mean of 0 and a standard deviation of σ2; Determine the 3D sampling points The unit normal vector Whether it meets: Among them, μ2 is the value sampled from the uniform distribution U(0,1); If satisfied Determine 3D sampling points Pass the secondary verification, otherwise, determine the three-dimensional sampling points Failed secondary verification.

5. The three-dimensional point cloud fusion method based on manifold embedding according to claim 4, characterized in that: Standard deviation σ1=0.5, standard deviation σ2=0.

5.

6. The three-dimensional point cloud fusion method based on manifold embedding according to claim 4, characterized in that: For the 3D sampling points participating in the updating process of the cumulative model, the 3D sampling points are updated from point to surface based on the voxelized point cloud data of the current frame, including: For each 3D sampling point involved in the update process of the cumulative model: use the implicit moving least squares surface function for fusion guidance, embed the 3D sampling points involved in the update process of the cumulative model into the smooth manifold established by the voxelized point cloud data of the current frame, and realize the point-to-surface fusion update.

7. The three-dimensional point cloud fusion method based on manifold embedding according to claim 6, characterized in that: The implicit moving least squares surface function is used for fusion guidance. The three-dimensional sampling points involved in the update process of the cumulative model are embedded into the smooth manifold established by the voxelized point cloud data of the current frame to achieve point-to-surface fusion update, including: For satisfaction The three-dimensional sampling points Determine the 3D sampling points Update increment in, For vector Projection to unit normal vector The directed projection distance on V k The center point of Based on update increment Combine the following formula to convert the three-dimensional sampling points Embedded into voxel V k In the corresponding smooth manifold, recursive optimization update is performed: Where F(x) is the implicit function of the moving least squares surface, whose null space Null(F) approximates the input surface S, which is the voxel V k The corresponding smooth manifold, p i ∈V k , V k ={p i ,i=1,2,…,n}, V k The center point, r k V k The radius of θ(|xp i |,r k ) is the weight function, n i For p i The unit normal vector of .

8. The three-dimensional point cloud fusion method based on manifold embedding according to claim 7, characterized in that: The weight function θ satisfies: θ(|x-p i |,r k )=exp(-|x-p i | 2 / r k 2 )。

Images