Point cloud restoration method based on microsampling

Abstract

The invention relates to a point cloud restoration method based on microsampling, and belongs to the technical field of point cloud reconstruction. According to the method, the advantages of the FPS and the differentiable down-sampling method are combined, so that down-sampling can have continuous differentiability required for gradient optimization and inherent comprehensive information in the FPS. The method disclosed by the invention shows extremely high effectiveness and generalization ability for various data, and obtains lower reconstruction error, chamfering distance (CD), Hausdorff distance (HD) and bulldozer distance (EMD) in various test sets compared with the existing method.

Description

Technical Field

[0001] The present invention belongs to the technical field of point cloud reconstruction, and in particular relates to a point cloud restoration method based on differentiable sampling. Background Art

[0002] Point cloud, as a 3D data representation with precise geographic coordinates and rich semantic information, plays a vital role in many fields. Point cloud has a wide range of applications, including but not limited to virtual reality, autonomous driving systems, medical modeling, and building maintenance. In practical applications, point cloud data either needs to be processed immediately or stored in large storage devices for subsequent analysis. When the cost of data processing and storage exceeds the cost of transmission, point cloud reconstruction technology becomes particularly important for economic benefits in practical applications. Especially in those scenarios that are limited by time or span large distances, point cloud reconstruction becomes the most feasible method.

[0003] In recent years, people have gradually begun to use deep learning methods to reconstruct point clouds, especially for optimizing the surface of 3D objects. Generally, methods for processing point clouds can be roughly divided into two categories: downsampling and upsampling. The role of downsampling technology is to extract a simplified version of the point cloud from the original point cloud to reduce the amount of data for easy transmission and storage; while the upsampling method is committed to refining the point cloud to restore more detailed structural details for the convenience of using point cloud data. Taking a specific technology as an example, PU-GCN [1] The shuffle unit is used to process the local features of the output of the graph convolution layer to generate a more representative point cloud; while the Dis-PU [2] A decoupled network was developed to focus on improving the optimization effects of local and global features.

[0004] Since the advent of point cloud upsampling methods for deep learning, these methods can be divided into three main types: shuffle-based, superposition-based, and implicit function-based methods. Among them, the high-order features of the shuffle-based point cloud upsampling method are converted into corresponding Cartesian coordinates using a specific upsampling rate. The direct shuffle-based upsampling may lead to phenomena beyond the surface of the object, so additional constraints need to be added, such as repulsion loss, uniformity loss, or other model-specific losses.

[0005] Traditional downsampling techniques, including the widely used Farthest Point Sampling (FPS), are the basic downsampling techniques in various point cloud processing methods. However, its limitation is that as a non-differentiable process, it only focuses on geometric parameters in low-dimensional space and lacks the continuous differentiability required for gradient optimization, which leads to insufficient optimization of sampling in downstream tasks. In order to learn these parameters, SampleNet [3] and PointSE [4]It is proposed to shuffle the high-order features to fit into the Euclidean space. Although this shuffling technique is reasonable, it fails to fully utilize the comprehensive information inherent in FPS.

[0006] [1] G. Qian, A. Abualshour, G. Li, A. Thabet, and B. Ghanem, "Pugcn: Point cloudupsampling using graph convolutional networks," in Proceedings of the IEEE / CVFConference on ComputerVision and Pattern Recognition, 2021, pp.11683–11692.

[0007] [2] R.Li,

[0008] [3]Lang, A.Manor, and S.Avidan, "Samplenet: Differentiable point cloudsampling," in Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition, 2020, pp.7578–7588.

[0009] [4] R. Li, X. Li, T.-T. Wong, and C.-W. Fu, "Point set self-embedding," IEEE Transactions on Visualization and Computer Graphics, 2022. Summary of the invention

[0010] 1. Technical issues to be resolved

[0011] The technical problem to be solved by the present invention is to design a point cloud restoration method to achieve downsampling that has the continuous differentiability required for gradient optimization and has the comprehensive information inherent in FPS.

[0012] (II) Technical solution

[0013] In order to solve the above technical problems, the present invention provides a point cloud restoration method based on differentiable sampling, comprising the following steps:

[0014] Step 1, obtaining input data based on one of two benchmark datasets, namely, the synthetic grid dataset ModelNet40 and the real-world dataset ScanObjectNN, and dividing the input data into training data and test data;

[0015] Step 2: The designed end-to-end model includes a downsampling module and an upsampling module. The downsampling module uses a differentiable sampling algorithm to process data, and the upsampling module further refines the differentiable point set output by the downsampling module, using a hybrid SPD method to generate dense points and upsample the differentiable point set to the original resolution.

[0016] Step 3: Train and optimize the entire end-to-end model based on the designed loss function and training data;

[0017] Step 4: Based on the trained end-to-end model, test the test data in the input data, downsample and upsample the input point cloud data to restore it, so as to obtain a reconstruction error lower than the preset value.

[0018] The present invention also provides a system for implementing the method.

[0019] (III) Beneficial effects

[0020] The present invention aims to combine the advantages of FPS and differentiable downsampling methods to achieve downsampling that has the continuous differentiability required for gradient optimization and the comprehensive information inherent in FPS. The present invention proposes a seed mixer for replacing the traditional rough seed generator with differentiable sampling, so that the comprehensive information inherent in FPS can be utilized and the point-to-point correspondence between the manifold and the non-differentiable sampling points can be learned, which enables the points to be repositioned more effectively to more comprehensively capture the overall morphology of the object. The present invention shows that the transfer mapping between adjacent submanifolds in the topological space is differentiable, and this differentiability extends to the corresponding local regions in the Euclidean space, making them homeomorphic. This analysis provides a theoretical basis for the effectiveness of the method proposed in the present invention.

[0021] Conventional FPS strategies have some difficulty in covering complex shapes. The present invention combines the advantages of FPS and differentiable downsampling methods to achieve downsampling with continuous differentiability required for gradient optimization and comprehensive information inherent in FPS. The method of the present invention shows strong effectiveness and generalization ability for various data, and has achieved lower reconstruction error, chamfer distance (CD), Hausdorff distance (HD) and earth mover distance (EMD) than existing methods in various test sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is an architecture diagram of the point cloud restoration method based on differentiable sampling of the present invention;

[0023] Figure 2 Experimental results comparing the reconstruction errors of the model designed for the present invention and other models for the data set;

[0024] Figure 3 Experimental results of the reconstruction visualization of the data set by the model designed for the present invention and other models. DETAILED DESCRIPTION

[0025] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0026] The present invention provides a point cloud restoration method based on differentiable sampling, which consists of two stages: downsampling and upsampling. In the downsampling stage, the present invention proposes a differentiable strategy as a point sampling algorithm, which is implemented through a feature extraction module and a seed mixer to replace the traditional seed generator. The present invention names this method DiffSample. In the upsampling stage, the present invention draws on SnowflakeNet [5] In Point Splitting Deconvolution (SPD), a hybrid SPD is proposed to generate dense points.

[0027] In order to improve the efficiency of point cloud data transmission and ensure that the output of the model retains more geometric features of the model input, the differentiable sampling algorithm in the downsampling stage proposed in this paper uses a learnable encoding method to make the downsampled point cloud retain as many features of the input data as possible: given a point set The present invention aims to learn a mapping function f that can transform p i ∈P is continuously transformed to represent the manifold M and subsets The shape of the encoding c is then reconstructed into output

[0028] The point cloud restoration method based on differentiable sampling provided by the present invention is as follows: Figure 1 As shown, the following steps are included:

[0029] Step 1: Get input data based on one of two common benchmark datasets, namely the synthetic grid dataset ModelNet40 [6] and the real-world dataset ScanObjectNN [7] , and divide the input data into training data and test data.

[0030] Furthermore, for step 1, the input data used in the present invention is one of the following three sets: (1) Uniform set: 2048 points are uniformly sampled from ModelNet40; (2) Random set: 2048 points are randomly sampled from ModelNet40; (3) Partial set: 2048 points in ScanObjectNN are directly used.

[0031] Step 2, the designed end-to-end model includes a downsampling module and an upsampling module: the downsampling module uses DiffSample to process data, and the upsampling module further refines the output of the downsampling model in detail. The present invention uses a hybrid point splitting deconvolution module to generate dense points based on the hybrid SPD method, and upsamples the differentiable point set to the original resolution;

[0032] Further, for step 2, specifically, in order to deeply understand the correspondence between the sampling point sets, the point set of the input data is input into the differentiable sampling module of the downsampling module to execute the differentiable sampling algorithm to predict the tiny displacement of each point, such as Figure 1 As shown in the architecture in , the differentiable sampling module consists of a feature extraction module and a seed mixer. In order to construct a manifold with boundaries, the feature extraction module uses the farthest point sampling (FPS) algorithm and the k-nearest neighbor (kNN) algorithm to sample an initial point set consisting of non-differentiable sampling points, and forms a one-dimensional vector shape encoding c, which is a global implicit code. At the same time, the seed mixer combines the shape encoding c with the non-differentiable sampling points generated by the feature extraction module, and proposes a Transformer, which takes the correspondence between local points and the global encoding, the point-by-point relationship in the neighborhood of non-differentiable sampling points, and the shape encoding itself as input to establish the point-to-point correspondence in the non-differentiable sampling point set. The local points are formed by searching the neighborhood based on the non-differentiable sampling points. The manifold in the topological space provides global guidance for the local area composed of the point set sampled by the FPS algorithm. Then, by fine-tuning the predicted subtle displacement from the implicit representation related to the shape encoding that integrates point-by-point and neighborhood information, the points can be moved to the position that best restores the original shape.

[0033] In the seed mixer of the present invention, represents the deconvolution feature in the manifold M, where PS(·) is the point splitting operation, that is, the one-dimensional deconvolution operation of the one-dimensional deconvolution layer in the seed mixer, then the shape encoding S that combines point-by-point and neighborhood information can be expressed as:

[0034]

[0035] in is the superimposed c to accommodate The dimension represents the global shape encoding attached to the point features.

[0036] Next, based on S, the present invention uses Transformer to learn the weights between sampling points with shape information and generate an attention vector:

[0037]

[0038] Among them, N(x i ) is point x i Neighbor, N Pl is the number of sampling points, δ = x i -x j , D is the dimension of the feature, and H is the point-by-point attention vector. ψ and η are feature transformations, γ is a mapping function, namely the multi-layer perceptron MLP, which is used to generate the attention vector H; ρ is softmax; ⊙ is the Einstein summation symbol.

[0039] The purpose of S can be intuitively understood as a force on the manifold, pushing the sampled points to a specific position to cover a more original shape, ultimately achieving the differentiability of the points previously sampled using the FPS algorithm.

[0040] Finally, the present invention inputs these point-by-point attention vectors H into another MLP to generate the displacement ΔP l , expressed as

[0041] ΔP l =tanh(MLP(H+S))

[0042] Then the displacement ΔP l Add back the non-differentiated sampled point P l Therefore, the differential sampling point P' l It can be expressed as:

[0043] P′ l =P l +ΔP l

[0044] Here, due to the addition of learnable displacement, P' l Can be in P lThe model is then modified to make slight changes to the original shape to get closer to the geometric details of the original shape.

[0045] Furthermore, for step 2, the upsampling module processes the sampled data P' l To restore. Specifically, the present invention draws on the point splitting deconvolution (SPD) in SnowflakeNet and proposes a hybrid SPD to generate dense points. Initially, there are three hybrid SPDs, which can use one-dimensional deconvolution to perform point splitting instead of traditional shuffling or superposition operations, ensuring that the generated points are smooth along their local graph structure. The present invention uses the differentiable sampling points generated by the seed mixer to strengthen the memory of shape features. These sampling points achieve differentiability of both input point cloud and shape encoding under the dual effects of both. This property enables the gradient to propagate smoothly from input to output. Therefore, the present invention only uses differentiable sampling points for splitting to avoid interfering with gradient-based optimization. The first hybrid SPD acts as a seed stabilizer and does not perform any upsampling operations. The subsequent two hybrid SPDs gradually restore the shape to the original resolution. In each hybrid SPD, the displacement of the point is predicted to assist the point in moving to the target position.

[0046] Step 3: Train and optimize the entire end-to-end model based on the designed loss function and training data.

[0047] Furthermore, for step 3, in order to effectively and efficiently learn geometric shapes, the present invention uses a relatively simple loss function. The reconstruction loss function is as follows:

[0048]

[0049] Where P and Q are the input point set of the downsampling stage and the output point set of the upsampling stage, respectively. Represents the Euclidean L2 norm of the chamfer distance (CD), and the reconstruction loss function:

[0050]

[0051] where Δp i is the predicted point-by-point displacement, Δq i is the output of the seed stabilizer, N l is the target number of sampling points, p i ∈P and q i ∈Q. To ensure that the Δp of the sampling point i and Δq at the recovery point i The displacements are similar.

[0052] Finally, the total loss function used for optimization is as follows:

[0053]

[0054] in Q l ∈{P' l ,Q 1 ,Q 2 ,Q 3}, Q 1 ,Q 2 ,Q 3 Output point cloud for each deconvolution layer.

[0055] In step 4, based on the trained end-to-end model, the test data in the input data is tested, and the input point cloud data is downsampled and upsampled to restore, so as to obtain lower reconstruction errors: chamfer distance (CD), Hausdorff distance (HD) and earth mover distance (EMD).

[0056] [5] P. Xiang,

[0057] [6] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao, "3d shapenets: A deep representation for volumetric shapes," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 1912–1920.

[0058] [7]MAUy, Q.-H.Pham, B.-S.Hua, T.Nguyen, and S.-K.Yeung, "Revisitingpoint cloud classification: A new benchmark dataset and classification model on real-world data," in Proceedings of the IEEE / CVF international conference oncomputervision, 2019, pp.1588–1597.

[0059] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A point cloud restoration method based on differentiable sampling, characterized in that: The following steps are involved: Step 1, obtaining input data based on one of two benchmark datasets, namely, the synthetic grid dataset ModelNet40 and the real-world dataset ScanObjectNN, and dividing the input data into training data and test data; Step 2: The designed end-to-end model includes a downsampling module and an upsampling module. The downsampling module uses a differentiable sampling algorithm to process data, and the upsampling module further refines the differentiable point set output by the downsampling module, using a hybrid SPD method to generate dense points and upsample the differentiable point set to the original resolution. Step 3: Train and optimize the entire end-to-end model based on the designed loss function and training data; Step 4: Based on the trained end-to-end model, test the test data in the input data, downsample and upsample the input point cloud data to restore it, so as to obtain a reconstruction error lower than the preset value.

2. The method according to claim 1, characterized in that The input data used in step 1 is one of the following three sets: (1) uniform set: 2048 points are uniformly sampled from ModelNet40; (2) random set: 2048 points are randomly sampled from ModelNet40; (3) incomplete set: 2048 points in ScanObjectNN are directly used.

3. The method according to claim 1, characterized in that In step 2, the point set of the input data is input into the differentiable sampling module of the downsampling module to predict point-by-point tiny displacements. The differentiable sampling module consists of a feature extraction module and a seed mixer. The feature extraction module uses the farthest point sampling FPS algorithm and the k-nearest neighbor kNN algorithm to sample an initial point set consisting of non-differentiable sampling points, and forms a one-dimensional vector shape code c, which is a global implicit code. The seed mixer combines the shape code c with the non-differentiable sampling points generated by the feature extraction module, and proposes a Transformer, which takes the correspondence between local points and global codes, the point-by-point relationship in the neighborhood of non-differentiable sampling points, and the shape code itself as input to establish a point-to-point correspondence in the non-differentiable sampling point set. The local points are formed by searching the neighborhood on the basis of the non-differentiable sampling points. The manifold in the topological space provides global guidance for the local area composed of the point set sampled by the FPS algorithm. Then, by fine-tuning the prediction of subtle displacements from the implicit representation related to the shape code that integrates point-by-point and neighborhood information, the point can be moved to the position that can best restore the original shape.

4. The method according to claim 3, characterized in that The operations performed by the seed mixer are as follows: make represents the deconvolution feature in the manifold M, where PS(·) is the point splitting operation, that is, the one-dimensional deconvolution operation of the one-dimensional deconvolution layer in the seed mixer. Then the shape encoding S that combines point-by-point and neighborhood information is expressed as: in, is the superimposed c to accommodate The dimension represents the global shape encoding attached to the point features; Next, based on S, Transformer is used to learn the weights between sampling points with shape information and generate attention vectors: Among them, N(x i ) is point x i Neighbor, N Pl is the number of sampling points, δ = x i -x j , D is the dimension of the feature, H is the point-by-point attention vector, ψ and η are feature transformations, γ is a mapping function, i.e., a multi-layer perceptron MLP, used to generate the attention vector H; ρ is softmax; ⊙ is the Einstein summation symbol; These point-wise attention vectors H are then fed into another MLP to generate the displacement ΔP l , expressed as: ΔP l =tanh(MLP(H+S)) Finally, the displacement ΔP l Add back the non-differentiated sampled point P l Therefore, the differential sampling point P' l It is expressed as: P′ l =P l +ΔP l , the differentiable point set is P' l composition.

5. The method according to claim 1, characterized in that In step 2, the up-sampling module outputs the data of the down-sampling module, that is, the sampled data P' l For restoration, we draw on the point splitting-based deconvolution SPD in SnowflakeNet and propose a hybrid SPD to generate dense points. There are three hybrid SPDs, which can use one-dimensional deconvolution to perform point splitting. The first hybrid SPD acts as a seed stabilizer and does not perform any upsampling operation. The subsequent two hybrid SPDs gradually restore the shape to the original resolution. In each hybrid SPD, the displacement of the point is predicted to assist the point in moving to the target position.

6. The method according to claim 1, characterized in that In step 3, the reconstruction loss function is designed as follows: Among them, P and Q are the input point set of the downsampling stage and the output point set of the upsampling stage respectively. Represents the Euclidean L2 norm of the chamfer distance CD; The reconstruction loss function is also designed as follows: Among them, Δp i is the predicted point-by-point displacement, Δq i is the output of the seed stabilizer, N l is the target number of sampling points, p i ∈P and q i ∈Q; Finally, the total loss function for optimization is as follows:

7. The method according to claim 1, characterized in that The reconstruction errors include: chamfer distance CD error, Hausdorff distance HD error and bulldozer distance EMD error.

8. The method according to claim 4, characterized in that S is generated to form a force on the manifold, pushing the sampled points to a specific position to cover a more original shape, and finally achieving the differentiability of the points sampled by the FPS algorithm.

9. The method according to claim 6, characterized in that To ensure the Δp of the sampling point i and Δq at the recovery point i The displacements are similar.

10. A system for implementing the method according to any one of claims 1 to 9.

Images