Fast upsampling method for large-scale point cloud based on implicit neural network and spatial hashing

Abstract

A large scene point cloud fast upsampling method based on implicit neural network and spatial hashing, comprising the following steps: step S1, large scene point cloud data set collection; step S2, point cloud implicit surface coding; step S3, spatial hash construction; step S4, adaptive spatial feature representation: using the spatial hash table, the spatial features around the query point are adaptively represented, according to the position and local geometric structure of the query point, the feature representation is dynamically adjusted, so as to better adapt to the point cloud density change of different regions; step S5, high-fidelity point cloud generation: according to the optimized query point position and adaptive spatial feature representation, new point cloud data is generated, the newly generated point cloud not only retains the geometric features of the original point cloud, but also has higher resolution and details. The present application can significantly improve the quality and efficiency of upsampling, and support real-time application.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of point cloud upsampling, and in particular to a large scene point cloud fast upsampling method based on implicit neural network and spatial hashing. BACKGROUND

[0002] With the rapid development of 3D scanning technology, LiDAR and depth camera sensors, point cloud data has been widely used in autonomous driving, virtual reality, augmented reality, medical imaging and other fields. However, due to the resolution limitations of acquisition devices, occlusion of view angle, motion blur and other factors, the actual acquired point cloud often has problems such as sparsity and irregular distribution, which seriously affects the accuracy and efficiency of subsequent 3D modeling, object recognition and scene understanding. In order to solve this problem, point cloud upsampling has become a key research direction, aiming to increase the number of points and optimize the distribution of points, improve the resolution and geometric details of the reconstructed model, and enhance the continuity and stability of the surface. Although traditional interpolation methods, optimization methods and deep learning-based methods have made some progress, they still face challenges such as low computational efficiency, dependence on a large number of parameter adjustments and high hardware resource requirements when dealing with large-scale, complex geometric structure data. Therefore, developing a point cloud upsampling method that is efficient, high-quality and suitable for large-scale real-world data is of great significance for promoting the development of 3D vision technology.

[0003] Although traditional existing point cloud upsampling methods have solved the problem of point cloud sparsity and irregular distribution to some extent, they still have significant limitations in practical applications. Traditional interpolation methods (such as radial basis function RBF, moving least squares MLS) are prone to overfitting or high computational complexity when dealing with complex geometric structures, resulting in low computational efficiency; optimization-based methods (such as Poisson reconstruction) can generate topologically correct surfaces, but have high computational cost, especially on large-scale data sets, efficiency becomes a bottleneck, and rely on normal information, while the normal estimation in actual point cloud may not be accurate, affecting the reconstruction effect. In addition, deep learning-based methods (such as DeepSDF, CONet) can learn complex geometric structures, but the training process is time-consuming and requires high hardware resources, and most of them focus on explicit surface reconstruction, and there is relatively less research on efficient representation and upsampling of implicit surfaces. These methods generally rely on a large number of parameter adjustments, increasing the difficulty of use and limiting their applicability in automated applications, making it difficult to effectively handle large-scale real-world data. SUMMARY

[0004] To overcome the shortcomings of existing technologies, this invention provides a fast upsampling method for large-scale point cloud data based on implicit neural networks and spatial hashing. It is suitable for the rapid processing of large-scale point cloud data, and can significantly improve the quality and efficiency of upsampling, especially when processing large-scale, real-world data, supporting real-time applications such as autonomous navigation and augmented reality.

[0005] The technical solution adopted by this invention to solve its technical problem is:

[0006] A fast upsampling method for large-scene point clouds based on implicit neural networks and spatial hashing includes the following steps:

[0007] Step S1: Large-scale scene point cloud dataset collection. The handheld device scans the real scene to obtain point cloud data, and then the obtained point cloud data is normalized.

[0008] Step S2, Implicit Surface Encoding of Point Cloud: The input sparse point cloud is converted into an implicit surface representation, and a deep learning model is used to encode the point cloud to capture its intrinsic geometric structure.

[0009] Step S3, Spatial Hash Construction: Based on the implicit surface representation of the point cloud, an efficient spatial hash table is built. This hash table is used to accelerate the fast lookup between a query point and its nearest neighbor.

[0010] Step S4, Adaptive Spatial Feature Representation: Using a spatial hash table, the spatial features around the query point are adaptively represented. The feature representation is dynamically adjusted according to the location and local geometry of the query point to better adapt to the changes in point cloud density in different regions.

[0011] Step S5: High-fidelity point cloud generation: Based on the optimized query point location and adaptive spatial feature representation, new point cloud data is generated. The newly generated point cloud not only retains the geometric features of the original point cloud, but also has higher resolution and detail.

[0012] Furthermore, the method also includes the following steps:

[0013] Step S6, Performance Evaluation and Optimization: Continuously optimize algorithm parameters and network structure to adapt to different application scenarios and datasets, ensuring optimal performance.

[0014] The main benefits of this invention are that it can significantly improve the quality and efficiency of upsampling and support real-time applications. Attached Figure Description

[0015] Figure 1 This is a flowchart of a fast upsampling method for large-scale point clouds based on implicit neural networks and spatial hashing. Detailed Implementation

[0016] The present invention will now be further described with reference to the accompanying drawings.

[0017] Reference Figure 1 A fast upsampling method for large-scene point clouds based on implicit neural networks and spatial hashing, the method includes the following steps:

[0018] Step S1: Large-scale scene point cloud dataset acquisition. The handheld device scans the real-world scene to obtain point cloud data, and then normalizes the obtained point cloud data. Given a point cloud... The hardware components used in this invention are as follows: Livox's latest generation 3D LiDAR Mid-360 and THUNDERROT-MIX host.

[0019] The following process is used to process it to achieve efficient and high-quality upsampling:

[0020] First, the original point cloud data is divided into a grid, dividing the point cloud space into a uniform small three-dimensional voxel grid; this allows for dense and uniform sampling of seed points in three-dimensional space, thereby obtaining corresponding dense and approximately uniform projection points.

[0021] Next, the original point cloud p is normalized, and the point cloud scale p is calculated. scale =max(X max -X min ,Y max -Y min Z max -Z min )and The normalized point cloud representation is as follows To deepen our understanding of point cloud structure, we will start from P nor A random sampling point is denoted as P. rs We use nearest neighbor search to calculate the distance between each point, denoted as P. true ;

[0022] Then, query the nearest neighbor distance d = P of each point to the 51 points around it. true ·query(P nor ,51), using random sampling and spatial indexing structures, the distances between points in the point cloud are calculated; this provides crucial data for subsequent processing; the more points there are, the more effective it is for subsequent field construction.

[0023] Step S2, implicit surface encoding of point clouds, is as follows:

[0024] Three methods were used for point cloud data mapping: Signed Distance Function (SDF), Single Resolution Feature Mesh, and Multi-Resolution Feature Mesh.

[0025] The SDF method represents geometry by learning the distance from each point to the nearest surface and improves point cloud density and quality by generating high-resolution point cloud data. Single-resolution feature meshes store feature vectors in a fixed-resolution 3D mesh, suitable for processing uniformly distributed data. Multi-resolution feature meshes capture features at different scales through multiple layers of meshes with varying resolutions, suitable for handling complex geometries and ensuring high-quality point cloud data both globally and locally. In the SDF method, the signed distance function SDF is used to represent scene geometry for a given 3D point P. rs This continuous function f returns the distance from the point to the nearest surface.

[0026]

[0027] Where p is a three-dimensional point and s is the corresponding SDF value;

[0028] Step S3: Spatial hash construction. Based on the implicit surface representation of the point cloud, an efficient spatial hash table is built. This hash table is used to accelerate the fast lookup between a query point and its nearest neighbors. It is achieved by dividing the three-dimensional space into several cells and assigning each point to a corresponding cell. The process is as follows:

[0029] Step S31, Spatial Hash Construction: The core idea of ​​spatial hashing is to divide a three-dimensional space into cubic cells of fixed size. Assume there is a three-dimensional space... Furthermore, define a cube cell with side length h. For any point p = (x, y, z), the cell it belongs to is calculated using the following formula.

[0030]

[0031] in, This indicates a round-down operation, where h is the side length of the cell, which is chosen based on the density and geometric complexity of the point cloud. A smaller h can improve query accuracy but increase memory consumption; a larger h can reduce memory usage but may lead to more hash collisions.

[0032] Step S32, Dynamically Adjust Cell Size: To adapt to point cloud regions with different densities, a mechanism for dynamically adjusting cell size can be introduced. The value of h is adaptively adjusted based on the local point cloud density. For each query point P, the number of points N within a certain radius (e.g., radius r) around it is calculated, and the cell size h is dynamically adjusted based on N using the following formula.

[0033]

[0034] Where V is the volume of the query region, and α is an adjustable parameter used to control the scaling ratio of the cell size;

[0035] Step 4: Adaptive spatial feature representation, the process is as follows:

[0036] Step 4.1, Progressive Optimization Strategy: To more accurately predict unsigned distance values ​​and learn more local details, this method proposes a progressive learning strategy, using the intermediate results of the moving query as additional priors. For the original point cloud with discrete surface representation, the closer the query position is to the given point cloud, the smaller the error in searching for the target point on the given point cloud. Based on this, two regions are established: a high-confidence region with small errors and a low-confidence region with large errors. Query points are sampled in the high-confidence region to help train the network, while auxiliary points are sampled in the low-confidence region. After the network converges in the current stage, it moves to the estimated surface position through the network gradient, and the moved auxiliary points are prioritized as the surface for the next stage.

[0037] Step 4.2, Hierarchical Representation Fusion of Feature Information: In previous SDF works, MLP (Multilayer Perceptron) was typically used to regress the distribution of object surfaces. MLP is a neural network architecture that can learn to map the spatial location of the input to values ​​related to the surface distance. However, traditional SDF methods mainly focus on generating an accurate representation of the object surface, and density information is usually not fully considered in this process. The method of this invention adds density information, making it related to spatial location. Considering density information not only provides information about the surface shape, but also provides the density distribution inside and outside the object, thus providing a more comprehensive object representation. In order to effectively demonstrate different implicit neural field methods, three different implicit neural field methods are designed: SDF, single-resolution feature grid, and multi-resolution feature grid.

[0038] In the SDF method, scene geometry is represented as a signed distance function (SDF), which is a continuous function f that, for a given 3D point, returns the distance from that point to the nearest surface.

[0039]

[0040] Where x is a 3D point and s is the corresponding SDF value, the SDF function is parameterized using a learnable parameter θ, and several different design choices to represent the function are investigated, including a dense grid with learnable SDF values, a hybrid MLP single-resolution or multi-resolution feature grid.

[0041] The most direct way to parameterize SDF is to store the SDF values ​​directly in the discrete volume. In each cell, the resolution is RH×RW×RD. To query the SDF value of any point x from the dense SDF mesh... Use interpolation operations

[0042]

[0043] The single-resolution feature grid method combines these two parameterization methods to use feature-conditional MLPf θ and resolution R 3 Feature mesh Φ θ Each grid cell stores a feature vector instead of directly storing SDF values;

[0044] Multi-resolution feature meshes use a single feature mesh Φ θ It can use a resolution of R l Multi-resolution feature mesh Sampling resolution in geometric space, combined with features of different frequencies.

[0045]

[0046] Among them, R min and R max These represent the coarsest and finest resolutions, respectively. As the total number of grid cells increases cubically, a fixed number of parameters are used to store the feature grid, and a spatial hash function is used to index the feature vectors at finer levels. Each grid contains a maximum of T feature vectors with dimension F. At the coarse level, the feature grid is densely stored; at a finer level, i.e. Use a hash function to index the corresponding feature vector.

[0047]

[0048] in, It is a bitwise XOR operation, π i It is the only large prime number; using the default values ​​Rmin=16, Rmax=2048, L=16, F=2, T=2 19 ,

[0049]

[0050] As the total number of grid cells increases cubically, a fixed number of parameters are used to store the feature grid, and a spatial hash function is used to index the feature vectors.

[0051] Step 5: High-fidelity point cloud generation: After optimizing the query point location and feature representation, new point cloud data is generated using methods such as upsampling to ensure that the new points are evenly distributed on the surface and to add more details; subsequently, post-processing steps such as smoothing and repairing holes are used to further improve the visual quality and integrity of the point cloud.

[0052] Step 6: The design of the loss function is crucial, as it not only determines the training direction of the model but also directly affects the quality and fidelity of the generated point cloud. This method introduces multiple loss functions to ensure that the model can effectively learn implicit surface representations and generate high-quality, high-resolution point clouds. The process is as follows:

[0053] Step 6.1, CD loss L1 and CD loss L2;

[0054] Point clouds A and B contain n and m points respectively. For each point z... i ∈A, find the point p that is closest to it. j ∈B, and calculate the distance between them. For each point p j ∈B, find the point z that is closest to it. i ∈A, and calculate the distance between them. ChamferL2 is defined as the average of these two sets of distances.

[0055]

[0056] Step 6.2, KL_loss and Geoloss;

[0057] Suppose we have two discrete probability distributions P and Q, corresponding to the distributions of two point clouds. In calculating the KL divergence between the two point clouds, the sample space can be viewed as a set of points, where P(i) represents the probability of a point in the first point cloud, and Q(i) represents the probability of the corresponding point in the second point cloud. KL The divergence loss measures the difference between two distributions; a larger value indicates a greater difference, and a smaller value indicates a smaller difference. The Sinkhorn distance is used as the loss function to calculate the distance between two point clouds.

[0058]

[0059] D s (P,Q)=min γ∈U(a,b) <γ,C>;

[0060] D s (P,Q) represents the Sinkhorn distance, γ is a joint probability distribution that has edge distributions of point clouds P and Q, U(a,b) is the set of joint probability distributions with edge distributions of point clouds P and Q, and C is the cost matrix that measures the loss from a point cloud in point cloud P to a point cloud in point cloud Q.

[0061] The embodiments described in this specification are merely examples of implementations of the inventive concept and are for illustrative purposes only. The scope of protection of this invention should not be considered limited to the specific forms described in these embodiments; rather, it extends to equivalent technical means conceived by those skilled in the art based on the inventive concept.

Claims

1. A fast upsampling method for large-scene point clouds based on implicit neural networks and spatial hashing, characterized in that, The method includes the following steps: Step S1: Large-scale scene point cloud dataset collection. The handheld device scans the real scene to obtain point cloud data, and then the obtained point cloud data is normalized. Step S2, Implicit Surface Encoding of Point Cloud: The input sparse point cloud is converted into an implicit surface representation, and a deep learning model is used to encode the point cloud to capture its intrinsic geometric structure. Step S3, Spatial Hash Construction: Based on the implicit surface representation of the point cloud, an efficient spatial hash table is built. This hash table is used to accelerate the fast lookup between a query point and its nearest neighbor. Step S4, Adaptive Spatial Feature Representation: Using a spatial hash table, the spatial features around the query point are adaptively represented. The feature representation is dynamically adjusted according to the location and local geometry of the query point to better adapt to the changes in point cloud density in different regions. Step S5: High-fidelity point cloud generation: Based on the optimized query point location and adaptive spatial feature representation, new point cloud data is generated. The newly generated point cloud not only retains the geometric features of the original point cloud, but also has higher resolution and detail. The process of step S3 is as follows: Step S31, Spatial Hash Construction: The core idea of ​​spatial hashing is to divide a three-dimensional space into cubic cells of fixed size. Assume there is a three-dimensional space... And define a side length of A cube cell, for any point The cell containing it is calculated using the following formula. ； in, This indicates a round-down operation. It is the side length of the cell, which is selected based on the density and geometric complexity of the point cloud; Step S32, Dynamically Adjust Cell Size: To adapt to point cloud regions with different densities, a mechanism for dynamically adjusting cell size is introduced, adaptively adjusting the size based on the local point cloud density. The value for each query point Calculate the number of points within a defined range around it. and according to Dynamically adjust cell size Use the following formula, ； in, It is the volume of the query region. It is an adjustable parameter used to control the scaling ratio of cell size; The process of step S4 is as follows: Step 4.1 Stepwise optimization strategy: Two regions were established: a high-confidence region with small error and a low-confidence region with large error. In the high-confidence region, query points were sampled to help train the network, and in the low-confidence region, auxiliary points were sampled. After the network converged in the current stage, it was moved to the estimated surface position through the network gradient. The moved auxiliary points were used as the surface prior for the next stage. Step 4.2: Feature information hierarchical representation fusion.

2. The method for fast upsampling of large-scene point clouds based on implicit neural networks and spatial hashing as described in claim 1, characterized in that, The method further includes the following steps: Step S6, Performance Evaluation and Optimization: Continuously optimize algorithm parameters and network structure to adapt to different application scenarios and datasets, ensuring optimal performance.

3. The method for fast upsampling of large-scene point clouds based on implicit neural networks and spatial hashing as described in claim 1 or 2, characterized in that, The process of step S1 is as follows: First, the original point cloud data is divided into a grid, dividing the point cloud space into a uniform small three-dimensional voxel grid; this allows for dense and uniform sampling of seed points in three-dimensional space, thereby obtaining corresponding dense and approximately uniform projection points. Next, the original point cloud p is normalized, and the point cloud ratio is calculated. and The normalized point cloud is represented as To deepen the understanding of point cloud structure, from Random sampling points, denoted as The nearest neighbor search is used to calculate the distance between each point, denoted as . ; Then, query the nearest neighbor distances of the 51 points surrounding each point. Using random sampling and a spatial indexing structure, the distances between points in the point cloud are calculated.

4. The method for fast upsampling of large-scene point clouds based on implicit neural networks and spatial hashing as described in claim 1 or 2, characterized in that, The process of step S2 is as follows: Point cloud data mapping adopts three methods: Symbolic Distance Field (SDF), single-resolution feature grid, and multi-resolution feature grid; the SDF method represents the geometry by learning the distance from each point to the nearest surface and improves the density and quality of point cloud by generating high-resolution point cloud data; the single-resolution feature grid stores feature vectors in a three-dimensional grid with a fixed resolution, which is suitable for processing uniformly distributed data; the multi-resolution feature grid captures features at different scales through multiple layers of grids with different resolutions, which is suitable for processing complex geometries and ensures that high-quality point cloud data can be obtained both globally and locally.

5. The method for fast upsampling of large-scene point clouds based on implicit neural networks and spatial hashing as described in claim 2, characterized in that, In step 6, by introducing multiple loss functions, it is ensured that the model can effectively learn implicit surface representations and generate high-quality, high-resolution point clouds. The process is as follows: Step 6.1, CD loss L1 and CD loss L2; Point clouds A and B contain n and m points respectively. For each point... Find the point closest to it. And calculate the distance between them. For each point Find the point closest to it. And calculate the distance between them. ChamferL2 is defined as the average of these two distances, denoted in the formula as... , ； ； Step 6.2, KL_loss and Geoloss; Suppose we have two discrete probability distributions P and Q, corresponding to the distributions of two point clouds. In calculating the KL divergence between the two point clouds, we can consider the sample space as a set of points, where P(i) represents the probability of a point in the first point cloud, and Q(i) represents the probability of the corresponding point in the second point cloud. The divergence loss measures the difference between two distributions; a larger value indicates a greater difference, and a smaller value indicates a smaller difference. The Sinkhorn distance is used as the loss function to calculate the distance between two point clouds. ； ； Indicates the Sinkhorn distance. It is a joint probability distribution that satisfies the marginal distributions of point clouds P and Q. C is the set of joint probability distributions of point clouds P and Q with edge distributions, and C is the cost matrix, which measures the loss from a point cloud in point cloud P to a point cloud in point cloud Q.

Images