Arbitrary multiplying power point cloud up-sampling method based on weight point interpolation

Abstract

The invention provides an arbitrary multiplying power point cloud up-sampling method based on weight point interpolation, which comprises the following steps: acquiring an original point cloud, and interpolating the original point cloud by using a weight point interpolation generator to obtain an interpolated point cloud; and refining the interpolation point cloud by using a distance refiner. In a weight point interpolation stage, calculation of the position of an interpolation point is fully considered, a deep learning method is considered to optimize the distribution weight of the position of the interpolation point, the point number and the uniformity of the interpolated point cloud are fully considered, and a farthest point sampling method is utilized to sample the interpolated point cloud; in the feature extraction stage, local features and global features of point clouds are fully considered, and better feature extraction is performed on the point clouds in combination with position information and an attention mechanism of the point clouds.

Description

Technical Field

[0001] The present invention relates to the field of upsampling technology, and in particular to an arbitrary-ratio point cloud upsampling method based on weighted point interpolation. Background Art

[0002] In view of the disorder of point cloud data, most researchers in the early days converted point cloud data into a collection of regular 3D voxel grids or images before processing them, but this would lead to problems such as large data volumes. In 2017, the PointNet model was proposed, which took advantage of the permutation invariance of the input points and directly used the disordered raw point cloud data as the input of the network architecture, providing a new idea for the processing of point cloud data and making the point cloud upsampling method based on deep learning gradually enter the public's field of vision.

[0003] Although there are different methods for upsampling point clouds based on deep learning, most models can be considered to consist of the following three components: feature extraction component, feature expansion (also called upsampling) component, and coordinate reconstruction component. In 2018, researchers proposed the first point cloud upsampling network PU-Net based on deep learning, which proposed the framework of "feature extraction-feature expansion-coordinate reconstruction", providing important ideas for the development of point cloud upsampling network architecture. With the deepening of research, there are also some methods based on the framework of "feature extraction-feature expansion-coordinate reconstruction", using methods such as cascade networks, generative adversarial networks, arbitrary rate upsampling, self-supervised training, etc. to improve the network architecture to obtain better upsampling effects.

[0004] However, the above prior art still has defects, as follows:

[0005] 1. The problem of upsampling efficiency. Among the current point cloud upsampling methods based on deep learning, most methods regard upsampling of different scales as independent tasks, that is, multiple networks need to be trained to achieve upsampling of different scales, which not only lacks flexibility but also increases the training cost.

[0006] 2. Problems with the feature extraction module. In the point cloud upsampling framework, the learning ability of the feature extraction module is critical, which affects the quality of the generated point cloud. Existing methods are mainly based on MLP and CNN, but they may not be able to fully utilize structural information or capture complex features. This causes the upsampled point cloud to contain noise, affecting shape regularity and local details. In addition, existing methods mainly focus on generating uniform point clouds, while ignoring the problem of local noise points, especially in complex geometric areas, which will seriously affect the quality of the point cloud. Summary of the invention

[0007] According to the technical problems raised above, a method for upsampling point clouds with arbitrary magnifications based on weighted key interpolation is provided. In the weighted key interpolation stage, the present invention fully considers the calculation of the interpolation point positions, and considers using deep learning methods to optimize the weights of the interpolation point position assignments, and fully considers the number of points and uniformity of the point cloud after interpolation, and uses the farthest point sampling method to sample the interpolation point cloud; in the feature extraction stage, the local and global features of the point cloud are fully considered, and the point cloud is better extracted by combining the position information of the point cloud with the attention mechanism.

[0008] The technical means adopted by the present invention are as follows:

[0009] An arbitrary-ratio point cloud upsampling method based on weighted point interpolation, comprising:

[0010] S1, obtain the original point cloud, and use the weighted point interpolation generator to interpolate the original point cloud to obtain the interpolated point cloud;

[0011] S2. Use the distance refiner to refine the interpolated point cloud.

[0012] Furthermore, step S1 specifically includes:

[0013] S11. Given an input point cloud P with N points, use KNN to find the K neighbors of each point and replicate P K times.

[0014] S12, initialize the weight w, and assign weights to neighbor points and copy points to find the position of the interpolation point;

[0015] S13. In order to remove duplicate interpolation points and control the number of points to rN, after the interpolation is completed, the interpolation point cloud is downsampled using the farthest point sampling method to finally obtain the interpolation point cloud Q′.

[0016] Furthermore, step S1 also includes a step of using deep learning to find a suitable weight w, as follows:

[0017] Use the chamfer distance loss function L CD To constrain the interpolated point cloud, make the interpolated point cloud closer to the ground truth, and then find the appropriate weight value w.

[0018] Further, step S2 specifically includes:

[0019] S21, use MLP to perform feature encoding on the interpolated point cloud;

[0020] S22, input the encoded features into the multi-level position feature fusion attention module to obtain multi-level local features;

[0021] S23, performing a maximum pooling operation on the local features to obtain global features;

[0022] S24. Input the multi-level local features, global features, and query points into the multi-level feature aggregation module, use MLP to predict the point-to-point distance, and refine the point-by-point position of the interpolated point cloud to generate a more refined point cloud Q.

[0023] Furthermore, in step S22, the multi-level position feature fusion attention module encodes the input point cloud to obtain a multi-scale point-by-point feature and Specifically include:

[0024] S221, given an input point cloud Q′ having N points, local features and global features of the input point cloud Q′ are obtained using MLP and global maximum pooling;

[0025] S222, inputting the global features and the local features into a global-local feature fusion module to obtain fused features;

[0026] S223, input the fusion feature into the multi-scale position feature fusion attention module, and obtain the multi-scale point-by-point feature F after being processed by the 6-layer position feature fusion attention module. init 、F L1 and F L2 .

[0027] Furthermore, in step S223, the position feature fusion attention module fuses the point position information with the feature information to generate a fusion feature containing rich representation information, and captures the dependency between the fusion features, thereby generating a richer feature representation containing contextual information, specifically including:

[0028] S2231, applying KNN to the input point cloud Q′ and feature F respectively to obtain the nearest neighbors of the two;

[0029] S2232, subtract the input point cloud Q′ from the nearest neighbor to obtain the relative position, then apply MLP to the relative feature and the feature F after constructing the neighbor information to align them to the same dimension and then multiply them to obtain a fusion feature containing rich representation information;

[0030] S2233. Send the fused features to the multi-feature aggregation module to learn the dependencies within and between features, thereby obtaining attention features containing contextual information.

[0031] Furthermore, in step S24, the multi-level feature aggregation module performs weighted interpolation on the multi-scale local features of the original point cloud through the relationship between the query point and the original point, and splices it with all the interpolation features and global features to obtain a comprehensive feature representation that integrates local and global context information, specifically including:

[0032] S241, using KNN to find the nearest neighbor point of each query point, and calculating the interpolation weight according to the distance between the query point and the nearest neighbor point;

[0033] S242, performing weighted calculation, i.e., interpolation operation, on the features of each scale;

[0034] S243. Concatenate the multi-scale interpolation features with the global features to obtain a comprehensive feature representation.

[0035] Compared with the prior art, the present invention has the following advantages:

[0036] 1. Most methods regard upsampling of different scales as independent tasks, that is, multiple networks need to be trained to achieve upsampling of different scales, which not only lacks flexibility but also increases training costs. To address this problem, the present invention provides an arbitrary-ratio point cloud upsampling method based on weighted key interpolation, which can use one network model to perform upsampling of different scales.

[0037] 2. In the point cloud upsampling task, due to the limited learning ability of the feature extraction module, the upsampled point cloud still contains some noise in local areas such as edges and corners. This paper designs a new attention-based feature extraction module to enhance the model's ability to extract features and provide high-quality point cloud features for subsequent modules.

[0038] 3. The present invention also designs a refiner to fine-tune the point position of the upsampled point cloud point by point, thereby generating a uniform, high-quality upsampled point cloud, which can show good robustness for real scenes and can provide better original data for downstream tasks such as point cloud classification and semantic segmentation.

[0039] Based on the above reasons, the present invention can be widely promoted in the fields of upsampling and the like. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0041] Figure 1 The figure is a flowchart of the method of the present invention. DETAILED DESCRIPTION

[0042] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0043] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0044] like Figure 1 As shown, the present invention provides an arbitrary-ratio point cloud upsampling method based on weighted point interpolation, comprising:

[0045] S1. Obtain the original point cloud, and use the weighted point interpolation generator to interpolate the original point cloud to obtain the interpolated point cloud;

[0046] S2. Use the distance refiner to refine the interpolated point cloud.

[0047] In specific implementation, as a preferred embodiment of the present invention, step S1 specifically includes:

[0048] S11. Given an input point cloud P with N points, use KNN to find the K neighbors of each point and replicate P K times.

[0049] S12, initialize the weight w, and assign weights to neighbor points and copy points to find the position of the interpolation point;

[0050] S13. In order to remove duplicate interpolation points and control the number of points to rN, after the interpolation is completed, the interpolation point cloud is downsampled using the farthest point sampling method to finally obtain the interpolation point cloud Q′.

[0051] In this embodiment, the weighted key interpolation method is used to upsample the point cloud, and the distance refiner is used for further refinement, and then a point cloud upsampling network based on weighted key interpolation is proposed. The design concept of the present invention: First, the network is designed using the "interpolation-refinement" design concept to decouple the upsampling task into two subtasks: Subtask 1: Generate an interpolated point cloud using weighted key interpolation, and Subtask 2: Adjust the interpolated point cloud to generate a refined point cloud. This design allows each subnet to focus more on its specific work and complement each other in their work; in the weighted key interpolation generator, a weight distribution module is designed and the farthest point sampling method is used to generate an interpolated point cloud with a uniform distribution of point sets; in the distance refiner, the network is used to minimize the distance between a point p in the interpolated point cloud and its nearest neighbor point in the ground truth point cloud. The Euclidean distance between .

[0052] In specific implementation, as a preferred embodiment of the present invention, step S1 specifically includes:

[0053] S11. Given an input point cloud P with N points, use KNN to find the K neighbors of each point and replicate P K times.

[0054] S12, initialize the weight w, and assign weights to neighbor points and copy points to find the position of the interpolation point;

[0055] S13. In order to remove duplicate interpolation points and control the number of points to rN, after the interpolation is completed, the interpolation point cloud is downsampled using the farthest point sampling method to finally obtain the interpolation point cloud Q′.

[0056] In this embodiment, if Figure 1 As shown, the structure and process of the method of the present invention are shown. Given a sparse input P with N points, it is first input into a de-weighted key interpolation generator to generate an interpolation point cloud Q' with rN points, where r is the upsampling rate.

[0057] In specific implementation, as a preferred embodiment of the present invention, step S1 also includes a step of using deep learning to find a suitable weight w, which is as follows:

[0058] Use the chamfer distance loss function L CD To constrain the interpolated point cloud, make the interpolated point cloud closer to the ground truth, and then find the appropriate weight value w.

[0059] In specific implementation, as a preferred embodiment of the present invention, step S2 specifically includes:

[0060] S21, use MLP to perform feature encoding on the interpolated point cloud;

[0061] S22, input the encoded features into the multi-positional feature fusion attention module (MPFA) to obtain multi-level local features;

[0062] S23, performing a maximum pooling operation on the local features to obtain global features;

[0063] S24. Input the multi-level local features, global features, and query points (obtained by adding noise to the interpolated point cloud) into the multi-feature aggregation module, use MLP to predict the point-to-point distance, and refine the point-by-point position of the interpolated point cloud to generate a more refined point cloud Q.

[0064] In this embodiment, continue to see Figure 1 Based on the above step S1, it is hoped that the interpolated point cloud Q′ can achieve accurate upsampling. So next, the interpolated point cloud Q′ is input into the distance refiner, which is further refined to generate a point-to-point distance to adjust the position of each point in the interpolated point cloud Q′, so that the points in the dense and refined output point cloud Q are more uniform and better covered on the surface of the object.

[0065] In specific implementation, as a preferred embodiment of the present invention, in step S22, the multi-positional feature fusion attention module (MPFA) encodes the input point cloud to obtain a multi-scale point-by-point feature and Specifically include:

[0066] S221, given an input point cloud Q′ having N points, local features and global features of the input point cloud Q′ are obtained using MLP and global maximum pooling;

[0067] S222, inputting the global features and the local features into a global-local feature fusion module (G-LFF) to obtain fused features;

[0068] S223, input the fusion feature into the multi-scale position feature fusion attention module, and obtain the multi-scale point-by-point feature F after being processed by the 6-layer position feature fusion attention module (PFA) init 、F L1 and F L2 .

[0069] In specific implementation, as a preferred embodiment of the present invention, in step S223, the position feature fusion attention module (PFA) fuses the point position information with the feature information to generate a fusion feature containing rich representation information, and captures the dependency between the fusion features, thereby generating a richer feature representation containing context information, specifically including:

[0070] S2231, applying KNN to the input point cloud Q′ and feature F respectively to obtain the nearest neighbors of the two;

[0071] S2232, subtract the input point cloud Q′ from the nearest neighbor to obtain the relative position, then apply MLP to the relative feature and the feature F after constructing the neighbor information to align them to the same dimension and then multiply them to obtain a fusion feature containing rich representation information;

[0072] S2233. Send the fused features to the multi-feature aggregation module to learn the dependencies within and between features, thereby obtaining attention features containing contextual information.

[0073] In this embodiment, the multi-scale position feature fusion module makes full use of the position information of the point cloud, effectively captures the geometric structure of the point cloud, and combines the multi-feature aggregation module to integrate the position information into the features. The obtained attention features can not only enhance the correlation between points, but also more accurately describe the local and global geometric relationships. In this way, the model will show stronger generalization ability in capturing complex shapes and details.

[0074] In specific implementation, as a preferred embodiment of the present invention, in step S24, the multi-level feature aggregation module (Multi-feature Aggregation) performs weighted interpolation on the multi-scale local features of the original point cloud through the relationship between the query point and the original point, and splices it with all interpolation features and global features to obtain a comprehensive feature representation that integrates local and global context information, specifically including:

[0075] S241, using KNN to find the nearest neighbor point of each query point, and calculating the interpolation weight according to the distance between the query point and the nearest neighbor point;

[0076] S242, performing weighted calculation, i.e., interpolation operation, on the features of each scale;

[0077] S243. Concatenate the multi-scale interpolation features with the global features to obtain a comprehensive feature representation.

[0078] In this embodiment, the multi-scale feature aggregation module can integrate the geometric structure into the feature expression, and can also fuse local and global information at the same time, thereby enhancing the model's ability to capture point cloud details and overall semantics at different scales.

[0079] Example

[0080] The present invention was tested using synthetic datasets PU1K and PUGAN, with input points of 512, 1024, and 2048, respectively, and an upsampling ratio of 4. Take 2048 input points as an example: 2048 input points means that 2048 points are used to represent a three-dimensional object. If the upsampling ratio is 4, 8192 points will be generated to represent the same three-dimensional object. Under the three input sizes of these two synthetic datasets, the effect of the present invention is excellent, and the obtained upsampled point cloud is uniform and noise-free, which is better than most methods in the three evaluation indicators of CD, HD, and P2F. The present invention was tested using the real scene dataset KITTI. Compared with other methods, it has obvious advantages.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for upsampling point clouds at arbitrary magnifications based on weighted interpolation, characterized in that: include: S1, obtain the original point cloud, and use the weighted point interpolation generator to interpolate the original point cloud to obtain the interpolated point cloud; S2. Use the distance refiner to refine the interpolated point cloud.

2. The method for upsampling point clouds at arbitrary magnifications based on weighted point interpolation according to claim 1, characterized in that: Step S1 specifically includes: S11. Given an input point cloud P with N points, use KNN to find the K neighbors of each point and replicate P K times. S12, initialize the weight w, and assign weights to neighbor points and copy points to find the position of the interpolation point; S13. In order to remove duplicate interpolation points and control the number of points to rN, after the interpolation is completed, the interpolation point cloud is downsampled using the farthest point sampling method to finally obtain the interpolation point cloud Q′.

3. The method for upsampling point clouds at arbitrary magnifications based on weighted point interpolation according to claim 2, characterized in that: Step S1 also includes the step of using deep learning to find a suitable weight w, as follows: Use the chamfer distance loss function L CD To constrain the interpolated point cloud, make the interpolated point cloud closer to the ground truth, and then find the appropriate weight value w.

4. The method for upsampling point clouds at arbitrary magnifications based on weighted interpolation according to claim 1, characterized in that: Step S2 specifically includes: S21, use MLP to perform feature encoding on the interpolated point cloud; S22, input the encoded features into the multi-level position feature fusion attention module to obtain multi-level local features; S23, performing a maximum pooling operation on the local features to obtain global features; S24. Input the multi-level local features, global features, and query points into the multi-level feature aggregation module, use MLP to predict the point-to-point distance, and refine the point-by-point position of the interpolated point cloud to generate a more refined point cloud Q.

5. The method for upsampling point clouds at arbitrary magnifications based on weighted point interpolation according to claim 4, characterized in that: In step S22, the multi-level position feature fusion attention module encodes the input point cloud to obtain multi-scale point-by-point features and Specifically include: S221, given an input point cloud Q′ having N points, local features and global features of the input point cloud Q′ are obtained using MLP and global maximum pooling; S222, inputting the global features and the local features into a global-local feature fusion module to obtain fused features; S223, input the fusion feature into the multi-scale position feature fusion attention module, and obtain the multi-scale point-by-point feature F after being processed by the 6-layer position feature fusion attention module. init 、F L1 and F L2 .

6. The method for upsampling point clouds at arbitrary magnifications based on weighted point interpolation according to claim 5, characterized in that: In step S223, the position feature fusion attention module fuses the point position information with the feature information to generate a fusion feature containing rich representation information, and captures the dependency between the fusion features, thereby generating a richer feature representation containing contextual information, specifically including: S2231, applying KNN to the input point cloud Q′ and feature F respectively to obtain the nearest neighbors of the two; S2232, subtract the input point cloud Q′ from the nearest neighbor to obtain the relative position, then apply MLP to the relative feature and the feature F after constructing the neighbor information to align them to the same dimension and then multiply them to obtain a fusion feature containing rich representation information; S2233. Send the fused features to the multi-feature aggregation module to learn the dependencies within and between features, thereby obtaining attention features containing contextual information.

7. The method for upsampling point clouds at arbitrary magnifications based on weighted interpolation according to claim 4, characterized in that: In step S24, the multi-level feature aggregation module performs weighted interpolation on the multi-scale local features of the original point cloud through the relationship between the query point and the original point, and splices it with all the interpolation features and global features to obtain a comprehensive feature representation that integrates local and global context information, specifically including: S241, using KNN to find the nearest neighbor point of each query point, and calculating the interpolation weight according to the distance between the query point and the nearest neighbor point; S242, performing weighted calculation, i.e., interpolation operation, on the features of each scale; S243. Concatenate the multi-scale interpolation features with the global features to obtain a comprehensive feature representation.

Images