A tree point cloud completion method based on deep learning

Abstract

The present invention discloses a tree point cloud completion method based on deep learning. The method consists of two stages: missing point cloud prediction and completion result optimization. In the prediction stage, we use an end-to-end neural network to predict the missing part by giving an incomplete tree point cloud. In this stage, only the point cloud of the missing part is predicted, and the input point cloud is kept unchanged. However, the predicted result is not well integrated with the input point cloud. To this end, we use a point cloud optimization network to further process the predicted results and the original input to optimize the overall distribution. Our results show that for the completion of incomplete tree point clouds, the combination of the prediction network and the optimization network has achieved better results. It has the characteristics of high completion efficiency, realistic morphology and wide application range. In order to avoid the high storage cost and loss of geometric information caused by voxelization, the present invention constructs and runs a network on the original point cloud, which can effectively repair the incomplete tree point cloud.

Description

Technical Field

[0001] The present invention relates to the field of point cloud data completion, and in particular to a tree point cloud completion method based on deep learning. Background Art

[0002] Trees naturally have complex geometric structures and are very common objects in daily life. Point cloud data in the real world is usually captured by using laser scanners, stereo cameras, or low-cost RGB-D depth cameras. When acquiring tree point clouds, the acquired point clouds are incomplete due to limitations such as occlusion, light reflection, and viewing angle.

[0003] As an important data source and research object for computer graphics and computer vision, 3D point clouds are widely used in 3D modeling, mapping, cultural relics protection and other fields due to their advantages such as small data volume, ability to effectively represent complex 3D shapes, and no need to maintain topological connections. Incomplete tree point clouds not only affect the display effect of the model, but also make many tasks based on complete point clouds impossible to carry out. Completing incomplete tree point clouds is a very meaningful task and a problem that needs to be solved urgently.

[0004] Traditional point cloud completion methods such as geometry-based methods, symmetry-based methods, and alignment-based methods are usually unable to repair large missing areas. Although three-dimensional shapes can be manually completed with the help of professional CAD software, due to the complexity of the model to be repaired, there are disadvantages such as low repair efficiency and high time cost. Most of the objects studied by existing learning-based point cloud completion methods are simple and regular. No network has been found for tree point cloud completion. Trees in nature have naturally complex structures. Existing learning-based methods are not suitable for missing tree point cloud completion, and there are phenomena such as the completion results are too smooth, lack of geometric details, and low completeness. Summary of the invention

[0005] The purpose of the present invention is to overcome the problems existing in the above-mentioned prior art and provide a tree point cloud completion method based on deep learning. In order to avoid the high storage cost and loss of geometric information caused by voxelization, a network is built and run on the original point cloud, which can effectively repair the incomplete tree point cloud.

[0006] To this end, the present invention provides a tree point cloud completion method based on deep learning, comprising the following steps:

[0007] A tree point cloud acquisition

[0008] Acquisition of real tree point cloud: Acquire the point cloud of trees in the forest area and segment the acquired point cloud of trees in the forest area into single trees, select the point cloud with complete tree structure, remove ground noise and outlier noise, and obtain the real tree point cloud;

[0009] Acquisition of virtual tree point cloud: Design and generate a tree model, which includes several single tree models. Use a virtual camera to scan the single tree model from the front, back, left, and right directions, record the camera parameters of a single scan, and store the scanned point cloud to obtain a virtual tree point cloud.

[0010] 2. Build and train the point cloud prediction network

[0011] The real tree point cloud and the virtual tree point cloud are combined to form a tree point cloud dataset, and the dataset is divided into a training set, a validation set, and a test set;

[0012] Establish a prediction network, crop each tree point cloud in the training set to obtain M', and input the remaining part I as the incomplete tree point cloud into the prediction network to obtain the predicted missing part M, and compare M' with M to correct the prediction network;

[0013] Three-point cloud optimization

[0014] Construct a point optimization network, whose input is the combination of the output M of the prediction network and the incomplete input I, and outputs the final optimized result P;

[0015] The point optimization network is a residual network, which generates residual values ​​of point clouds through learning, thereby obtaining the P.

[0016] Furthermore, the prediction network uses a multi-resolution encoder and a joint multi-layer perception CMLP mechanism in the PF-Net network;

[0017] The incomplete tree point cloud I is converted through a multi-resolution encoder;

[0018] The transformed incomplete tree point cloud I is downsampled by the farthest point sampling algorithm FPS to obtain incomplete tree point clouds I represented by different resolutions;

[0019] Three independent CMLPs map the corresponding incomplete tree point clouds I with different resolutions into three latent vectors, and connect these three latent vectors and integrate them into the final feature vector F1;

[0020] The feature vector F1 is input to the decoder module, and after full connection, convolution function and transformation operations, the predicted point cloud of the missing part is output, which is the predicted missing part M.

[0021] Furthermore, it also includes calculating the loss of the prediction network, where the loss of the prediction network is the chamfer distance error L CD :

[0022] Calculate the chamfer distance error L between the output point cloud S1 and the ground truth point cloud S2 CD ;

[0023]

[0024] Among them, L CD It represents the calculation formula of chamfer distance error, x is the point in S1, y is the point in S2, |||| solves the Euclidean distance, and min means selecting the minimum value;

[0025] Wherein, in this formula, the output point cloud S1 is the M, and the ground truth point cloud S2 is the M'.

[0026] Furthermore, it also includes calculating the loss of the point optimization network, the loss of the point optimization network is the soil moving distance error L EMD :

[0027] Calculate the earth moving distance error L between the output point cloud S1 and the ground truth point cloud S2 EMD ;

[0028]

[0029] Among them, L EMD represents the calculation formula of earth moving distance error, x is a point in S1, y is a point in S2, |||| solves the Euclidean distance, min means to select the minimum value, φ is a bijective function between the point sets S1 and S2;

[0030] Among them, in this formula, the output point cloud S1 is the P, and the ground truth point cloud S2 is the tree point cloud GT.

[0031] Furthermore, combined with the chamfer distance error L CD and the earth moving distance error L EMD The obtained error L;

[0032] L=μ·L CD (M',M)+L EMD (P,GT)

[0033] Among them, μ is a coefficient used to control the error ratio of the prediction network during training.

[0034] The present invention provides a tree point cloud completion method based on deep learning, which has the following beneficial effects:

[0035] (1) A point cloud completion method for sparse trees. A learning-based point cloud completion method that can run directly on 3D point clouds without intermediate voxelization.

[0036] (2) By combining the prediction network (predicting missing point clouds) and the optimization network (optimizing the overall completion effect), the “clustering” of the completed tree point clouds can be avoided, ensuring the natural form of the completed tree point clouds.

[0037] (3) The missing tree point cloud can be completed in a short time, and the completed result is more realistic and consistent with the growth form of the tree.

[0038] The present invention is divided into two stages: missing point cloud prediction and completion result optimization. In the prediction stage, we use an end-to-end neural network to predict the missing part by giving the incomplete tree point cloud. In this stage, only the point cloud of the missing part is predicted, and the input point cloud is kept unchanged. However, the predicted result is not well integrated with the input point cloud. For this reason, we use a point cloud optimization network to further process the predicted results and the original input to optimize the overall distribution. Our results show that for the completion of incomplete tree point clouds, the combination of the prediction network and the optimization network has achieved better results. It has the characteristics of high completion efficiency, realistic morphology, and a wide range of applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic block diagram of the overall process of the present invention;

[0040] Figure 2 Point cloud image of real tree point cloud;

[0041] Figure 3 A point cloud image of a virtual tree point cloud;

[0042] Figure 4 Point cloud of trees sampled for the farthest point for visualization;

[0043] Figure 5 It is the structure diagram of the decoder;

[0044] Figure 6 Diagram of the architecture for a point-optimized network. DETAILED DESCRIPTION

[0045] A specific embodiment of the present invention is described in detail below in conjunction with the accompanying drawings, but it should be understood that the protection scope of the present invention is not limited to the specific embodiment.

[0046] Specifically, Figure 1-6 As shown, an embodiment of the present invention provides a tree point cloud completion method based on deep learning, comprising the following steps:

[0047] A tree point cloud acquisition

[0048] Acquisition of real tree point cloud: Acquire the point cloud of trees in the forest area and segment the acquired point cloud of trees in the forest area into single trees, filter out the point cloud with complete tree structure, remove ground noise and outlier noise, and obtain the real tree point cloud.

[0049] The present invention uses airborne laser radar to obtain forest tree point clouds. The marker-controlled watershed segmentation algorithm is used to segment the obtained forest tree point clouds into individual trees. Due to the density of trees, not all trees can be segmented intact during segmentation, and it is necessary to manually screen out point clouds with complete tree structures.

[0050] The proposed voxel-based upward filter method is used to remove ground noise. The statistical filter method is used to remove outlier noise around the tree point cloud. Figure 2 is the processed point cloud, Figure 2 In the figure: tree point cloud segmentation (left), ground point removal (center), and outlier noise removal (right).

[0051] Acquisition of virtual tree point cloud: Design and generate a tree model, which includes several single tree models. Use a virtual camera to scan the single tree model from the front, back, left, and right directions, record the camera parameters of a single scan, and store the scanned point cloud to obtain a virtual tree point cloud.

[0052] To enhance the richness of the dataset, this solution generates point clouds of beech and cherry trees through "simulation". The specific steps are as follows:

[0053] The specific steps are as follows:

[0054] Design and generate tree models using Blender and its plugin The Grove;

[0055] Use a virtual camera in Blensor to scan a single tree model from the front, back, left, and right directions, record the camera parameters for a single scan, and store the point cloud data obtained from the scan.

[0056] Use the camera parameters to calculate the transformation matrix C (Formula 1) for each scan, multiply the corresponding stored point cloud data by the inverse matrix of matrix C (Formula 2) to obtain the transformed point cloud data, and finally combine them into a single complete tree point cloud, such as Figure 3 As shown, Figure 3 Included: tree model (left), single scan result (middle), and synthesized complete tree point cloud (right).

[0057] C=R x ·R y ·R z ·T (1)

[0058] Where R x , R y , R z are the rotation matrices around the three coordinate axes, and T is the translation matrix. The relevant parameters used to calculate these matrices are the camera parameters stored during scanning.

[0059] P'=C -1 ·P scan (2)

[0060] Where P scan is the point cloud data obtained by a single scan, C -1 is the inverse matrix of the transformation matrix C, and P' is the point cloud data after affine transformation.

[0061] 2. Build and train the point cloud prediction network

[0062] The real tree point cloud and the virtual tree point cloud are combined to form a tree point cloud dataset, and the dataset is divided into a training set, a validation set, and a test set in a ratio of 6:2:2.

[0063] A prediction network is established, each tree point cloud in the training set is cropped to obtain M', and the remaining part I is input into the prediction network as the incomplete tree point cloud to obtain the predicted missing part M, and M' is compared with M to correct the prediction network.

[0064] In order to better train the network model and reduce the requirements on computer hardware, for a single tree point cloud, a smaller number of point clouds are used as much as possible without losing the details of the tree's morphology. When sampling feature points of the original point cloud, the advantage of the farthest point sampling algorithm over the random sampling algorithm is that it can cover all points in the space as much as possible. Figure 4 In , we visualize the effect of farthest point sampling. Considering the balance between the number of point clouds and their representation information, we use 4096 point clouds to represent the morphology of a single tree in the experiment.

[0065] In order to take into account both high-resolution and low-resolution point cloud features, the prediction network in the present invention uses a multi-resolution encoder in the PF-Net network and a joint multi-layer perception CMLP mechanism. In terms of extracting point cloud features, CMLP is better than MLP that only outputs the last layer of features.

[0066] The incomplete tree point cloud I is converted through a multi-resolution encoder; the converted incomplete tree point cloud I is downsampled through the farthest point sampling algorithm FPS to obtain incomplete tree point clouds I represented by different resolutions; three independent CMLPs map the corresponding incomplete tree point clouds I of different resolutions into three latent vectors, and connect these three latent vectors and integrate them into the final feature vector F1; the feature vector F1 is input to the decoder module, and after full connection, convolution function and transformation operations, the missing part prediction point cloud is output, which is the predicted missing part M.

[0067] The result of PF-Net for tree point cloud completion lacks detailed distribution. This is due to the multi-level structure of its point pyramid decoder. This solution redesigns the decoder module. The modified decoder can better utilize the point cloud feature information, making the prediction result more consistent with the point cloud morphology of the actual missing part. The decoder structure diagram is shown in the figure below. Figure 5 shown.

[0068] The present invention also calculates the loss of the prediction network, which is the chamfer distance error L CD :

[0069] Calculate the chamfer distance error L between the output point cloud S1 and the ground truth point cloud S2 CD ;

[0070]

[0071] Among them, L CD It represents the calculation formula of chamfer distance error, x is the point in S1, y is the point in S2, |||| solves the Euclidean distance, and min means selecting the minimum value;

[0072] Wherein, in this formula, the output point cloud S1 is the M, and the ground truth point cloud S2 is the M'.

[0073] Three-point cloud optimization

[0074] A point optimization network is constructed, whose input is a combination of the output M of the prediction network and the incomplete input I, and outputs the final optimized result P; the point optimization network is a residual network, which generates the residual value of the point cloud through learning, thereby obtaining the P.

[0075] In the experiment, the output result M of the prediction network may have different degrees of "aggregation" phenomenon, or may not be well integrated with the original input I, and there may be visible gaps between the two. In order to solve the above undesirable phenomena, the prediction results need to be further optimized. This scheme implements the point optimization network, and takes the prediction result M and the incomplete input I as the input of the optimization network. The optimization network controls the optimization process by calculating the error between the optimization result and the real model. The results show that the point optimization network is effective in solving the "aggregation" and "gap" problems.

[0076] The input of the point optimization network is the combination of the output M of the prediction network and the incomplete input I. The point optimization network outputs the final optimized result P (P is calculated based on the two-part point cloud M and I, and P is the final output result of the completion network). The essence of the point optimization network is a residual network. The displacement value that each point needs to move is calculated. The displacement value is applied to the input of the residual network to obtain the final completion output P.

[0077] The architecture of the point optimization network is as follows Figure 6As shown in the figure. Its essence is a residual network, which generates residual values ​​of point clouds through learning and is used to calculate the final optimization output. In order to distinguish the two parts of the point cloud I and M, labels are added in the fourth dimension. A '0' label is assigned to the incomplete tree point cloud I belonging to the original input, and a '1' label is assigned to the points belonging to the predicted result M. The point optimization network uses this information to determine which points need further improvement.

[0078] The residual value can be understood as the displacement value that each point needs to move calculated based on the input of the residual network. The displacement value is applied to the input of the residual network to obtain the final completion output P.

[0079] The present invention also calculates the loss of the point optimization network, and the loss of the point optimization network is the soil moving distance error L EMD ,Unlike the chamfer distance, the earthmoving distance measures the minimum cost of converting one distribution to another.

[0080] Calculate the earth moving distance error L between the output point cloud S1 and the ground truth point cloud S2 EMD ;

[0081]

[0082] Among them, L EMD It represents the calculation formula of earth moving distance error, x is a point in S1, y is a point in S2, |||| solves the Euclidean distance, min means selecting the minimum value, φ is a bijective function between point sets S1 and S2; φ makes the two point sets correspond one to one, and the sum of the Euclidean distances calculated by the two is minimized.

[0083] Among them, in this formula, the output point cloud S1 is the P, and the ground truth point cloud S2 is the tree point cloud GT.

[0084] The use of chamfer distance often leads to uneven density distribution and blurred details. During the optimization phase, we use the earthmoving distance to evaluate the similarity between the optimized result P and the real tree point cloud GT. The earthmoving distance forces the output to have the same density distribution as the ground truth, so it is more discriminative for local details and density distribution. The smaller its value, the higher the similarity between P and GT.

[0085] The input of the point optimization network is the combination of the output M of the prediction network and the incomplete input I. The point optimization network outputs the final optimized result P (P is calculated based on the two-part point cloud M and I, and P is the final output result of the completion network). The essence of the point optimization network is a residual network. The displacement value that each point needs to move is calculated. The displacement value is applied to the input of the residual network to obtain the final completion output P.

[0086] The loss of the completion network in this solution includes two parts: prediction network loss and point optimization network loss. CD and the earth moving distance error L EMD The obtained error L;

[0087] L=μ·L CD (M',M)+L EMD (P,GT)

[0088] Among them, μ is a coefficient used to control the error ratio of the prediction network during training.

[0089] The chamfer distance (CD) is differentiable compared to the earth moving distance (EMD) and has high computational efficiency. In the completion phase, this scheme focuses on predicting the overall effect of the part and uses the chamfer distance to measure the difference between M and M'. The smaller the value, the smaller the difference between M and M'.

[0090] The above disclosures are only several specific embodiments of the present invention. However, the embodiments of the present invention are not limited thereto. Any changes that can be conceived by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. A tree point cloud completion method based on deep learning, characterized in that: The steps include: A tree point cloud acquisition Acquisition of real tree point cloud: Acquire the point cloud of trees in the forest area and segment the acquired point cloud of trees in the forest area into single trees, select the point cloud with complete tree structure, remove ground noise and outlier noise, and obtain the real tree point cloud; Acquisition of virtual tree point cloud: Design and generate a tree model, which includes several single tree models. Use a virtual camera to scan the single tree model from the front, back, left, and right directions, record the camera parameters of a single scan, and store the scanned point cloud to obtain a virtual tree point cloud.

2. Build and train the point cloud prediction network The real tree point cloud and the virtual tree point cloud are combined to form a tree point cloud dataset, and the dataset is divided into a training set, a validation set, and a test set; Establish a prediction network, crop each tree point cloud in the training set to obtain M', and input the remaining part I as the incomplete tree point cloud into the prediction network to obtain the predicted missing part M, and compare M' with M to correct the prediction network; Three-point cloud optimization Construct a point optimization network, whose input is the combination of the output M of the prediction network and the incomplete input I, and outputs the final optimized result P; The point optimization network is a residual network, which generates residual values ​​of point clouds through learning, thereby obtaining the P.

2. A tree point cloud completion method based on deep learning as claimed in claim 1, characterized in that: The prediction network uses a multi-resolution encoder and a joint multi-layer perception CMLP mechanism in the PF-Net network; The incomplete tree point cloud I is converted through a multi-resolution encoder; The transformed incomplete tree point cloud I is downsampled by the farthest point sampling algorithm FPS to obtain incomplete tree point clouds I represented by different resolutions; Three independent CMLPs map the corresponding incomplete tree point clouds I with different resolutions into three latent vectors, and connect these three latent vectors and integrate them into the final feature vector F1; The feature vector F1 is input to the decoder module, and after full connection, convolution function and transformation operations, the predicted point cloud of the missing part is output, which is the predicted missing part M.

3. The tree point cloud completion method based on deep learning as claimed in claim 1, characterized in that: The method also includes calculating the loss of the prediction network, where the loss of the prediction network is the chamfer distance error L CD : Calculate the chamfer distance error L between the output point cloud S1 and the ground truth point cloud S2 CD ; Among them, L CD It represents the calculation formula of chamfer distance error, x is the point in S1, y is the point in S2, |||| solves the Euclidean distance, and min means selecting the minimum value; In this formula, the output point cloud S1 is the M, and the ground truth point cloud S2 is the M'.

4. A tree point cloud completion method based on deep learning as claimed in claim 3, characterized in that: The method also includes calculating the loss of the point optimization network, where the loss of the point optimization network is the soil moving distance error L EMD : Calculate the earth moving distance error L between the output point cloud S1 and the ground truth point cloud S2 EMD ; Among them, L EMD represents the calculation formula of earth moving distance error, x is a point in S1, y is a point in S2, |||| solves the Euclidean distance, min means to select the minimum value, φ is a bijective function between the point sets S1 and S2; Among them, in this formula, the output point cloud S1 is the P, and the ground truth point cloud S2 is the tree point cloud GT.

5. The tree point cloud completion method based on deep learning as claimed in claim 4, characterized in that: Combined with the chamfer distance error L CD and the earth moving distance error L EMD The obtained error L; L=μ·L CD (M',M)+L EMD (P,GT) Among them, μ is a coefficient used to control the error ratio of the prediction network during training.

Images