A method for completing a three-dimensional single-tree point cloud based on deep learning

Abstract

The application belongs to the field of forestry remote sensing, and particularly relates to a three-dimensional single tree point cloud completion method based on deep learning. The application combines deep learning technology to build a TC-Net network model suitable for single tree structure completion; inspired by the penetration attenuation of electromagnetic waves in homogeneous medium, a dataset with uneven density point cloud loss is designed and made, so that the training of the model is more in line with the application scenario; then the model is trained in the open source large dataset, and the trained model is used as the initial model for retraining of tree completion, so that the model can converge faster and the effect is better; finally, the single tree point cloud structure completion of the real scene is realized in the self-collected vehicle-mounted laser point cloud dataset, and the details of the missing point cloud are accurately restored. The application can provide more complete point cloud data for researchers in the aspects of calculating three-dimensional greenness and counting urban greening rate in the field of forestry remote sensing, and plays an important role in subsequent research.

Description

Technical Field

[0001] This invention belongs to the field of forestry remote sensing, specifically relating to a method for completing three-dimensional single-tree point clouds based on deep learning. Background Technology

[0002] Analyzing key biological attributes of trees based on 3D point clouds (such as 3D green volume, biomass, tree height, diameter at breast height, and crown diameter) or performing accurate 3D modeling both require and rely on the integrity of the vegetation point cloud structure. However, the spatial distribution of features in urban areas is complex, with significant mutual occlusion; sensor accuracy and penetration capabilities are limited; and sensors are constrained by the observation angle of the acquisition equipment. All these factors contribute to missing tree point cloud structures, severely affecting the accuracy of subsequent steps.

[0003] Common methods for solving this problem in the field of forestry remote sensing can be broadly classified into three categories: point cloud fusion from multiple perspectives / sensors, correction based on prior knowledge or modeling results, and structural completion based on feature-based original single-tree point cloud data.

[0004] Multi-view / multi-sensor point cloud fusion relies on additional data, is limited by site and experimental conditions, and significantly increases costs. Examples include multi-view fusion of ground-based or backpack lidar, and weighted fusion of airborne lidar point clouds and backpack lidar. Drones can only operate at specific altitudes and need to navigate around tall man-made structures, while vehicles can only travel along roads. This results in a significant lack of complete perspective observation of trees, leading to missing canopy point clouds and affecting the accuracy of 3D green volume calculations. Figure 1 As shown, from left to right, the data are: vehicle-mounted LiDAR point cloud, UAV-mounted LiDAR point cloud, and UAV-based oblique image reconstructed point cloud. The vehicle-mounted LiDAR data is missing tree canopy point cloud that is far from the data acquisition vehicle, while the UAV-mounted LiDAR point cloud and the UAV-based oblique image reconstructed point cloud are missing tree trunk point cloud that is close to the ground.

[0005] Methods based on prior knowledge or modeling corrections, such as those by Xu H et al., first obtain a rough tree skeleton from the original missing point cloud, then generate branches of the skeleton based on the canopy structure, and finally assign leaves to the corresponding positions on the skeleton to complete the modeling. Zhang X et al. first obtain a visible skeleton from the original missing point cloud, then generate an invisible skeleton based on layered canopy feature points, and finally combine the two parts using a particle flow method to obtain the final tree model. A feature-based structural completion method for original single-tree point cloud data first extracts key points from the missing point cloud using the L1-Median algorithm, and then completes the point cloud based on the principal direction of the key points and the point cloud density distribution.

[0006] Point cloud structure completion is an emerging problem in the field of deep learning-based point cloud processing. For example, L-GAN uses an autoencoder model combined with generative adversarial networks (GANs) to complete point cloud structures. FoldingNet designs a folding-based decoder to better reconstruct the 3D surface information of objects. PCN combines the advantages of L-GAN and FoldingNet, and incorporates post-processing operations to smooth the results, achieving even better completion performance. RL-GAN-Net combines reinforcement learning and conditional generative adversarial networks, enabling it to stably complete large areas of missing point clouds.

[0007] PF-Net preserves the original point cloud and predicts the detailed geometry of the missing region based on it. This type of method can effectively restore the detailed features of the missing point cloud. Specifically, PF-Net designs a multi-scale pyramid structure feature encoder-decoder to perform hierarchical estimation of the missing point cloud. A multi-scale completion loss function is designed to complete backpropagation, and corresponding discriminators and adversarial loss functions are designed to achieve a better training process. TopNet designs a hierarchical tree structure encoder to achieve completion. DPGG-Net designs two modules based on generative adversarial networks, transforming the point cloud completion task into an adversarial training task between the global features of the point cloud generated by the decoder based on the missing point cloud and the global features of the complete point cloud. PMP-Net++ treats the point cloud completion task as a deformation task, predicting the complete point cloud by moving the missing point cloud three times. During the movement, the total movement distance is minimized, and the result of each movement is used as the output of the next movement.

[0008] In summary, multi-view / multi-sensor point cloud fusion relies on additional data, is limited by site and experimental conditions, and significantly increases costs. Structural completion based on prior or modeling corrections, as well as feature-based original single-tree point cloud data, cannot achieve accurate completion and often results in missing details in the completed portion. Therefore, existing methods for completing 3D single-tree point clouds all have certain limitations, making this a problem that urgently needs to be solved in practical applications. Summary of the Invention

[0009] To address the aforementioned problems and shortcomings, and to resolve the limitations of existing 3D single-tree point cloud completion methods due to limitations in scene conditions, cost, and accuracy, this invention introduces deep learning technology and applies it to the field of single-tree structure completion, providing a deep learning-based 3D single-tree point cloud completion method.

[0010] A method for completing 3D single-tree point clouds based on deep learning includes the following steps:

[0011] Step 1. Use a vehicle-mounted LiDAR to scan the urban vegetation to obtain raw point cloud data;

[0012] Step 2. Separate the point cloud of the tree part from the original point cloud dataset collected in Step 1, and select the point cloud data with complete point cloud structure to construct complete-incomplete single tree point cloud pairs.

[0013] Step 3. Combine Self-Supervised, Multi-Scale Encoder-Decoder, TransformerBlock, and Self-Attention deep learning techniques to build an end-to-end, data-driven deep learning network model, TC-Net. TC-Net (Tree Completion Net) is suitable for single-tree structure completion, such as... Figure 3 As shown.

[0014] Using a self-supervised approach based on the complete-incomplete single-tree point cloud pairs constructed in step 2, the model learns to predict the missing structures of single-tree point clouds in a data-driven manner, enabling the model to generate corresponding complete single-tree point clouds based on incomplete single-tree point clouds.

[0015] A Multi-Scale Encoder-Decoder was used to capture semantic features of single-tree point clouds at different spatial scales, and these features were then used to progressively predict the missing parts of the point cloud.

[0016] By utilizing the Transformer Block and employing the Self-Attention mechanism, different weights are applied to each part of the input data based on its importance, capturing long-distance dependencies between different locations in the 3D point cloud. This significantly improves the model's ability to express and understand the characteristics of large spatial scale and complex relationships between features in urban 3D point clouds. At the same time, the Transformer has the property of sequence permutation invariance, making it suitable for 3D point clouds with spatially dispersed and disordered characteristics.

[0017] Step 4. Pre-train the network model built in Step 3 on the large open-source dataset ShapeNet-Part, and save the final model parameters.

[0018] Step 5. To address the issue of uneven density and missing data in the point cloud data obtained from LiDAR scanning, a dataset with uneven density and missing data is constructed and used to further train the model obtained in Step 4 (as the initial value) to obtain the final model.

[0019] Step 6. Use the final model obtained in Step 5 to complete the structure of the incomplete point cloud.

[0020] Furthermore, step 3 specifically includes:

[0021] Step 3.1. Set up the Encoder component:

[0022] First, we obtain three downsampled incomplete point cloud X values ​​at three resolutions using FPS (Farthest Point Sampling). i (i = 1, 2, 3, with the number of points being N, N / k, and N / k respectively) 2 ) and the corresponding missing point cloud Y i (i = 1, 2, 3, with points M, M / k, and M / k respectively) 2 Where X represents an incomplete point cloud with the number of points N, N / k, and N / k respectively. 2 Y represents the corresponding missing point cloud, with point counts of M, M / k, and M / k respectively. 2 i = 1, 2, 3, N is the number of points in the incomplete point cloud, M is the number of points in the corresponding missing point cloud, and k is a manually set downsampling parameter. Compared to random sampling, the point cloud obtained by FPS can better represent the overall distribution of missing point clouds.

[0023] Subsequently, hierarchical semantic features are extracted from the point clouds with three different sampling rates using a multi-resolution encoder. The multi-resolution point clouds first pass through a Point Transformer layer to obtain feature F. i (F represents the feature output of the Point Transformer layer, and i corresponds to i during downsampling, i.e., X) i F is obtained through the Point Transformer layer. i Then, the features are fused using MLP to obtain V, where V represents a vector of global spatial semantic features.

[0024] Furthermore, the Point Transformer module works as follows:

[0025] The Point Transformer module employs the Offset-Attention mechanism to calculate the semantic similarity between different point cloud features to achieve semantic modeling, while predicting residual blocks rather than the features themselves, thus achieving better training results. Furthermore, Offset-Attention models the entire system and possesses sequence permutation invariance, making it suitable for processing spatially dispersed and disordered 3D point clouds. Let Query, Key, and Value be Q, K, and V respectively. The principle of Offset-Attention is as shown in equation (1):

[0026] (Q,K,V)=F in ·(W q W k W v (1)

[0027] in For the learnable linear transformations shared by this layer; d e =C j d a =d e / R, where R is an adjustable hyperparameter; N j and C j These represent the number of feature points and the number of dimensions for each spatial scale layer, respectively. The input F to the Attention layer... out The calculation is shown in equation (2):

[0028] A = Softmax(Q·K) T )

[0029] F out =LBR(F in -A·V)+F in (2)

[0030] A represents the Attention Score, and LBR represents the combination of a linear layer, a BathNorm layer, and a ReLU layer. The attention mechanism can establish global connections between feature points, and is particularly good at capturing long-distance dependencies between different locations in a 3D point cloud. Adding the attention mechanism can effectively improve the model's perception of the overall tree structure. Tree structure completion depends on the relative spatial and semantic relationships between vegetation point clouds and surrounding ground features. Adding the attention mechanism can effectively improve the accuracy of tree structure completion tasks.

[0031] Step 3.2. Building the Decoder Part: The Decoder part uses a multi-scale generative network to progressively predict the missing part of the point cloud.

[0032] Starting from V, we obtain three feature layers FC through linear layers. i The deepest FC1 layer is transformed into the first layer of predicted point cloud Y using a Point Transformer layer. primary FC2 with Y primary Predict the second layer point cloud Y centered on each point. secondary The relative coordinates of the corresponding points in FC3. secondary Predict the third layer point cloud Y centered on each point. detail The relative coordinates of the corresponding points in the middle, and at the same time, Y detail This will be used as the final output to predict the missing part of the point cloud structure. primary Both Y1 and Y1 have the same number of points M, Y secondary Both Y2 and Y2 have points M / k, Y detail Both Y3 and Y4 have a point count of M / k. 2 .

[0033] TC-Net's loss function is a multi-scale completion loss, which is derived from the predicted point cloud (Y). detail ,Y secondary, Y primary The equation (3) is composed of a weighted Chamfer Distance (CD) of (Y1,Y2,Y3). CD is a distance metric between point clouds that can be used to evaluate the similarity between two point clouds. The basic idea is to calculate the minimum distance from each point in one point set to another point set and then average the minimum distances. As shown in equation (3),

[0034] L com =d CD (Y detail ,Y1)+αd CD (Y secondary ,Y2)+α 2 d CD (Y primary ,Y3) (3)

[0035] Where α is an adjustable hyperparameter, and CD value d CD The specific calculation is shown in equation (4):

[0036]

[0037] S1 and S2 are two point clouds. This indicates Euclidean distance.

[0038] TC-Net first obtains features of different spatial scales of incomplete point clouds based on a multi-scale encoder, and then fuses them to obtain global features. Then, it uses a multi-scale decoder to gradually transform the global features into missing part point clouds, enabling the model to learn to generate the corresponding complete single-tree point cloud based on the incomplete single-tree point cloud, that is, to complete the single-tree structure.

[0039] Furthermore, step 5 involves constructing a dataset with unevenly distributed missing data, and the specific steps are as follows:

[0040] Step 5.1. To simulate the point cloud data loss caused by the penetration defect of lidar data, this invention draws on the characteristic that the electric field strength of electromagnetic waves attenuates exponentially in a uniform medium, as shown in formula (5).

[0041] I = I0 × e -μd (5)

[0042] Where I is the current electric field strength, I0 is the original electric field strength, in V / m; μ is the propagation attenuation coefficient of the homogeneous medium (related to the physical properties of the homogeneous medium and the frequency and velocity of the electromagnetic wave, in m). -1 ); d is the electromagnetic wave propagation distance relative to the original position (in meters).

[0043] Step 5.2. Applying Equation (5) to the vehicle-mounted LiDAR scenario, considering the main attenuation as the attenuation of electromagnetic waves emitted from the LiDAR in the tree canopy, and relating the attenuation to the point cloud missing probability, Equation (6) is proposed:

[0044]

[0045] Where p represents the probability that the target point cloud P appears in the missing dataset, and p0 is 1. Given the attenuation parameter, d is the distance through which the line connecting the vehicle-mounted lidar and the target point cloud passes through the tree canopy. Overall, it represents the highest probability of occurrence when there is no canopy obstruction between the target point cloud and the vehicle-mounted lidar, and then the probability decreases exponentially with the distance through the canopy.

[0046] Considering that the vehicle-mounted lidar will travel along the road when collecting data, d is set as the distance between the target point cloud and the perpendicular line representing the vehicle's travel line passing through the tree canopy. For example... Figure 4 As shown, the point cloud in the lower left corner represents the vehicle being collected, and the corresponding gray line represents the vehicle's travel line. The gray point cloud on the right represents the tree point cloud. P is the target point cloud, and the white solid line represents the perpendicular line between the target point cloud and the vehicle's travel line. d is the distance the perpendicular line passes through the tree canopy.

[0047] Step 5.3. Assume the input complete point cloud contains m points and the missing point cloud contains n points. Find the perpendicular point T corresponding to the driving line P, and then find the length of the perpendicular segment. Find all points m Afterwards, We use linear normalization (restricting the value to between 0 and 1) to obtain d, and use d to approximate the distance across the canopy.

[0048] To better fit the real-world situation while accelerating computation, a piecewise probability approach is adopted.

[0049] First, calculate d for all points. Then, keep d directly in the first α point clouds (α∈[0,mn); add the missing part to the next β point clouds (β∈[0,n); normalize the distance d in the middle γ point cloud (γ=m-α-β).

[0050] Then, according to formula (6), it is converted into probability p (different decay parameters). The change of probability p under the value is as follows Figure 5 (As shown), then subtract it from a vector T of length γ that is uniformly distributed in [0,1) to obtain the probability.

[0051] Finally The n-β point clouds with the lowest probability are added to the missing part.

[0052] Different hyperparameter choices for density-discrete missing methods significantly affect the TC-Net structure completion performance. Density-discrete missing methods consider two sets of hyperparameters: lane line selection and missing parameters α, β, and... The choice.

[0053] Furthermore, the specific steps of step 5.3, finding the optimal hyperparameters, are as follows:

[0054] Step 5.2.1: Normalize the dataset to occupy a cube space with all sides of length 1, and the xyz coordinates are all in the range of 0 to 1.

[0055] Step 5.2.2: Set up three lane line schemes.

[0056] Lane Line Scheme 1: Randomly selected from the straight lines containing the four base edges of the cube. Lane Line Scheme 2: Randomly selected from the straight lines containing the four base edges of the cube, plus a random translation of 0 to 3, performed on the bottom surface of the cube, in a direction away from the center of the bottom surface. Lane Line Scheme 3: Randomly selected from the straight lines containing all twelve edges of the cube and the straight lines containing the perpendicular bisectors of opposite edges on all faces (two per face, 12 in total).

[0057] Step 5.2.3: Set up three missing parameter schemes.

[0058] Missing parameter solution 1: β is directly set to m, meaning the m points furthest from the lane line are directly eliminated. Hyperparameter solution 2: α, β, and Take fixed values ​​(based on the actual number of point clouds input; generally, α is half the actual number of point clouds input, and β is one-eighth of the actual number of point clouds input). Choose from 1 to 4). Hyperparameter scheme 3: If half of the actual number of input point clouds is n, α takes a random integer within the range of (n±250), and β takes... random integers within the range Select a random integer between 1 and 4.

[0059] Step 5.2.4: Take pairwise combinations of the three lane line schemes and the three missing parameter schemes, for a total of nine schemes, to test the results under different missing parameters for uneven density. Use the CD value between the actual missing point cloud and the predicted missing point cloud to measure the predicted missing and actual missing results.

[0060] Step 5.2.5: Use the nine methods from Step 5.2.4 to perform density missing processing on the training set of ShapeNet-Part, and use the processed dataset to train the network model TC-Net constructed in Step 3, obtaining a pre-trained model. The nine methods correspond to nine pre-trained models.

[0061] Model training: Pre-training refers to training a general-purpose model on a large-scale dataset and then fine-tuning it on specific downstream tasks to improve its performance and generalization ability. This method was first used in the field of computer vision because computer vision tasks often have many similar image features, so a model trained on a large dataset can be easily transferred to other tasks. Currently, there are many pre-trained models in the CV field, such as ResNet, ViT, and Swin Transformer, which are pre-trained on large supervised image datasets such as ImageNet.

[0062] Step 5.2.6: Use the pre-trained model obtained in Step 5.2.5 to test the test set and calculate the average CD value for each scheme on the test set. Based on the test results, select the pre-trained model corresponding to the optimal scheme from the nine schemes as the optimal pre-trained model.

[0063] The optimal pre-trained model was selected, solving the problem of the model's inability to successfully complete missing data due to mismatched missing patterns in real-world scenarios. The parameters of the optimal pre-trained model were used as initial parameters, and the model was further trained using the dataset collected in step 2. The trained model was termed the model that conforms to the real-world scenario. Pre-training techniques were used here to compensate for the limited amount of data available for tree completion. This step also verified the effectiveness of equation (6).

[0064] This invention combines deep learning techniques such as Self-Supervised and Multi-Scale Encoder-Decoder to build a TC-Net network model suitable for single-tree structure completion. Inspired by the attenuation of electromagnetic waves in a homogeneous medium, a dataset simulating real-world vehicle-mounted point cloud gaps (density-uniform point cloud gaps) is designed and created. The model is first trained on a large open-source dataset, and then retrained using this trained model as the initial model for tree completion. Finally, single-tree point cloud structure completion in real-world scenes is achieved on a self-collected vehicle-mounted laser point cloud dataset. The effectiveness of combining TC-Net and density-based point cloud gap methods is tested and verified on the open-source CAD model point cloud dataset ShapeNet-Part. The structure completion results are also compared with point clouds reconstructed from oblique images with relatively complete canopies to quantitatively demonstrate the completion effect.

[0065] In summary, the three-dimensional single-tree point cloud completion method of the present invention is less restricted by site and experimental conditions, has low cost, and provides accurate and efficient completion results; it effectively overcomes the various defects of existing three-dimensional single-tree point cloud completion methods and provides a more superior technical means for practical applications. Attached Figure Description

[0066] Figure 1 This is a schematic diagram showing the missing point cloud structure of a single tree.

[0067] Figure 2 This is a technical roadmap of the present invention;

[0068] Figure 3 For the specific structure of TC-Net;

[0069] Figure 4 This is a schematic diagram illustrating the uneven density loss pattern.

[0070] Figure 5 For different attenuation parameters The change of probability p under the given value;

[0071] Figure 6 A comparison of random spherical defects and the density-dissipative defects achieved in this invention;

[0072] Figure 7 The effects of different missing solutions;

[0073] Figure 8 For self-collecting point clouds from vehicle-mounted lidar;

[0074] Figure 9 The test results are for TC-Net trained using a density-deficient approach. Detailed Implementation

[0075] To intuitively illustrate the advantages of this invention, an implementation case of single-tree point cloud structure completion based on the TC-Net model is described in conjunction with actual data, experimental results, and accompanying figures. The specific implementation process is as follows:

[0076] A method for completing 3D single-tree point clouds based on deep learning includes the following steps:

[0077] Step 1: Acquisition of raw point cloud data.

[0078] Step 1.1. Data is collected using a 128-line iScan-SZ lidar. The lidar is fixed on the roof of the data collection vehicle, and the supporting equipment includes a vehicle speed sensor, a point cloud box, and a static differential base station deployed on the ground.

[0079] Step 1.2. The data acquisition vehicle travels along urban roads to collect raw point cloud data, GNSS (Global Navigation Satellite System) data, odometry data, and IMU (Inertial Measurement Unit) data. The raw GNSS data is then converted using StaticToRinex64 software.

[0080] Step 1.3. Input the experimental parameters into the Inertial Explorer software, including the geographic coordinates of the control points, the installation data of the on-board equipment, and the POS (Position and Orientation System) interval, and use the converted GNSS data, IMU data, and odometer data to perform IE (Inertial-Exterior Solution) to obtain the POS file of the driving path.

[0081] Step 1.4. Input the POS file and raw point cloud data into the mmsconvert software. Adjust the corresponding parameters according to the experiment to obtain the final static full-scene point cloud of the entire scene. Then, crop the required survey area data. The obtained survey area acquisition data is as follows: Figure 8 As shown.

[0082] Step 2: Construct the training and test sets.

[0083] The software CloudCompare is used to separate trees from the point cloud data, obtaining individual tree point cloud data. Data collected by LiDAR suffers from uneven point cloud density due to limited viewing angle; that is, the point cloud density decreases with distance from the vehicle's trajectory due to leaf or branch occlusion and the limited penetration of LiDAR, eventually leading to structural gaps—i.e., uneven density gaps. The network model in this invention aims to fill in these missing parts. Point cloud data of trees at road corners or parking areas have relatively complete viewing angles. We will select point cloud data with relatively complete point cloud structures to manually construct complete-incomplete pairs for training the end-to-end network model TC-Net in step 3. The dataset information constructed in this embodiment is shown in Table 1.

[0084] Table 1. Detailed number of each type in the training and test sets.

[0085]

[0086] Step 3: Build the end-to-end, data-driven deep learning network model TC-Net. Specific details have been elaborated in the invention description and will not be repeated here.

[0087] Step 4: Pre-train the network model built in Step 3 on ShapeNet-Part and save the final model parameters.

[0088] Step 5: Construct a dataset with uneven density and missing data, and use it to further train the model obtained from the pre-training in Step 4.

[0089] A dataset with uneven density and missing data was constructed. The specific research details have been elaborated in the invention content. Here, we only describe the parameter selection results.

[0090] The choice of different hyperparameters for uneven density missing methods significantly affects the TC-Net structure completion performance. Uneven density missing methods mainly involve two sets of hyperparameters: lane selection and missing parameters α, β, and... The choice.

[0091] Three hyperparameter schemes are set: Hyperparameter Scheme 1: β is directly set to 512, that is, the 512 points farthest from the lane line are directly eliminated; Hyperparameter Scheme 2: α, β and The values ​​are fixed at 1024, 256, and 1 respectively; Hyperparameter scheme 3: α is a random integer ranging from 768 to 1280, β is a random integer ranging from 128 to 384, Select a random integer between 1 and 4.

[0092] The process of determining hyperparameters is as follows:

[0093] Step 5.2.1: Take pairwise combinations of the three lane line schemes and the three missing parameter schemes, for a total of nine schemes, to test the results under different missing parameters for uneven density. Use the CD value between the actual missing point cloud and the predicted missing point cloud to measure the predicted missing and actual missing results.

[0094] Step 5.2.2: Nine schemes are used to process the training set of ShapeNet-Part, and the processed dataset is used to train the network model TC-Net constructed in Step 3. The training configuration is as follows: the number of incomplete region points N in TC-Net is set to 1536, the number of missing region points M is set to 512, the downsampling rate k is set to 2, and the hyperparameter α of the multi-scale completion loss function is set to 0.1; the entire code project is built based on PyTorch; a 40GB NVIDIA A100 graphics card is used to complete the entire training and testing; Batch Normalization and ReLU activation functions are used; the optimizer is Adam, the initial learning rate is lr = 0.001, and the weight decay rate is 10. -4 The batch size is 32. Each training session runs for 200 epochs, and the model from the last epoch is used as the final weights. The model from the last epoch is called the pre-trained model. There are nine pre-trained models corresponding to the nine schemes.

[0095] Step 5.2.3: Use the pre-trained model to test the test set constructed in Step 2, and calculate the average CD value of each scheme on the test set, as shown in Table 3. Some results are shown below. Figure 7 As shown.

[0096] Step 5.2.4: The closer the missing point cloud data matches the missing data data encountered when scanning objects with a LiDAR scanner in reality, the better the model can learn to predict missing point clouds. Figure 6 The figures show the random spherical deletion method and the density-dissimilar deletion method implemented in this invention (parameters: m = 2048, n = 512, α = 1024, β = 256, decay parameter). By comparing with 2), it can be seen that the density unevenness missing mode implemented by the present invention is closer to real data.

[0097] The pseudocode for this method is shown in Table 2.

[0098] Table 2. Density Inhomogeneity Missing - Pseudocode

[0099]

[0100]

[0101] This invention investigates two sets of hyperparameters on the large open-source dataset ShapeNet-Part. ShapeNet-Part is a richly annotated, large-scale 3D shape dataset that includes the PartNet dataset. PartNet contains semantic-level, instance-level, and hierarchical 3D object labels, with 573,585 part instances from 26,671 3D models, covering 24 object categories. It can support research such as classification, dynamic 3D scene modeling, and semantic segmentation.

[0102] Step 5.2.5: Analyze the results. It can be seen that the predicted CD value for lane line scheme 1 is significantly smaller than that for lane line schemes 2 and 3, meaning the actual missing point cloud and the predicted missing point cloud are closer. This is because the missing point cloud pattern obtained by lane line scheme 1 is simpler and easier to predict, while the other two lane line schemes have more scattered distribution characteristics and more irregular missing points and outliers (such as...). Figure 7 The parts marked with boxes in the text (the missing parts) can lead to chaotic model gradients and difficulty in convergence. For the missing parameter schemes, the CD values ​​of schemes 2 and 3 are smaller than those of scheme 1, and the CD values ​​of schemes 2 and 3 are close. The main reason is that the geometric characteristics of the missing point clouds generated by schemes 2 and 3 are smoother, which is more conducive to model training and learning. Considering that scheme 3 can include more distributions of missing point clouds, scheme 3 is more suitable as a subsequent tree completion scheme.

[0103] Based on the above, this embodiment selects a combination of lane line scheme 1 and scheme 3 with missing hyperparameters (i.e., the hyperparameters are as follows: lane line scheme 1 is selected; α is a random integer ranging from 768 to 1280, and β is a random integer ranging from 128 to 384). A random integer (ranging from 1 to 4) is selected as the hyperparameter scheme for single-tree completion. At the same time, the pre-trained model corresponding to this scheme (called the optimal pre-trained model) is retained as the initial training parameters for tree structure completion on the collected dataset in the next step.

[0104] Table 3. Average CD value of different lane marking schemes and lane marking schemes × 100

[0105]

[0106]

[0107] In step 5 of this embodiment, the model selects a combination of lane line scheme 1 and hyperparameter scheme 3 as the hyperparameter scheme for single-tree structure completion. The network model is trained on the large open-source dataset ShapeNet-Part, obtaining the optimal pre-trained model. This solves the problem of incompatibility between missing patterns and the model's inability to complete tree structures in real-world scenarios. The parameters of the optimal pre-trained model are used as initial parameters, and the model is further trained using the dataset collected in step 2. This pre-training technique compensates for the limited amount of data available for tree completion. The hyperparameter configuration for training is the same as in step 5.2.2, and the model from the last epoch is used as the final weights; this model is referred to as the model that conforms to the real-world scenario.

[0108] Step 6: Evaluate the model: Measure the predicted missing and true missing results using the CD value between the real missing point cloud and the predicted missing point cloud. Test the model on the test set to assess the accuracy of the model in a real-world scenario, and visualize some results to enhance persuasiveness.

[0109] The CD value is 9.20 × 10 on the 33 point clouds in the test set. -3 Select four visualizations (such as...) Figure 9 As shown in the figure, it can be seen that the model that matches the real scene on the test set can complete the completion task well. It is worth noting that the CD value of the density-based missing method is greater than that of the random spherical missing method. This is because the density-based missing method contains more of the missing point cloud distribution, which is closer to the missing in the real scene. It is inherently more difficult to complete, which is reflected in the larger final CD value. However, this also ensures that the model can recognize the single missing model in the real scene and successfully complete the structural completion.

[0110] Tree point cloud completion presents two main challenges: first, how to train the model to learn and predict the missing structures of individual tree point clouds to complete the structure; and second, the need to create pairs of missing and complete point clouds for model training. In computer vision, a common method for creating missing point clouds is random spherical deletion, where a center is selected in space, and the portion of the point cloud closest to that center is randomly eliminated. However, real-world vehicle point cloud data does not exhibit spherical deletion; rather, the point cloud density is uneven due to limited viewing angle. Specifically, the point cloud density decreases with increasing distance from the vehicle's trajectory due to occlusion by leaves or branches, coupled with the limited penetration of LiDAR.

[0111] As demonstrated by the above embodiments, this invention combines deep learning technology to build a TC-Net network model suitable for single-tree structure completion. Inspired by the penetration and attenuation of electromagnetic waves in a homogeneous medium, a dataset with missing point clouds of uneven density was designed and created, making the model training more suitable for application scenarios. The model was first trained on a large open-source dataset, and then retrained using this trained model as the initial model for tree completion, enabling faster convergence and better results. Finally, single-tree point cloud structure completion in real-world scenes was achieved on a self-collected vehicle-mounted laser point cloud dataset, accurately restoring the detailed features of missing point clouds. This invention has been tested in real-world scenarios, and the test results demonstrate the feasibility of the method. It can provide researchers with more complete point cloud data for calculating 3D green volume and statistically analyzing urban greening rates, playing an important role in subsequent research.

Claims

1. A method for completing 3D monocular point clouds based on deep learning, characterized in that, The steps include the following: Step 1: Use a vehicle-mounted LiDAR to scan the urban vegetation to obtain raw point cloud data; Step 2: Separate the point cloud of the tree part from the original point cloud dataset collected in Step 1, and select the point cloud data with complete point cloud structure to construct complete-incomplete single tree point cloud pairs. Step 3: Combine Self-Supervised, Multi-Scale Encoder-Decoder, Transformer Block and Self-Attention to build an end-to-end, data-driven deep learning network model TC-Net; Using self-supervised pairs of complete and incomplete single-tree point clouds built in step 2, we learn to predict the missing structures of single-tree point clouds in a data-driven manner. Multi-Scale Encoder-Decoder is used to capture semantic features of single tree point clouds at different spatial scales, and these features are then used to progressively predict the missing parts of the point cloud. By using the Transformer Block and employing the Self-Attention mechanism, different weights are applied according to the importance of each part of the input data to capture long-distance dependencies between different locations in the 3D point cloud. Step 4: Pre-train the network model built in Step 3 on ShapeNet-Part and save the final model parameters; Step 5: Construct a dataset with uneven density and missing data, and use it to train the model obtained from the pre-training in Step 4 to obtain the final model. Step 5.1: Drawing on the characteristic that the electric field intensity of electromagnetic waves attenuates exponentially in a uniform medium, the point cloud data loss caused by the penetration defect of lidar data is simulated, as shown in formula (5): (5) Where I is the current electric field strength and I0 is the original electric field strength, in V / m; The propagation attenuation coefficient in a homogeneous medium, in meters (m). -1 ; d represents the electromagnetic wave propagation distance relative to the original position, in meters; Step 5.2: Apply formula (5) to the vehicle-mounted lidar scenario, consider the main attenuation as the attenuation of electromagnetic waves emitted from the lidar in the tree canopy, and correlate the attenuation with the point cloud missing probability to propose formula (6): (6) Where p represents the probability that the target point cloud P appears in the missing dataset. =1, Given the attenuation parameter, d is the distance through which the line connecting the vehicle-mounted lidar and the target point cloud passes through the tree canopy. Overall, it represents the highest probability of occurrence when there is no canopy obstruction between the target point cloud and the vehicle-mounted lidar, and then the probability decreases exponentially with the distance through the canopy. Considering that the vehicle-mounted lidar will travel along the road when collecting data, d is set as the distance between the target point cloud and the perpendicular line representing the vehicle's travel line passing through the tree canopy. Step 5.3: Assume the input complete point cloud contains m points and the missing point cloud contains n points; find the perpendicular point T corresponding to the driving line P, and then find the length of the perpendicular segment. Find all points m Afterwards, Linear normalization is used to limit the value to between 0 and 1 to obtain d, and d is used to approximate the distance through the canopy; Implemented using piecewise probability: First, calculate d for all points, then find the point with d at the beginning. Each point cloud is directly preserved. ;back The missing parts are directly added to the point cloud. ;middle The point cloud of each element will be normalized to the distance d. ; Then convert it to probability according to formula (6). Then, evenly distributed in The length is Subtracting the vector T yields the probability. ; Finally The one with the lowest probability Add missing parts to the point cloud; Different hyperparameter choices for density-discrepancy missing methods can significantly affect the TC-Net structure completion performance; density-discrepancy missing methods consider two sets of hyperparameters: lane line selection and missing parameters. , and The choice; Step 6: Use the final model obtained in Step 5 to complete the structure of the incomplete point cloud.

2. The method for completing 3D monocular point clouds based on deep learning as described in claim 1, characterized in that, Step 3 specifically involves: Step 3.1: Set up the Encoder component: First, obtain the downsampled incomplete point cloud X at three resolutions using FPS. i And the corresponding missing point cloud Y i ; Where X represents an incomplete point cloud, with the number of points being N, N / k, and N / k respectively. 2 Y represents the corresponding missing point cloud, with point counts of M, M / k, and M / k respectively. 2 i=1,2,3, N is the number of points in the incomplete point cloud, M is the number of points in the corresponding missing point cloud, and k is the downsampling parameter set manually. Subsequently, hierarchical semantic features are extracted from the point clouds with three different sampling rates using a multi-resolution encoder. The multi-resolution point clouds first pass through a Point Transformer layer to obtain feature F. i F represents the feature output by the Point Transformer layer, X i F is obtained through the Point Transformer layer. i Then, the features are fused using MLP to obtain V, where V represents a vector of global spatial semantic features; The Point Transformer module uses the Offset-Attention mechanism to calculate the semantic similarity between different point cloud features to achieve semantic modeling, while predicting residual blocks; let Query, Key and Value be Q, K and V respectively, and the Offset-Attention principle is as shown in equation (1): in ; ; , This is a learnable linear transformation shared by this layer; , R is an adjustable hyperparameter; and These represent the number of feature points and the number of dimensions for each spatial scale layer; the Attention layer output... The calculation is shown in equation (2): （2） A represents Attention Score, and LBR represents a combination of linear layer, BathNorm layer and ReLU layer; Step 3.2: Building the Decoder Part: The Decoder part uses a multi-scale generative network to progressively predict the missing part of the point cloud; Starting from V, we obtain three feature layers FC through linear layers. i The deepest FC1 layer is transformed into the first layer of predicted point cloud Y using a Point Transformer layer. primary ;FC2 with Y primary Predict the second layer point cloud Y centered on each point. secondary The relative coordinates of the corresponding points in FC3; FC3 in Y secondary Predict the third layer point cloud Y centered on each point. detail The relative coordinates of the corresponding points in the middle, and at the same time, Y detail The point cloud structure that will be used as the final output to predict the missing parts; Y primary Both Y1 and Y1 have the same number of points M, Y secondary Both Y2 and Y2 have points M / k, Y detail Both Y3 and Y4 have points M / k. 2 ; TC-Net's loss function is a multi-scale completion loss, derived from the predicted point cloud Y. detail, Y secondary, Y primary The weighted CD of Y1, Y2, and Y3 is composed of a distance metric between point clouds, which is used to evaluate the similarity between two point clouds. It calculates the minimum distance from each point in one point set to another point set and averages the minimum distances, as shown in equation (3). in, For adjustable hyperparameters, CD value The specific calculation is shown in equation (4): in, and For two point clouds, Indicates Euclidean distance; TC-Net first obtains features of different spatial scales of incomplete point clouds based on a multi-scale encoder, and then fuses them to obtain global features. Then, it uses a multi-scale decoder to gradually transform the global features into missing part point clouds, enabling the model to learn to generate the corresponding complete single-tree point cloud based on the incomplete single-tree point cloud, that is, to complete the single-tree structure.

3. The method for completing 3D monocular point clouds based on deep learning as described in claim 1, characterized in that, The specific steps of finding the optimal hyperparameters in step 5.3 are as follows: Step 5.2.1: Normalize the dataset to occupy a cube space with all sides of length 1, and the xyz coordinates are all in the range of 0 to 1; Step 5.2.2: Set up three lane marking schemes: Lane line scheme 1: Randomly selected from one of the four straight lines containing the base edges of the cube; Lane line scheme 2: Randomly select one of the four bottom edges of the cube as a straight line, and add a random translation of 0 to 3. The translation is carried out on the bottom surface of the cube, and the direction is away from the center of the bottom surface of the cube. Lane line scheme 3: Randomly selected from the straight lines containing all twelve edges of the cube and the straight lines containing the perpendicular bisectors of opposite edges on all faces; Step 5.2.

3. Set three missing parameter schemes; Missing parameter solution 1: Take m directly, that is, directly eliminate the m points that are farthest from the lane line; Hyperparameter scheme 2: Values ​​are determined based on the actual number of point clouds input. The actual number of point clouds input is half. One-eighth of the actual number of point clouds input. Choose from 1 to 4. , and Take fixed values ​​respectively; Hyperparameter scheme 3: If half of the actual number of input point clouds is n, Pick random integers within the range Pick random integers within the range Select a random integer between 1 and 4; Step 5.2.4: Take two pairs of the three lane line schemes and the three missing parameter schemes, for a total of nine schemes, to test the results under different missing parameters due to uneven density. Use the CD value between the real missing point cloud and the predicted missing point cloud to measure the predicted missing and real missing results. Step 5.2.5: Use the nine schemes in step 5.2.4 to perform density missing processing on the training set of ShapeNet-Part, and use the processed dataset to train the network model TC-Net constructed in step 3 to obtain a pre-trained model. The nine schemes correspond to nine pre-trained models. Step 5.2.6: Use the pre-trained model obtained in step 5.2.5 to test the test set and calculate the average CD value of each scheme on the test set; Based on the test results, the pre-trained model corresponding to the optimal solution among the nine solutions is selected as the optimal pre-trained model.

Images