A large-scale garden scene point cloud geometry completion method based on semantic segmentation

Abstract

The application provides a large-scale garden scene point cloud geometry completion method based on semantic segmentation, comprising the following steps: 1, scene point cloud semantic segmentation: using a point cloud semantic segmentation network trained on a garden point cloud semantic segmentation dataset to perform semantic segmentation on input garden point cloud, and then performing intra-class clustering on each semantic category to obtain instance objects; 2, object point cloud completion: after normalization, the instance objects are used for network training, fitting is performed to obtain a latent vector corresponding to a single object and a generator, and a completed instance object is further obtained; 3, scene point cloud assembly: according to the object size and coordinates saved during normalization of the instance objects, the completed instance objects are restored and assembled to obtain a completed garden point cloud.

Description

Technical Field

[0001] This invention relates to a point cloud geometric completion method, and more particularly to a large-scale garden scene point cloud geometric completion method based on semantic segmentation. Background Technology

[0002] The geometric completion method for scene point clouds aims to fill in holes or missing parts in scene point clouds according to their categories or the semantic features of surrounding objects, thereby obtaining a more complete scene point cloud. Holes or missing parts in scene point clouds are usually caused by occlusion between objects during device scanning or by incomplete shooting perspectives during multi-view reconstruction. Large-scale garden scenes are a special type of scene in scene point clouds, characterized by their large spatial scale and extremely large number of points. The objects within the scene are numerous, densely and complexly distributed, and contain many missing parts. Furthermore, some object categories possess numerous and detailed local structures. These factors make geometric completion of large-scale garden scene point clouds a challenging problem.

[0003] For large-scale garden scenes, existing technologies have proposed some solutions, such as patent publication 1: Wu Qincheng. A method and system for scene construction based on BIM technology and its application in digital modeling of garden landscapes [P]. Guangdong Province: CN110298136A, 2019-10-01.; Patent publication 2: Zhang Qingping, Ding Mingjing, Liang Huilin. A method for digital mapping and three-dimensional visualization of private garden space [P]. Jiangsu Province: CN109945845A, 2019-06-28. These methods collect garden scene image data and then use software to process the data to obtain the three-dimensional data of the scene. However, these methods can only reconstruct three-dimensional data from two-dimensional images of garden scenes, and cannot guarantee that the reconstruction results will not contain gaps or missing parts. Furthermore, they cannot perform geometric completion for large-scale garden scenes. Patent Document 3: Li Changhui, Research and Application of Key Technologies for 3D Laser Measurement Modeling of Ancient Architecture and Gardens. Guangdong Province, Guangzhou Urban Planning Survey and Design Institute, 2015-01-05. Directly performing 3D laser measurement modeling on garden scenes cannot guarantee that there will be no occlusion during measurement, which may lead to voids or missing parts in the results.

[0004] To address the geometric completion problem of scene point clouds, some existing methods complete the scene point cloud geometrically by voxelizing the point cloud. Examples include: Reference 1: Cheribier I, Schonberger JL, Oswald MR, et al. Learning priors for semantic 3d reconstruction [C] / / Proceedings of the European conference on computer vision (ECCV). 2018:314-330; Reference 2: Dai A, Ritchie D, Bokeloh M, et al. Scancomplete: Large-scale scene completion and semantic segmentation for 3dscans [C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern. Recognition.2018:4578-4587. This type of method performs semantic completion on the voxel representation of the scene to complete the geometric completion. However, this type of method has two problems when applied to large-scale garden scenes: First, large-scale garden scenes are widely distributed in terms of spatial scale, and processing after voxelization will bring huge time and computational costs; Second, some objects in the garden scene have a lot of fine local structures, and voxelization will lose these local structures, resulting in insufficient precision of the completion result.

[0005] Existing single-object point cloud completion methods can be divided into supervised and unsupervised methods. Supervised methods use complete object point clouds as supervision to train the completion network, such as in reference 3: Zhang W, Yan Q, Xiao C. Detail preserved point cloud completion via separated feature aggregation[C] / / Computer Vision–ECCV 2020:16th European Conference, Glasgow,UK,August 23–28,2020,Proceedings,Part XXV 16.Springer International Publishing,2020:512-528. However, these methods require a large amount of complete object point clouds as strong supervision information. Since garden scenes do not have complete object point clouds, this type of method cannot be used for object point cloud completion. Unsupervised methods do not require a complete object point cloud as supervision. They complete the object point cloud by designing a network structure or introducing priors, such as in reference 4: Zhang J, Chen X, Cai Z, et al. Unsupervised 3d shape completion through gan inversion[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2021:1768-1777. This method introduces prior information by using pre-trained generator and discriminator models, learns latent vectors and generator models by optimizing the distance loss between the generated object and the incomplete object, and then uses the learning results to generate a complete object point cloud. Summary of the Invention

[0006] Purpose of the invention: The technical problem to be solved by the present invention is to provide a geometric completion method for point clouds of large-scale garden scenes based on semantic segmentation, which addresses the shortcomings of the existing technology.

[0007] To address the aforementioned technical problems, this invention discloses a method for geometric completion of point clouds in large-scale garden scenes based on semantic segmentation, comprising the following steps:

[0008] Step 1, Scene point cloud semantic segmentation: Use the point cloud semantic segmentation network pre-trained on the garden point cloud semantic segmentation dataset to perform semantic segmentation on the input garden scene point cloud, and then obtain instance objects by performing intra-class clustering on each semantic category.

[0009] Furthermore, the aforementioned scene point cloud semantic segmentation specifically includes the following steps:

[0010] Step 1-1: Train the semantic segmentation network using the garden point cloud semantic segmentation dataset to obtain the semantic segmentation network F. s ;

[0011] Furthermore, the semantic segmentation network F s as follows:

[0012] Using existing supervised semantic segmentation methods for processing scene point clouds, inputting a garden scene point cloud P∈R N×6 , where R N×6 Represents an N×6 dimensional real number field, where N is the number of points in the garden scene point cloud. Each point contains XYZ axis coordinate information and RGB color information, totaling 6 dimensions. Output the semantic segmentation result S. sem ={S1, S2, ..., S cls}, where S cls This represents the cls-th semantic segmentation result, where cls is the number of semantic categories in the garden scene. Let N be the point cloud of the semantic category of class cls. cls denoted as the number of points in the point cloud for the semantic category cls.

[0013] Steps 1-2: Utilize the semantic segmentation network F s Semantic segmentation is performed on the input point cloud of a garden scene to obtain segmentation results for different semantic categories;

[0014] Steps 1-3 involve performing intra-class clustering on the point clouds of each semantic category to obtain instance-level object point clouds. Among them ins n The number of instance objects, For the first ins n An instance object.

[0015] Furthermore, the aforementioned intra-class clustering specifically includes the following steps:

[0016] Step 1-3-1, Input semantic segmentation result S sem ={S1, S2, ..., S cls}, neighborhood radius eps, eps>0, minimum number of neighborhood samples minPts, minPts>1;

[0017] Step 1-3-2, if the semantic segmentation result S to be clustered sem If the result is Φ, then proceed to step 1-3-8, where Φ represents the empty set; otherwise, proceed from the semantic segmentation result S to be clustered. sem Take any point cloud S = {s1, s2, ..., s} t}, s t ∈R 3, where s t This represents the t-th point in the point cloud S, where t is the number of points in point cloud S; update S sem =Ss em -{S}; where, during the update, each point in the point cloud to be clustered uses only the 3D XYZ axis coordinate information;

[0018] Step 1-3-3: Initialize the current category label K = 0, initialize the queue to be processed L = {}, initialize the point-by-point category C = {c1, c2, ..., ct}, c t = -1, where c t This indicates the category of the t-th point, where t is the number of points in the point cloud S;

[0019] Step 1-3-4: If the point cloud to be clustered S = Φ, execute step 1-3-2, where Φ represents the empty set; otherwise, randomly select a point s from the point cloud S and update S = S - {s}; calculate the Euclidean distance of point s to all other points, and select points whose distance is less than the neighborhood radius eps as the neighborhood of point s; if |Neighbors| < minPts, then execute step 1-3-4 again, otherwise execute step 1-3-5, where |·| represents the number of elements in the statistical set ·.

[0020] Steps 1-3-5: Update K = K+1, update the category label c = K for point s, and update the queue L = Neighbors.

[0021] Step 1-3-6: If the queue to be processed L = Φ, then return to step 1-3-4; otherwise, take a point p from the queue to be processed L, update the queue L = L - {p}, and if the category label c of point p is... p =-1, then update c p =K, proceed to step 1-3-7; otherwise, re-execute step 1-3-6;

[0022] Step 1-3-7: Calculate the Euclidean distances of point p to all other points, and select points whose distances are less than the neighborhood radius eps as the neighborhoods of point p. p If |Neighbors p If |≥minPts, then update queue L=L∪Neighbors p Otherwise, no operation will be performed, and the process will return to step 136 to continue.

[0023] Steps 1-3-8 involve segmenting points with cluster labels greater than 0 across all semantic categories according to their cluster labels to obtain individual instance objects. These instance objects from all semantic categories are then merged to obtain a set of all instance objects after semantic segmentation of the garden scene point cloud.

[0024] Step 2, object point cloud completion: After normalizing the instance objects, the completion network is trained to fit the latent vector and generator corresponding to a single instance object, and the latent vector and generator are used to further obtain the completed instance object.

[0025] Furthermore, the object point cloud completion process specifically includes the following steps:

[0026] Step 2-1, for the collection of instance objects I ins Each instance object in the set is normalized to obtain a normalized set of instance objects. And the size of each object: scale = {sc1, sc2, ..., sc n} and the original coordinates coord={co1, co2, …, co n}; where sc n Co represents the size of the nth object. n This represents the original coordinates of the nth object. Indicates the norm n A normalized instance object, norm n Indicates the number of normalized instance objects;

[0027] Furthermore, the normalization operation is as follows:

[0028] From the initial collection of instance objects I ins Take any object I∈R m×3 Where m represents the number of points in object I, and the minimum value of each coordinate axis in its XYZ coordinate system is used as the original coordinate (x... min y min , z min The calculation method is as follows:

[0029] x min =min(I[0,:])

[0030] y min =min(I[1,:])

[0031] z min =min(I[2,:])

[0032] Where min represents the minimum value among all data in the statistical matrix, I[0,:] represents the x-axis coordinate of all points in object I, I[1,:] represents the y-axis coordinate of all points in object I, and I[2,:] represents the z-axis coordinate of all points in object I;

[0033] Then, based on the statistical values, the object's coordinates (x, y, z) are translated to the origin. The calculation method is as follows:

[0034] x = I[0,:] - x min

[0035] y = I[1,:] - y min

[0036] z = I[2,:] - z min

[0037] Next, calculate the maximum value across all coordinate axes for the translated object and use it as the object size. Then, scale the object to the range [0, 1]. The calculation method is as follows:

[0038] sc = amax(x, y, z)

[0039] x norm =x / sc

[0040] y norm =y / sc

[0041] z norm =z / sc

[0042] Where, amax represents the maximum element among all input matrices; x norm Represents the scaled x-axis coordinates and y-axis coordinates of all points in object I. norm Represents the scaled y-axis coordinates of all points in object I, z-axis coordinates. norm This represents the scaled z-axis coordinates of all points in object I;

[0043] The scaled object is the normalized object, and we obtain the object's size sc and its original coordinates co = [x] min y min , z min ]; For the initial set of instance objects I ins This process is performed on each object to obtain a normalized set of instance objects I. norm .

[0044] Step 2-2: Train the completion network for each normalized instance object separately, and fit the latent vector Z = {z1, z2, ..., z} of each object. n} and generator model G = {g1, g2, ..., g n};

[0045] Furthermore, the aforementioned process of training the completed network includes the following steps:

[0046] Step 2-2-1, Input normalized object I g ∈R m×3 Input the number of iterations epoch, epoch > 50, input the number of retained points k, k > 0, and randomly initialize the latent vector z and generator g;

[0047] Step 2-2-2: Input the latent vector z into the generator g to generate the point cloud T;

[0048] Step 2-2-3, from normalized object I g Choose any point i I ∈R 3 Calculate the distance between it and all points in the point cloud T, and obtain the index idx of the k smallest points. The calculation method is as follows:

[0049] idx = argmin(sum(pow(i I -T,2),1),k)

[0050] Among them, pow(i I -T,2) indicates that for i I Each element of the -T matrix is ​​squared, sum(pow, 1) represents summing the second dimension of the result of the operation on pow, and argmin(sum, k) represents taking the index of the first k minimum values ​​of the result of the operation on sum.

[0051] Step 2-2-4, for normalized object I g Repeat steps 2-2-3 for each point in the cloud T. After obtaining the indices of all points, merge them and select a subset T from the point cloud T based on the merged indices. map ;

[0052] Step 2-2-5, calculate subset T map and normalized object I g Chamfer distance d between CD ;

[0053] Furthermore, the Chamfer distance d CD The specific calculation method is as follows:

[0054]

[0055] in, This indicates that the Euclidean distance between two points x and y is calculated, min indicates that the minimum distance is selected, and Σ indicates that the summation is performed over all elements.

[0056] Step 2-2-6, set the Chamfer distance d CD The latent vector z and generator g are fed back as losses, and gradient descent is used to update the parameters. Steps 2-2-2 to 2-2-6 are repeated epoch times to obtain the optimized latent vector z and generator g.

[0057] Step 2-2-7: Perform the above operation on all normalized objectives to obtain the latent vector Z = {z1, z2, ..., z} of each object.n} and generator model G = {g1, g2, ..., g n}

[0058] Step 2-3: The latent vectors Z = {z1, z2, ..., zn} of each object obtained in Step 2-2 are... n} Input the corresponding generator model G = {g1, g2, ..., g} respectively n}, generate the completed instance object in, Indicates the comp-th n A completed instance object, comp n This indicates the number of instances after completion.

[0059] Step 3, Scene Point Cloud Assembly: Based on the object size and coordinates saved during instance object normalization, the completed instance objects are restored and assembled to obtain the completed garden point cloud.

[0060] Furthermore, the specific process of assembling the scene point cloud includes the following steps:

[0061] Step 3-1: Use the dimensions scale = {sc1, sc2, ..., sc...} of each object obtained in Step 2-1. n} and the original coordinates coord={co1, co2, …, co n The dimensions and coordinates of each completed object are restored to obtain the restored completed object. Among them, sc n Co represents the size of the nth object. n This represents the original coordinates of the nth object. Indicates the fc-th n A restored and completed object, fc n Indicates the number of objects to be completed after restoration;

[0062] Furthermore, the method for restoring the dimensions and coordinates is as follows:

[0063] For the completed instance object I comp Any object in Utilizing its size sc i and original coordinates co i Restore it to the restored completion object The calculation method is as follows:

[0064]

[0065] For the completed instance object I comp Perform the above operations on all objects.

[0066] Step 3-2, By assembling all the objects in the image, a complete point cloud of the garden scene is obtained. comp ∈R N×3 .

[0067] Beneficial effects:

[0068] 1. To address the problem of geometric completion of point clouds in large-scale garden scenes, this invention proposes a semantic segmentation-based method. After segmenting the garden semantics into multiple instance objects, the complete scene is obtained by completing each individual object.

[0069] 2. This invention uses an unsupervised method for object point cloud completion. Since there is no complete object point cloud as supervision in the garden scene, the use of an unsupervised method can avoid the time and money costs caused by manual completion and annotation. Attached Figure Description

[0070] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0071] Figure 1 This is a schematic diagram of the processing flow of the present invention.

[0072] Figure 2 It is the visualization result of the input garden point cloud.

[0073] Figure 3 It is a visualization of the before and after completion of a certain instance object.

[0074] Figure 4 This is the final visualization result of the garden scene after geometric completion. Detailed Implementation

[0075] This invention uses a semantic segmentation-based method to perform geometric completion on large-scale garden scenes. First, the garden scene is semantically segmented and semantic intra-class clustered to divide the scene into multiple instance objects. Then, each object is completed using an unsupervised object point cloud completion method. Finally, the completed objects are assembled to obtain a complete geometrically completed scene.

[0076] This invention proposes a geometric completion method for point clouds in large-scale garden scenes based on semantic segmentation, comprising the following steps:

[0077] Step 1, Scene point cloud semantic segmentation: Use the point cloud semantic segmentation network trained on the garden point cloud semantic segmentation dataset to perform semantic segmentation on the input garden point cloud, and then obtain instance objects by performing intra-class clustering on each semantic category.

[0078] Step 2, Object Point Cloud Completion: Normalize the instance objects and use them to complete the network training, fit the latent vector and generator corresponding to a single object, and further obtain the completed instance objects;

[0079] Step 3: Scene point cloud assembly. Based on the object size and coordinates saved during instance object normalization, the completed instance objects are restored and assembled to obtain the completed garden point cloud.

[0080] The scene point cloud semantic segmentation process in step 1 includes the following steps:

[0081] Step 11: Train a semantic segmentation network using a garden point cloud semantic segmentation dataset to obtain a semantic segmentation network F capable of processing large-scale garden scene point clouds. s ;

[0082] Step 12, using the semantic segmentation network F s Semantic segmentation is performed on the input point cloud of the garden scene to obtain segmentation results for multiple semantic categories;

[0083] Step 13: Perform intra-class clustering on the point clouds of multiple semantic categories to obtain instance-level object point clouds I. ins ={I1, I2, ..., I n}, where n is the number of instance objects.

[0084] The semantic segmentation network used in steps 11 and 12 adopts the network described in reference 5: Xie E, Wang W, Yu Z, et al. SegFormer: Simple and efficient design for semantic segmentation with transformers[J]. Advances in Neural Information Processing Systems, 2021, 34: 12077-12090. The input garden scene point cloud P∈R N×6 Output semantic segmentation result S sem ={S1, S2, ..., S cls} Where N is the number of points, each point contains XYZ coordinate information and RGB color information, totaling 6 dimensions, and cls is the number of semantic categories in the garden scene. i∈[1,cls] represents the point cloud of the i-th semantic category, N i Let be the number of point clouds for the i-th semantic category.

[0085] Step 13, the intra-class clustering process, includes the following steps:

[0086] Step 131, input semantic segmentation result S sem={S1, S2, ..., S cls}, neighborhood radius eps, eps>0, minimum number of neighborhood samples minPts, minPts>1.

[0087] Step 132, if the semantic segmentation result S to be clustered sem =Φ, jump to step 138, where Φ represents the empty set; otherwise, from S sem Take any point cloud S = {s1, s2, ..., s} t}, s i ∈R 3 Update S sem =S sem -{S}. Each point in the point cloud to be clustered only requires 3D XYZ coordinate information;

[0088] Step 133: Initialize the current category label K = 0, initialize the queue to be processed L = {}, and initialize the point-by-point category C = {c1, c2, ..., c}. n}, c i =-1;

[0089] Step 134: If the point cloud to be clustered, S = Φ, jump to step 132, where Φ represents the empty set; otherwise, randomly select a point s from S and update S = S - {s}. Calculate the Euclidean distance of s to all other points, and select points whose distance is less than the neighborhood radius eps as the neighborhoods of s. If |Neighbors| < minPts, repeat step 134; otherwise, proceed to step 135. Here, |·| represents the number of elements in the statistical set ·.

[0090] Step 135: Update K = K+1, update the category label c = K of s, and update the queue to be processed L = Neighbors;

[0091] Step 136: If the queue to be processed L = Φ, return to step 134; otherwise, take a point p from the queue to be processed L, update the queue L = L - {p}, and if the category label c of p is... p =-1, then update c p =K, proceed to step 137; if p's category label c p If the value is not equal to -1, then repeat step 136.

[0092] Step 137: Calculate the Euclidean distances of p to all other points, and select points whose distances are less than the neighborhood radius eps as the neighborhoods of p. p If |Neighbors p If |≥minPts, then update queue L=L∪Neighbors pOtherwise, no action will be taken. Return to step 136 to continue execution;

[0093] Step 138: For all semantic categories, the points with cluster label c > 0 are segmented according to the cluster label to obtain individual instance objects. The instance objects of all semantic categories are merged to obtain the set I of all instance objects after semantic segmentation of the garden point cloud. ins ={I1, I2, ..., I n}

[0094] Step 2, the object point cloud completion process, includes the following steps:

[0095] Step 21: Perform normalization operations on each instance object to obtain a normalized set of instance objects I. norm ={I1, I2, ..., I n}, and the size of each object: scale = {sc1, sc2, ..., sc n} and the original coordinates coord={co1, co2, …, co n};

[0096] Step 22: Train the completion network for each normalized instance object separately, and fit the latent vector Z = {z1, z2, ..., z} of each object. n} and generator model G = [g1, g2, ..., g n};

[0097] Step 23, take the latent vector Z = {z1, z2, ..., zn} of each object obtained in step 22. n} Input the corresponding generator model G = {g1, g2, ..., g} respectively n}, generate the completed instance object I comp ={I1, I2, ..., I n}

[0098] The instance object normalization operation in step 21 is as follows:

[0099] From the initial collection of instance objects I ins Take any object I∈R m×3 The minimum value of each coordinate axis in its XYZ coordinate system is used as the original coordinate. The calculation method is as follows:

[0100] x min =min(I[0,:])

[0101] y min =min(I[1,:])

[0102] z min =min(I[2,:])

[0103] Where min represents the minimum value among all data in the statistical matrix.

[0104] Then, based on the statistical values, the object's coordinates are shifted back to the origin. The calculation method is as follows:

[0105] x = I[0,:] - x min

[0106] y = I[1,:] - y min

[0107] z = I[2,:] - z min

[0108] Next, calculate the maximum value across all coordinate axes for the translated object and use it as the object size. Then, scale the object to the range [0, 1]. The calculation method is as follows:

[0109] sc = amax(x, y, z)

[0110] x norm =x / sc

[0111] y norm =y / sc

[0112] z norm =z / sc

[0113] Here, amax represents the maximum value element among all matrices in the statistical input.

[0114] The scaled object is the normalized object, and we obtain the object's size sc and its original coordinates co = [x] min y min , z min ] For the initial collection of instance objects I ins This process is performed on each object to obtain a normalized set of instance objects I. norm .

[0115] The completion network in step 22 adopts the network described in reference 4: Zhang J, Chen X, Cai Z, et al. Unsupervised 3dshape completion through gan inversion[C] / / Proceedings of the IEEE / CVFConference on ComputerVision and Pattern Recognition.2021:1768-1777. The difference between this and the original paper is that the garden scene cannot be pre-trained to obtain the generator and discriminator as priors. This invention removes the discriminator and completes the network training by jointly optimizing the latent vector and the generator. The training process includes the following steps:

[0116] Step 221, Input the normalized object I∈R m×3 Input the number of iterations epoch, epoch > 50, input the number of retained points k, k > 0, and randomly initialize the latent vector z and generator g;

[0117] Step 222: Input the latent vector z into the generator g to generate the point cloud T;

[0118] Step 223, randomly select a point i∈R from I. 3 Calculate the distance between it and all points in T, and obtain the indices of the k points with the smallest distances. The calculation method is as follows:

[0119] idx=argmin(sum(pow(iT, 2), 1), k)

[0120] Here, pow(iT, 2) represents the result of calculating the square of each element of the iT matrix, sum(pow, 1) represents the summation of the second dimension of the result of pow, and argmin(sum, k) represents the index of the first k minimum values ​​of the result of sum.

[0121] Step 224: Repeat step 223 for each point in I, merge all the indices, and select a subset T from T based on the indices. map ;

[0122] Step 225, calculate T map The Chamfer distance between I and I is calculated as follows:

[0123]

[0124] in, This indicates that the Euclidean distance between x and y is calculated, min indicates that the minimum distance is selected, and ∑ indicates that the summation is performed over all elements.

[0125] Step 226, d CD The latent vector z and generator g are fed back as losses, and gradient descent is used to update the parameters. Steps 222 to 226 are repeated epoch times to obtain the optimized latent vector z and generator g.

[0126] Step 227: Perform the above operation on all normalized objectives to obtain the latent vector Z = {z1, z2, ..., z} of each object. n} and generator model G = {g1, g2, ..., g n}

[0127] Step 3, the scene point cloud assembly process, includes the following steps:

[0128] Step 31, use the dimensions of each object obtained in step 21: scale = {sc1, sc2, ..., sc n} and the original coordinates coord={co1, co2, …, co n The dimensions and coordinates of each completed object are restored to obtain the restored completed object I. fc ={I1, I2, ..., I n};

[0129] Step 32, I fc ={I1, I2, ..., I n All objects are assembled to obtain the complete point cloud P of the garden scene. comp ∈R N×3 .

[0130] The object restoration method in step 31 is as follows:

[0131] For any object I, it is reconstructed using its size sc and original coordinates co, calculated as follows:

[0132] I = I * sc + co

[0133] Example:

[0134] One embodiment of the present invention is as follows: Figure 1 As shown, the present invention discloses a method for geometric completion of point clouds in large-scale garden scenes based on semantic segmentation, which specifically includes the following steps:

[0135] Step 1, Scene point cloud semantic segmentation: Use the point cloud semantic segmentation network trained on the garden point cloud semantic segmentation dataset to perform semantic segmentation on the input garden point cloud, and then obtain instance objects by performing intra-class clustering on each semantic category.

[0136] Step 2, Object Point Cloud Completion: Normalize the instance objects and use them to complete the network training, fit the latent vector and generator corresponding to a single object, and further obtain the completed instance objects;

[0137] Step 3: Scene point cloud assembly. Based on the object size and coordinates saved during instance object normalization, the completed instance objects are restored and assembled to obtain the completed garden point cloud.

[0138] The scene point cloud semantic segmentation process in step 1 includes the following steps:

[0139] Step 11: Train a semantic segmentation network using a garden point cloud semantic segmentation dataset to obtain a semantic segmentation network F capable of processing large-scale garden scene point clouds. s ;

[0140] Step 12, using the semantic segmentation network F s Semantic segmentation is performed on the input point cloud of the garden scene to obtain segmentation results for multiple semantic categories;

[0141] Step 13: Perform intra-class clustering on the point clouds of multiple semantic categories to obtain instance-level object point clouds I. ins ={I1, I2, ..., I n}, where n is the number of instance objects.

[0142] The semantic segmentation network used in steps 11 and 12 adopts the network described in reference 5: Xie E, Wang W, Yu Z, et al. SegFormer: Simple and efficient design for semantic segmentation with transformers[J]. Advances in Neural Information Processing Systems, 2021, 34: 12077-12090. The input garden scene point cloud P∈R N×6 ,like Figure 2 The point cloud of the garden scene is used to output the semantic segmentation result S. sem ={S1, S2, ..., S cls} Where N is the number of points, each point contains XYZ coordinate information and RGB color information, totaling 6 dimensions, and cls is the number of semantic categories in the garden scene. Let N be the point cloud of the i-th semantic category. i Let be the number of point clouds for the i-th semantic category.

[0143] Step 13, the intra-class clustering process, includes the following steps:

[0144] Step 131, input semantic segmentation result S sem ={S1, S2, ..., S cls}, neighborhood radius eps, minimum number of neighborhood samples minPts.

[0145] Step 132, if the semantic segmentation result S to be clustered sem =Φ, jump to step 138, where Φ represents the empty set; otherwise, from S sem Take any point cloud S = {s1, s2, ..., s} t}, s i ∈R 3 Update S sem =S sem -{S}. Each point in the point cloud to be clustered only requires 3D XYZ coordinate information;

[0146] Step 133: Initialize the current category label K = 0, initialize the queue to be processed L = {}, and initialize the point-by-point category C = {c1, c2, ..., c}. n}, c i =-1;

[0147] Step 134: If the point cloud to be clustered, S = Φ, jump to step 132, where Φ represents the empty set; otherwise, randomly select a point s from S and update S = S - {s}. Calculate the Euclidean distance of s to all other points, and select points whose distance is less than the neighborhood radius eps as the neighborhoods of s. If |Neighbors| < minPts, repeat step 134; otherwise, proceed to step 135. Here, |·| represents the number of elements in the statistical set ·.

[0148] Step 135: Update K = K+1, update the category label c = K of s, and update the queue to be processed L = Neighbors;

[0149] Step 136: If the queue to be processed L = Φ, return to step 134; otherwise, take a point p from the queue to be processed L, update the queue L = L - {p}, and if the category label c of p is... p =-1, then update c p =C, proceed to step 137; if the category label of p is c p If the value is not equal to -1, then repeat step 136.

[0150] Step 137: Calculate the Euclidean distances of p to all other points, and select points whose distances are less than the neighborhood radius eps as the neighborhoods of p. p If |Neighbors p If |≥minPts, then update queue L=L∪Neighbors p Otherwise, no action will be taken. Return to step 136 to continue execution;

[0151] Step 138: For all semantic categories, the points with cluster label c > 0 are segmented according to the cluster label to obtain individual instance objects. The instance objects of all semantic categories are merged to obtain the set I of all instance objects after semantic segmentation of the garden point cloud. ins ={I1, I2, ..., I n}

[0152] Step 2, the object point cloud completion process, includes the following steps:

[0153] Step 21: Perform normalization operations on each instance object to obtain a normalized set of instance objects I. norm ={I1, I2, ..., I n}, and the size of each object: scale = {sc1, sc2, ..., sc n} and the original coordinates coord={co1, co2, …, co n};

[0154] Step 22: Train the completion network for each normalized instance object separately, and fit the latent vector Z = {z1, z2, ..., z} of each object. n} and generator model G = {g1, g2, ..., g n};

[0155] Step 23, take the latent vector Z = {z1, z2, ..., zn} of each object obtained in step 22. n} Input the corresponding generator model G = {g1, g2, ..., g} respectively n}, generate the completed instance object I comp ={I1, I2, ..., I n} Figure 3 It is a visualization of the before and after completion of a certain instance object.

[0156] The instance object normalization operation in step 21 is as follows:

[0157] From the initial collection of instance objects I ins Take any object I∈R m×3 The minimum value of each coordinate axis in its XYZ coordinate system is used as the original coordinate. The calculation method is as follows:

[0158] x min =min(I[0,:])

[0159] y min =min(I[1,:])

[0160] z min =min(I[2,:])

[0161] Where min represents the minimum value among all data in the statistical matrix.

[0162] Then, based on the statistical values, the object's coordinates are shifted back to the origin. The calculation method is as follows:

[0163] x = I[0,:] - x min

[0164] y = I[1,:] - y min

[0165] z = I[2,:] - z min

[0166] Next, calculate the maximum value across all coordinate axes for the translated object and use it as the object size. Then, scale the object to the range [0, 1]. The calculation method is as follows:

[0167] sc = amax(x, y, z)

[0168] x norm =x / sc

[0169] y norm =y / sc

[0170] z norm =z / sc

[0171] Here, amax represents the maximum value element among all matrices in the statistical input.

[0172] The scaled object is the normalized object, and we obtain the object's size sc and its original coordinates co = [x] min y min , z min ] For the initial collection of instance objects I ins This process is performed on each object to obtain a normalized set of instance objects I. norm .

[0173] The completion network in step 22 adopts the network described in reference 4: Zhang J, Chen X, Cai Z, et al. Unsupervised 3dshape completion through gan inversion[C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition.2021:1768-1777. The difference between this and the original paper is that the generator and discriminator cannot be pre-trained as priors for the garden scene. This invention removes the discriminator and completes the network training by jointly optimizing the latent vector and the generator. The training process includes the following steps:

[0174] Step 221, Input the normalized object I∈R m×3 Input the number of iterations (epoch), the number of retained points (k), and randomly initialize the latent vector z and the generator g;

[0175] Step 222: Input the latent vector z into the generator g to generate the point cloud T;

[0176] Step 223, randomly select a point i∈R from I. 3 Calculate the distance between it and all points in T, and obtain the indices of the k points with the smallest distances. The calculation method is as follows:

[0177] idx=argmin(sum(pow(iT, 2), 1), k)

[0178] Here, pow(iT, 2) represents the result of calculating the square of each element of the iT matrix, sum(pow, 1) represents the summation of the second dimension of the result of pow, and argmin(sum, k) represents the index of the first k minimum values ​​of the result of sum.

[0179] Step 224: Repeat step 223 for each point in I, merge all the indices, and select a subset T from T based on the indices. map ;

[0180] Step 225, calculate T map The Chamfer distance between I and I is calculated as follows:

[0181]

[0182] in, This indicates that the Euclidean distance between x and y is calculated, min indicates that the minimum distance is selected, and ∑ indicates that the summation is performed over all elements.

[0183] Step 226, d CD The latent vector z and generator g are fed back as losses, and gradient descent is used to update the parameters. Steps 222 to 226 are repeated epoch times to obtain the optimized latent vector z and generator g.

[0184] Step 227: Perform the above operation on all normalized objectives to obtain the latent vector Z = {z1, z2, ..., z} of each object. n} and generator model G = {g1, g2, ..., g n}

[0185] Step 3, the scene point cloud assembly process, includes the following steps:

[0186] Step 31, use the dimensions of each object obtained in step 21: scale = {sc1, sc2, ..., sc n} and the original coordinates coord={co1, co2, …, co n The dimensions and coordinates of each completed object are restored to obtain the restored completed object I. fc ={I1, I2, ..., I n};

[0187] Step 32, I fc ={I1, I2, ..., I n All objects are assembled to obtain the complete point cloud P of the garden scene.comp ∈R N×3 . Figure 4 This is the final visualization result of the garden scene after geometric completion.

[0188] The object restoration method in step 31 is as follows:

[0189] For any object I, it is reconstructed using its size sc and original coordinates co, calculated as follows:

[0190] I = I * sc + c

[0191] In its specific implementation, this application provides a computer storage medium and a corresponding data processing unit. The computer storage medium is capable of storing a computer program, which, when executed by the data processing unit, can run the invention's content regarding a method for geometric completion of point clouds in large-scale garden scenes based on semantic segmentation, as well as some or all of the steps in various embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0192] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented using computer programs and their corresponding general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of computer programs, i.e., software products. These computer program software products can be stored in a storage medium and include several instructions to cause a device containing a data processing unit (which may be a personal computer, server, microcontroller, MUU, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0193] This invention provides an approach and method for geometric completion of point clouds in large-scale garden scenes based on semantic segmentation. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A semantic segmentation-based large-scale garden scene point cloud geometry completion method, characterized in that, Comprising the following steps: Step 1, scene point cloud semantic segmentation: using the point cloud semantic segmentation network pre-trained on the garden point cloud semantic segmentation dataset to perform semantic segmentation on the input garden scene point cloud, and then performing intra-class clustering on each semantic class to obtain instance objects; Step 2, object point cloud completion: after normalizing the instance objects, training the completion network, fitting to obtain the latent vector and generator corresponding to the single instance object, and further obtaining the completed instance object using the latent vector and the generator; Step 3, scene point cloud assembly: according to the object size and coordinates saved during the normalization of the instance objects, restoring and assembling the completed instance objects to obtain the completed garden point cloud; The object point cloud completion of step 2 comprises the following steps: Step 2-1, for the collection of instance objects Each instance object in the set is normalized to obtain a normalized set of instance objects. and the dimensions of each object. and original coordinates ;in, Indicates the first The size of the object, Indicates the first The original coordinates of the objects Indicates the first A normalized instance object, Indicates the number of normalized instance objects; Step 2-2, respectively, for each normalized instance object, the completion network training, fitting to get each object latent vector and generator model The specific process includes the following steps: Step 2-2-1, input normalizing instance object , input iteration number , , input number of reserved points , , randomly initialize latent vector and generator ; Step 2-2-2, to generate a latent vector input generator generating a point cloud ; Step 2-2-3, from any point in , calculate its distance with all points in the point cloud , and get the index of the minimum points , the calculation is as follows: ； wherein, represents the operation result of each element of the matrix, represents the operation result of the second dimension of the operation result of represents the operation result of taking the first minimum value of the operation result of Step 2-2-4, to Step 2-2-3 is repeated for each point, and the indices of all points are merged after being obtained, and a subset is selected from the point cloud ;​ Step 2-2-5, calculating the subset and Chamfer distance between ; Step 2-2-6, calculate Chamfer distance Back to the latent vector as loss respectively And the generator , gradient descent update parameters, repeat steps 2-2-2 to step 2-2-6 Times, get the optimized latent vector And the generator ; Step 2-2-7, the above operation is performed on all normalized objects to obtain the latent vector of each object and generator model ; Step 2-3, obtaining the potential vector of each object in step 2-2 Input the corresponding generator model respectively , to generate the completed instance object ; wherein, represents the th completed instance object, represents the number of completed instance objects; The scene point cloud assembly of step 3 comprises the following steps: Step 3-1, using the size of each object obtained in Step 2-1 and the original coordinates restore the size and coordinates of each completed object to obtain the restored completed object ; wherein, denotes the size of the th object, denotes the original coordinates of the th object, denotes the restored completed object of the th object, denotes the number of restored completed objects. Step 3-2, assembling all the objects to obtain the completed garden scene point cloud all the objects to obtain the completed garden scene point cloud .

2. The method of claim 1, wherein, The scene point cloud semantic segmentation of step 1 comprises the following steps: Step 1-1, semantic segmentation network training is performed using the garden point cloud semantic segmentation dataset to obtain a semantic segmentation network ; Step 1-2, using a semantic segmentation network The input garden scene point cloud is subjected to semantic segmentation to obtain segmentation results of different semantic categories. Step 1-3, respectively, the point cloud of each semantic category is clustered within the class, and the instance-level object point cloud is obtained wherein is the number of instance objects, is the instance object.

3. The method of claim 2, wherein, the semantic segmentation network described in step 1-1 as follows: Using an existing supervised semantic segmentation method for processing scene point clouds, inputting garden scene point clouds wherein represents a real number field of dimension , the number of points of the garden scene point cloud, each point containing XYZ axis coordinate information and RGB color information, a total of 6 dimensions, and outputting a semantic segmentation result wherein represents the th semantic segmentation result , the number of semantic categories of the garden scene, the th semantic category point cloud , the number of points in the th semantic category point cloud.

4. The method of claim 3, wherein, The intra-class clustering of steps 1-3 comprises the following steps: Step 1-3-1, input semantic segmentation result Neighborhood radius , Minimum number of neighborhood samples , ; Step 1-3-2, if the semantic segmentation result of the clustering to be performed... Then proceed to steps 1-3-8, where Represent the empty set; otherwise, use the semantic segmentation results from the clustering process. Choose any point cloud ,in, Point cloud The first in One point, For point clouds Number of midpoints; Update During the update, each point in the point cloud to be clustered uses only 3D XYZ axis coordinate information. Step 1-3-3, initialize current class label , initialize queue to be processed , initialize point-wise class where denotes the class of the th point, is the number of points in the point cloud . Step 1-3-4, if the point cloud to be clustered , execute step 1-3-2, wherein represents an empty set; otherwise, take a point from the point cloud , , update ; calculate the Euclidean distance of the point from all other points, and select points with a distance less than the neighborhood radius from the point as the neighborhood of the point ; if , re-execute step 1-3-4, otherwise execute step 1-3-5, wherein represents the number of elements in the statistical set· Step 1-3-5, update , update point of the category label , update pending queue ; Steps 1-3-6, if the queue to be processed If the condition is met, return to steps 1-3-4; otherwise, remove the process from the pending queue. Take a little from Update queue If point Category tags Then update If yes, proceed to step 1-3-7; otherwise, repeat step 1-3-6. Steps 1-3-7, calculate points Euclidean distances to all other points, and selecting those with distances less than the neighborhood radius. The point is used as a point neighborhood ;like Then update the queue. Otherwise, no operation will be performed, and the process will return to step 136 to continue. Step 1-3-8, for all semantic categories, the points with cluster labels greater than 0 are divided according to the cluster labels to obtain single instance objects, and the instance objects of all semantic categories are combined to obtain a set of all instance objects after semantic segmentation of the garden scene point cloud .

5. The method of claim 4, wherein, The normalization operation of step 2-1 is as follows: from the initial set of instance objects any object wherein represents the number of points of the object , the minimum value of each coordinate axis in the XYZ axis coordinates of the points is counted as the original coordinate , and the calculation method is as follows: ； ； ； wherein, represents the minimum value in all data of the statistical matrix, represents the object all points of axis coordinates, represents the object all points of axis coordinates, represents the object all points of axis coordinates; The object coordinates are then translated to the origin according to the statistical values in the following way: ； ； ； Then, the maximum value in all coordinate axes of the translated object coordinates is calculated as the object size, and the object is scaled to between them, the calculation is as follows: ； ； ； ； wherein, represents the maximum value element in all the inputted matrices; represents the object all points after scaling axis coordinates, represents the object all points after scaling axis coordinates, represents the object all points after scaling axis coordinates; The scaled object is the normalized object, and the object size and the original coordinates of the object ; for each object in the initial instance object set , the process is performed to obtain the normalized instance object set .

6. The method of claim 5, wherein, The size and coordinate restoration of step 3-1 is as follows: For any of the completed instance objects , the completed instance object is restored to a restored completed instance object using its dimensions and original coordinates in the following manner: ； All objects in the completed instance object are operated on as above.

7. The method of claim 6, wherein, Chamfer distance as described in steps 2-2-5 The specific calculation is as follows: ； wherein, represents calculating the Euclidean distance between two points , represents selecting the smallest distance, represents summing over all elements.

Images