Patch segmentation-based point cloud edge reconstruction method

Abstract

The invention discloses a surface patch segmentation-based point cloud edge reconstruction method. The method comprises the following steps of 1, performing surface patch segmentation, boundary point detection and normal prediction on a three-dimensional point cloud by using a point cloud edge reconstruction network; 2, for each pair of adjacent boundary point pairs belonging to different patches, generating a straight line segment formed by two points and formed by intersecting local patches; 3, the generated straight line segments are used for fitting point cloud edges, including three curve edges of a straight line, a circle and a B-spline curve; step 4, detecting angular points of each plane by utilizing the segmentation surface patches, and obtaining an initial guess wireframe with a topological structure by combining end points of previous fitting edges; and step 5, optimizing the initial guess wireframe by using the fitting edge to obtain a final point cloud edge result.

Description

Technical Field

[0001] The invention relates to the field of computer graphics, and in particular to a three-dimensional point cloud edge reconstruction method based on facet segmentation. Background Art

[0002] Object edges are important structural features represented by 3D modeling software models, especially for the many sharp edges contained in man-made objects. Reconstructing high-quality model edges has extremely important application value in virtual reality, digital twins and other fields. With the rapid development of hardware equipment, LiDAR and other equipment are becoming more and more popular, and the acquisition of 3D point cloud data is becoming easier and easier. Therefore, it is necessary to develop a reconstruction method that can automatically detect the edges in the scanned point cloud data and represent them parametrically.

[0003] Some previous methods fit a parametric curve after classifying each point in the point cloud, but the number of points that can be accurately sampled on the edge is too small, which leads to an increase in fitting errors and does not fully utilize the information provided by a large number of non-boundary points. Other methods use neural networks to predict the distance from each point to its nearest edge, but they require too dense input points to accurately predict this distance in order to obtain candidate boundary points for reliable edge reconstruction.

[0004] The boundary points of an object are usually located at the intersection of adjacent facets, so edge detection can be promoted by facet segmentation. Therefore, a facet segmentation-based edge reconstruction method can be designed to accurately obtain the edge from the input scanning point cloud that does not carry normal information, which is more conducive to the construction of a high-precision parameterized edge model. Summary of the invention

[0005] The purpose of the invention is to provide a point cloud edge reconstruction method based on patch segmentation in order to address the shortcomings of existing point cloud edge detection technology.

[0006] The present invention discloses a point cloud edge reconstruction method based on facet segmentation, comprising the following steps:

[0007] Step 1: Use the point cloud edge reconstruction network to perform face segmentation, boundary point detection and normal prediction on the 3D point cloud;

[0008] Step 2: For each pair of adjacent boundary points belonging to different patches, generate a straight line segment where the local patches constructed by the two points intersect;

[0009] Step 3: Use the generated straight line segments to fit the edge of the point cloud to obtain fitting edges, including three types of curve edges: straight line, circle, and B-spline curve;

[0010] Step 4: Use the segmented face to detect the corner points of each plane, and combine them with the endpoints of the fitted edge in step 3 to obtain an initial guess wireframe with a topological structure;

[0011] Step 5: Use the fitted edge to optimize the initial guess wireframe to obtain the final point cloud edge result.

[0012] The point cloud edge reconstruction network is composed of a dynamic graph convolutional neural network, a point cloud encoder, a point cloud decoder for edge features, a point cloud decoder for segmentation features, a normal decoder and a normal encoder;

[0013] The point cloud encoder is a deep learning encoder for extracting point cloud features;

[0014] The point cloud decoder for edge features is a deep learning decoder that obtains point cloud edge features through convolution operations;

[0015] The point cloud decoder for segmentation features is a deep learning decoder that obtains point cloud segmentation features through convolution operations;

[0016] The normal encoder is a deep learning encoder that extracts normal features of point clouds through convolution operations;

[0017] The normal decoder is a deep learning decoder that restores the feature representation generated by the encoder to the target normal output.

[0018] The step 1 comprises:

[0019] Step 1-1: Use the dynamic graph convolutional neural network in the point cloud edge reconstruction network to extract features from the input original 3D point cloud through the point cloud encoder, and then input the extracted features into the point cloud decoder for edge features and the point cloud decoder for segmentation features to obtain point cloud edge features and point cloud segmentation features respectively;

[0020] Step 1-2: A normal decoder is used to predict the normal of the input 3D point cloud, and then a normal encoder is used to extract the normal as a normal feature, and the edge feature and the segmentation feature are fused with the normal feature in an element-wise manner to obtain the final edge fusion feature and segmentation fusion feature;

[0021] Step 1-3: For the final edge fusion feature, the edge point is obtained by decoding through the point cloud decoder. For the final segmentation fusion feature, the face segmentation result is obtained by decoding through the point cloud decoder using the mean shift clustering method.

[0022] Step 1-4: The point cloud edge reconstruction network uses a joint loss function for supervised learning, including edge point regression loss function, normal loss function and surface segmentation embedding loss function; the joint loss function L is the weighted sum of the edge point regression loss function, the normal loss function and the surface segmentation embedding loss function, and the calculation method is as follows:

[0023] L=L edge +μL normal +vL emb

[0024] Among them, μ and v are weighting coefficients, both of which are positive real numbers, and the range of values ​​used is between 0 and 5, which are used to balance the contribution of different loss terms; when μ or v is close to 0, the contribution of the corresponding normal loss or surface segmentation embedding loss will decrease, and the network may focus more on the learning of edge points. When μ or v increases, the contribution of these loss terms will increase, which may improve the performance of the corresponding task, but at the same time the learning of edge points will be weakened, so it is often necessary to select an appropriate value to achieve a balance between all loss terms. Edge point regression loss function L edge , normal loss function L normal and surface segmentation embedding loss function L emb The calculation methods are:

[0025]

[0026] Among them, for each point cloud, when point p i When located at the edge of the point cloud, the weight w of the point i =1, when point p i When in the non-edge area of ​​the point cloud, the weight N e represents the number of edge points, N represents the total number of points; the true edge label of each point is e(p i ), the predicted edge label of each point is when β represents w i The standard deviation of the marginal regression loss term of the changed point In other cases, i =|e(p i )- represents the predicted normal, n i represents the real normal; K represents the total number of faces that need to be segmented. represents the real k-th part in the point cloud, Indicates p iThe embedding vector representation of , the distance parameters δ1, δ2, respectively correspond to the lower limit of the distance of the same type of patches and the upper limit of the distance of different types of patches during segmentation, and are both positive real numbers, with a range of values ​​between 0 and 2, which are used to control the distance threshold between the embedding vectors of the segmented patches; when δ1 and δ2 are close to 0, the distance threshold between the embedding vectors will become very strict, and the network may be too sensitive, resulting in overfitting or instability, and when δ1 and δ2 increase, these distance thresholds will become loose, and the model may relax the requirements for the distance of the embedding vector, which may affect the discrimination between the segmented patches. Therefore, it is necessary to comprehensively select appropriate values ​​to achieve a balance between the embedding vector distance thresholds.

[0027] The step 2 generates a straight line segment where the local patches constructed by two points intersect. The specific method is as follows: for adjacent boundary point pairs belonging to different patches, two bottom circles perpendicular to the normal of the point are constructed respectively in combination with the normal predicted by the point cloud edge reconstruction network, and the intersection line of the pair of circles is calculated to obtain the straight line segment.

[0028] In step 3, the generated straight line segment is fitted with a straight line, a circle, and a B-spline curve in sequence using the least squares method, and the residuals of the three types of curves are calculated, and the curve type with the smallest residual is used.

[0029] In step 4, the specific method of detecting the corner points of each plane is: perform a plane check on each segmented face, and if it belongs to a plane, use the medial axis transformation to detect the corner points of the segmented face. The corner points detected on each plane and the endpoints of each previous fitted edge together constitute the corner points of the initial guess wireframe, connect all corner point pairs, and trim the redundant lines with few edge points to obtain the initial guess wireframe.

[0030] In step 5, the initial guess wireframe and fitted edge results are iteratively optimized by minimizing energy to reconstruct accurate edges and corners with complete topological connections, i.e., the final point cloud edge reconstruction result. The minimum energy function is specifically:

[0031] E=dist(WF(p),Ψ)

[0032] Where p represents the entire point cloud, WF(p) represents the initial guess wireframe of the point cloud, Ψ represents the fitted edge of the point cloud, dist(,) represents the distance between two sets of edges. The specific calculation method of the distance is to sample two point sets from the two edges respectively, calculate the chamfer distance of the two point sets, and use it as the distance between the two edges.

[0033] Beneficial effects:

[0034] 1) This method uses a point cloud edge reconstruction network to integrate normal features, edge features and segmentation features to accurately obtain the surface segmentation and boundary point results of the point cloud, and at the same time obtain the segmentation label and boundary point label of each point. This boundary reconstruction neural network has excellent characteristics such as high robustness, easy training, and low computing resource overhead.

[0035] 2) This method uses boundary point pairs to construct the local patch intersection of two points to obtain straight line segments, thereby fitting the edge curve to obtain a more realistic and complete edge.

[0036] 3) This method uses the accurate information segmented by the point cloud edge reconstruction network and combines it with the extracted corner point information to obtain the edge reconstruction result. This method can handle point clouds with complex morphology, and the reconstructed edge result can well reflect the three-dimensional topological structure of the object, thereby providing a more detailed and accurate geometric description. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Flow chart of the method of the present invention.

[0038] Figure 2 Schematic diagram of the structure of the point cloud edge reconstruction network.

[0039] Figure 3 It is the input point cloud image of the embodiment of this method.

[0040] Figure 4 is the reconstructed edge map of the embodiment of the method. DETAILED DESCRIPTION

[0041] In deep learning, the main task of the encoder is to convert the input data into a latent and highly representative feature representation. It is a feature extraction module that can capture the pattern, context and hidden information of the data. The encoder accepts the original input (such as images, text, graph structure data, etc.), extracts features layer by layer, and finally represents it as a vector of fixed length or structure (i.e. latent representation).

[0042] The task of the decoder is to restore the target output (such as the translated sentence, predicted label, generated image, etc.) based on the feature representation (Z) generated by the encoder. The decoder can be regarded as a generation module. The decoder accepts the feature representation output by the encoder and gradually decodes the generated result according to the target task.

[0043] Dynamic Graph Convolutional Neural Network (DGCNN) is an innovative algorithm for processing and analyzing point cloud data. It proposes a new module called EdgeConv, which is suitable for high-level tasks based on Convolutional Neural Network (CNN), such as classification and segmentation.

[0044] The present invention proposes a point cloud edge reconstruction method based on patch segmentation, which integrates dynamic graph convolutional neural network and codec. The core of the method is to use Point Cloud Edge Reconstruction Network (PCER-Net) to make full use of the information of edge points and non-edge points to perform patch segmentation and boundary point detection on three-dimensional point clouds. For each pair of adjacent boundary points belonging to different patches, a straight line segment is generated where the local patches constructed by the two points intersect. The generated straight line segment is then used to fit the edge of the point cloud, including three types of curve edges: straight line, circle, and B-spline curve. The segmented patches are used to detect the corner points of each plane, and the initial guess wireframe with a topological structure is obtained together with the endpoints of the previously fitted edges. Finally, the initial guess wireframe is optimized using the fitted edges to obtain accurate and complete point cloud edges. The method has the advantages of high precision, strong robustness, easy training, and low overhead.

[0045] The present invention extracts the edge features and segmentation features of the point cloud respectively, and integrates the normal features, making full use of the information contained in the edge points and non-edge points of the point cloud. Then the local patch intersection straight line segments constructed by the boundary point pairs are calculated, fitted as preliminary edge results, and finally corner points are added for guidance and optimized to the final reconstruction results.

[0046] Example:

[0047] like Figure 1 As shown, a point cloud edge reconstruction method based on facet segmentation specifically includes the following steps:

[0048] Step 1: Use the point cloud edge reconstruction network to perform face segmentation, boundary point detection and normal prediction on the 3D point cloud;

[0049] Step 2: For each pair of adjacent boundary points belonging to different patches, generate a straight line segment where the local patches constructed by the two points intersect;

[0050] Step 3: Use the generated straight line segment to fit the edge of the point cloud to obtain the fitting edge;

[0051] Step 4: Use the segmented face to detect the corner points of each plane, and combine them with the endpoints of the fitted edge in step 3 to obtain an initial guess wireframe with a topological structure;

[0052] Step 5: Use the fitted edge to optimize the initial guess wireframe to obtain the final point cloud edge result.

[0053] The point cloud edge reconstruction network described in step 1 simultaneously performs face segmentation, boundary point recognition and normal prediction on the 3D point cloud. The deep neural network model is called PCER-Net, and the data set used by the network includes a training set of 20,000 point clouds, a validation set of 4,000 point clouds and a test set of 4,000 point clouds, each with 10,000 points. These point clouds are all from part models and various furniture models of commonly used 3D modeling software, and each point contains the real edge label, the face category to which it belongs and the real normal.

[0054] The point cloud edge reconstruction network model structure is as follows Figure 2 As shown, the following steps are included:

[0055] Step 1-1: Use the dynamic graph convolutional neural network in the point cloud edge reconstruction network to extract features from the input original 3D point cloud through the point cloud encoder, and then input these features into the point cloud decoder for edge features and the point cloud decoder for segmentation features to obtain point cloud edge features and point cloud segmentation features respectively;

[0056] Step 1-2: A normal decoder is used to predict the normal of the input 3D point cloud, and then a normal encoder is used to extract the normal as a normal feature, and the edge feature and the segmentation feature are fused with the normal feature in an element-wise manner to obtain the final edge fusion feature and segmentation fusion feature;

[0057] Step 1-3: For the final edge fusion feature, the edge point is obtained by decoding through the point cloud decoder. For the final segmentation fusion feature, the patch segmentation result is obtained by decoding through the point cloud decoder using the mean shift clustering method.

[0058] Step 1-4: The point cloud edge reconstruction network uses a joint loss function for supervised learning, including edge point regression loss function, normal loss function and surface segmentation embedding loss function. The joint loss function L is the weighted sum of the edge point regression loss function, the normal loss function and the surface segmentation embedding loss function, and the calculation method is as follows:

[0059] L=L edge +μL normal +vL emb

[0060] Among them, μ and v are weighted coefficients, with values ​​of μ = 0.25 and v = 1. The edge point regression loss function L edge , normal loss function L normal and surface segmentation embedding loss function L emb The calculation methods are:

[0061]

[0062]

[0063] Among them, for each point cloud, when point p i When located at the edge of the point cloud, the weight w of the point i =1, when point p i When in the non-edge area of ​​the point cloud, the weight N e represents the number of edge points, N represents the total number of points; the true edge label of each point is e(p i ), the predicted edge label of each point is when β represents w i The standard deviation of In other cases, represents the predicted normal, n i represents the real normal; K represents the total number of faces that need to be segmented. represents the real k-th part in the point cloud, Indicates p i The embedding vector of is represented by , and the distance parameters δ1 = 0.5 and δ2 = 1.5 correspond to the lower limit of the distance of the same category of patches and the upper limit of the distance of different categories of patches during segmentation.

[0064] The specific method for generating the straight line segment of the intersection of the local patches constructed by two points described in step 2 is as follows: for the pair of adjacent boundary points belonging to different patches, two bottom circles perpendicular to the normal of the point are constructed respectively in combination with the predicted normal, and the intersection line of the pair of circles is calculated, which is the straight line segment.

[0065] In step 3, the least square method is used to fit the straight line segment obtained in turn to fit the straight line, circle, and B-spline curve, and the residuals of the three curve types are calculated, and the curve type with the smallest residual is used.

[0066] In step 4, a plane check is performed on each segmented face. If it belongs to a plane, the medial axis transform (MAT) is used to detect the corner points of the segmented face.

[0067] In step 4, the corner points detected on each plane and the endpoints of each previously fitted edge together constitute the corner points of the initial guess wireframe. All corner point pairs are connected, and redundant lines with few edge points are trimmed to obtain the initial guess wireframe.

[0068] In step 5, the initial guess wireframe and fitted edge results are iteratively optimized using energy minimization to reconstruct accurate edges and corners with complete topological connections, i.e., the final point cloud edge reconstruction result.

[0069] In step 5, the minimum energy function used is:

[0070] E=dist(WF(p),Ψ)

[0071] Where WF(p) represents the initial guess wireframe, Ψ represents the fitted edge, and dist(,) represents the distance between the two sets of edges. The specific calculation method of the distance is to sample two point sets of 5000 points from the two edges respectively, calculate the chamfer distance of the two point sets, and use it as the distance between the two edges.

[0072] In step 5, the final point cloud edge reconstruction result is fitted based on minimization iterative optimization.

[0073] Figure 3 and Figure 4 This is a schematic diagram of the implementation effect of this method. Figure 3 is the input point cloud, Figure 4 The generated accurate reconstruction result with edges and corners. Figure 3 The input original point cloud data is displayed, including three part models and three furniture (bed, cabinet, sofa) models. Figure 4 The reconstruction result generated by applying the reconstruction method of the present application to the corresponding model is presented, and the result accurately reflects the edge and corner details of the original point cloud. For parts, it can be observed that the generated edge lines are continuous and accurate, and the contours and internal structures of the parts are truly reproduced. In terms of furniture models, the boundaries of large objects such as beds, cabinets, and sofas are clearly visible as a whole, and complex details such as armrests, legs and feet can also be accurately extracted. In general, this method can handle point clouds with complex morphology, and the reconstructed edge results constructed can well reflect the three-dimensional topological structure of the object, thereby providing a more detailed and accurate geometric description.

[0074] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium can store a computer program, and when the computer program is executed by the data processing unit, the invention content of a point cloud edge reconstruction method based on facet segmentation provided by the present invention and some or all of the steps in each embodiment can be executed. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0075] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of computer programs and their corresponding general hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention are essentially or partly contributed to the prior art can be embodied in the form of a computer program, i.e., a software product, which can be stored in a storage medium and includes several instructions for enabling a device including a data processing unit (which can be a personal computer, a server, a single-chip microcomputer, a MUU or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0076] The present invention provides a method and idea for point cloud edge reconstruction based on face segmentation. There are many methods and approaches to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented using existing technologies.

Claims

1. A point cloud edge reconstruction method based on patch segmentation, characterized in that: The following steps are involved: Step 1: Use the point cloud edge reconstruction network to perform face segmentation, boundary point detection and normal prediction on the 3D point cloud; Step 2: For each pair of adjacent boundary points belonging to different patches, generate a straight line segment where the local patches constructed by the two points intersect; Step 3: Use the generated straight line segment to fit the edge of the point cloud to obtain the fitting edge; Step 4: Use the segmented patch to detect the corner points of each plane, and combine the endpoints of the fitted edge in step 3 to obtain an initial guess wireframe with a topological structure; Step 5: Use the fitted edge to optimize the initial guess wireframe to obtain the final point cloud edge result.

2. The point cloud edge reconstruction method based on patch segmentation according to claim 1, characterized in that: The point cloud edge reconstruction network is composed of a dynamic graph convolutional neural network, a point cloud encoder, a point cloud decoder for edge features, a point cloud decoder for segmentation features, a normal decoder and a normal encoder; The point cloud encoder is a deep learning encoder for extracting point cloud features; The point cloud decoder for edge features is a deep learning decoder that obtains point cloud edge features through convolution operations; The point cloud decoder for segmentation features is a deep learning decoder that obtains point cloud segmentation features through convolution operations; The normal encoder is a deep learning encoder that extracts normal features of point clouds through convolution operations; The normal decoder is a deep learning decoder that restores the feature representation generated by the encoder to the target normal output.

3. The point cloud edge reconstruction method based on facet segmentation according to claim 1, characterized in that: Step 1 includes: Step 1-1: Use the dynamic graph convolutional neural network in the point cloud edge reconstruction network to extract features from the input original 3D point cloud through the point cloud encoder, and then input the extracted features into the point cloud decoder for edge features and the point cloud decoder for segmentation features to obtain point cloud edge features and point cloud segmentation features respectively; Step 1-2: A normal decoder is used to predict the normal of the input 3D point cloud, and then a normal encoder is used to extract the normal as a normal feature, and the edge feature and the segmentation feature are fused with the normal feature in an element-wise manner to obtain the final edge fusion feature and segmentation fusion feature; Step 1-3: For the final edge fusion feature, the edge point is obtained by decoding through the point cloud decoder. For the final segmentation fusion feature, the face segmentation result is obtained by decoding through the point cloud decoder using the mean shift clustering method. Step 1-4: The point cloud edge reconstruction network uses a joint loss function for supervised learning, including edge point regression loss function, normal loss function and surface segmentation embedding loss function; the joint loss function L is the weighted sum of the edge point regression loss function, the normal loss function and the surface segmentation embedding loss function, and the calculation method is as follows: L=L edge +μL normal +vL emb Among them, μ and v are weighted coefficients, both of which are positive real numbers, and the range of values ​​used is between 0 and 5. The edge point regression loss function L edge , normal loss function L normal and surface segmentation embedding loss function L emb The calculation methods are: Among them, for each point cloud, when point p i When located at the edge of the point cloud, the weight w of the point i =1, when point p i When in the non-edge area of ​​the point cloud, the weight N e represents the number of edge points, N represents the total number of points; the true edge label of each point is e(p i ), the predicted edge label of each point is when β represents w i The standard deviation of the marginal regression loss term at this point In other cases, represents the predicted normal, n i represents the real normal; K represents the total number of faces that need to be segmented. represents the real k-th part in the point cloud, Indicates p i The embedding vector of is represented by the distance parameters δ1 and δ2, which correspond to the lower limit of the distance of the same type of patches and the upper limit of the distance of different types of patches during segmentation, respectively. Both of them are positive real numbers, and the range of values ​​used is between 0 and 2.

4. The point cloud edge reconstruction method based on facet segmentation according to claim 1, characterized in that: In step 2, the point cloud edges include three types of curve edges: straight line, circle, and B-spline curve.

5. The point cloud edge reconstruction method based on patch segmentation according to claim 1, characterized in that: The specific method for generating the straight line segment where the local patches constructed by two points intersect is as follows: for adjacent boundary point pairs belonging to different patches, two bottom circles perpendicular to the normal of the point are constructed respectively in combination with the normal predicted by the point cloud edge reconstruction network, and the intersection line of the pair of circles is calculated, which is the straight line segment.

6. The point cloud edge reconstruction method based on facet segmentation according to claim 1, characterized in that: In step 3, the least square method is used to fit the generated straight line segment to a straight line, a circle, and a B-spline curve in turn, and the residuals of the three curve types are calculated, and the curve type with the smallest residual is used.

7. The point cloud edge reconstruction method based on patch segmentation according to claim 1, characterized in that: In step 4, the specific method for detecting the corner points of each plane is: performing a plane check on each segmented face patch, and if it belongs to a plane, using the medial axis transformation to detect the corner points of the segmented face patch.

8. The point cloud edge reconstruction method based on patch segmentation according to claim 1, characterized in that: In step 4, the corner points detected on each plane and the endpoints of each previously fitted edge together constitute the corner points of the initial guess wireframe. All corner point pairs are connected, and redundant lines with few edge points are trimmed to obtain the initial guess wireframe.

9. The point cloud edge reconstruction method based on patch segmentation according to claim 1, characterized in that: In step 5, the initial guess wireframe and fitted edge results are iteratively optimized using energy minimization to reconstruct accurate edges and corners with complete topological connections, i.e., the final point cloud edge reconstruction result.

10. The point cloud edge reconstruction method based on facet segmentation according to claim 9, characterized in that: The minimum energy function is specifically: E=dist(WF(p),Ψ) Where p represents the entire point cloud, WF(p) represents the initial guess wireframe of the point cloud, Ψ represents the fitted edge of the point cloud, dist(,) represents the distance between two sets of edges. The specific calculation method of the distance is to sample two point sets from the two edges respectively, calculate the chamfer distance of the two point sets, and use it as the distance between the two edges.

Images