Point cloud feature recognition and labeling algorithm based on improved pointnet++

Abstract

Disclosed in the present invention are a point cloud feature recognition and labeling algorithm based on improved PointNet++. The algorithm comprises a supervoxel growth method suitable for a deep learning network, and a deep learning network suitable for large-component scan data. In the method, PointNet++ is improved, a deep learning network suitable for large-scale point cloud recognition is constructed, and point clouds in regions are processed, the interrelationship between the point clouds in the regions are also taken into consideration, so that the technical problem of conventional PointNet series networks being not capable of processing large-scale point clouds is solved. Provided is a supervoxel growth method which is suitable for determining feature parameters within voxels that are highly distinctive, computationally efficient and highly robust, so as to realize fast, efficient and meaningful voxelized point cloud calibration.

Description

A point cloud feature recognition and annotation algorithm based on improved PointNet++ Technical Field

[0001] The present invention belongs to the fields of three-dimensional data processing and deep learning, and specifically relates to a point cloud feature recognition and annotation method suitable for large components. More specifically, the present invention relates to a point cloud feature recognition and annotation algorithm based on an improved PointNet++. Background Art

[0002] Point cloud recognition based on unordered points: PointNet is a pioneer in this area. Based on the principle of order matters, this approach, designed for raw point clouds, employs a deep network that is independent of unordered input, maintains local point meaning, and is immune to rigid body transformations. This network's construction differs from convolutional networks. Its base layer relies on a multi-layer perceptron network to approximate a symmetric function acting on the entire input. For spatial transformations, a transformation network (T-Net) is introduced, and finally, a segmentation network is integrated to generate global features. This network does not rely on convolutional generalizations, but the validity of its various steps remains to be determined. Furthermore, the initial weights are equal to the number of points in the point cloud, which can be computationally prohibitive for measurement objects with large numbers of points. Compared to PointNet, PointNet++ incorporates a hierarchical structure to extract features at different scales. This is similar to a pyramid of features in an image: the receptive field increases with scale.

[0003] Point cloud simplification: There are two main methods for downsampling point clouds: uniform sampling and non-uniform sampling. As the name suggests, uniform sampling results in neatly spaced points and a uniform density, ensuring that every region is sampled. Non-uniform sampling, on the other hand, results in varying distances between points, and the sampling result for each region depends on the point density.

[0004] A supervoxel is a collection whose elements are voxels. Essentially, they are small blocks. The purpose of supervoxel clustering is not to segment specific objects. Instead, it performs over-segmentation on the point cloud, dividing it into many small blocks and studying the relationships between each block.

[0005] Existing related patents only process a small number of point clouds, and a lot of point cloud information is lost during the processing. In the patent "3D Point Cloud Segmentation Method and Apparatus Based on Moving Least Squares and Supervoxels", it voxelizes the point cloud data after octree processing, dividing it into voxels of equal size, each voxel is a cube voxel, and the centroid of all points in the cube is used to replace the original point cloud, resulting in oversimplification of the point cloud and only targeting small-scale point clouds. Summary of the Invention

[0006] The present invention discloses a point cloud feature recognition and annotation algorithm based on improved PointNet++, which includes two parts: "a supervoxel method suitable for deep learning networks" and "a deep learning network suitable for large component scanning data".

[0007] 1. Supervoxel Growth Method for Deep Learning Networks

[0008] Based on the current voxel growth theory, this method voxelizes the point cloud and predefines the characteristic parameters of each voxel. In order to comprehensively consider the distribution state of the point cloud within the voxel, the characteristic parameters are selected, the growth standard and growth threshold are formulated, and each growth region is grown, while ensuring that the number of point clouds in each growth region is consistent. This method is suitable for determining the characteristic parameters within the voxel with high resolution, ease of calculation, and strong robustness, achieving fast, efficient, and clear voxelized point cloud calibration.

[0009] 2. Deep Learning Network for Large Parts Scan Data

[0010] The design is based on the PointNet++ network and is divided into a pre-processing module, the original PointNet++ network backbone module, and a post-processing module.

[0011] 1. Pre-processing module:

[0012] Each growth region grown by the method described in step 1 is placed in batches into a pre-processing module, wherein the pre-processing module is composed of three SetAbstraction layers;

[0013] 2. Original PointNet++ network backbone module:

[0014] The point cloud features obtained after the pre-processing module and the centroid information of each growth area grown in step 1 are put into the original PointNet++ network backbone module, which consists of three Set Abstraction layers and three Feature Propagation layers, and finally a new point cloud feature dimension is obtained.

[0015] 3. Post-processing module:

[0016] The point cloud feature information obtained by the original PointNet++ network backbone module and the point cloud feature information obtained after processing by the pre-processing module are sequentially placed into the post-processing module according to the batches when each growth area is placed into the pre-processing module; the post-processing module consists of three feature propagation layers, three fully connected layers (FC), and two regularization layers (Dropout);

[0017] Regarding the technical solution above, further: the algorithm for completing region growing includes the following steps: first, the voxels are traversed using the PCA method and the quadratic surface equation, and the characteristic parameters of each voxel are obtained through the three-dimensional coordinate information, color, and number of the point cloud within the voxel; then, a seed voxel is randomly generated, and the neighboring voxels are searched from the seed voxel. According to the set growth standard and growth threshold, it is determined whether to include it in the growth region. The method terminates when all voxels in the region have completed growth and the growth process cannot continue.

[0018] Regarding the above technical solution, further: the randomly generated seed voxels can be single seed growth or multi-seed parallel growth.

[0019] For the above technical solution, further: the characteristic parameters include 14 parameters: 3 normal vector parameters, 1 curvature parameter, 3 color parameters, 1 density parameter, and 6 quadratic surface fitting parameters.

[0020] Regarding the above technical solution, further: the SetAbstraction layer in the pre-processing module first forms a local point cloud cluster by downsampling and grouping the input growth area, and then puts each local point cloud cluster into a multi-layer perceptron to increase the feature dimension of the point cloud, and finally performs local maximum pooling to obtain local global features.

[0021] Regarding the above technical solution, further: each growth area described in the pre-processing module contains the three-dimensional coordinates, semantic information, etc. of the points.

[0022] Regarding the above technical solution, further: since the point cloud is downsampled in the SetAbstraction layer, the Feature Propagation layer first upsamples the point cloud through linear interpolation to restore the number of point clouds in each growth area to the number before downsampling by the SetAbstraction layer, and then uses a multi-layer perceptron to reduce the feature dimension of the point cloud; finally, three fully connected layers and two regularization layers are used to obtain the category to which each point belongs.

[0023] Regarding the above technical solution, further: the regularization layer can solve the problem that the model has too many parameters and too few training samples, and the trained model is prone to overfitting.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] (1) The present invention proposes a supervoxel growing method suitable for determining characteristic parameters within voxels with strong resolution, easy calculation and strong robustness, so as to achieve fast, efficient and meaningful voxelized point cloud calibration.

[0026] (2) This paper improves PointNet++ and constructs a deep learning network suitable for large-scale point cloud recognition. While processing the point clouds of each region, it also takes into account the relationship between the point clouds of each region.

[0027] (3) Solve the technical problem that the traditional PointNet series networks cannot process large-scale point clouds. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] FIG1 is a schematic diagram of six adjacent voxels used in an embodiment of the present invention.

[0029] FIG2 is a schematic diagram of the supervoxel growth method used in an embodiment of the present invention.

[0030] FIG3 is a schematic diagram of a deep learning network suitable for large component scan data in an embodiment of the present invention.

[0031] FIG4 is a diagram showing the prediction effect of a deep learning network applied to large component scan data in an embodiment of the present invention.

[0032] FIG5 is a diagram showing the prediction effect of a deep learning network applicable to large component scanning data in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The following is a complete description of the embodiments of the present invention, providing detailed implementation methods and specific calculation processes, but the protection scope of the present invention is not limited to the following embodiments.

[0034] The present invention uses a supervoxel growing method suitable for deep learning networks.

[0035] (1) Based on the current voxel growth theory, the point cloud is voxelized and the characteristic parameters of each voxel are predefined.

[0036] Step 1: Find the maximum value on the X, Y, and Z axes based on the point cloud coordinate information 、 、 and minimum value 、 、 .

[0037] Step 2: Set the side length of the voxel 、 、 , divide the original point cloud into cubic voxels of size Lx ×Ly ×Lz, and calculate the number of voxels in the point cloud on the X, Y, and Z axes 、 、 ,

[0038] ;

[0039] Step 3: Calculate the voxel index h of each point in the point cloud, put the points with the same index into the same voxel, and save its voxel index, thereby obtaining the information of the points in each voxel.

[0040] ;

[0041] Step 4: Use the PCA method to calculate the normal vector and curvature of each voxel through the three-dimensional coordinates of the points in each voxel; use the quadratic surface equation to calculate the quadratic surface parameters of each voxel; calculate the color information of each voxel through the color information of the points in each voxel; calculate the density of each voxel through the number of points in each voxel.

[0042] (2) In order to comprehensively consider the distribution state of point clouds within voxels, feature parameters are selected, growth standards and growth thresholds are formulated, each growth area is grown, and the number of point clouds in each growth area is guaranteed to be consistent.

[0043] Step 5: Select a voxel as the seed voxel. Starting from the seed voxel, search for a neighborhood. The neighborhood consists of the six voxels in front, behind, left, right, above, and below the seed voxel. As shown in Figure 1, these are the six coplanar voxels of the seed voxel in space. If the difference between the normal vector angle, curvature difference, color difference, density difference, and quadratic surface fitting parameters of the adjacent voxel and the seed voxel meets the set threshold, it is included in the growing region. Otherwise, it is discarded. The algorithm terminates when all voxels in the region have completed growing and the growing process cannot continue. Through region growing, a single voxel is transformed into a region consisting of a series of voxels. Because some components are at 90-degree angles, region growing cannot grow them together, so a secondary growth process is required. First, the region after the first growth is divided, separating regions with fewer voxels from regions with more voxels. Regions with more voxels are not subjected to secondary growth, while regions with fewer voxels are subjected to secondary growth. The criterion for secondary growth is that if there are neighboring voxels in the two regions, for example, if region 1 contains voxel A and region 2 contains voxel B, and voxels A and B are adjacent, the two regions are merged into one. Each secondary growth merges the two regions with neighboring voxels until the number of regions obtained by the n+1th secondary growth is the same as that obtained by the nth secondary growth, and the growth stops. The growth effect is shown in Figure 2, which shows the process of first voxelizing the original point cloud, then performing the initial growth according to the established growth criteria, and then merging the regions that meet the secondary growth criteria through the above-mentioned secondary growth.

[0044] Step 6: Randomly sample the points in the voxels in each region and downsample them to N points. Limit the number of regions to , if the number of regions after supervoxel growth is greater than , randomly selected from regions; if the number of regions after supervoxel growth is less than , randomly copy several areas and add to area.

[0045] The design is based on the PointNet++ network and is divided into a pre-processing module, the original PointNet++ network backbone module, and a post-processing module.

[0046] 1. Pre-processing module

[0047] The point cloud information in each growth area grown by the method described in (2) is put into the pre-processing module in batches. The pre-processing module consists of three SetAbstraction layers;

[0048] Deep learning network processes The sample size is , B is the number of samples input in each batch, is the number of regions, N is the number of points in each region, The three-dimensional coordinate information and other semantic information of the representative point. First, the sample is transformed into Size, the point cloud is input into the pre-processing module k times, each input contains Point cloud of different sizes, As shown in Figure 3, after SetAbstraction1(SA1), the output l1_xyz size is 、l1_points size is , is the number of points after downsampling by SA1, and 128 is the number of features; l1_xyz and l1_points are input into SA2, and the output l2_xyz after SA2 is , l2_points size is , is the number of points after downsampling by SA2, and 256 is the number of features; l2_xyz and l2_ points are input into SA3, and the output l3_xyz after SA3 is 、l3_points size is , 1 is the number of points after SA3 downsampling, and 512 is the number of features. The calculation process of the SA layer takes SA1 as an example. First, N points are divided into N1 groups by sampling and grouping the farthest point. Each group has several points, for example, X points, and the size of l1_xyz is 、l1_points size is , and then convert l1_points into Put it into the multi-layer perceptron for feature transformation, l1_points becomes , and finally perform local maximum pooling, l1_points becomes , and then becomes .

[0049] 2. Original PointNet++ network backbone module:

[0050] The point cloud features obtained after the pre-processing module and the centroid information of each growth area grown in (2) are put into the original PointNet++ network backbone module, which consists of three SetAbstraction layers and three Feature Propagation layers, and finally a new point cloud feature is obtained.

[0051] Merge the l3_points obtained from the point cloud input k times to get l3_gather, which is considered to be the feature of the center point of each area, with a size of , and then converted into , and at the same time find the center point of each growth area to get pc_xyz, the size is As shown in Figure 3, pc_xyz and l3_gather are put into the original PointNet++ network backbone module, and after SA4, the output l4_xyz size is 、l4_points size is , is the number of points after downsampling by SA4, and 128 is the number of features; l4_xyz and l4_points are input into SA5, and the output l5_xyz after SA5 is 、l5_points size is , is the number of points after downsampling by SA5, and 256 is the number of features; l5_xyz and l5_points are input into SA6, and after SA6, the output l6_xyz size is (B, 1,3) and the l6_points size is , 1 is the number of points after SA6 downsampling, and 1024 is the number of features. Input l5_xyz, l5_points, l6_xyz, and l6_points into FP6, calculate the distance between the points in l5_xyz and l6_xyz, find the three points closest to each point in l5_xyz and l6_xyz, and give these three points a weight based on the principle that the farther the distance, the smaller the weight. Multiply the weight by its feature to obtain the feature l5_points2 of the interpolation point, and then concatenate l5_points2 with l5_ points to obtain the concatenated l5_points2, which is then put into the multi-layer perceptron to obtain the new feature. The final size of l5_points2 is Similarly, input l4_xyz, l4_points, l5_xyz, and l5_ points2 into FP5, and the size of l4_points2 is Similarly, input pc_xyz, l3_gather, l4_xyz, and l4_points2 into FP4, and the size of l0_points is .

[0052] 3. Post-processing module:

[0053] The point cloud feature information obtained by the original PointNet++ network backbone module and the point cloud feature information obtained after processing by the pre-processing module are sequentially placed into the post-processing module according to the batches when each growth area is placed into the pre-processing module; the post-processing module consists of three feature propagation layers, three fully connected layers (FC), and two regularization layers (Dropout);

[0054] As shown in Figure 3, since the pre-processing module uses batch input, the post-processing module inputs the point cloud information l2_xyz, l2_points, l3_xyz, and l3_points obtained in the corresponding pre-processing module of the same batch into FP3, and the size of l2_points2 is obtained. ; Input l1_xyz, l1_points, l2_xyz, and l2_points2 into FP2, and the size of l1_points2 is ; Separate l0_points into k times in sequence and transform them into , and then becomes , l0_xyz, l0_points, l1_xyz, l1_points2 are input into FP1. The purpose of adding l0_points here is to consider the connection between the sample areas, and the size of l0_points2 is .

[0055] Then it is put into the first fully connected layer (FC), regularization layer (DP), second fully connected layer (FC), regularization layer (DP) and third fully connected layer in sequence, and the resulting point cloud size is , class is the number of classifications. Finally, the l0_points2 obtained by inputting the point cloud k times is merged and then transformed to obtain the data , that is, the category of each point is predicted.

[0056] The network's prediction results are shown in Figures 4 and 5. In Figure 4, arrow 1 points to plate, arrow 2 to bulb flat steel, arrow 3 to T-shaped steel, and arrow 4 to flat steel. In Figure 5, arrow 1 points to plate, arrow 2 to bulb flat steel, and arrow 3 to flat steel. This network can effectively classify dense point clouds of large components into several categories based on their component types, enabling efficient recognition and annotation of large-scale point data sets.

[0057] The above embodiment elaborates in detail the design of a point cloud feature recognition and labeling algorithm based on improved PointNet++ claimed in the present invention. This method ensures the feasibility of applying deep learning methods in dense point set data of large components through super voxel growth and deep learning networks, and can realize efficient recognition and labeling of large-scale point set data.

[0058] The foregoing description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be readily conceived by those skilled in the art within the technical scope disclosed herein are intended to be encompassed within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A point cloud feature recognition and annotation algorithm based on improved PointNet++, characterized in that: The algorithm consists of the following steps:

1. Supervoxel Growth Method for Deep Learning Networks Based on the current voxel growth theory, the point cloud is voxelized and the characteristic parameters of each voxel are predefined. In order to comprehensively consider the distribution state of the point cloud in the voxel, the characteristic parameters are selected, the growth standard and growth threshold are formulated, and each growth area is grown, and the number of point clouds in each growth area is guaranteed to be consistent.

2. Deep Learning Network for Large Parts Scanning Data The design is based on the PointNet++ network and is divided into a pre-processing module, an original PointNet++ network backbone module, and a post-processing module.

1. The pre-processing module includes: Each growth region grown by the method described in step 1 is put into a pre-processing module in batches, and the pre-processing module is composed of three SetAbstraction layers; 2. The original PointNet++ network backbone modules include: The point cloud features obtained after the pre-processing module and the centroid information of each growth area grown in step 1 are put into the original PointNet++ network backbone module, which consists of three Set Abstraction layers and three Feature Propagation layers, and finally a new point cloud feature dimension is obtained; 3. Post-processing module includes: The point cloud feature information obtained by the original PointNet++ network backbone module and the point cloud feature information obtained after processing by the pre-processing module are placed in the post-processing module in batches according to the growth areas when they are placed in the pre-processing module; the post-processing module consists of three Feature Propagation layers, three fully connected layers (FC), and two regularization layers (Dropout).

2. The algorithm according to claim 1, characterized in that The algorithm for completing regional growth includes the following steps: first, the voxels are traversed using the PCA method and the quadratic surface equation, and the characteristic parameters of each voxel are obtained through the three-dimensional coordinate information, color, and quantity of the point cloud in the voxel; then, a seed voxel is randomly generated, and the neighboring voxels are searched from the seed voxel. According to the set growth standard and growth threshold, it is determined whether to include it in the growth area. The method terminates when all voxels in the region have completed growth and the growth process cannot continue.

3. The algorithm according to claim 1, characterized in that The randomly generated seed voxels may be single seed growth or multi-seed parallel growth.

4. The algorithm according to claim 1, characterized in that The characteristic parameters include 14 parameters: 3 normal vector parameters, 1 curvature parameter, 3 color parameters, 1 density parameter, and 6 quadratic surface fitting parameters.

5. The algorithm according to claim 1, characterized in that The SetAbstraction layer in the pre-processing module first forms a local point cloud cluster by downsampling and grouping the input growth area, then puts each local point cloud cluster into a multi-layer perceptron to increase the feature dimension of the point cloud, and finally performs local maximum pooling to obtain local global features.

6. The algorithm according to claim 1, characterized in that Each growth region described in the pre-processing module contains the three-dimensional coordinates and semantic information of the points.

7. The algorithm according to claim 1, characterized in that Since the point cloud is downsampled in the SetAbstraction layer, the Feature Propagation layer first upsamples by linear interpolation to restore the number of point clouds in each growth area to the number before downsampling by the SetAbstraction layer, and then uses a multi-layer perceptron to reduce the feature dimension of the point cloud; finally, three fully connected layers and two regularization layers are used to obtain the category to which each point belongs.

8. The algorithm according to claim 1, characterized in that The regularization layer can solve the problem that the model has too many parameters and too few training samples, and the trained model is prone to overfitting.

Images