Point cloud sampling method for perception reconstruction

Abstract

The application provides a point cloud sampling method for perceptual reconstruction, comprising the following steps: S1, point cloud sampling network sampling: a residual block based on graph convolution is used for local feature extraction of the point cloud, and the sampled point cloud is obtained through mapping according to the point cloud features; S2, point cloud up-sampling network recovery perception: the sampled point cloud is up-sampled to obtain the point cloud data after perceptual reconstruction; S3, point cloud classification network identifies the up-sampled point cloud: the point cloud data after perceptual reconstruction is input into the point cloud classification network, and the training of the point cloud sampling network and the point cloud up-sampling network is supervised through the point cloud classification network. The method can obtain the sampled point cloud which is easier to recover perception.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of point cloud sampling, in particular to a point cloud sampling method for perception reconstruction. BACKGROUND

[0002] 3D point cloud has become a popular data structure for representing real-world objects, with wide application prospects such as autonomous driving, satellite images and mixed reality. In these applications, point cloud sampling is a key technology, and the point cloud after sampling processing can usually meet the requirements of data transmission and storage. However, sampling may discard redundant points, and it is inevitable to lose some cases containing perception information. Therefore, sparse point cloud has certain difficulties in machine recognition, especially when applied in subsequent classification tasks. Directly training a model on a sparse point cloud dataset is prone to catastrophic forgetting problem, that is, the performance of the model at the original point cloud resolution will decrease. In addition, retraining the model for different down-sampling scales to adapt to sparse point cloud is not a flexible method. Due to the limitation of the number of points, the model is prone to information bottleneck problem when learning sparse point cloud representation. Therefore, the quality of point cloud sampling is very important to the accuracy of downstream tasks. In terms of image classification network DGCNN, a 4-fold random sampling can cause its performance to decrease by about 60%.

[0003] Therefore, good point cloud sampling can obtain a delicate representation of point cloud, while also ensuring good perception in downstream tasks. This technology aims to sample representative points from relatively dense point clouds. Therefore, sampling algorithms are used to simplify data and speed up computation. There are many classic sampling methods, such as random sampling (RS), farthest point sampling (FPS) and inverse density importance sampling (IDIS). These methods can quickly process large-scale point clouds. However, due to the limited semantic information, they cannot well serve downstream tasks. The learning-based point cloud sampling method has problems in optimization goals, that is, the optimization goal in the process of point cloud sampling is to make the down-sampled point cloud have better recognition accuracy.

[0004] According to the research of the present application, for most point cloud classification networks, the classification performance of the point cloud will be severely damaged after sampling. This is because the distance calculation result in the K-nearest neighbor algorithm in the deep network will change greatly in the process of local point loss, and at the same time, the feature distribution extracted locally will produce a certain deviation. With the continuous increase of the sampling rate, this deviation will be continuously amplified, and finally the sampled point cloud cannot be directly applied to downstream tasks. Therefore, in the process of sampling, how to supplement the points must be considered, that is, the strategy of combining sampling and up-sampling is adopted, so that the original perception information can be reconstructed.

[0005] The difficulties for solving the above problems and defects are: 1, how to design the sampling plus the framework of the sampling to help reconstruct the original perception information; 2, how to design the sampling network and the up-sampling network structure to ensure good downstream classification task performance; 3, how to train the sampling network and the up-sampling network to ensure that the result after sampling has good generalization.

[0006] The significance of solving the above problems and defects is: 1, the network framework combining sampling and up-sampling is first proposed to better help the perception reconstruction of the sampled point cloud; 2, a more effective and reliable structure graph convolution structure is provided for feature extraction of the point cloud in the sampling and up-sampling network. This network structure can be widely applied to various graph convolution-based networks; 3, a new training strategy is explored, which makes the joint training of the sampling and up-sampling tasks more stable and robust in performance, laying a training foundation for future related research. SUMMARY

[0007] The application provides a point cloud sampling method for perception reconstruction, which can obtain a better perception recovery sampling point cloud.

[0008] The technical scheme of the application is as follows:

[0009] The point cloud sampling method for perception reconstruction comprises the following steps: S1. Point cloud sampling network sampling: a point cloud feature extractor based on graph aggregation convolution is used to extract point cloud features with semantic representation, and a sampled point cloud is obtained by mapping according to the point cloud features; S2. Point cloud up-sampling network recovery perception: a residual block based on graph convolution is used for point cloud local feature extraction; S3. Point cloud classification network identifies the up-sampled point cloud: the perception reconstructed point cloud data is input into the point cloud classification network, and the training of the point cloud sampling network and the point cloud up-sampling network is supervised by the point cloud classification network.

[0010] Optionally, in the above point cloud sampling method for perception reconstruction, in step S1, a point cloud is given The point cloud sampling network can obtain a sampled sparse point cloud Wherein n represents the number of sampling points, 3 represents the space x, y, z coordinates of the point cloud, and r represents the sampling rate, which is input into the up-point cloud sampling network to obtain the sampling result.

[0011] Optionally, in the above point cloud sampling method for perception reconstruction, in step S1, the point cloud sampling network structure comprises a convolution and activation function module, three residual blocks, a full connection layer and a dimension transformation module; wherein each residual block comprises a plurality of alternating graph convolution and activation function, and a global pooling layer.

[0012] Optionally, in the above point cloud sampling method for perceptual reconstruction, in step S1, the dense point cloud passes through a first convolution and activation function module to obtain a c1-dimensional feature, and then the point cloud is input into a residual block composed of a graph convolution and an activation function to extract local features, and the output of the residual block is also a c1-dimensional feature, after the last residual block, a global pooling layer is used to convert the rnxc1 feature into a 1xc1 feature, and then several fully connected layers are used to obtain a 1x3n feature, and the feature is put into a dimension transformation module, and a conversion function is used here Then the farthest point sampling result of the dense point cloud is added to the output after the dimension transformation through a priori connection, to obtain a network output with prior perception.

[0013] Optionally, in the above point cloud sampling method for perceptual reconstruction, in step S2, the point cloud upsampling network includes a convolution and activation function module, four residual blocks, an upsampling module, a convolution and activation function module, and a convolution module; each residual block includes multiple alternating graph convolutions and activation functions, and a global pooling layer.

[0014] Optionally, in the above point cloud sampling method for perceptual reconstruction, in step S2, four residual blocks are used, and the feature extraction through these residual blocks obtains an nxc2-dimensional feature, in the upsampling module, a gradual point shuffling method is used to shuffle the F a Upscale to a specified multiple, and a dimension expansion weight is used for each level of gradual shuffling The 256-dimensional feature is converted into a 512-dimensional feature, and the shuffling operation The feature dimension is converted into a spatial dimension, and the upsampling module is represented as:

[0015]

[0016] Where p is the number of upsampling, which is represented as p=logr, and the result after upsampling is added to the result after several layers of convolution and sampling of the nearest neighbor of the point cloud to obtain the upsampled point cloud.

[0017] According to the technical scheme of the present application, the beneficial effects are:

[0018] The application combines sampling and up-sampling, provides a point cloud sampling method for perception reconstruction, and constructs a network framework for recovering point cloud perception after point cloud sampling. The application explores and discovers that the sparse point cloud obtained by the conventional sampling method is not conducive to the downstream point cloud classification task, so the application considers combining point cloud sampling, up-sampling and classification task network to improve the robustness of the downstream task to the sampled point cloud; the application skillfully uses prior information, obtains prior information by farthest point sampling of the point cloud, and then adds the prior sampling information to the point cloud up-sampling network, which improves the perception reconstruction effect; the application designs a new training scheme, uses multiple loss functions to optimize the training of the supervised sampling network and the up-sampling network, so that the sampled point cloud has better representation; after the point cloud is transmitted and stored, it can be better recovered in the downstream classification task.

[0019] In order to better understand and illustrate the concept, working principle and effect of the application, the application will be described in detail below by combining the drawings and specific examples. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the specific embodiments of the application or the technical solutions in the prior art, the drawings needed to be used in the specific embodiments or the prior art description will be briefly introduced.

[0021] Figure 1 is a flowchart of the point cloud sampling method for perception reconstruction of the application;

[0022] Figure 2 is a comparison diagram of the network framework of the point cloud sampling for perception reconstruction of the application and the traditional point cloud sampling framework;

[0023] Figure 3 is a structural schematic diagram of the point cloud sampling network and the point cloud up-sampling network. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical method and advantages of the application clearer, the application will be further described in detail below by combining the drawings and specific examples. These examples are only illustrative, and are not limiting to the application.

[0025] Figure 2 The (a) traditional point cloud sampling method in is directly for the downstream task network, this method usually uses a sampling network to sample the point cloud, and then inputs the sampling result directly into the point cloud classification network in the downstream task; the training of the sampling network uses the error of the classification result of the classification network and the true data label as the supervision information. And Figure 2The method of the application is oriented to perceptual reconstruction, and the sampling result of the sampling network is input into the up-sampling network, and then the up-sampled point cloud is input into the classification network of the downstream task; the sampling network and the up-sampling network in the application are trained together, and the EMD distance between the up-sampled point cloud and the real point cloud, the feature similarity and the category error are used as the supervision information.

[0026] The point cloud sampling method for perceptual reconstruction of the application is used for sampling a dense point cloud into a sparse point cloud, while ensuring that the sampled point cloud can be better recovered for perception.

[0027] The working principle of the application is that the input point cloud is sampled by the point cloud sampling method, at the same time, the input point cloud is sampled by the farthest point, and the two sampling results are added and input into the up-sampling network to recover the point cloud, and then the recovered point cloud is input into the classification network.

[0028] As shown in (b) of Figure 1 and Figure 2 The point cloud sampling method for perceptual reconstruction of the application comprises the following steps:

[0029] S1. Point cloud sampling network sampling: a residual block based on graph convolution is used for point cloud local feature extraction, and the sampled point cloud is obtained by mapping according to the point cloud features.

[0030] In this step, in the sampling stage, the main part of the sampling network adopts a dynamic graph convolution network based on residual connection, which improves the ability of the sampling network to obtain local information of the point cloud, and uses the prior information of the dense point cloud to improve the performance of the sampling network.

[0031] Specifically, given a point cloud The point cloud sampling network can obtain a sampled sparse point cloud Where n represents the number of sampling points, 3 represents the space x, y, z coordinates of the point cloud, and r represents the sampling rate, which is input into the point cloud sampling network to obtain the sampling result. Figure 3As shown in (a) of FIG. 1, the point cloud sampling network structure includes a convolution and activation function module, three residual blocks, a full connection layer, and a dimension transformation module; wherein each residual block includes multiple alternating graph convolution and activation functions, and a global pooling layer. The dense point cloud passes through the first convolution and activation function module to obtain a c1-dimensional feature. Then the point cloud is input into the residual block composed of graph convolution and activation function to extract local features, and the output of the residual block is also a c1-dimensional feature. After the last residual block, a global pooling layer is used to convert the rnxc1 feature into a 1xc1 feature, and then several full connection layers are used to obtain a 1x3n feature, which is put into the dimension transformation module. Here, a conversion function is used. After that, the farthest point sampling result of the dense point cloud is added to the output after the dimension transformation through the prior connection to obtain the network output with prior perception. Since the network output is discrete, a matching mapping (point matching) between the result and the original dense point cloud is needed to ensure that the generated point cloud is the sampling point cloud among the original point cloud. Figure 3

[0032] S2. Point cloud up-sampling network restores perception: up-sampling the sampled point cloud to obtain the point cloud data after perception reconstruction.

[0033] In this step, a dual network similar to the point cloud sampling network in step S1 is used, which uses a residual connection-based graph convolution block for feature extraction and uses point shuffling technology to improve the resolution of point cloud features to ensure that it can up-sample the sampled point cloud by a certain multiple. The result of step S1 can be combined with the prior information of the farthest point sampling of the original point cloud to restore the number of points through up-sampling.

[0034] The structure diagram of the point cloud up-sampling network is shown in (b) of FIG. 1. Figure 3 The sampled point cloud needs to be put into the point cloud up-sampling network to obtain the semantic reconstruction representation. The structure of the point cloud up-sampling network is similar to that of the sampling network, including a convolution and activation function module, four residual blocks, an up-sampling module, a convolution and activation function module, and a convolution module; wherein each residual block includes multiple alternating graph convolution and activation functions, and a global pooling layer. Four residual blocks are used, and through feature extraction of these residual blocks, a nxc2-dimensional feature is obtained. In the up-sampling module, a progressive point shuffling method is used to enlarge the F a to a specified multiple. This shuffling method can convert the dimension to a spatial representation. Specifically, a dimension expansion weight is used for each level of progressive shuffling to convert a 256-dimensional feature to a 512-dimensional feature, and the shuffling operation can convert the feature dimension to a spatial dimension, so the up-sampling module can be represented as: ​

[0035]

[0036] where p is the number of upsampling steps, which is denoted as p=logr. The result of upsampling is added to the result of the nearest neighbor of the sampled point cloud (nearest neighbor interpolation) after several layers of convolution to obtain the upsampled point cloud.

[0037] S3. The point cloud classification network identifies the upsampled point cloud: the perceptually reconstructed point cloud data is input into the point cloud classification network, and the training of the point cloud sampling network and the point cloud upsampling network is supervised by the point cloud classification network.

[0038] In this step, the upsampled point cloud (i.e., the perceptually reconstructed point cloud data) is directly used as the input of the point cloud classification network. In this way, the point cloud sampling network, the point cloud upsampling network, and the point cloud classification network are organically connected in series, ensuring that the point cloud classification network supervises the training of the point cloud sampling network and the point cloud upsampling network; ensuring that the sampled point cloud can be better reconstructed by the point cloud upsampling network according to specific perception, so that the sampled point cloud has a high classification accuracy after being upsampled. Placing the upsampled point cloud in the point cloud classification network can directly improve the classification accuracy of the point cloud classification network on the sampled point cloud.

[0039] The dense point cloud after point cloud upsampling contains more perception than the sampled point cloud, so compared with directly inputting the sampled point cloud into the point cloud classification network, the present application can improve the classification accuracy of the point cloud classification network. The present application uses two classical point cloud classification networks, PointNet and DGCNN, to verify the method proposed by the present application. Here, the present application uses three loss functions to optimize the point cloud upsampling network and the upsampling network together; they are EMD (Earth Mover's Distance) loss, feature similarity loss, and class loss. The EMD loss is represented as follows:

[0040]

[0041] where P u is the upsampled point cloud. For the feature similarity loss, it is necessary to capture high-level feature representations of the classification network, of which the most important is to capture the global features of these classification networks, i.e., the output after global max pooling. For the dense point cloud and the corresponding sparse point cloud, their global features need to be extracted and the feature-level similarity loss needs to be calculated:

[0042]

[0043] By the loss function, the application establishes a high-level similarity representation of the up-sampled point cloud and the dense point cloud. In addition, the application also uses another category loss as follows:

[0044]

[0045] where y i is the label P i is the probability of the category output by the classification network. For a point cloud classification network such as PointNet, only the feature similarity loss needs to be used. For a network such as DGCNN, all three loss functions need to be used, and the corresponding weights are 100, 0.1 and 0.001. The point cloud sampling network and the up-sampling network need to be trained for a total of 120 rounds and use cosine learning rate decay. The learning rate of 0.001 and 0.00001 is used for the sampling network and the up-sampling network, respectively. To ensure the fairness of the test results, the application uniformly uses PyTorch as the code implementation and uses the Adam optimizer to optimize the model.

[0046] The application performs experimental verification on the classic ModelNet40 dataset, and uses the average classification accuracy, the class average accuracy, and the CD distance between the real point cloud as evaluation indexes when different point cloud sampling methods are input into the downstream point cloud classification network under the corresponding test set. The application verifies the feasibility of the scheme under different sampling rates, including 2 times, 4 times, 8 times and 16 times, as shown in Table 1, wherein CD-1 and accuracy-1 represent the performance of the sampling method on the downstream classification network PointNet, and CD-2 and accuracy-2 represent the performance of the sampling method on the downstream classification network DGCNN. The smaller the CD distance, the closer the up-sampled point cloud is to the real dense point cloud, indicating that the result of spatial upsampling is better, and the higher the classification accuracy, the higher the perception of the point cloud processed by different methods.

[0047] Table 1

[0048]

[0049]

[0050] As can be seen from Table 1, the application achieves advanced sampling effect, that is, the sampling method proposed by the application is more conducive to perceptual reconstruction representation.

[0051] The above description is based on the best embodiment of the application concept and working principle. The above embodiment should not be understood as limiting the scope of protection of the present application, and other embodiments and combinations of implementation modes according to the application concept all belong to the protection scope of the application.

Claims

1. A point cloud sampling method for perceptual reconstruction, characterized in that, The method comprises the following steps: S1. Point cloud sampling network sampling: feature extraction is performed on the input dense point cloud using a sampling network comprising residual blocks with graph convolution-activation function alternation, discrete sampling indexes are obtained through dimension transformation, and point matching is performed to make the output sampling point set a subset of the original point set; wherein the farthest point sampling (FPS) result of the dense point cloud is taken as a priori, which is fused with the output after dimension transformation in a priori connection manner to form a sampling result with priori constraints; The point cloud sampling network structure comprises a convolution and activation function module, three residual blocks, a full connection layer, and a dimension transformation module; wherein each residual block comprises a plurality of alternating graph convolution and activation function, and a global pooling layer, Given a dense point cloud , the point cloud sampling network obtains a sampled sparse point cloud , where denotes the number of sampled points, denotes the space where the point cloud is located coordinates, denotes the sampling rate; The dense point cloud is subjected to a convolution and an activation function module to obtain a feature of an r n ×c1 dimension, and then the point cloud is input into a residual block composed of a graph convolution and an activation function to extract local features, and an output of the residual block is a feature of an r n ×c1 dimension, after the last residual block, a global pooling layer is used to convert the feature of the r n ×c1 dimension into a feature of a 1 ×c1 dimension, and then a fully connected layer is used to obtain a feature of a 1 ×3n dimension, and the feature is placed into the dimension transformation module, a conversion function is used, and then the farthest point sampling result of the dense point cloud is added to the output after the dimension transformation through a prior connection to obtain a network output with prior perception. S2. Point cloud up-sampling network restores perception: input the sampled point cloud into the up-sampling network set in step S1, the network comprises a convolution and activation function module, four residual blocks, an up-sampling module, a convolution and activation function module, and a convolution module in sequence; wherein each residual block comprises a plurality of alternating graph convolution and activation function, and a global pooling layer, wherein the up-sampling module is used to expand the feature dimension step by step and shuffle the spatial coordinates; the up-sampling result is fused with the nearest neighbor interpolation result based on the sampling point cloud to obtain the up-sampling point cloud to restore the point number; S3. Point cloud classification network identifies the up-sampling point cloud: input the up-sampled point cloud into the point cloud classification network, and perform end-to-end joint training on the networks of steps S1 and S2 with multiple joint losses; the joint losses comprise EMD loss, feature similarity loss, and category loss.

2. The method of claim 1, wherein, In step S2, using the 4 residual blocks, feature extraction of the residual blocks is performed to obtain dimensional features.

Images