A point cloud downsampling method based on partial attention mechanism and mutual supervision loss

Abstract

The present invention discloses a point cloud downsampling method based on a partial attention mechanism and mutual supervision loss, and relates to the field of 3D sensing technology. The present invention comprises the following steps: using Simplify to simplify the original point cloud into a simplified point cloud; using Pointnet to extract the semantic features of the original point cloud and the simplified point cloud; finding the local neighbors of the simplified point cloud in the original point cloud by the KNN algorithm; performing feature fusion of the offset attention mechanism; using a similarity projection convolutional neural network to generate a similarity score according to the fused features, and using the distance between them and the score to correct the probability position distribution. The present invention extracts features while considering the features of neighbors, so that the simplified point cloud is more robust; using the partial attention mechanism can capture local contexts through point cloud sampling, and at the same time can improve the effectiveness of point cloud sampling; correcting the distribution of projection points by multiple supervisions, can obtain more reliable sampling points and can be better applied to registration work.

Description

Technical Field

[0001] The present invention belongs to the field of 3D sensing technology, and in particular relates to a point cloud downsampling method based on partial attention mechanism and mutual supervision loss. Background Art

[0002] In recent years, 3D sensing devices have become increasingly popular. These devices usually capture the surface features of the target in the form of point clouds and generate a collection of massive points. Point clouds have also been applied to more and more tasks, such as classification, segmentation, registration, and shape reconstruction. However, as the size of point clouds gradually increases, the computational requirements of these tasks are also increasing. The massive amount of point cloud data often brings processing difficulties due to its large number. A possible solution is to downsample the point cloud to reduce the amount of data and improve the efficiency of point cloud processing. Among the commonly used downsampling algorithms, geometric sampling accumulates more sampling points in places with large curvature of the point cloud; grid sampling discretizes the three-dimensional space into grid points and then samples a point at each grid point; uniform sampling can evenly extract points in the sample. The most commonly used is FPS (furthest point sampling), which minimizes geometric errors by continuously iteratively selecting the farthest point from the existing sampling point set. However, the above methods still have some problems: (1) Lack of consideration of geometric features. (2) The distribution of projection points cannot be effectively trained through observation coefficients. Therefore, the effectiveness and robustness of point cloud sampling are poor. Based on this, we developed and disclosed a point cloud downsampling method based on partial attention mechanism and mutual supervision loss of the present invention. Summary of the invention

[0003] The purpose of the present invention is to provide a point cloud downsampling method based on partial attention mechanism and mutual supervision loss to solve the existing problems: the existing technology lacks consideration of geometric features and cannot effectively train the distribution of projection points through observation coefficients, resulting in poor effectiveness and robustness of point cloud sampling.

[0004] In order to solve the above technical problems, the present invention is achieved through the following technical solutions:

[0005] A point cloud downsampling method based on partial attention mechanism and mutual supervision loss includes the following steps:

[0006] Use Samplenet's Simplify to simplify the original point cloud into a simplified point cloud;

[0007] Use Pointnet to extract the semantic features of the original point cloud and simplify the point cloud;

[0008] Find the local neighbors of the simplified point cloud in the original point cloud through the KNN algorithm;

[0009] The semantic features of the simplified point cloud and its local neighbors are fused using the offset attention mechanism;

[0010] A similarity projection convolutional neural network is used to generate a similarity score based on the fused features, and the distance between them is used in conjunction with the score to correct the probability position distribution.

[0011] Further: Among them, the adjusted Pointnet is used to extract the features of the original point cloud and the simplified point cloud, mainly including:

[0012] The points are sent to the point alignment module to adjust the points;

[0013] Send to LBR layer;

[0014] Align structures by features;

[0015] Enter another LBR layer and output the K-dimensional semantic features of each point.

[0016] Further: Among them, feature fusion mainly includes:

[0017] The point set of the original point cloud is The point set of the simplified point cloud is Remember p s are the s local neighbors of Q in P;

[0018] Find the neighbor points of Q and construct a graph G(Q,E) whose vertices are Q={q1,...,qm} and edges are

[0019] Maintain the structural connections between vertices and their neighbors and map them to new features by learning functions. The formula is as follows;

[0020]

[0021] Among them, s represents the index of the neighbor point of q in p, f and f s Represents q and p respectively s The geometric characteristics of ff s is the feature difference between the simplified point and its neighbors, ||qp s ||2 represents the Euclidean distance between two points, [;] represents the connection, M α Multilayer Perceptron representing fused features.

[0022] Further: Among them, a similarity projection convolutional neural network is used to generate a similarity score according to the fusion features, and the distance between them is used in conjunction with the score to correct the probability position distribution, and the loss function formula is:

[0023]

[0024] Among them, r is a hyperparameter used to control the distribution and supervise q and p s The distance between them, M refers to the number of vertices, S(q,p s ) refers to the similarity score.

[0025] The present invention has the following beneficial effects:

[0026] ① The present invention proposes a robust point cloud sampling network for feature extraction, which solves the problem of massive points. The present invention uses Samplenet to simplify 3D point clouds, and uses the adjusted Pointnet to extract the features of the original point cloud and the simplified point cloud, and then uses KNN to extract the features of the local neighbors of the simplified point cloud in the original point cloud, which solves the problem that the previous network did not consider the further processing of the sampling points. The simplified point cloud extracts features through a deep neural network while considering the features of the neighbors, making the simplified point cloud more robust.

[0027] ② This paper proposes a biased attention mechanism for feature fusion to improve feature expression ability. The biased attention mechanism used in Pointnet can capture local context through point cloud sampling and improve the effectiveness of point cloud sampling.

[0028] ③ The present invention proposes a similarity projection method to correct the distribution of projection points through multiple supervision. The present invention evaluates the possible positions of projection points through mutual supervision loss instead of observation values. Updating the positions of projection points through mutual supervision loss can obtain more reliable sampling points and can be better applied to registration work. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0030] Figure 1 It is the overall flow chart of the present invention;

[0031] Figure 2 Schematic diagram of the adjustable Pointnet structure for feature extraction;

[0032] Figure 3 Find and construct a feature description diagram for Q's neighbors;

[0033] Figure 4 Schematic diagram of the partial attention mechanism structure;

[0034] Figure 5Schematic diagram of feature structure transformation and similarity projection;

[0035] Figure 6 To extract different size feature curves through the PointNet module;

[0036] Figure 7 A schematic diagram of the sampling and registration effect on massive point clouds;

[0037] Figure 8 Schematic diagram of sampling point registration of FPS, SampleNet and the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] The present invention is a point cloud downsampling method based on partial attention mechanism and mutual supervision loss. The overall process is as follows: Figure 1 , specifically including the following steps:

[0040] 1. First is the point set Includes the coordinates of n 3D point clouds, point sets is a subset of P after simplification.

[0041] Pointnet is used to extract the features of P and Q. Then, the s local neighbors of Q are found in P through the KNN algorithm, and a distance matrix of shape m×s×3 and a feature matrix of shape m×s×k are generated. Therefore, in addition to the distance and geometric features of Q itself, the features of its neighbors are also transmitted to the feature fusion module based on the partial attention mechanism. Finally, the present invention uses the similarity score and the distance between Q and its local neighbors to generate the projection point R as the output of the present invention. In order to obtain reliable sampling points, the present invention calculates the possible positions of the projection points through mutual supervision loss instead of observation values.

[0042] 2. Feature extraction: The present invention uses the adjusted Pointnet to realize feature extraction. Pointnet is a classic deep learning network that directly extracts features from unordered point sets. Figure 2 Shown Figure 1 The feature extraction module in , where the source point cloud and the simplified point cloud use the same structure. Figure 2In the example, n points are sent to the point alignment module to adjust the points to ensure the invariance of spatial transformation. Then they are sent to the LBR layer, which is the Linear, BatchNorm and ReLU layers. Then they pass through the feature alignment structure similar to the point alignment structure, and finally enter another LBR layer to output the K-dimensional semantic features of each point.

[0043] 3. Feature fusion: Although Pointnet can provide rich feature descriptions, it ignores local features. Therefore, the present invention uses a partial attention mechanism to process feature descriptions. The specific steps are as follows. First, Q finds its neighbor points and constructs a graph G(Q,E). The vertices are Q = {q1,...,qm}, and the edges are It is noteworthy that the present invention maintains the structural connections between vertices and their neighbors, i.e., the directed edges connecting each other, and maps them into new features through learning functions. Figure 3 The corresponding details are shown, and the specific formula is as follows:

[0044]

[0045] Where s represents the index of the neighbor point of q in p. f and f s Represents q and p respectively s The geometric characteristics of ff s is the difference in features between the simplified point and its neighbors. ||qp s ||2 represents the Euclidean distance between two points, and [;] represents a connection. α Multilayer Perceptron representing fused features.

[0046] 4. The d-dimensional semantic description is learned through the offset attention mechanism. Figure 4 The detailed steps are shown, and more specific instructions are as follows:

[0047] (F q ,F k ,F v )=h*(W q ,W k ,W v )

[0048]

[0049] h a =A*F v

[0050] h o =LBR(hh a )+h

[0051] from Figure 4 It can be seen that W is formed through the learning of linear transformation q,W k ,W v , which transforms the feature description into the query matrix F q , the bond matrix F k , and the value matrix F v .d a is the dimension of the query and vector, which is used to normalize the dot product of the query matrix and the key matrix, and also obtains the weight of the self-attention mechanism. Inspired by the Laplacian matrix L = DE, the module uses subtraction to calculate the difference between the input features and the attention features. After passing through the LBR layer, this difference is added to the input features. Figure 5 An accurate description is provided, where Q has found its neighbors through the KNN algorithm. First, the distance features and geometric features between the two point sets are connected. Then the output of the feature descriptor and the offset attention mechanism are connected. After that, the present invention applies a 1*1 2D convolution to the fused features, which is similar to using a multi-layer perceptron to enhance the feature expression of each position.

[0052] 5. Finally, similarity projection. The simplified set may not be a subset of the input points, so similarity projection aims to optimize the initial simplified point cloud to the vicinity of the source point set. To this end, the present invention verifies a supervised method to approximate the projection points. To further explore this concept, the present invention uses a neural network to generate a similarity score based on the fused features, and then uses the distance between them to match the score to correct the probability position distribution. The present invention uses similarity projection convolution to generate a similarity score, and is supervised by the distance of the two sets. The closer the distance, the more likely it is a real point. This part can be defined as follows:

[0053]

[0054] Here r is used as a hyperparameter to control the distribution to gain more insights for supervising q and p s If the result is much larger than r, it will not construct the correspondence between them, nor will it affect the distribution of the projection points.

[0055] The following are the technical effects of the present invention:

[0056] According to the Samplenet network, the 1024 points of the car category in MoudelNet40 are rotated by [-45,45] degrees. In addition, the present invention uses 100 source template pairs from the test set for performance evaluation. At the same time, the present invention uses PCRNet as the registration network (such as Figure 7), which uses rotation matrix and translation vector to collect point cloud in the same coordinate system, so the present invention measures the performance of registration through four indicators, namely, mean rotation error (MRE), standard rotation error (SRE), mean consistency error (MCE) and standard consistency error (SCE). The following experiments can prove the effectiveness of the present invention.

[0057] First, the present invention is tested at different scales through different extraction modules. Since sampling is to improve the efficiency of the task, high-dimensional features may consume a lot of time and computing resources. Figure 6 The MRE and SRE in 256-dimensional and 512-dimensional respectively present the best results, with MRE of 6.58 in 256-dimensional and SRE of 6.61 in 512-dimensional. However, the MRE in 1024-dimensional is 8.57 and the SRE is 7.6. Therefore, high-dimensional feature descriptors may not bring the best results. In the experiment of the present invention, the present invention selects a 256-dimensional feature descriptor.

[0058] Then, the present invention uses three sampling methods as baselines, namely random sampling, FPS and Samplenet to evaluate the registration results of the simplified point cloud. The present invention found that the official implementation of Samplenet does not provide versions of SRE, MCE, and SCE. Therefore, the present invention uses the official code to retrain and verify and supplement the relevant data. The detailed information is shown in Table 1.

[0059] It can be found from the table that the SRE of the method with biased attention mechanism of the present invention remains at 6.85, while the MRE value is 6.58, ranking second. Despite this, the gap between the method of the present invention and Samplenet is only 0.19, and the performance of the other two techniques is lower than that of the present invention. In particular, random sampling has achieved disappointing results. The MRE of random sampling is almost four times that of the method of the present invention, and the SRE is almost twice that of the method of the present invention. There is no significant difference in MCE and SCE among the four methods. Then the MRE of the present invention increased by 1.25 without the biased attention mechanism.

[0060] Table 1 Effectiveness of various sampling methods in registration

[0061]

[0062] In addition, we compare the SREs of different categories to the baseline. Table 2 provides the full details. The unique properties of the different subsets create different effects. Overall, our method achieves good results, especially in the door category, where the number is at the lowest position at 3.35. There is a clear contrast between random sampling and our method in the table category, where the value of random sampling is twice that of our method.

[0063] Table 2 Comparison of SRE in each category

[0064] method airplane chair desk door guitar Rondom 18.82 24.44 22.53 7.48 17.55 FPS 8.46 15.52 11.01 3.91 16.68 Samplenet 10.55 16.39 12.03 3.98 11.74 Ours 8.75 15.34 10.70 3.35 10.71

[0065] In addition, as shown in Table 3, the present invention analyzes the SRE of different sampling point methods in different dimensions of the automobile category. It should be pointed out that the present invention has fewer sampling points and has certain advantages. For example, when the present invention samples 64 points, the SRE of Samplenet and the present invention are 9.87 and 6.85 respectively, with a significant difference. The data of Samplenet is 3.02 more than that of the present invention.

[0066] Table 3 SRE in different dimensions

[0067] method 64 128 256 512 Rondom 16.23 10.94 9.75 8.99 FPS 12.07 9.29 9.57 8.86 Samplenet 9.87 8.43 8.31 8.40 Ours 6.85 6.19 6.14 6.04

[0068] Finally, if Figure 8 Figure 2 shows the visualization of three registration examples with 64 sampling points. The top row shows the unregistered points, while the sampling points are marked in blue and green, respectively. The bottom row shows the point cloud registration effect after different downsampling methods.

[0069] Therefore, based on the above, we can know that:

[0070] ① The present invention proposes a robust point cloud sampling network for feature extraction, which solves the problem of massive points. The present invention uses Samplenet to simplify the 3D point cloud and uses the adjusted Pointnet (such as Figure 2 ) to extract the features of the original point cloud and the simplified point cloud, and then use KNN (such as Figure 3 ) Extract the features of the local neighbors of the simplified point cloud in the original point cloud. This solves the problem that the previous network did not consider the further processing of the sampling points. The simplified point cloud extracts features through a deep neural network while considering the features of the neighbors, making the simplified point cloud more robust.

[0071] ② This paper proposes a biased attention mechanism to perform feature fusion to improve feature expression capabilities. Although Pointnet can provide rich feature descriptions, it ignores local features. The biased attention mechanism used in this paper (such as Figure 4 ) can capture local context through point cloud sampling and improve the effectiveness of point cloud sampling.

[0072] ③ The present invention proposes a similarity projection method to correct the distribution of projection points through multiple supervision. The present invention evaluates the possible positions of projection points through mutual supervision loss rather than observation values. Updating the positions of projection points through mutual supervision loss can obtain more reliable sampling points and can be better applied to registration work.

[0073] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A point cloud downsampling method based on partial attention mechanism and mutual supervision loss, characterized by: The following steps are involved: Use Samplenet's Simplify to simplify the original point cloud into a simplified point cloud; Use Pointnet to extract the semantic features of the original point cloud and simplify the point cloud; Find the local neighbors of the simplified point cloud in the original point cloud through the KNN algorithm; The semantic features of the simplified point cloud and its local neighbors are fused using the offset attention mechanism to obtain the fused features; A similarity projection convolutional neural network is used to generate a similarity score between a simplified point cloud and its local neighbors in the original point cloud based on the fused features. The distance between the simplified point cloud and its local neighbors is combined with the similarity score to correct the probability position distribution of the projection points.

2. The point cloud downsampling method according to claim 1, characterized in that: in, Use the adjusted Pointnet to extract the features of the original point cloud and the simplified point cloud, mainly including: Send the point cloud to the point alignment module to adjust the points; Send the point cloud adjusted by the point alignment module to the LBR layer; Send the point cloud after the LBR layer to the feature alignment structure; The point cloud that has undergone the feature alignment structure is sent to another LBR layer, and then the K-dimensional semantic features of each point are output.

3. The point cloud downsampling method according to claim 1, characterized in that: in, Feature fusion mainly includes: The point set of the original point cloud is ; The point set of the simplified point cloud is ,remember are the s local neighbors of Q in P; turn up The neighbor points of and construct the graph G(Q,E), whose vertices are , side is ; Maintain the structural connection between the vertex and its neighbors, and map the vertex and its neighbors to new features through learning functions. The formula is as follows: ， in, express exist The indices of the neighboring points in , and Respectively represent and The geometric characteristics of is the difference in features between the simplified point and its neighbors, represents the Euclidean distance between two points, Represents connection, Multilayer Perceptron representing fused features.

4. The point cloud downsampling method according to claim 3, characterized in that: The similarity projection convolutional neural network is used to generate similarity scores based on the fusion features. The distance between the simplified point cloud and its local neighbors is combined with the similarity score to correct the probability position distribution of the projection point. The loss function formula is: ， in, is a hyperparameter used to control the distribution and supervise and The distance between refers to the number of vertices, Refers to the similarity score.

Images