A point cloud registration method based on graph-structured local feature descriptors

Abstract

The present invention relates to the fields of computer vision and robotics, and in particular to a point cloud registration method based on graph-structured local feature descriptors. The method comprises the following steps: S1: performing patch extraction and normalization processing on training sample data and data to be registered; S2: building a deep neural network model based on a graph structure, so that the model is suitable for processing point cloud data; S3: using the preprocessed training sample data in step S1 to train the deep neural network built in step S2; S4: inputting the preprocessed data to be registered into the deep neural network trained in step S3, where the data to be registered is a pair of point clouds to be registered, and obtaining feature descriptor vectors of corresponding points respectively; S5: using the feature descriptor vectors of corresponding points generated in step S4, calculating the rigid transformation matrix [R, t], and using the final rigid transformation matrix [R, t] to achieve point cloud registration, where R represents a rotation matrix and t represents a translation vector.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and robotics technology, and in particular to a point cloud registration method based on graph structure local feature descriptors. Background Art

[0002] With the development and continuous improvement of robotics technology, when it is impossible to receive available satellite signals for positioning and navigation in large and complex disaster environments, simultaneous localization and mapping (SLAM) methods have become a means of perceiving location and environmental information. Point clouds, as a commonly used 3D representation format, provide robots with an opportunity to better understand their surroundings. In SLAM systems, when multiple robots are used for environmental perception, they are generally limited by viewing angles and object occlusions. Multi-view scanning is required to fully cover the entire three-dimensional scene. However, point cloud data obtained from different viewpoints is in different coordinate systems. To obtain a complete three-dimensional scene map, a point cloud registration method is needed to unify the point cloud data from different viewpoints into the same coordinate system.

[0003] Extracting effective 3D point cloud descriptors to improve the accuracy of point cloud registration is gaining increasing attention. Point cloud registration descriptors can be categorized as either traditional handcrafted or deep learning-based methods. While the classic iterative closest point (ICP) algorithm and its variants achieve good registration accuracy, they are overly sensitive to initial position and are therefore inadequate for large-scale 3D point cloud registration. Subsequently, some researchers proposed manually designed descriptors such as FPFH, SHOT, and ROPS for point cloud registration. These methods require only the computation of low-dimensional features such as curves, corners, and handcrafted statistical histograms, but are not robust enough for real-world 3D point clouds.

[0004] Deep learning has achieved significant breakthroughs in computer vision in recent years. However, due to the sparsity, irregularity, and disorder of point clouds, early deep learning-based point cloud registration methods required converting the point clouds into regular geometric forms. These point cloud conversion approaches suffered from information loss and high computational complexity, resulting in unsatisfactory results. It wasn't until 2017 that Charles et al. proposed PointNet and PointNet++, which addressed the disordered nature of point clouds as input to convolutional networks using max pooling. Subsequent work shifted attention to networks that directly process raw 3D point clouds without intermediate conversion operations. While PointNet and PointNet++ achieved promising results in classification and segmentation tasks, they failed to fully consider local geometric information. Consequently, their use for point cloud registration suffers from poor generalization and difficulty adapting to new 3D scenes. Therefore, building deep network models to learn universal and stable local feature descriptors is a crucial challenge in point cloud registration. Summary of the Invention

[0005] In order to solve the above problems, the present invention provides a point cloud registration method based on graph structure local feature descriptors.

[0006] The present invention adopts the following technical solution: a point cloud registration method based on graph structure local feature descriptors, comprising the following steps:

[0007] S1: Patch extraction and normalization of training sample data and data to be registered;

[0008] S2: Build a deep neural network model based on a graph structure to make the model suitable for processing point cloud data;

[0009] S3: Using the pre-processed training sample data in step S1 to train the deep neural network built in step S2;

[0010] S4: Input the pre-processed data to be registered into the deep neural network trained in step S3, where the data to be registered is a pair of point clouds to be registered, and obtain the feature descriptor vectors of the corresponding points respectively;

[0011] S5: Using the feature descriptor vectors of the corresponding points generated in step S4, calculate the rigid transformation matrix [R, t], and use the final rigid transformation matrix [R, t] to achieve point cloud registration, where R represents the rotation matrix and t represents the translation vector.

[0012] Step S1 specifically includes the following sub-steps:

[0013] S11: Using the true transformation matrix T, the overlapping area of ​​the two sets of point clouds in the training sample data is extracted;

[0014] S12: Downsampling is performed using the farthest point sampling in the overlapping area;

[0015] S13: Take out all points contained in the local patch with the sampling point as the center of mass and a radius of r, which is defined as follows:

[0016]

[0017] in is the point in the input patch, Contains all points in the Patch, a is the center of mass, and r is the radius;

[0018] S14: Use the points in the patch to calculate the corresponding local reference coordinate system LRF. The three coordinate axes of LRF are combined into a rotation matrix ;

[0019] S15: Use LRF to rotate the patch and randomly sample from the patch;

[0020] S16: The coordinates of the random sampling points are expressed relative to the center of the patch and normalized to have a unit radius.

[0021] Step S2 specifically includes the following sub-steps:

[0022] S21: Build a feature extraction model based on the graph structure network, so that the points in the patch can be directly input into the graph structure network, and finally map the extracted features in the form of a fully connected network;

[0023] S22: Build a twin network model based on the graph structure network, so that the model branches act on different point clouds respectively, and output the feature descriptors of the corresponding points after full connection mapping.

[0024] The specific process of building the feature extraction model in step S21 is:

[0025] S211: First, the input Patch is passed through a trainable conversion network Spatial T-Net, which takes the original point cloud as input and regresses to a The affine transformation matrix A aligns the coordinates of the points in the Patch. Spatial T-Net uses the PointNet network architecture.

[0026] S212: This is then fed into four groups of GraphConv modules to extract the local geometric features of each point. Each GraphConv module first uses Ball Query to select K neighboring points to form the graph range. A shared-weight multilayer perceptron (64, 64, 28, 256) is then used to extract edge information from the Ball Query graph structure. After feature extraction using graph convolution, k edge features are obtained. This information is then pooled using Maxpooling to aggregate the k edge features and obtain the feature vector of the centroid a.

[0027] S213: Then the output feature vectors of each group of graph convolutions are connected to extract multi-scale features, a layer of Mlp is used to aggregate the multi-scale features, and Maxpooling is used to obtain the global features of the point cloud patch.

[0028] S214: Finally, a set of MLPs outputs a d-dimensional feature vector, and a local response normalization layer is used to generate a unit-length descriptor.

[0029] In step S3, the normalized patch is randomly rotated by [0,π / 30] to expand the training samples for model training.

[0030] In step S5, the RANSAC strategy is used to calculate the rigid transformation matrix [R, t], and the final rigid transformation matrix [R, t] is used to realize the registration of the point cloud, where R represents the rotation matrix and t represents the translation vector.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] 1. This paper extracts local patches and calculates the local coordinate system (LRF) for normalization, fully leveraging the advantages of traditional LRF-based point cloud feature extraction methods and increasing the flexibility of designing local feature extraction algorithms based on deep learning, making the learned descriptors rotationally invariant.

[0033] 2. The present invention uses a deep graph structure network to mine the spatial correlation between point clouds, so that a more powerful feature representation of the input point cloud can be learned, making the learned descriptor more accurate and more robust.

[0034] 3. This paper uses a large amount of data to train a graph-based twin network model to determine matching relationships, making matching more reliable. In addition, data augmentation is used to expand the sample, making the learned descriptors more generalizable and better adaptable to different scenarios.

[0035] 4. Compared with the existing deep neural network based on PointNet as the backbone, the present invention is more robust to rotation transformation, occlusion, and changes in application scenarios, and has better alignment effect due to the use of a graph structure network. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of the registration process of the present invention;

[0037] Figure 2 Schematic diagram of the result of extracting Patch;

[0038] Figure 3 Schematic diagram of the results of calculating LRF;

[0039] Figure 4 Schematic diagram of the network model structure based on the deep graph structure;

[0040] Figure 5 This is a schematic diagram of the structure of Spatial T-Net;

[0041] Figure 6 This is a schematic diagram of the structure of the GraphConv module;

[0042] Figure 7 A twin network model for determining the corresponding relationship;

[0043] Figure 8It is the public 3DMatch dataset;

[0044] Figure 9 It is a public ETH dataset;

[0045] Figure 10 and Figure 11 is a pair of point clouds to be tested on the 3DMatch dataset;

[0046] Figure 12 The test results of the method of the present invention on the 3DMatch dataset;

[0047] Figure 13 and Figure 14 is a pair of point clouds to be tested on the ETH dataset;

[0048] Figure 15 This is the test result of the method of the present invention in the ETH dataset. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments described herein are only used to illustrate and explain the present invention and are not intended to limit the present invention.

[0050] This invention specifically relates to a point cloud registration method based on graph-structured local feature descriptors, designed to address point cloud registration tasks in diverse scenarios, including indoor scene point clouds and outdoor scanned point clouds. To address the difficulty of registration in unseen scenes, a two-stage local feature descriptor extraction method is employed. In the first stage, local patches are extracted from the input point cloud and a local reference frame is computed for normalization, making it robust to rotational transformations. In the second stage, local feature descriptors are learned using a deep learning graph-structured network to exploit spatial correlations between local point clouds.

[0051] Specifically, if Figure 1 As shown, a point cloud registration method based on graph structure local feature descriptors includes the following steps:

[0052] S1. Preprocess the training sample data and the data to be registered:

[0053] In order to convert the point cloud into a set of regular "local structures" and prepare for input into the deep neural network, the training sample data and the data to be registered are first patch extracted and normalized.

[0054] S11. Use the true transformation matrix T to extract the overlapping area of ​​the training sample pairs. Since the matching feature points of point clouds from different perspectives only exist in the overlapping area between point clouds, using the overlapping area of ​​the point cloud pair for feature point extraction and matching can effectively improve configuration efficiency and accuracy.

[0055] S12. In the overlapping area, the farthest point sampling is used to downsample n = 256 points. In order to reduce the amount of data processing, the use of farthest point sampling can cover all points in the space as evenly as possible and has good robustness to the density distribution differences of the point set.

[0056] S13, take out all points contained in the local patch with sampling point a as the center of mass and radius r, see Figure 2 , the black points in the figure are points in the Patch, which can be defined as follows:

[0057]

[0058] in To input Patch The point in Contains all points within the Patch.

[0059] S14. In order to normalize the points in the point cloud, the points in the patch are calculated using a covariance analysis-based method to calculate the corresponding local reference coordinate system (LRF). The three coordinate axes of the LRF can be combined into a rotation matrix .

[0060] S15, such as Figure 3 As shown, LRF is used to rotate the patch and randomly sample from the patch. There are a large number of points in , in order to speed up the calculation of the network, In the random sampling, b=256 points are randomly sampled. The set of randomly sampled points is defined as follows:

[0061] .

[0062] S16. The coordinates of the random sample points are expressed relative to the center of the patch and normalized to have a unit radius. This process is defined as follows:

[0063] .

[0064] S2. Build a deep neural network model based on graph structure to make the model suitable for processing point cloud data.

[0065] S21. Build a feature extraction model based on the graph structure network, so that the points in the patch can be directly input into the graph structure network, and finally map the extracted features in the form of a fully connected network.

[0066] S211: First, the input Patch is passed through a trainable conversion network Spatial T-Net, which takes the original point cloud as input and regresses to a The affine transformation matrix A aligns the coordinates of the points in the Patch. Spatial T-Net uses the PointNet network architecture.

[0067] S212: The local geometric features of each point are then continuously input into four groups of GraphConv modules. Each group of GraphConv modules first uses Ball Query to select K neighboring points to form the range of the graph. A shared weighted multi-layer perceptron (MLP) (64, 64, 28, 256) is used to extract the edge information of the Ball Query graph structure. After feature extraction through graph convolution, k edge feature information is obtained. This information is then subjected to Maxpooling to aggregate the k edge features and obtain the feature vector of the centroid a.

[0068] S213: The output feature vectors of each group of graph convolutions are then connected to extract multi-scale features, a layer of Mlp (1024) is used to aggregate the multi-scale features and Maxpooling is used to obtain the global features of the point cloud patch.

[0069] S214: Finally, a set of MLPs (512, 256, d) are used to output a d-dimensional feature vector, and a local response normalization layer is used to generate a unit-length descriptor.

[0070] The main part of the model is shown in Figure 4 First, the input Patch is passed through the trainable conversion network Spatial T-Net to learn an affine transformation matrix A for coordinate alignment. The processing flow of Spatial T-Net is as follows Figure 5 As shown in the figure, the PointNet network architecture is used. It is then continuously input into four groups of GraphConv (four fully connected layers 64, 64, 128, 256) modules to extract the local geometric features of each point. In order to extract multi-scale features, the output feature vectors of each group of GraphConv are connected, followed by a layer of Mlp (1024) to aggregate multi-scale features and use Maxpooling to obtain the global features of each patch. Finally, a group of Mlp (512, 256, d) is used to output d-dimensional features, and a local response normalization layer (LRN) is used to generate a unit-length descriptor to improve the stability and convergence of network training. In this method, a feature descriptor of d = 32 dimensions is output.

[0071] The detailed structure of GraphConv is as follows Figure 6 As shown, Ball Query is first used to construct the local graph G. BallQuery represents a point a randomly sampled in the Patch i is the center of the sphere, with a radius of Select the closest one to the center of the sphere in the spherical neighborhood of points In the present invention, = 0.3, K = 20. Then vertex a is learned through graph convolution i and each neighboring point Edge features between . A learnable parameter The nonlinear function Find:

[0072]

[0073] in , The edge feature dimension of the layer output. Represents a i The difference between the feature vectors of the points and their neighbors constitutes the local neighborhood information. Then an aggregation function is used to aggregate the features of each directed edge. Perform aggregation operation to get point a i One dimensional eigenvector:

[0074]

[0075] This method selects the aggregation operation based on Maxpooling because Maxpooling is not affected by the order of K nearest neighbors and can extract the most important features from all edges.

[0076] In order to obtain more accurate feature descriptors, the graph structure G is recalculated in each GraphConv by dynamically updating the graph, that is, the K nearest neighbors of a point change with the feature situation in different network layers. l After the edge features of the layer are aggregated, the input representation of the next layer GraphConv is obtained as follows:

[0077] .

[0078] S22. Build a twin network model based on the graph structure network, such as Figure 7 As shown, a Siamese structure with two input branches and weight sharing is used to obtain the feature descriptor used for matching.

[0079] S3, using the pre-processed training sample data in step S1 to train the deep neural network built in step S2.

[0080] The normalized patches are randomly rotated by [0, π / 30] to expand the training sample for model training. Chamfer loss is used to learn an affine transformation to align the two patches, and Hardest-contrastive loss is used to reduce the cost between the two matching patches. The network parameters are learned by minimizing the linear combination of these two loss functions. Hyperparameters such as the learning step size, learning rate, and batch size of the input data are then adjusted based on the specific training situation.

[0081] Chamferloss is used on the affine transformation matrix A to minimize the distance between corresponding points in two patches. The smaller the distance, the better the alignment effect. Chamferloss is defined as follows:

[0082]

[0083] Where, is the number of points in the set, A and A' are the affine transformation matrices of patchX and patchY learned by TNet respectively.

[0084] The Hardest-contrastive loss is used to measure the similarity between two patches, making the distance between two matching patches as small as possible and the distance between two non-matching patches as large as possible. The Hardest-contrastive loss is defined as follows:

[0085]

[0086] In the formula, is the set of all positive sample pairs on the features extracted by GraphConv in minnibatch, and It is a random subset in the feature space used for negative example mining. p and m n are the boundaries of positive sample pairs and negative sample pairs, respectively. The distance between positive sample pairs must be less than m p , the distance between negative sample pairs must be greater than m n , Indicates taking the positive part. In this method, set m p =0.1,m n =1.4.

[0087] The total loss function is defined as: .

[0088] S4. Input the two point clouds to be registered into the deep neural network trained in step S3 to extract the local geometric features of the corresponding points in the two point clouds.

[0089] S5. Using the feature descriptors of the corresponding points generated in step S4, the RANSAC strategy is used to calculate the rigid transformation matrix [R, t], and the final rigid transformation matrix [R, t] is used to achieve point cloud registration to obtain a complete three-dimensional point cloud scene graph.

[0090] In order to facilitate understanding of the above technical solutions of the present invention, the above technical solutions of the present invention are described in detail below through specific embodiments.

[0091] Example

[0092] To verify the effectiveness and generalization of the present invention, the present invention is compared with several other point cloud registration algorithms on the 3DMatch and ETH datasets to verify the superiority of the present invention in different application scenarios.

[0093] The 3DMatch dataset and ETH dataset are used to test the model. Figure 8 As shown in Figure 3, the 3DMatch dataset contains data from 62 real indoor scenes, of which 54 scenes are used for training and 8 scenes are used for evaluation. Figure 9 The ETH dataset consists of some outdoor vegetation acquired by a static ground scanner. The superiority of the method is verified on point cloud pairs with an overlap rate greater than 30%.

[0094] This paper uses a common feature point matching evaluation metric, Feature Matching Recall, to assess the descriptive power of the extracted features. After registration, the quality of the predicted rigid transformation matrix [R, t] is assessed using Registration Recall, which measures the ratio of successfully matched pairs to the total number of point clouds.

[0095] The specific experimental process includes the following steps:

[0096] (1) Model training: The method of the present invention is trained on the training set of the public dataset 3DMatch, using the SGD optimizer with an initial learning rate of 0.01. The training is performed for 10 epochs, and the learning rate is reduced by 10% every 3 epochs.

[0097] (2) Model testing: The model is tested on the 3DMatch test set and the ETH dataset to verify the effectiveness and generalization of the model on these two datasets. Registration is achieved using RANSAC.

[0098] (3) Experimental results: The performance on the two evaluation indicators is shown in the following table:

[0099] Table 1 Comparison of feature matching recall rates on two datasets

[0100]

[0101] Table 2 Comparison of registration recall rates on the 3DMatch dataset

[0102]

[0103] From the above experimental results, it can be seen that the method of the present invention has achieved better performance compared with some classic comparison algorithms, whether on the 3DMatch or ETH datasets. In particular, in Table 1, although PerfectMatch and FCGF showed similar results to the method of the present invention on the 3DMatch dataset, when they were tested on ETH, the feature matching recall rate dropped sharply, while the method of the present invention can still reach 94.4%. This shows that the method has good generalization. Figure 12 , Figure 15 As shown, the method of the present invention can better detect the 3DMatch dataset ( Figure 10 ,11) and the ETH dataset ( Figure 13 ,14) Upper registration.

[0104] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A point cloud registration method based on graph-structured local feature descriptors, characterized by: The following steps are included: S1: Patch extraction and normalization of training sample data and data to be registered; S2: Build a deep neural network model based on a graph structure to make the model suitable for processing point cloud data; Step S2 specifically includes the following sub-steps: S21: Build a feature extraction model based on the graph structure network, so that the points in the patch can be directly input into the graph structure network, and finally map the extracted features in the form of a fully connected network; The specific process of building the feature extraction model in step S21 is: S211: First, the input patch is passed through a trainable transformation network, Spatial T-Net, which takes the original point cloud as input and regresses it to a 3×3 affine transformation matrix A to align the coordinates of the points within the patch. S212: Then, the local geometric features of each point are extracted by continuously inputting into four groups of GraphConv modules. Each group of GraphConv modules first uses Ball Query to select K neighboring points to form the range of the graph. Then, a multi-layer perceptron (64, 64, 128, 256) with shared weights is used to extract the edge information of the Ball Query graph structure. After feature extraction through graph convolution, k edge feature information is obtained. Then, the information is subjected to Maxpooling to aggregate the k edge features and obtain the feature vector of the centroid a. S213: Then the output feature vectors of each group of graph convolutions are connected to extract multi-scale features, a layer of MLP is used to aggregate the multi-scale features, and Maxpooling is used to obtain the global features of the point cloud patch; S214: Finally, a set of MLPs are used to output a d-dimensional feature vector, and a local response normalization layer is used to generate a unit-length descriptor; S22: Build a twin network model based on a graph structure network, so that the model branches act on different point clouds respectively, and output the feature descriptors of the corresponding points after full connection mapping; S3: Using the pre-processed training sample data in step S1 to train the deep neural network built in step S2; S4: Input the pre-processed data to be registered into the deep neural network trained in step S3, where the data to be registered is a pair of point clouds to be registered, and obtain the feature descriptor vectors of the corresponding points respectively; S5: Using the feature descriptor vectors of the corresponding points generated in step S4, calculate the rigid transformation matrix [R, t], and use the final rigid transformation matrix [R, t] to achieve point cloud registration, where R represents the rotation matrix and t represents the translation vector.

2. The point cloud registration method based on graph structure local feature descriptors according to claim 1, characterized in that: The step S1 specifically includes the following sub-steps: S11: Using the true transformation matrix T, the overlapping area of ​​the two sets of point clouds in the training sample data is extracted; S12: Downsampling is performed using the farthest point sampling in the overlapping area; S13: Take out all points contained in the local patch with the sampling point as the center of mass and a radius of r, which is defined as follows: in is the point in the input patch, Contains all points in the Patch, a is the center of mass, and r is the radius; S14: Use the points in the patch to calculate the corresponding local reference coordinate system LRF. The three coordinate axes of LRF are combined into a rotation matrix ; S15: Use LRF to rotate the patch and randomly sample from the patch; S16: The coordinates of the random sampling points are expressed relative to the center of the patch and normalized to have a unit radius.

3. The point cloud registration method based on graph structure local feature descriptors according to claim 1, characterized in that: In step S3, the normalized patch is randomly rotated by [0,π / 30] to expand the training samples for model training.

4. The point cloud registration method based on graph structure local feature descriptors according to claim 1, characterized in that: In step S5, the RANSAC strategy is used to calculate the rigid transformation matrix [R, t], and the final rigid transformation matrix [R, t] is used to achieve point cloud registration, where R represents the rotation matrix and t represents the translation vector.

Images