A point cloud matching method based on masked reconstruction assisted training network

Abstract

The present invention provides a point cloud matching method based on a masked reconstruction auxiliary training network. During training, the masked reconstruction auxiliary training network independently utilizes the encoded representation of each point cloud obtained from the backbone network to reconstruct the complete point cloud in coordinate space. After training, the masked reconstruction auxiliary training network is separated, thereby avoiding additional inference time. Unlike previous methods that only perform mask reconstruction on the visible part, the masked reconstruction auxiliary training network fully utilizes the transformation between point clouds to avoid early leakage of position information. Therefore, this method directly utilizes the contextual information obtained from the two point clouds, designing a simple method that can model the relationship between point clouds and perform single-shot model training. In addition, this design enables the auxiliary training network to guide the contextual features in the backbone network to capture the geometric details and overall structure of the point cloud pair.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional computer vision, and in particular relates to a point cloud matching method based on an occlusion reconstruction auxiliary training network. Background Art

[0002] Point cloud registration is a fundamental problem in computer vision and robotics, aiming to compute the optimal point cloud registration transformation. In recent years, learning-based methods have made significant progress in 3D point representation, using neural networks to extract point features and establish point correspondences. However, the lack of interaction between point clouds makes it difficult to accurately register partially visible point clouds. Inspired by recent advances in transformers, transformer-based methods use transformers for information exchange and contextual information encoding, demonstrating significant improvements in registration accuracy and robustness. However, low-overlap point clouds share fewer common features, leading to ambiguity in identifying common structures, which reduces the performance of these methods.

[0003] Recently proposed methods attempt to measure pairwise consistency by introducing specially designed encoding methods, enabling better identification of common structure and generation of reliable correspondences. These methods have shown promising results in low-overlap scenarios. However, these modifications may have limitations in generalizability. Furthermore, these methods require additional encoding computations, which incurs additional inference time costs. Summary of the Invention

[0004] In view of this, the object of the present invention is to provide a point cloud matching method based on masked reconstruction auxiliary training network, which can achieve efficient and accurate alignment.

[0005] A point cloud matching method based on masked reconstruction assisted training network, including a training phase and a registration phase;

[0006] In the training phase, for a given source point cloud and target point cloud The backbone network uses kernel point convolutional network to obtain super points and its characteristics Then, a transformer encoder is used to learn context information and extract features In the mask reconstruction auxiliary training network, based on the feature Reconstruct the complete point cloud; at the same time, in the matching network, based on the feature Generate the corresponding point cloud And predict the superposition score Finally, the optimal transformation is estimated based on the predicted correspondences and the superposition scores

[0007] After the training of the joint network model consisting of the backbone network, matching network and mask reconstruction auxiliary training network is completed, in the registration stage, the two point clouds to be registered are input into the backbone network, and after the features are extracted, they are sent to the matching network to obtain the pose transformation.

[0008] The occlusion reconstruction auxiliary training network includes a patch generation module, an occlusion reconstruction decoder, a prediction head and a loss function module;

[0009] The point patch generation module is used for point cloud First, use the farthest point sampling to sample it into g center points Then, the nearest neighbor algorithm is used to select The k nearest points in the , to construct g point blocks The generation process can be expressed as:

[0010]

[0011] For point clouds Construct g point blocks

[0012]

[0013] The masked reconstruction decoder consists of L d layers, each layer includes a self-attention sublayer and a feedforward sublayer; the input of the decoder is the complete set of tags and These tags are encoded feature sets and and the mask marker set and where each mask tag is a shared, learnable vector representing a point patch to be predicted;

[0014] In each masked reconstruction decoder layer, the position encoding of the marker is obtained by sinusoidal position encoding PE and added to it by direct addition Marking, getting marked

[0015] Positional encoding Position encoding including coded features and the positional encoding of the mask markers The position encoding of the encoded feature is generated directly from the center point of the point cloud, while the position encoding of the mask mark is generated using the transformed center point of another point cloud:

[0016]

[0017]

[0018] in, and They represent the true value transformation from Y to X and from X to Y respectively;

[0019] The overall process of the mask reconstruction decoder is as follows:

[0020]

[0021]

[0022] The prediction head is used to reconstruct each point patch of another aligned point cloud in the coordinate space; it consists of a two-layer perceptron; the prediction head first obtains the decoded mark Get the decoded mask mark and and project it into a vector with the number of output channels equal to the total number of coordinates in the patch; these vectors are then reconstructed into the predicted point patch in a reciprocal manner

[0023]

[0024]

[0025] Reshape means transforming the matrix shape, MLP means multi-layer perceptron, which consists of linear layer and activation function;

[0026] The loss function module is used to construct the loss function for training the backbone network and the masking and reconstruction auxiliary training network, specifically:

[0027] Each mask marker is responsible for predicting the corresponding point patch in the other aligned point cloud, specified by its position encoding; given the ground-truth point patch and and from and Obtained prediction point patch and Reconstruction loss Use l2Chamfer distance representation; in Defined as:

[0028]

[0029] in, express The cardinality of the set, and Represents the squared error between a and b.

[0030] Preferably, the matching network includes an error correction module and a posture solving module;

[0031] The error correction module is used to: and Generate corresponding points using superpoints and That is, the corresponding point Obtained by projecting the center position of each superpoint into another point cloud:

[0032]

[0033] Similarly, get the corresponding points

[0034]

[0035] Among them, softmax represents the normalized exponential function;

[0036] For each predicted point coordinate in the corresponding point, it is concatenated with the encoded feature and mapped through two layers of fully connected layers to obtain the compensation value required to correct the deviation. Correction corresponding point Expressed as:

[0037]

[0038] Similarly, Correction corresponding point It can be expressed as:

[0039]

[0040] Among them, Concat represents the matrix connection operation;

[0041] Subsequently, a single FC layer with a sigmoid activation function is used to generate the overlap score. and

[0042] The pose solving module is used to:

[0043] First, the predicted transformed positions in both directions are concatenated to obtain a final set of M′+N′ corresponding points:

[0044]

[0045] The required rigid transformation is estimated by solving the following problem from the estimated corresponding points:

[0046]

[0047] in yes The element in row i.

[0048] Preferably, the backbone network includes a kernel point convolution module; composed of blocks similar to residual networks and strided convolutions for downsampling and feature extraction; the kernel point convolution backbone downsamples the point cloud arrive And extract relevant features, which are again linearly projected to obtain features

[0049] Preferably, the backbone network includes a converter encoder;

[0050] Super Point and and their related characteristics and Input L e Layer converter encoder; the converter encoder performs information exchange and context information extraction to obtain encoding features and Each transformer encoder layer consists of a self-attention sublayer and a criss-cross attention sublayer, followed by a feedforward network (FFN). Both the self-attention layer and the criss-cross attention layer belong to the multi-head attention layer (MA). The three inputs of the self-attention layer are the same, while those of the criss-cross attention layer are different. The overall flow of the transformer encoder is as follows, with the self-attention layer, criss-cross attention layer, and feedforward network performed in sequence:

[0051]

[0052]

[0053]

[0054] The multi-head attention layer is shown in the following formula. Each attention equation obtains the attention map through the scaled dot product operation, and then multiplies the map with V to aggregate information. Subsequently, the results are concatenated and used W O Project to get the final value:

[0055]

[0056]

[0057] Among them, d K is the dimension of key K.

[0058] The present invention has the following beneficial effects:

[0059] The present invention provides a point cloud matching method based on a masked reconstruction auxiliary training network. During training, the masked reconstruction auxiliary training network independently utilizes the encoded representation of each point cloud obtained from the backbone network to reconstruct the complete point cloud in coordinate space. After training, the masked reconstruction auxiliary training network is separated, thereby avoiding additional inference time. Unlike previous methods that only perform mask reconstruction on the visible part, the masked reconstruction auxiliary training network fully utilizes the transformation between point clouds to avoid early leakage of position information. Therefore, this method directly utilizes the contextual information obtained from the two point clouds, designing a simple method that can model the relationship between point clouds and perform single-shot model training. In addition, this design enables the auxiliary training network to guide the contextual features in the backbone network to capture the geometric details and overall structure of the point cloud pair. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is the overall framework of the point cloud matching method based on the masked reconstruction auxiliary training network of the present invention. DETAILED DESCRIPTION

[0061] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0062] Given a source point cloud and target point cloud The goal of point cloud registration is to predict a rotation matrix and a translation vector Align the source point cloud with the target point cloud.

[0063] The network model of the present invention includes a backbone network, a matching network and an occlusion reconstruction auxiliary training network; the backbone network includes a kernel point convolution network and a converter encoder; the matching network includes an error correction module and a posture solution module.

[0064] The overall process of the point cloud matching method based on masked reconstruction auxiliary training network of the present invention is as follows: Figure 1 The training phase begins with the backbone network, which uses a kernel point convolutional network to obtain super points and its characteristics Then, a transformer encoder is used to learn context information and extract features These features are used to assist in training the mask reconstruction network to reconstruct the complete point cloud. Used to generate the corresponding point cloud in the error correction module And predict the superposition score Finally, the optimal transformation is estimated based on the predicted correspondences and the superposition scores

[0065] In the registration stage, the two point clouds to be registered are input into the backbone network, and after the features are extracted, they are sent to the matching network to obtain the pose transformation. Among them, since the auxiliary training network is separated, no additional inference time is introduced.

[0066] 1. Kernel point convolution module

[0067] The kernel point convolution backbone consists of residual network-like blocks and strided convolution for downsampling and feature extraction. arrive And extract relevant features, which are again linearly projected to obtain features

[0068] 2. Converter Encoder

[0069] Super Point and and their related characteristics and Input L e Layer transformer encoder. The transformer encoder performs information exchange and context information extraction to obtain encoding features and Each transformer encoder layer consists of a self-attention sublayer and a criss-cross attention sublayer, followed by a feed-forward network (FFN). To incorporate position information, a sinusoidal positional encoding is added to the input of each attention sublayer. The self-attention mechanism allows each point to interact with all points in the same point cloud, while the criss-cross attention mechanism allows one point cloud to perceive another.

[0070] The self-attention layer and the criss-cross attention layer are both multi-head attention (MA) layers. The three inputs to the self-attention layer are the same, while those to the criss-cross attention layer are different. The overall flow of the converter encoder is shown below. The self-attention layer, the criss-cross attention layer, and the feedforward network are performed in sequence.

[0071]

[0072]

[0073]

[0074] The multi-head attention layer is shown below. Each attention equation obtains an attention map through a scaled dot product operation, and then multiplies the map with V to aggregate information. Subsequently, the results are concatenated and used W O Projection is performed to obtain the final value.

[0075]

[0076]

[0077] Among them, d K is the dimension of key K.

[0078] 3. Occlusion Reconstruction Auxiliary Training Network

[0079] The masked reconstruction auxiliary training network consists of four key components: a point patch generation module, a masked reconstruction decoder, a prediction head, and a loss function module. Initially, each point cloud is downsampled to its center point and a point patch is generated. These are used to generate the position encoding of the mask markers and the reconstruction target of the masked reconstruction auxiliary training network, respectively. Next, the masked reconstruction decoder aggregates the information obtained from the encoded features of each point cloud, and the prediction head predicts the point patch in the other aligned point cloud.

[0080] Point patch generation module: Point cloud has the characteristics of sparseness and disorder. In order to deal with this characteristic, the present invention adopts the farthest point sampling (FPS) (Qi, CR, Yi, L., Su, H., et al. (2017). Pointnet++) and the nearest neighbor algorithm (KNN) to generate center points and point patches. Specifically, for point cloud The present invention first uses the farthest point sampling to sample it into g center points Then, the nearest neighbor algorithm is used to select the The k nearest points in the , to construct g point blocks The generation process can be expressed as:

[0081]

[0082] For point clouds In a similar way, construct g point blocks

[0083]

[0084] Occlusion reconstruction decoder: Occlusion reconstruction decoder is composed of L d layers, each layer includes a self-attention sublayer and a feedforward sublayer. The input of the decoder is the complete set of tags and These tags are encoded feature sets and and the mask marker set and Each mask token is a shared, learnable vector representing a point patch to be predicted.

[0085] Due to the disordered nature of the attention mechanism, it is crucial to incorporate positional encoding to specify the point patch that each masked marker is responsible for predicting. To this end, in each masked reconstruction decoder layer, the positional encoding of the marker is obtained via sinusoidal positional encoding (PE) and added to it by direct addition Marking, getting marked

[0086] Positional encoding Position encoding including coded features and the positional encoding of the mask markers The position encoding of the encoded feature is generated directly from the center point of the point cloud, while the position encoding of the mask mark is generated using the transformed center point of another point cloud:

[0087]

[0088]

[0089] in and They represent the true value transformation from Y to X and from X to Y respectively.

[0090] Next, the present invention uses the multi-head self-attention mechanism (MSA) to aggregate the information extracted from the encoded features. The self-attention mechanism establishes The relationship between the markers in the mask From the coded features In addition, the self-attention mechanism effectively avoids direct interaction between the two point clouds in the decoder stage, promoting the backbone network to capture contextual features, thereby promoting the capture of geometric details and overall structure.

[0091] The overall process of the mask reconstruction decoder is as follows:

[0092]

[0093]

[0094] Prediction head: The prediction head is used to reconstruct each point patch of another aligned point cloud in the coordinate space. It consists of a two-layer perceptron. The prediction head first extracts the decoded markup from the Get the decoded mask mark and And project it into a vector with the number of output channels equal to the total number of coordinates in the patch. These vectors are then reconstructed into the predicted point patch in a mutual manner

[0095]

[0096]

[0097] Loss function module: The method of the present invention additionally considers the overlapping area in the other point cloud. This guides the backbone network to capture the differences in the overlapping areas between the two point clouds at the super-point level and model their geometric details more accurately. Therefore, the reconstruction goal of the present invention is to restore each point patch in the other aligned point cloud, thereby achieving more accurate registration. The specific definition of the loss function is as follows: Each mask marker is responsible for predicting the corresponding point patch in the other aligned point cloud, specified by its position encoding. Given a true point patch and and from and Obtained prediction point patch and Reconstruction loss It is represented using l2Chamfer distance. in Defined as:

[0098]

[0099] in express The cardinality of the set, and represents the squared error between a and b. To facilitate convergence during training, point patches are represented as normalized coordinates relative to their center points.

[0100] 4. Error correction module

[0101] While reconstructing the task, the encoded features extracted from the encoder and is used to establish the correspondence. In order to achieve efficient registration, and Generate corresponding points using superpoints without upsampling and Specifically, the corresponding points It can be obtained by projecting the center position of each superpoint into another point cloud.

[0102]

[0103] Similarly, get the corresponding points

[0104]

[0105] However, super point matching is sparse and imprecise, which hinders the accurate prediction of corresponding points. Since the point cloud pairs are rich in detail information, especially the context features preserved in the overlapping areas, the spatial deviation between the predicted corresponding points can be predicted and corrected. Specifically, for each predicted corresponding point, the point coordinates are concatenated with the encoded features and mapped through two fully connected layers to obtain the compensation value required to correct the deviation. Therefore, Correction corresponding point It can be expressed as:

[0106]

[0107] Similarly, Correction corresponding point It can be expressed as:

[0108]

[0109] Subsequently, the overlap score and These scores indicate the probability that a point lies in the overlapping region and are generated by a single FC layer with a sigmoid activation function:

[0110]

[0111] 5. Posture calculation module

[0112] First, the predicted transformed positions in both directions are concatenated to obtain a final set of M′+N′ corresponding points:

[0113]

[0114] The required rigid transformation can be estimated by solving the following problem from the estimated corresponding points:

[0115]

[0116] in yes The element in row i;

[0117] 6. Other loss functions

[0118] The method of the present invention also uses three other loss functions to train the backbone network and the matching network: overlap loss Corresponding losses and feature loss By introducing coefficients λc and λf, the final loss function is constructed and expressed as:

[0119]

[0120] Overlap loss: Used to measure ground truth label overlap Overlap score with prediction consistency between them. in Defined as:

[0121]

[0122] Overlapping labels The threshold r is set by aligning the nearest point distance of point cloud X,Y o To obtain, where o X ,o Y It is obtained by calculating the distance between the nearest points in the point cloud X and Y.

[0123] Corresponding losses: For l-based 1 The loss is used to measure the accuracy of the predicted corresponding point cloud in the overlapping area. The corresponding loss Defined as:

[0124]

[0125] in is a true transformation from X to Y.

[0126] Feature loss: Used to measure the discriminative ability of extracted features based on InfoNCE loss. in Defined as:

[0127]

[0128]

[0129] in, Indicates that The corresponding point set in represents the extracted features of point x; p x and n x Indicates that according to the positive and negative boundaries (r p ,r n ) The selected positive and negative points are W f is a learnable linear transformation.

[0130] Table 1 Performance of various methods on 3DMatch and 3DLoMatch, where the units of RRE, RTE, and RR are °, m, and %, respectively.

[0131]

[0132]

[0133] 3DMatch: To demonstrate the performance of our method for real-world point cloud matching, experiments were conducted on 3DMatch {Huang, 2021#68}. The 3DMatch dataset is a real-world pairwise registration dataset with 46 scenes designed for training and the remaining 16 scenes evenly divided between validation and testing. Comparison methods were evaluated on the 3DMatch (overlap > 30%) {Zeng, 2017#136} and 3DLoMatch (overlap 10%-30%) {Huang, 2021#68} benchmarks.

[0134] Comparison Methods: Our method is compared with the state-of-the-art methods RegTR{Yew,2022#134}, Lepard{Li,2022#64}, and SC$^2$PCR{Chen,2022#67}. In addition, the comparison methods also include representative methods on 3DMatch: 3DSN{Gojcic,2019#127}, FCGF{Choy,2019#128}, CG-SAC{Quan,2020#124}, D3Fea{Bai,2020#129}, DGR{Choy,2020#62}, PCAM{Cao,2021#133}, MNet{Xu,2021#131}, DHVR{Lee,2021#126}, Predator{Huang,2021#68}, CoFiNet{Yu,2021#113}.

[0135] Evaluation method: The performance of each method is evaluated using the following evaluation methods: (1) relative rotation error (RelativeRotationError, RRE) is the distance between the estimated and true value rotation matrix, relative translation error (RelativeTranslationError, RTE) RTE (evaluates the Euclidean distance between the predicted value and the true value) and matching recall (RegistrationRecall, RR) evaluates the percentage of successful matching pairs, where a successful matching pair is defined as a matching pair with a correspondence root mean square error below 0.2).

[0136] Comparison results analysis: The comparison results are shown in Table 1. The proposed method can accurately align real-world point cloud pairs even at low overlap rates and outperforms other methods on 3DMatch and 3DLoMatch. Specifically, the proposed method achieves 95.1% RR on the 3DMatch benchmark, surpassing all the compared methods. Even compared with Lepard and SC2 Compared to PCR, our method achieves at least a 1.6% improvement in RR and reduces both RRE and RTE by 33.8%-46.7%. These results demonstrate that our method effectively captures the overall structure of point cloud pairs, enhancing its ability to identify overlapping regions and predict corresponding points. Consequently, our method is able to accurately align real-world point clouds with superior accuracy and RR values.

[0137] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A point cloud matching method based on masked reconstruction assisted training network, characterized in that: It includes training phase and registration phase; In the training phase, for a given source point cloud and target point cloud The backbone network uses kernel point convolutional network to obtain super points and its characteristics Then, a transformer encoder is used to learn context information and extract features In the mask reconstruction auxiliary training network, based on the feature Reconstruct the complete point cloud; at the same time, in the matching network, based on the feature Generate the corresponding point cloud And predict the superposition score Finally, the optimal transformation is estimated based on the predicted correspondences and the superposition scores After the training of the joint network model consisting of the backbone network, matching network and mask reconstruction auxiliary training network is completed, in the registration stage, the two point clouds to be registered are input into the backbone network, and after the features are extracted, they are sent to the matching network to obtain the pose transformation. The occlusion reconstruction auxiliary training network includes a patch generation module, an occlusion reconstruction decoder, a prediction head and a loss function module; The patch generation module is used for point cloud First, use the farthest point sampling to sample it into g center points Then, the nearest neighbor algorithm is used to select The k nearest points in the , to construct g point blocks The generation process can be expressed as: For point clouds Construct g point blocks Assume that the mask reconstruction decoder is composed of L d layers, each layer includes a self-attention sublayer and a feedforward sublayer; the input of the decoder is the complete set of tags and These tags are encoded feature sets and and the mask marker set and where each mask tag is a shared, learnable vector representing a point patch to be predicted; In each masked reconstruction decoder layer, the position encoding of the marker is obtained by sinusoidal position encoding PE and added to it by direct addition Marking, getting marked Positional encoding Position encoding including coded features and the positional encoding of the mask markers The position encoding of the encoded feature is generated directly from the center point of the point cloud, while the position encoding of the mask mark is generated using the transformed center point of another point cloud: in, and They represent the true value transformation from Y to X and from X to Y respectively; The overall process of the mask reconstruction decoder is as follows: The prediction head is used to reconstruct each point patch of another aligned point cloud in the coordinate space; it consists of a two-layer perceptron; the prediction head first obtains the decoded mark Get the decoded mask mark and and project it into a vector with the number of output channels equal to the total number of coordinates in the patch; these vectors are then reconstructed into the predicted point patch in a reciprocal manner Reshape means transforming the matrix shape, MLP means multi-layer perceptron, which consists of linear layer and activation function; The loss function module is used to construct the loss function for training the backbone network and the masking and reconstruction auxiliary training network, specifically: Each mask marker is responsible for predicting the corresponding point patch in the other aligned point cloud, specified by its position encoding; given the ground-truth point patch and and from and Obtained prediction point patch and Reconstruction loss Use l2Chamfer distance representation; in Defined as: in, express The cardinality of the set, and Represents the squared error between a and b.

2. The point cloud matching method based on masked reconstruction auxiliary training network according to claim 1, characterized in that: The matching network includes an error correction module and a posture solution module; The error correction module is used to: and Generate corresponding points using superpoints and That is, the corresponding point Obtained by projecting the center position of each superpoint into another point cloud: Similarly, get the corresponding points Among them, softmax represents the normalized exponential function; For each predicted point coordinate in the corresponding point, it is concatenated with the encoded feature and mapped through two layers of fully connected layers to obtain the compensation value required to correct the deviation. Correction corresponding point Expressed as: Similarly, Correction corresponding point It can be expressed as: Among them, Concat represents the matrix connection operation; Subsequently, a single FC layer with a sigmoid activation function is used to generate the overlap score. and The pose solving module is used to: First, the predicted transformed positions in both directions are concatenated to obtain a final set of M′+N′ corresponding points: The required rigid transformation is estimated by solving the following problem from the estimated corresponding points: in yes The element in row i.

3. The point cloud matching method based on masked reconstruction auxiliary training network according to claim 2, characterized in that: The backbone network includes a kernel point convolution module; it is composed of blocks similar to residual networks and strided convolution for downsampling and feature extraction; the kernel point convolution backbone downsamples the point cloud arrive And extract relevant features, which are again linearly projected to obtain features 4. The point cloud matching method based on masked reconstruction auxiliary training network according to claim 1, characterized in that: The backbone network includes a converter encoder; Super Point and and their related characteristics and Input L e layer transformer encoder; The transformer encoder performs information exchange and context information extraction to obtain encoding features and Each transformer encoder layer consists of a self-attention sublayer and a cross-attention sublayer, followed by a feed-forward network (FFN). Both the self-attention layer and the cross-attention layer belong to the multi-head attention layer. The three inputs of the self-attention layer are the same, while the three inputs of the cross-attention layer are different. The overall flow of the transformer encoder is as follows, with the self-attention layer, the cross-attention layer, and the feed-forward network in sequence: The multi-head attention layer is shown in the following formula. Each attention equation obtains the attention map through the scaled dot product operation, and then multiplies the map with V to aggregate information. Subsequently, the results are concatenated and used W O Project to get the final value: Among them, d K is the dimension of key K.

Images