A point cloud non-rigid registration method based on point cloud consistency relationship reinforcement

Abstract

The application belongs to the technical field of point cloud registration, and relates to a point cloud non-rigid registration method based on point cloud consistency relation reinforcement, comprising the following steps: acquiring point cloud data, randomly dividing and pairing the point cloud data to obtain paired point cloud data; clustering and feature coding the paired point cloud to obtain high-dimensional features; performing self-attention weighting on the high-dimensional features; calculating a correlation matrix and performing desmoothing processing on the correlation matrix; rearranging the point cloud data and the high-dimensional features according to the desmoothed correlation matrix; performing feature extraction on the rearranged high-dimensional features by using an optimized agent task reinforcement network; splicing and fusing all the extracted features; performing symmetric deformation on the fused features by using a symmetric deformation driving network; and registering the point cloud data according to the symmetric deformation features; the application performs uneven attention weighting on the high-dimensional features of the extracted point cloud data, thereby improving the robustness of the network to data of symmetric or similar parts.

Description

Technical Field

[0001] The present invention belongs to the technical field of point cloud registration, and in particular relates to a point cloud non-rigid registration method based on point cloud consistency relationship enhancement. Background Art

[0002] Point cloud data has been widely used in recent years due to its intuitive and accurate ability to reflect the geometric features of an object's surface, as well as the recent improvements in the performance of various sensors and other hardware used for point cloud acquisition and processing. For example, point cloud data is used as a key carrier in areas such as autonomous driving, surgical navigation, and augmented reality. Point cloud registration is an upstream task involved in various technologies for processing point cloud data, and its effectiveness directly impacts downstream tasks such as point cloud classification, segmentation, and reconstruction. Therefore, achieving more accurate registration results plays a vital role in various tasks involving point cloud data processing.

[0003] Similar to image registration, point cloud registration is divided into rigid and non-rigid registration based on whether or not non-rigid deformation is involved before and after registration. Non-rigid point cloud registration involves more degrees of freedom, making it more challenging than rigid registration and subject to limited research. Determining point-by-point correspondences between point clouds, or rather, dense correspondences, is a key area of ​​non-rigid registration and has garnered significant interest among researchers. The accuracy of these dense correspondences has become a crucial metric for evaluating non-rigid point cloud registration. With the advancement of deep learning technology, numerous excellent methods have emerged for processing point cloud data, providing promising solutions for determining dense correspondences in point clouds. However, deep learning is highly data-driven, and obtaining labeled data for dense point cloud relationships is extremely difficult, significantly limiting its performance. Therefore, training networks using unsupervised learning can significantly alleviate the network's demand for labeled point cloud data, thus providing a solution for non-rigid point cloud registration.

[0004] In unsupervised learning, due to the lack of constraints from corresponding label information, deep networks have low learning efficiency, significantly higher model convergence difficulty, and the final model performance may be significantly reduced. Therefore, mining the inherent interrelationships hidden in point clouds and designing corresponding proxy tasks to assist network training can not only alleviate the network's demand for labeled data, but also improve the network's learning efficiency and the accuracy of the network's prediction of point cloud correspondences. Therefore, by designing appropriate proxy tasks, it is possible to effectively learn dense correspondences between point clouds without the assistance of labeled data, which will play an unparalleled role in the field of non-rigid registration of point clouds. Summary of the Invention

[0005] In order to solve the problems existing in the above-mentioned prior art, the present invention proposes a point cloud non-rigid registration method based on point cloud consistency relationship enhancement, the method comprising: acquiring point cloud data, randomly dividing the point cloud data, pairing the randomly divided point cloud data in pairs to obtain paired point cloud data; clustering and feature encoding the paired point clouds to obtain high-dimensional features; performing self-attention weighting on the high-dimensional features; calculating the correlation matrix based on the weighted high-dimensional features, and performing desmoothing on the correlation matrix; rearranging the point cloud data and the high-dimensional features according to the desmoothed correlation matrix; optimizing the proxy task reinforcement network based on point cloud consistency, and using the optimized proxy task reinforcement network to extract features from the rearranged high-dimensional features; splicing and fusing all extracted features; using a deformation-driven network to symmetrically deform the fused features; and registering the point cloud data based on the symmetrically deformed features.

[0006] Beneficial effects of the present invention:

[0007] The network framework proposed in this paper uses deformation to drive network training and learns to accurately predict dense correspondences between point clouds in an unsupervised manner. By exploiting the inherent characteristics of point clouds and their arrangement consistency and permutation invariance, proxy tasks are designed to improve network learning efficiency and prediction accuracy. Furthermore, without the supervision constraints of labeled data, the network effectively learns to accurately predict dense correspondences in point clouds in an end-to-end manner using a symmetric network. Furthermore, the high-dimensional features extracted from the point cloud data are unevenly weighted with attention, improving the network's robustness to data with symmetric or similar parts. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a flow chart of the method of the present invention;

[0009] Figure 2 This is a flow chart of feature extraction of high-dimensional features of the present invention;

[0010] Figure 3 The self-attention encoding and weighting flow chart of high-dimensional features of the present invention;

[0011] Figure 4 Schematic diagram of correspondence matrix calculation and optimization of the present invention;

[0012] Figure 5 This is a schematic diagram of the consistency agent task of the present invention;

[0013] Figure 6 It is a schematic diagram of the feature splicing and fusion of the present invention;

[0014] Figure 7 It is a structural diagram of the decoder of the present invention;

[0015] Figure 8 This is the overall network structure diagram of the present invention;

[0016] Figure 9 This is a visualization diagram of the network output result of the present invention. DETAILED DESCRIPTION

[0017] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0018] A point cloud non-rigid registration method based on point cloud consistency relationship enhancement, such as Figure 1 As shown, the method includes: obtaining point cloud data, randomly dividing the point cloud data, pairing the randomly divided point cloud data in pairs to obtain paired point cloud data; clustering and feature encoding the paired point clouds to obtain high-dimensional features; self-attention weighting the high-dimensional features; calculating the correlation matrix based on the weighted high-dimensional features, and performing desmoothing on the correlation matrix; rearranging the point cloud data and the high-dimensional features according to the desmoothed correlation matrix; optimizing the proxy task reinforcement network based on point cloud consistency, and using the optimized proxy task reinforcement network to extract features from the rearranged high-dimensional features; splicing and fusing all extracted features; using a symmetrical deformation driven network to symmetrically deform the fused features; and aligning the point cloud data according to the symmetrically deformed features.

[0019] A non-rigid registration method for point clouds based on point cloud consistency relationship enhancement, the method comprising: acquiring paired point cloud data; performing local clustering and feature encoding on point clouds; weighting high-dimensional feature self-attention; calculating and smoothing the correlation matrix; rearranging the order of point clouds and their high-dimensional features; proxy task-enhanced network correlation calculation; feature splicing and fusion; and symmetric deformation-driven network optimization.

[0020] In this embodiment, the process of acquiring paired point cloud data includes: randomly pairing the point cloud data in pairs; dividing the point cloud data into training, validation, and test sets. The training set is used to train the model, the validation set is used to verify the model, and the test set is used to test the trained model.

[0021] In this embodiment, the local clustering and feature encoding of the point cloud are as follows: Figure 2 Shown, including:

[0022] Step 1: Calculate the Euclidean distance between a point and all other points in the point cloud;

[0023] Step 2: For any point, select the k points with the smallest distance to it as its neighborhood;

[0024] Step 3: Use Edge-Conv to encode the features of each point and its neighborhood to obtain the high-dimensional features f of each point i ,like Figure 2 (b)

[0025] Step 4: Repeat steps 2 and 3 to increase the network depth;

[0026] Step 5: Perform maximum pooling and average pooling operations on the high-dimensional feature matrix of the point cloud respectively; concatenate and fuse the feature vectors obtained from the two operations, and use the fused features as the global features of the point cloud.

[0027] Preferably, when performing k-nn clustering on the points in the point cloud, the value of k is 20; the process from inputting the point cloud to performing local clustering and feature encoding is as follows: Figure 2 As shown in (a).

[0028] The high-dimensional feature of the midpoint in layer l is denoted as f i l After passing Edge-Conv Block, we can further obtain the high-dimensional features of the points. As shown in the following formula:

[0029]

[0030] in, represents the k-dimensional neighborhood of point i in layer l, max represents the maximum pooling operation, MLP represents the multi-layer perceptron, and cat represents the feature concatenation operation. By repeatedly stacking Edge_Conv Blocks (here 5 times), the feature of each point is increased from 3 dimensions to 128 dimensions.

[0031] The high-dimensional feature self-attention weighted process is as follows Figure 3 Shown, including:

[0032] Step 1: The obtained 128-dimensional point-by-point high-dimensional features are mapped through a linear mapping layer, i.e. Figure 3 The FP layer in the ,formation forms a 1024-dimensional feature vector;

[0033] Step 2: Using the learnable weight matrix W q 、W k With W v Map the high-dimensional features of each point into q, k, v vectors;

[0034] Step 3: Based on the spatial coordinates of the input point cloud, calculate the Euclidean distance between each point and all other points. Then, use the k-nn clustering method to obtain the neighborhood points of each point and the index of the neighborhood points corresponding to each point. Figure 3 The KNN part in .

[0035] Step 4: Based on the results of k-nn local clustering, the neighborhood index of each point is used to cluster the q, k, and v vectors generated in the high-dimensional space to obtain a patch-like structure to better capture the spatial geometric information near the point.

[0036] Step 5: Based on the results of k-nn local clustering, make a difference between each point and all points in its neighborhood, and use the learnable position encoding function δ, that is, Figure 3 The PE part in is linearly mapped to achieve vector position encoding;

[0037] Step 6: Calculate the correlation between the q and k vectors by subtraction, add it to the position encoding result, and pass the structure into the Feed Forward layer, that is, Figure 3 The FF part in , to obtain the patch-to-patch attention weight;

[0038] Step 7: Add the position encoding result to the v vector and add it to the attention weight, i.e. Figure 3 The output of the middle FF layer is multiplied element by element to realize the attention weighted fusion process of the v vector;

[0039] Step 8: Through another linear mapping layer, the v vector after attention weighted fusion is reduced from 1024 dimensions to 128 dimensions, which is equal to the dimension of the encoded high-dimensional feature vector, and added to it to obtain the final high-dimensional feature that integrates the global information of the uneven point cloud.

[0040] Preferably, the length of the q, k, and v vectors is 1024; the value of k in k-nn is 20;

[0041] Preferably, for any point i in the input point cloud, its high-dimensional feature x i The process of generating q, k, v vectors can be expressed as follows:

[0042] q i =W q φ(x i ), k i =W k φ(x i ), v i =W v φ(x i )

[0043] Where φ(·) represents a linear mapping function that transforms x i From 128 dimensions to 1024 dimensions, W q , W k , W v is the learnable weight matrix.

[0044] Preferably, different from ordinary scalar attention, the attention used in the present invention is vector attention, and its calculation and weighted fusion process can be expressed as follows:

[0045]

[0046] in, is the neighborhood of point i, δ is a learnable position encoding function; β is the difference operation; ψ, α represent the generation of the above q, k, v respectively; γ represents the Feed Forward layer; ρ represents the linear mapping function.

[0047] Preferably, the correlation matrix calculation and smoothing process are as follows Figure 4 ,include:

[0048] Step 1: Calculate any point in the input point cloud S in high-dimensional space All points of the input point cloud T The reciprocal of the Euclidean distance between the two points is the correlation c between the two point clouds. i,j , we can preliminarily get the correlation matrix P c ;

[0049] Step 2: Use learnable parameters to adjust the correlation matrix P c Optimize row by row until there is only one dominant element in each row and column;

[0050] Step 3: Use the Dual-Softmax function to sharpen the matrix as a whole to obtain the final correspondence matrix P c ′.

[0051] The correlation formula between two points is as follows:

[0052]

[0053] Dual-Softmax is used. Softmax is first performed on the input correlation matrix column by column, and then the result is multiplied by the original input matrix element by element, and then Softmax is performed on it row by row, so as to amplify the influence of the dominant elements and curb the non-dominant elements, and obtain the deep sharpening correspondence matrix P. c ′, making it closer to the permutation matrix. The optimization process of this matrix is ​​as follows Figure 4 The expression is as follows:

[0054] P′ c =Softmax((Softmax(P c (·,j))·P c )(i,·))

[0055] Optionally, the Sinkhorn algorithm can be used to sharpen the correspondence matrix, but its iterative update calculation form affects the learning efficiency of the network.

[0056] Preferably, the consistency-based proxy task design is as follows Figure 5 ,include:

[0057] Step 1: Get a random permutation matrix P t , and acts on the input point cloud, that is, the spatial coordinate matrix of the point cloud is right-multiplied by the permutation matrix P t , so that the point cloud is in accordance with P t rearrange the form;

[0058] Step 2: Randomly rotate the rearranged point cloud;

[0059] Step 3: Send the original matrix and the rearranged matrix to the feature encoding module, the sub-attention weighting module and the correspondence matrix prediction and desmoothing module respectively to obtain the predicted correspondence matrix P for the proxy task t ';

[0060] Step 4: Based on Focal Loss, all the used permutation matrices P t As the correspondence matrix P of label pair prediction t ’ for optimization.

[0061] Preferably, since the point cloud has the property of being invariant to permutation, the point cloud after rearrangement still retains the spatial geometric characteristics of the original point cloud. In order to avoid the high-dimensional feature matrix obtained subsequently having only a row-permutation relationship with the high-dimensional feature matrix of the original point cloud, the input point cloud data is randomly rotated around the origin in the range of 30° to -30° for each axis as a data enhancement method.

[0062] Preferably, the Focal Loss function L is used Focal , whose expression is:

[0063] L Focal = -λ(1-p) τ log(p)

[0064] Among them, p is the predicted correspondence matrix P t The specific value principles of the elements in ′ are shown in the following formula:

[0065]

[0066] here Represents the permutation matrix P used t , that is, the element in the i-th row and j-th column of the label participating in the training; τ and λ are hyperparameters set empirically to control L Focal The impact on the total loss, here we set τ = 2, λ is a full-1 vector with a dimension equal to the number of categories, that is, 1024.

[0067] The process of feature splicing and fusion is as follows Figure 6 ,include:

[0068] Step 1: Use the optimized correspondence matrix p′c to transform the spatial coordinate matrices S and T of the input point cloud, and the corresponding high-dimensional feature matrix F′ that integrates global information S and F′ T to rearrange;

[0069] Step 2: Rearrange the spatial coordinate matrix and High-dimensional feature matrix F′ that integrates global information S and F′ T , and the global feature vector v of the deformed template point cloud T With v S , perform the splicing operation.

[0070] Preferably, the correspondence matrix P c The action process of ' can be expressed as the following formula:

[0071]

[0072] Preferably, before concatenating the global eigenvectors, copy them until they match the dimensions of the spatial coordinate matrix and the high-dimensional feature matrix. Here, we copy them 1024 times to obtain the concatenated feature matrix F. S_T With F T_S , as shown below:

[0073]

[0074] Among them, cat(·) represents the feature concatenation operation.

[0075] Preferably, the steps of optimizing the symmetrical deformation driven network include:

[0076] Step 1: The concatenated feature matrix is ​​fed into a deformer composed of MLP to deform the template point cloud and directly predict the spatial coordinates of each point in the deformed point cloud.

[0077] Step 2: Calculate the overall loss L of the networktotal , calculate the gradient, update the parameters, and repeat all the above steps until the network converges and the training ends.

[0078] Preferably, the deformer used in the present invention is as follows Figure 7 As shown, the input feature matrix F S _ T With F T _ S The deformation results T with the template point cloud as the target are obtained respectively d , which can be expressed as follows:

[0079]

[0080] in, It shows the deformer used in the present invention.

[0081] Preferably, the total loss L total It consists of three parts: the supervision loss L in the consistency proxy task Focal , the final deformation loss L of the network D And the network regularization loss L Reg , where L D is represented as:

[0082] L D =|||SS d || F +||TT d || F

[0083] in,‖·‖ F Represents the Frobenius norm of the matrix. Network regularization loss L Reg It also includes two items: L Rank With L mfd , which are respectively expressed as:

[0084]

[0085] Among them, P′ c is the correspondence matrix predicted by the network, i.e., the dense correspondence of the point cloud; and Represent the matrix P′ respectively c The i-th row and i-th column of ; and They represent the k-nn neighborhood of the i-th point in the input point clouds S and T respectively; n is the number of points in the point cloud.

[0086] Preferably, the overall network framework of the present invention is as follows Figure 8As shown. It has been verified that with the participation of unlabeled data, the network architecture proposed in this invention has been fully trained and verified on the synthetic dataset SHREC and the real scan dataset MPIDynamic FAUST, respectively. The accuracy of predicting the dense correspondence relationship of point clouds reached 85.25% and 57.92%, respectively, both exceeding the current best methods of the same type, indicating that the network framework proposed in this invention can make effective and accurate predictions of the dense correspondence relationship of this type of data; the network output result visualization diagram is shown in the figure below. Figure 9 shown.

[0087] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation plans of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A point cloud non-rigid registration method based on point cloud consistency relationship enhancement, characterized in that: include: Obtain point cloud data, randomly divide the point cloud data, pair the randomly divided point cloud data to obtain paired point cloud data; cluster and feature encode the paired point clouds to obtain high-dimensional features; Perform self-attention weighting on high-dimensional features; calculate the correlation matrix based on the weighted high-dimensional features and perform desmoothing on the correlation matrix; rearrange the point cloud data and high-dimensional features according to the desmoothed correlation matrix; optimize the proxy task reinforcement network based on point cloud consistency, and use the optimized proxy task reinforcement network to extract features from the rearranged high-dimensional features; splice and fuse all extracted features; use a symmetric deformation drive network to symmetrically deform the fused features; and align the point cloud data based on the symmetrically deformed features. The optimized proxy task reinforcement network is used to extract the rearranged high-dimensional features, including: the input point cloud is permuted according to the given matrix P t to rearrange; Randomly rotate the rearranged point cloud; send the point cloud before and after scrambling to the encoding module, self-attention weighting module and correspondence matrix prediction and desmoothing module respectively to obtain the predicted correspondence matrix P for the proxy task t ′ ; Based on Focal Loss, the permutation matrix P used t As the correspondence matrix P of label pair prediction t ′ Optimize.

2. The point cloud non-rigid registration method based on point cloud consistency relationship enhancement according to claim 1, characterized in that: Clustering and feature encoding of paired point clouds include: calculating the Euclidean distance between a point in the point cloud and all other points; for any point cloud, selecting the k points with the smallest distance to it as its neighborhood; using Edge-Conv to perform feature encoding on each point cloud and its neighborhood to obtain high-dimensional features of each point; repeating the above process until all point clouds are encoded; and performing Max-Pooling and Avg-Pooling operations on the high-dimensional features globally to obtain the global features of the point cloud.

3. The point cloud non-rigid registration method based on point cloud consistency relationship enhancement according to claim 2, characterized in that: Using Edge-Conv to encode features for each point cloud and its neighborhood includes: performing k-nn clustering based on the Euclidean distance between each point in the point cloud and all other points to obtain clustering results; splicing and fusing the features of the center point in the clustering results and each point in its neighborhood; mapping each spliced ​​feature vector to a high-dimensional space to obtain a local feature matrix; performing Max-Pooling operations on the local feature matrix to obtain a high-dimensional feature vector for each point; performing the above operations on all clustering areas to obtain a point-by-point high-dimensional feature matrix for the point cloud; repeating the above operations to obtain the final point-by-point high-dimensional feature matrix for the point cloud.

4. The point cloud non-rigid registration method based on point cloud consistency relationship enhancement according to claim 2, characterized in that: Max-Pooling and Avg-Pooling operations on high-dimensional features include: performing maximum pooling operations and average pooling operations on the high-dimensional feature matrix of the point cloud respectively; concatenating and fusing the feature vectors obtained from the two operations, and using the fused features as the global features of the point cloud.

5. The point cloud non-rigid registration method based on point cloud consistency relationship enhancement according to claim 1, characterized in that: Self-attention weighting for high-dimensional features includes: using a learnable weight matrix W q 、W k With W v The high-dimensional features of each point are mapped to q, k, and v vectors; based on the spatial coordinates of each point and its neighborhood points in the point cloud, k-nn clustering is performed on the q, k, and v vectors to obtain a patch structure; for the spatial coordinates of each point and its neighborhood points, a learnable position encoding function δ is used to encode their positions; the attention weights between patches are calculated; the v vectors are weightedly fused based on the self-attention weights, and the mapping function is used to regain the point-by-point high-dimensional features of the point cloud.

6. The point cloud non-rigid registration method based on point cloud consistency relationship enhancement according to claim 1, characterized in that: Calculating the correlation matrix and smoothing the matrix include: in high-dimensional space, calculating the inverse distance between each point and all other points as the correlation between points, and preliminarily obtaining the correlation matrix P c ; Use learnable parameters to optimize the correspondence matrix row by row; Use the Dual-Softmax function to optimize the matrix as a whole to obtain the final correspondence matrix P c ′ .

7. The point cloud non-rigid registration method based on point cloud consistency relationship enhancement according to claim 1, characterized in that: The splicing and fusion of all extracted features include: using the optimized correspondence matrix P c ′ Rearrange the input matrix; concatenate the rearranged spatial coordinates, high-dimensional features, and global features of the template point cloud in sequence.

8. The point cloud non-rigid registration method based on point cloud consistency relationship enhancement according to claim 1, characterized in that: The symmetrical deformation driven network is used to symmetrically deform the fused features, including: sending the feature matrix obtained after splicing into the deformer composed of MLP, deforming the template point cloud, and directly predicting the spatial coordinates of each point in the deformed point cloud.

Images