A point cloud registration method based on semantic and structural collaborative guidance

Abstract

The invention belongs to the field of point cloud registration, and relates to a point cloud registration method based on semantic and structural collaborative guidance, which includes obtaining point cloud semantic information, wherein the point cloud semantic information includes semantic information of outdoor radar point clouds and semantic information of indoor reconstructed point clouds; sampling point clouds to obtain super points; extracting geometric structural features of super points; performing super point semantic feature representation processing on point cloud semantics to obtain super point semantic features; adopting a multimodal feature fusion module to fuse super point semantic features and geometric structural features to obtain enhanced super point features; performing super point matching according to the enhanced super point features, performing point matching according to the super point matching results, and completing point cloud registration; the invention considers both semantic information and structural information, and assists point cloud registration under the synergistic effect of the two, and the registration performance is significantly improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of point cloud registration, and in particular relates to a point cloud registration method based on semantic and structural collaborative guidance. Background Art

[0002] Current point cloud registration methods can be roughly divided into traditional methods and learning-based methods. Traditional methods usually rely on hand-crafted feature descriptors to establish correspondences between point clouds, starting the initial registration to obtain the initial transformation matrix. Subsequently, through iterative optimization, they minimize the distance between point clouds or maximize the similarity between matching point pairs. In addition, they also apply pose estimators RANSAC and SVD to refine the transformation matrix based on correspondences. However, hand-crafted descriptors heavily rely on expert knowledge and have limited flexibility.

[0003] Due to the booming development of deep learning in various fields, people have made concerted efforts to learn point cloud feature descriptors in a data-driven manner. These data-driven methods usually rely on training data to ensure generalizability. Most methods follow a correspondence-based approach, first extracting correspondences and then recovering the transformation using a robust pose estimator. In order to seek more accurate correspondences, many recent works have deeply studied the structural information in point clouds. However, sometimes the extracted structural features lack discriminability in certain scenes due to repetitive structures and low-geometry areas. These problems pose significant challenges to achieving reliable registration by accurately locating point correspondences. Summary of the invention

[0004] In order to solve the problems existing in the above prior art, the present invention proposes a point cloud registration method based on semantic and structural collaborative guidance, which includes:

[0005] Acquire point cloud semantic information, wherein the point cloud semantic information includes semantic information of outdoor radar point cloud and semantic information of indoor reconstructed point cloud; adopt point cloud semantic segmentation model to obtain semantic information of outdoor radar point cloud; indoor reconstructed point cloud data includes source point cloud data, target point cloud data, image corresponding to source point cloud and image corresponding to target point cloud; input the image corresponding to source point cloud and the image corresponding to target point cloud into Semantic-SAM image segmentation model to obtain image pixel semantics; project image pixel semantics through projection module to obtain semantic information of indoor reconstructed point cloud;

[0006] The point cloud is sampled to obtain super points; the geometric structure features of the super points are extracted using an encoder, the angle and distance between two super points are calculated, and the angle and distance between the super points are embedded into the geometric structure features;

[0007] The point cloud semantics is represented by super-point semantic features to obtain super-point semantic features. The super-point semantic features and geometric structure features are fused using a multimodal feature fusion module to obtain enhanced super-point features. Super-point matching is performed based on the enhanced super-point features, and point matching is performed based on the super-point matching results to complete point cloud registration.

[0008] Beneficial effects of the present invention:

[0009] The present invention considers both semantic information and structural information, and assists in point cloud registration under the synergy of the two, and the registration performance is significantly improved, especially in the case of low overlap. In addition, previous methods using semantics all use point cloud semantic segmentation models to obtain semantics, and have not explored the situation where there is a lack of point cloud semantic segmentation models. Therefore, when the point cloud has a corresponding image, the present invention uses a large image model to perform zero-sample segmentation on the image to assist in obtaining the semantics of the point cloud. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is the overall flow chart of the present invention;

[0011] Figure 2 This is a network structure diagram of the present invention. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0013] A point cloud registration method based on semantic and structural collaborative guidance, such as Figure 1 and Figure 2 As shown, the method includes: obtaining point cloud semantic information, and proposing two approaches for different types of data: 1) using a point cloud semantic segmentation model to obtain semantic information of outdoor radar point clouds; 2) indoor reconstructed point cloud data, specifically point cloud data including source point cloud data, target point cloud data, an image corresponding to the source point cloud, and an image corresponding to the target point cloud; inputting the image corresponding to the source point cloud and the image corresponding to the target point cloud into a Semantic-SAM image segmentation model to obtain image pixel semantics; projecting the image pixel semantics through a projection module to obtain semantic information of indoor reconstructed point clouds.

[0014] The point cloud is sampled to obtain super-points; the encoder is used to extract the geometric structure features of the super-points, the angle and distance between two super-points are calculated, and the angle and distance between the super-points are embedded into the geometric structure features; the point cloud semantics are represented by super-point semantic features to obtain super-point semantic features; the super-point semantic features and geometric structure features are fused using a multimodal feature fusion module to obtain enhanced super-point features; super-point matching is performed based on the enhanced super-point features, and point matching is performed based on the super-point matching results to complete point cloud registration.

[0015] In this embodiment, a super-point semantic feature representation module is used to describe the semantic information of the super-point using the original point semantic information of the super-point neighborhood, fuse the semantic information of multiple categories, and reduce the loss of semantic information during the downsampling process. In order to further explore the semantic and structural correspondences and global correlations within and between point clouds, a multimodal feature fusion module is proposed. First, by integrating the structural features and semantic features using the self-attention mechanism, the network can better understand the global correlations within the semantic space and structural space of the point cloud during the representation learning process. Secondly, a cross-attention mechanism is used to enhance the interaction between the semantics and structures between point clouds, so that the network can automatically focus on similar semantic and structural areas. Finally, a super-point feature with discriminability is obtained, and the corresponding relationship with high confidence is selected through the feature similarity score.

[0016] Sampling point cloud data includes two point clouds P and Q with different perspectives but partial overlap in a scene. First, the two original related features are passed through a downsampling network to obtain super-points and local structural features of super-points. The next layer of super-points in the downsampling network is sampled with a resolution of 1 / 2 of the previous layer, and at the same time, the features of the previous layer of super-points will be aggregated to the super-points in the next layer that are closest in Euclidean space.

[0017] The encoder module will output two point cloud super points and the characteristics of super points It is expressed as follows:

[0018] Obtaining the original point cloud semantic information includes: For different types of data, two approaches are proposed, and the following uses P point cloud as an example. First, for outdoor radar point cloud data, the original point cloud semantic information is obtained using the point cloud semantic segmentation model. Second, for the point cloud data of indoor reconstruction, the image pixel semantics is obtained using the image large model Then use the projection formula (1) to obtain the semantic information of the original point cloud The projection formula is:

[0019]

[0020] in is the camera external parameter including rotation and translation matrix and is the camera internal parameter. f(·) represents a homogeneous function. (w a ,h b ) represents the i-th point in the point cloud th Points p i The pixel position in the image. |W||H| represents the pixel position. Indicates the image (w a ,h b )’s pixel semantics. Indicates the i-th point in the point cloud th Points p i semantics.

[0021] The downsampled features of superpoints effectively capture the local structural features of the point cloud, but they ignore the geometric structural features of the distance and angle between superpoints. The geometric structural features are obtained by geometric structural embedding. The following uses the P point cloud as an example. The i-th superpoint in the P point cloud and the jth superpoint Two embeddings are applied. The first is the pairwise distance embedding, calculated as shown in Formula 2:

[0022]

[0023] where S(·) is a sine function, σ d is a hyperparameter used to adjust the sensitivity of distance changes. represents the Euclidean distance between two superpoints. The other is the three-element angle embedding, calculated as shown in Formula 3:

[0024]

[0025] in yes k nearest neighbors, σ a It is the sensitivity parameter that controls the angle change.

[0026] The geometric structure characteristics of the final superpoint are:

[0027]

[0028] in These are two embedded projection matrices.

[0029] In this embodiment, in order to maximize the use of semantic information in the original point cloud, a super-point semantic feature representation module is introduced to obtain fine-grained semantic features of super-points. The source point cloud P is used as an example to explain the representation process. Since downsampling involves merging the original points in the patch into super-points The semantic information of the original point in the patch is used to express the semantic features of this super point, which is expressed as The calculation method is:

[0030]

[0031] Where n is the number of points in the i-th superpoint, and k is the number of categories in the dataset. BOW(·) represents the bag-of-words model, which counts the internal category and frequency of each superpoint and encodes it into semantic features.

[0032] Since global context has been shown to be crucial in many computer vision tasks, transformers have been used to exploit global contextual information for point cloud registration. However, existing methods often use transformers to implicitly or explicitly fuse geometric structures, ignoring semantic information. This results in low distinguishability of learned features in structurally similar scenes, leading to severe matching ambiguity and many wrong correspondences, especially in low overlap cases. A simple approach is to explicitly inject semantic information to ensure that the learned features are not only structurally relevant but also semantically relevant.

[0033] In this embodiment, in order to solve the problem of global environmental features, the present invention designs a multimodal feature fusion module, which not only encodes implicit structural information and explicit structural information to achieve structural consistency within the point cloud, but also explicitly captures the semantic relevance within the point cloud and between point clouds. The multimodal feature fusion module consists of two parts: structural semantic self-attention, which learns the geometric structure transformation invariance and semantic relevance within the point cloud; structural semantic cross-attention, which learns the structural and semantic consistency between point clouds.

[0034] There is a lot of contextual information inside the point cloud. Therefore, the present invention provides a structure-semantic self-attention mechanism to learn the global correlation between super-points in the feature space, structure space and semantic space, and embed the structural and semantic contextual information into the features of the super-points. This process is illustrated using the example of the source point cloud P. For two super-points and The matrices of their query, key, value, structure and semantics are calculated as follows:

[0035]

[0036]

[0037]

[0038]

[0039] The superscript s represents the self-attention mechanism, W and b are the parameters shared in each feature self-attention mechanism, and It is the downsampled local structural feature of the j-th superpoint and the i-th superpoint in the P point cloud. is the semantic feature of the jth superpoint, is the geometric structure feature of the jth superpoint.

[0040] Calculate superpoint and The attention score of the semantic structure feature between The formula is as follows:

[0041]

[0042] Where softmax is the probability distribution function that transforms the output to [0,1], and the superscript T is the transpose of the matrix.

[0043] By learning features from all other super-points in the same point cloud By performing weighted summation, we can obtain enhanced structural semantic features. The calculation formula is as follows:

[0044]

[0045] in represents the number of super points in the P point cloud, It is the enhanced structural semantic feature of the i-th super point of the P point cloud obtained through the self-attention mechanism. Enhanced super point feature is also obtained in a similar way.

[0046] Enhanced super-point features in the obtained point cloud and Finally, the goal of the present invention is to extract discriminative super-point features, and then select the correspondence between super-points based on the feature similarity between the original point cloud and the target point cloud super-points. Because the present invention not only promotes the interaction of structural and semantic information within the point cloud, but also extends this interaction between two frames of point clouds. Therefore, a cross-attention mechanism is used to interact structural information and semantic information.

[0047] For any superpoint in the source point cloud Cross-Attention Mechanism Calculation All super points of the target point cloud The attention score of the features between , and the final feature is obtained by weighted summing of the super-point features of the target point cloud. Specifically, their query matrix, key matrix, value matrix and semantic matrix are calculated as follows:

[0048]

[0049]

[0050]

[0051]

[0052] The superscript c represents the cross-attention mechanism, W and b are the parameters shared in the cross-attention mechanism for each feature, It is the downsampled local structural feature of the jth superpoint in the Q point cloud. and They are the semantic features of the i-th superpoint of the P point cloud and the semantic features of the j-th superpoint of the Q point cloud, respectively. It is the enhanced feature of the i-th super point of the P point cloud obtained through the self-attention mechanism.

[0053] Calculate superpoint and The weight score of the structural semantic features between according to Learn features for all superpoints in another point cloud Perform weighted addition to obtain enhanced features. These formulas are expressed as follows:

[0054]

[0055]

[0056] Where softmax is the probability distribution function that converts the output to [0,1], and the superscript T is the transpose of the matrix. represents the number of super points in the Q point cloud, It is the enhanced structural semantic feature of the i-th super point of the P point cloud obtained through the self-attention mechanism. Enhanced super point feature is also obtained in a similar way.

[0057] The multimodal feature fusion module outputs the super-point features of the structural semantic fusion of P and Q point clouds through a layer of MLP:

[0058]

[0059]

[0060] Using the optimal transfer model to achieve super-point matching, given the structural semantic fusion of the super-point features of the P and Q point clouds and The affinity matrix is ​​calculated as shown in formula 20:

[0061]

[0062] Where W is a learnable matrix, and each element Z in the affinity matrix Z ij Represents two super points and However, the corresponding superpoints cannot be found in the non-overlapping regions. In order to reduce the impact of superpoint correspondence retrieval, the similarity matrix An extra row and column are added as relaxation terms, which are expressed as follows:

[0063]

[0064] Among them, a is a learnable parameter. Similarity matrix of rows and columns is called the relaxation term. The optimal transport problem is solved using the Sinkhorn algorithm, a differentiable algorithm that normalizes the affinity matrix alternately along rows and columns. Finally, the top K correspondences with the highest scores are selected as the final results of super-point matching, where the final results of super-point correspondence are expressed as follows:

[0065]

[0066] Where i and j are the indices of the super-points in the P point cloud and the Q point cloud, respectively, and the similarity matrix The rows and columns of Indicates the correspondence between two superpoints. topk() means selecting the top K with the highest scores.

[0067] After superpoint matching, the superpoint correspondences are established and the decoder module is used to reconstruct the superpoint features. This decoding process reflects the symmetric steps taken in the encoding stage. First, the point correspondence score matrix Z within each superpoint association is calculated based on the decoded point features. Like superpoint matching, an extra row and column are added to the similarity matrix Z, and the Sinkhorn algorithm is used to obtain the point correspondence set C. Finally, the top k correspondences with the highest scores are selected:

[0068] C n,m ={(p n ,q m )|Z n,m ∈topk(Z)} (23)

[0069] Where n and m are the index of the P point cloud point and the Q point cloud point in a pair of superpoint correspondences, as well as the rows and columns of the similarity matrix Z. i,j Indicates the correspondence between two points. topk() means selecting the top K points with the highest scores.

[0070] In order to obtain the transformation accurately and effectively, we first fuse the structural information and semantic information from both within and between point clouds based on the super-point features to obtain high-confidence regional matching. For a super-point matching, a transformation matrix can be calculated, and the corresponding Ci,j To calculate the rotation R and translation t:

[0071]

[0072] Where (i, j) represents the correspondence between the i-th super point in the source point cloud and the j-th super point in the target point cloud, and k represents the k-th point correspondence in the (i, j) super point correspondence. The number of superpoint correspondences in the superpoint matching. k is the weight corresponding to the point, Computes the squared Euclidean distance between two points.

[0073] The transformation matrix calculated by matching all superpoints is used to obtain the final transformation matrix through the following formula. The calculation formula of the final transformation matrix is:

[0074]

[0075] Where |C n,m | is the number of point correspondences, and (n,m) represents the correspondence between the nth point of the source point cloud and the mth point of the target point cloud, R (i,j) ,t (i,j) is the transformation matrix calculated from the correspondence between the i-th superpoint in the source point cloud and the j-th superpoint in the target point cloud, Calculate the square of the Euclidean distance between two points, τ a is the distance threshold.

[0076] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A point cloud registration method based on semantic and structural collaborative guidance, characterized in that: include: Acquire point cloud semantic information, wherein the point cloud semantic information includes semantic information of outdoor radar point cloud and semantic information of indoor reconstructed point cloud; adopt point cloud semantic segmentation model to obtain semantic information of outdoor radar point cloud; indoor reconstructed point cloud data includes source point cloud data, target point cloud data, image corresponding to source point cloud and image corresponding to target point cloud; input the image corresponding to source point cloud and the image corresponding to target point cloud into Semantic-SAM image segmentation model to obtain image pixel semantics; project image pixel semantics through projection module to obtain semantic information of indoor reconstructed point cloud; Sampling the source point cloud data and the target point cloud data to obtain super points; The encoder is used to extract the geometric structure features of the superpoints, the angle and distance between two superpoints are calculated, and the angle and distance between the superpoints are embedded into the geometric structure features; The point cloud semantics is represented by super-point semantic features to obtain super-point semantic features; the super-point semantic features and geometric structure features are fused using a multimodal feature fusion module to obtain enhanced super-point features; Perform super-point matching based on the enhanced super-point features, perform point matching based on the super-point matching results, and complete point cloud registration; The super-point semantic feature representation processing of the point cloud semantics includes: aggregating points in a local area of ​​the point cloud image to form a super-point; The categories of each point in the super-point area and the frequency of occurrence of each category are counted; the categories of the points and the frequency of occurrence of each category are encoded to obtain the semantic features of the super-point.

2. The point cloud registration method based on semantic and structural collaborative guidance according to claim 1, characterized in that: The distance between the super points is calculated as: Where S(·) is a sine function, σ d is a hyperparameter used to adjust the sensitivity of distance changes. is the Euclidean distance between two superpoints, is the i-th superpoint, is the jth super point; The angle between two superpoints is: in, is the i-th superpoint k nearest neighbors, σ a It is the sensitivity parameter that controls the angle change.

3. The point cloud registration method based on semantic and structural collaborative guidance according to claim 1, characterized in that: The geometric structure characteristic expression is: in, These are two embedded projection matrices.

4. The point cloud registration method based on semantic and structural collaborative guidance according to claim 1, characterized in that: The Semantic-SAM image segmentation model processes the input image including: The Semantic-SAM image segmentation model processes the input image including: the Semantic-SAM image segmentation model consists of two branches, namely the mask branch and the semantic branch; in the mask branch, the image is input into the image encoder to generate a one-time image embedding; the image embedding is input into the mask decoder to obtain the segmentation mask; in the semantic branch, the image is input into the semantic segmenter to obtain the category assigned to each pixel in the image; the segmentation mask and the pixel assignment category are input into the semantic voting module to obtain the final semantic category of the mask, where the final semantic category is the image pixel semantics.

5. The point cloud registration method based on semantic and structural collaborative guidance according to claim 1, characterized in that: Projecting image pixel semantics includes: in, is the camera external parameter including rotation and translation matrix, is the camera internal parameter, f(·) represents the homogeneous function, Indicates the image (w a ,h b )’s pixel semantics, Indicates the i-th point in the point cloud th Points p i semantics.

6. The point cloud registration method based on semantic and structural collaborative guidance according to claim 1, characterized in that: The multimodal feature fusion module is used to fuse the super-point semantic features and geometric structure features, including: using the structure-semantic self-attention mechanism to learn the super-point semantic features and geometric structure features to obtain the global correlation between super-points; specifically including: calculating the super-points in the two source point clouds and The query matrix, key matrix, value matrix, structure matrix and semantic matrix of the query matrix, key matrix, value matrix, structure matrix and semantic matrix are used to calculate two super points. and The attention score of the semantic structural features between the source and target point clouds is calculated; the learned features of all other super-points in the same point cloud are weighted summed according to the attention score to obtain enhanced structural semantic features; the calculation method for super-point feature enhancement in the target point cloud is the same as that for the source point cloud; the structural information and semantic information between the source point cloud and the target point cloud are interacted by using the structural-semantic cross-attention mechanism; the interacted super-point features of the target point cloud and the source point cloud are input into the MLP layer, and the super-point features of the semantic and structural fusion are output.

7. The point cloud registration method based on semantic and structural collaborative guidance according to claim 1, characterized in that: The super-point matching includes: constructing an affinity matrix based on the enhanced super-point features; each element Z in the affinity matrix Z ij Represents two super points and similarity score; an extra row and column are added to the similarity matrix Z as relaxation terms; the Sinkhorn algorithm is used to solve the optimal transmission problem; the top K correspondences with the highest scores are selected as the final result of super-point matching.

8. The point cloud registration method based on semantic and structural collaborative guidance according to claim 1, characterized in that: Point matching includes: constructing super-point correspondences based on the final results of super-point matching; reconstructing super-point features using a decoder module; calculating the point correspondence score matrix Z within each super-point association based on the decoded point features; adding an extra row and column to the similarity matrix Z, and using the Sinkhorn algorithm to obtain a point correspondence set C; and selecting the top k correspondences with the highest scores in the correspondence set as the point matching results.

Images