Three-dimensional point cloud registration method

Abstract

A three-dimensional point cloud registration method, which takes a first point cloud, a first image and a second point cloud, a second image as input, respectively takes the geometric features of the two point clouds as the respective point cloud features, respectively takes the semantic or texture features of the two images as the respective image features, obtains the mapping relationship of the point cloud features and the image features, forms a resampling feature by resampling the mapping relationship and the image features, calculates and maps the resampling feature weight and the point cloud weight, performs point multiplication on the resampling feature and the resampling feature weight to obtain a resampling result, performs multiplication on the point cloud feature and the point cloud weight to obtain a point cloud result, and takes the sum of the resampling result and the point cloud result as a fusion feature; the first resampling feature and the first point cloud feature, and the second resampling feature and the second point cloud feature each have a respective fusion feature; the two fusion features are first matched in features, and then a transformation matrix is estimated, and the data of the two perspectives are registered into an integral point cloud through the transformation matrix.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a registration method, apparatus, computer equipment, and storage medium for fusing two point cloud data into one point cloud. Background Technology

[0002] Point cloud registration is a crucial task in 3D vision, playing a vital role in navigation, localization, pose estimation, and 3D reconstruction. Given two point cloud datasets that only partially overlap, the goal is to estimate the transformation matrix for aligning them. In a typical solution, the geometric point clouds are first encoded into feature descriptors using their structural features. Then, a correspondence is established between the two point clouds by matching the most similar feature descriptors, and finally, the RANSAC method is used to solve the matrix.

[0003] Most current 3D feature descriptors utilize the geometric structure information of the point cloud itself to obtain feature descriptors. PPFNet first constructs point pair features (PPFs) based on the geometric structure information of the point cloud, then inputs the PPFs into the network for learning and associates them with global context information; the recent state-of-the-art method Geo uses a transformer to learn the angle and distance information between multiple points in the point cloud scene. However, in the pair of point clouds to be paired, there are often parts with weak geometric structure information (ground, walls, ceiling) and repeatable structural combinations, which can easily lead to the final matching failure. Summary of the Invention

[0004] The purpose of this invention is to provide a three-dimensional point cloud registration method, apparatus, computer device, and storage medium that can fuse semantic texture information of an image and point cloud geometric structure information to achieve adaptive learning of point cloud features.

[0005] A three-dimensional point cloud registration method is provided, which takes a first point cloud, a first image, a second point cloud, and a second image as inputs. Each point cloud comes from its own point cloud acquisition device. The two point clouds are data representations of the same object from different viewpoints. The first point cloud and the first image are from the same viewpoint, and the second point cloud and the second image are from the same viewpoint. The geometric features of the two point clouds are used as their respective point cloud features. The semantic or texture features of the two images are used as their respective image features.

[0006] Obtain a first mapping relationship between the first point cloud features and the first image features, and obtain a second mapping relationship between the second point cloud features and the second image features; resample the first mapping relationship and the first image features to form a first resampled feature, and resample the second mapping relationship and the second image features to form a second resampled feature; the expression form of the resampled feature is the same as that of the point cloud features;

[0007] The resampled features and point cloud features are used as inputs. The resampled feature weights and point cloud weights are obtained through calculation and mapping. The resampled features and resampled feature weights are multiplied to obtain the resampled result. The point cloud features and point cloud weights are multiplied to obtain the point cloud result. The sum of the resampled result and the point cloud result is used as the fusion feature. The first resampled feature and the first point cloud feature, the second resampled feature and the second point cloud feature each have their own fusion features.

[0008] The first and second fusion features are used as inputs and are enhanced by a geometric relation attention mechanism. The two fusion features are first matched and then the transformation matrix is ​​estimated. The data from the two perspectives are registered into a whole point cloud by the transformation matrix.

[0009] Furthermore, one of the point clouds mentioned above is called the source point cloud, and the other point cloud is called the target point cloud. The purpose of this method is to transform the source point cloud into the coordinate system of the target point cloud to form a global point cloud; or, the purpose of this method is to transform the target point cloud into the coordinate system of the source point cloud to form a global point cloud; or, the purpose of this method is to transform both the source point cloud and the target point cloud into the same world coordinate system to form a global point cloud.

[0010] Furthermore, the method for obtaining the mapping relationship between point clouds and images includes: downsampling the point cloud to obtain superpoints, performing perspective projection on the superpoints, and calculating the mapping from the superpoints to image pixels; the mapping relationship between superpoints and image pixels should satisfy: ;where [X i ,Y i Z i [u] represents the coordinates of the i-th superpoint; i ,v i ] represents pixel coordinates; K represents the camera's intrinsic parameter matrix, and T represents the camera's extrinsic parameter matrix.

[0011] Furthermore, the method for forming resampled features includes: the arrangement of the resampled image features is the same as the arrangement of the point cloud features; if the point cloud features are mapped outside the image space, empty features are filled in.

[0012] Furthermore, the semantic features of the image are resampled using a resampling function. The resampling function used is `torch.nn.functional.grid_sample`. Alternatively, other functions capable of rearranging semantic features can also be used to implement resampling.

[0013] Furthermore, the method for obtaining the resampled feature weights and point cloud weights includes: convolving the resampled features and point cloud features into the same dimension, and then obtaining the overall weights of the resampled features and point cloud features; mapping the overall weights using a linear layer to divide them into image weights W.img and point cloud weight W PC , .

[0014] Furthermore, normalize the overall weights to the range [0,1]. This can be done using the softmax function or other existing normalization functions.

[0015] Furthermore, fusion features The method for obtaining it is as follows:

[0016]

[0017]

[0018] Select two super points within the point cloud and In the formula k is The number of nearest neighbors, where c is the number of feature channels. for The angles of the surrounding area and k points, and These are hyperparameters that control geosensitivity; Indicates exceeding the limit and The encoding distance parameter between them, 2x represents the distance at the 2nd...

[0019] Calculations are performed along the x-dimensional axis. Indicates exceeding the limit and The encoded distance parameter between them, 2x+1 indicates that it is calculated in the 2x+1th dimension; Indicates exceeding the limit and The encoded angle parameter between the k nearest neighbor superpoints, 2x indicates that it is calculated in the 2x dimension; Indicates exceeding the limit and The encoding angle parameter between the k nearest neighbor superpoints, 2x+1 indicates that the calculation is performed in the 2x+1th dimension.

[0020] A second aspect of the present invention aims to provide a three-dimensional point cloud registration method, apparatus, computer device, and storage medium applicable to scenes with weak geometric structures.

[0021] A three-dimensional point cloud registration method is provided, which takes a first point cloud, a first image, a second point cloud, and a second image as inputs. Each point cloud comes from its own point cloud acquisition device. The two point clouds are data representations of the same object from different perspectives. The first point cloud and the first image are from the same perspective, and the second point cloud and the second image are from the same perspective.

[0022] Semantic features of the first and second images are obtained respectively. The semantic features of the images are resampled to form resampled image features with the same representation as the point cloud. Gaussian correlation matrix is ​​calculated for the two resampled image features. , Where P' represents the set of source point cloud superpoints and Q' represents the set of target point cloud superpoints. Represents the lowest layer feature of the first resampling The lowest layer feature representing the second resampling This represents the Gaussian correlation between the i-th point in the first resampled image and the j-th point in the second resampled image;

[0023] Obtain the Gaussian correlation matrix of the first point cloud and the second point cloud respectively. ,in , Superpoint features representing the source point cloud Superpoint features representing the target point cloud This represents the Gaussian correlation between the i-th superpoint in the source point cloud and the j-th superpoint in the target point cloud;

[0024] After normalizing the two Gaussian correlation matrices, perform a dot product to obtain the combined Gaussian correlation matrix. The top-k algorithm is used to obtain the point pair correspondence relationship from the comprehensive Gaussian correlation matrix to achieve feature matching of the first point cloud and the second point cloud. Then, the transformation matrix is ​​estimated, and the data from the two perspectives are registered into a whole point cloud through the transformation matrix.

[0025] Furthermore, one of the point clouds mentioned above is called the source point cloud, and the other point cloud is called the target point cloud. The purpose of this method is to transform the source point cloud into the coordinate system of the target point cloud to form a global point cloud; or, the purpose of this method is to transform the target point cloud into the coordinate system of the source point cloud to form a global point cloud; or, the purpose of this method is to transform both the source point cloud and the target point cloud into the same world coordinate system to form a global point cloud.

[0026] A third aspect of the present invention aims to provide a three-dimensional point cloud registration method, apparatus, computer device, and storage medium that can be applied to situations with weak scene geometry, integrates semantic texture information of images and point cloud geometric information, and achieves adaptive learning of point cloud features.

[0027] A three-dimensional point cloud registration method is proposed. The method takes a first point cloud, a first image, a second point cloud, and a second image as inputs. Each point cloud comes from its own point cloud acquisition device. The two point clouds are data representations of the same object from different perspectives. The first point cloud and the first image are from the same perspective, and the second point cloud and the second image are from the same perspective.

[0028] Obtain a first mapping relationship between the first point cloud features and the first image features, and obtain a second mapping relationship between the second point cloud features and the second image features; resample the first mapping relationship and the first image features to form a first resampled feature, and resample the second mapping relationship and the second image features to form a second resampled feature; the expression form of the resampled feature is the same as that of the point cloud features;

[0029] The resampled features and point cloud features are used as inputs. The resampled feature weights and point cloud weights are obtained through calculation and mapping. The resampled features and resampled feature weights are multiplied to obtain the resampled result. The point cloud features and point cloud weights are multiplied to obtain the point cloud result. The sum of the resampled result and the point cloud result is used as the fusion feature. The first resampled feature and the first point cloud feature, the second resampled feature and the second point cloud feature each have their own fusion features.

[0030] The first and second fusion features are taken as inputs and calculated using a geometric relation attention mechanism to enhance the first and second fusion features respectively.

[0031] Calculate the Gaussian correlation matrix for the two fused features. , Where P' represents the set of source point cloud superpoints and Q' represents the set of target point cloud superpoints. Indicates the first resampled feature Indicates the second resampling feature This represents the Gaussian correlation between the i-th point in the first resampled image and the j-th point in the second resampled image;

[0032] Obtain the Gaussian correlation matrix of the first point cloud and the second point cloud respectively. ,in , Superpoint features representing the source point cloud Superpoint features representing the target point cloud This represents the Gaussian correlation between the i-th superpoint in the source point cloud and the j-th superpoint in the target point cloud;

[0033] After normalizing the two Gaussian correlation matrices, perform a dot product to obtain the combined Gaussian correlation matrix. The top-k algorithm is used to obtain the point pair correspondence relationship from the comprehensive Gaussian correlation matrix to achieve feature matching of the first point cloud and the second point cloud. Then, the transformation matrix is ​​estimated, and the data from the two perspectives are registered into a whole point cloud through the transformation matrix.

[0034] Furthermore, the KPConv architecture is used to downsample the first and second point clouds respectively while extracting point cloud features.

[0035] Furthermore, we obtain the super-point set and the correspondence. Then, the downsampled point cloud is obtained. and The point-to-point correspondence in P 1 The characteristics are Q 1 The characteristics are Calculate the cost matching matrix :

[0036]

[0037] With learnable parameters Fill the cost matching matrix with a new row and a new column to obtain the soft matching matrix. Calculate the soft matching matrix Then delete the last row and last column that were added earlier to restore it to its original state. , The specific form and M i same, For the matching matrix; As the confidence matrix for selecting dense point pairs, the top-k algorithm is used to select the optimal point pair correspondence:

[0038]

[0039] In the formula, Represents the matching matrix The value in the x-th row and y-th column of the array; the calculated dense point pairs are merged into a set. : ; It is a set of dense point pair correspondences, serving as the registered point cloud. C contains all the corresponding point relationships predicted by the algorithm.

[0040] The advantages of this invention are: 1. It proposes a general point cloud registration method with adaptive fusion of multimodal features at the superpoint level, which significantly improves the uniqueness of point cloud descriptors and can be applied to situations with weak scene geometry.

[0041] 2. A multi-level joint similarity judgment module is proposed, which can better filter the correspondences in the point cloud based on diverse features. Attached Figure Description

[0042] Figure 1 This is a flowchart of the third 3D point cloud registration method.

[0043] Figure 2 This is a flowchart of feature extraction.

[0044] Figure 3 This is a flowchart of feature matching. Detailed Implementation

[0045] In some embodiments, a three-dimensional point cloud registration method is provided, which takes a first point cloud, a first image, a second point cloud, and a second image as inputs. Each point cloud comes from its respective point cloud acquisition device. The two point clouds are data representations of the same object from different viewpoints. The first point cloud and the first image are from the same viewpoint, and the second point cloud and the second image are from the same viewpoint. The geometric features of the two point clouds are used as their respective point cloud features. The semantic or texture features of the two images are used as their respective image features.

[0046] Obtain a first mapping relationship between the first point cloud features and the first image features, and obtain a second mapping relationship between the second point cloud features and the second image features; resample the first mapping relationship and the first image features to form a first resampled feature, and resample the second mapping relationship and the second image features to form a second resampled feature; the expression form of the resampled feature is the same as that of the point cloud features;

[0047] The resampled features and point cloud features are used as inputs. The resampled feature weights and point cloud weights are obtained through calculation and mapping. The resampled features and resampled feature weights are multiplied to obtain the resampled result. The point cloud features and point cloud weights are multiplied to obtain the point cloud result. The sum of the resampled result and the point cloud result is used as the fusion feature. The first resampled feature and the first point cloud feature, the second resampled feature and the second point cloud feature each have their own fusion features.

[0048] The first and second fusion features are used as inputs and are enhanced by a geometric relation attention mechanism. The two fusion features are first matched and then the transformation matrix is ​​estimated. The data from the two perspectives are registered into a whole point cloud by the transformation matrix.

[0049] In some embodiments, one of the point clouds is referred to as the source point cloud and the other as the target point cloud. The purpose of this method is to transform the source point cloud into the coordinate system of the target point cloud to form a global point cloud; or, the purpose of this method is to transform the target point cloud into the coordinate system of the source point cloud to form a global point cloud; or, the purpose of this method is to transform both the source point cloud and the target point cloud into the same world coordinate system to form a global point cloud.

[0050] In some embodiments, the method for obtaining the mapping relationship between point clouds and images includes: downsampling the point cloud to obtain superpoints, performing perspective projection on the superpoints, and calculating the mapping from the superpoints to image pixels; the mapping relationship between superpoints and image pixels should satisfy: ;where [X i ,Y i Z i [u] represents the coordinates of the i-th superpoint; i ,v i ] represents pixel coordinates; K represents the camera's intrinsic parameter matrix, and T represents the camera's extrinsic parameter matrix.

[0051] In some embodiments, the method for forming resampled features includes: arranging the resampled image features in the same way as the point cloud features, and filling empty features if the point cloud features are mapped outside the image space.

[0052] In some embodiments, a resampling function is invoked to resample the semantic features of the image. The resampling function is `torch.nn.functional.grid_sample`. Alternatively, other functions capable of rearranging semantic features can also be used to implement resampling.

[0053] In some embodiments, the method for obtaining resampled feature weights and point cloud weights includes: convolving resampled features and point cloud features into the same dimension, and then obtaining the overall weights of the resampled features and point cloud features; mapping the overall weights using a linear layer to divide them into image weights W. img and point cloud weight W PC , .

[0054] In some embodiments, the overall weights are normalized to the range [0,1]. Normalization is performed using the softmax function, or other existing normalization functions.

[0055] In some embodiments, fusion features The method for obtaining it is as follows:

[0056]

[0057]

[0058] Select two super points within the point cloud and In the formula k is The number of nearest neighbors, where c is the number of feature channels. for The angles of the surrounding area and k points, and These are hyperparameters that control geosensitivity; Indicates exceeding the limit and The encoding distance parameter between them, 2x represents the distance at the 2nd...

[0059] Calculations are performed along the x-dimensional axis. Indicates exceeding the limit and The encoded distance parameter between them, 2x+1 indicates that it is calculated in the 2x+1th dimension; Indicates exceeding the limit and The encoded angle parameter between the k nearest neighbor superpoints, 2x indicates that it is calculated in the 2x dimension; Indicates exceeding the limit and The encoding angle parameter between the k nearest neighbor superpoints, 2x+1 indicates that the calculation is performed in the 2x+1th dimension.

[0060] In some embodiments, a three-dimensional point cloud registration method is provided, which takes a first point cloud, a first image, a second point cloud, and a second image as inputs. Each point cloud comes from its respective point cloud acquisition device. The two point clouds are data representations of the same object from different viewpoints. The first point cloud and the first image are from the same viewpoint, and the second point cloud and the second image are from the same viewpoint.

[0061] Semantic features of the first and second images are obtained respectively. The semantic features of the images are resampled to form resampled image features with the same representation as the point cloud. Gaussian correlation matrix is ​​calculated for the two resampled image features. , Where P' represents the set of source point cloud superpoints and Q' represents the set of target point cloud superpoints. Represents the lowest layer feature of the first resampling The lowest layer feature representing the second resampling This represents the Gaussian correlation between the i-th point in the first resampled image and the j-th point in the second resampled image;

[0062] Obtain the Gaussian correlation matrix of the first point cloud and the second point cloud respectively. ,in , Superpoint features representing the source point cloud Superpoint features representing the target point cloud This represents the Gaussian correlation between the i-th superpoint in the source point cloud and the j-th superpoint in the target point cloud;

[0063] After normalizing the two Gaussian correlation matrices, perform a dot product to obtain the combined Gaussian correlation matrix. The top-k algorithm is used to obtain the point pair correspondence relationship from the comprehensive Gaussian correlation matrix to achieve feature matching of the first point cloud and the second point cloud. Then, the transformation matrix is ​​estimated, and the data from the two perspectives are registered into a whole point cloud through the transformation matrix.

[0064] In some embodiments, one of the point clouds is referred to as the source point cloud and the other as the target point cloud. The purpose of this method is to transform the source point cloud into the coordinate system of the target point cloud to form a global point cloud; or, the purpose of this method is to transform the target point cloud into the coordinate system of the source point cloud to form a global point cloud; or, the purpose of this method is to transform both the source point cloud and the target point cloud into the same world coordinate system to form a global point cloud.

[0065] In some embodiments, a three-dimensional point cloud registration method is provided, which takes a first point cloud, a first image, a second point cloud, and a second image as inputs. Each point cloud comes from its respective point cloud acquisition device. The two point clouds are data representations of the same object from different viewpoints. The first point cloud and the first image are from the same viewpoint, and the second point cloud and the second image are from the same viewpoint.

[0066] Obtain a first mapping relationship between the first point cloud features and the first image features, and obtain a second mapping relationship between the second point cloud features and the second image features; resample the first mapping relationship and the first image features to form a first resampled feature, and resample the second mapping relationship and the second image features to form a second resampled feature; the expression form of the resampled feature is the same as that of the point cloud features;

[0067] The resampled features and point cloud features are used as inputs. The resampled feature weights and point cloud weights are obtained through calculation and mapping. The resampled features and resampled feature weights are multiplied to obtain the resampled result. The point cloud features and point cloud weights are multiplied to obtain the point cloud result. The sum of the resampled result and the point cloud result is used as the fusion feature. The first resampled feature and the first point cloud feature, the second resampled feature and the second point cloud feature each have their own fusion features.

[0068] The first and second fusion features are taken as inputs and calculated using a geometric relation attention mechanism to enhance the first and second fusion features respectively.

[0069] Calculate the Gaussian correlation matrix for the two fused features. , Where P' represents the set of source point cloud superpoints and Q' represents the set of target point cloud superpoints. Indicates the first resampled feature Indicates the second resampling feature This represents the Gaussian correlation between the i-th point in the first resampled image and the j-th point in the second resampled image;

[0070] Obtain the Gaussian correlation matrix of the first point cloud and the second point cloud respectively. ,in , Superpoint features representing the source point cloud Superpoint features representing the target point cloud This represents the Gaussian correlation between the i-th superpoint in the source point cloud and the j-th superpoint in the target point cloud;

[0071] After normalizing the two Gaussian correlation matrices, perform a dot product to obtain the combined Gaussian correlation matrix. The top-k algorithm is used to obtain the point pair correspondence relationship from the comprehensive Gaussian correlation matrix to achieve feature matching of the first point cloud and the second point cloud. Then, the transformation matrix is ​​estimated, and the data from the two perspectives are registered into a whole point cloud through the transformation matrix.

[0072] In some embodiments, the KPConv architecture is used to downsample the first point cloud and the second point cloud respectively while extracting point cloud features.

[0073] In some embodiments, the superpoint set and correspondence are obtained. Then, the downsampled point cloud is obtained. and The point-to-point correspondence in P 1 The characteristics are Q 1 The characteristics are Calculate the cost matching matrix :

[0074]

[0075] With learnable parameters Fill the cost matching matrix with a new row and a new column to obtain the soft matching matrix. Calculate the soft matching matrix Then delete the last row and last column that were added earlier to restore it to its original state. , The specific form and M i same, For the matching matrix; As the confidence matrix for selecting dense point pairs, the top-k algorithm is used to select the optimal point pair correspondence:

[0076]

[0077] In the formula, Represents the matching matrix The value in the x-th row and y-th column of the array; the calculated dense point pairs are merged into a set. : ; It is a set of dense point pair correspondences, serving as the registered point cloud. C contains all the corresponding point relationships predicted by the algorithm.

[0078] Example 1

[0079] Given two point clouds and Our goal is to obtain a transformation matrix. This allows P and Q to have consistent spatial positions and orientations in the same coordinate system, where , Many existing methods rely on optimization algorithms to obtain one-to-one matching point pairs between point clouds, i.e., constructing a set of point pairs. ,in and It should satisfy the following:

[0080]

[0081] Where R,t is the true value of the transformation between point cloud pairs, and N is the maximum number of point pairs that can be found in the point cloud pairs.

[0082] Figure 1 This paper illustrates the basic framework of the invention. It takes a pair of point clouds to be registered and their corresponding RGB images as input. The point cloud network uses the KPConv architecture to downsample the input point clouds and extract point-by-point features. The image network uses the ResNet34 framework to extract semantic information from the images. A coarse-to-fine approach is used, first performing coarse matching at the lowest level of superpoints, and then using the next lower level of downsampled points to extract densely corresponding matching points. The PTALK module is used to adaptively fuse point cloud structural information and image semantic information. The IGCM module uses the semantic information of the image and fused feature information to select the most suitable overlapping parts of the coarse matching superpoints at multiple levels. Finally, based on the obtained point pair relationships, the RANSAC algorithm can be used to obtain the predicted transformation matrix.

[0083] We follow the adopted KPConv architecture to downsample the original point cloud while extracting multi-level point cloud structural feature information. We perform coarse matching on the superpoints of the pair of point clouds to be registered, i.e., the lowest resolution point cloud features. The coordinates of these superpoints are denoted as follows: and Their corresponding features are as follows and Similarly, we follow the ResNet34 architecture to extract semantic features from the input RGB image, where the semantic features of the image are: and .

[0084] Unlike other methods that simultaneously utilize point cloud structural information and image semantic information, we argue that projecting dense point clouds onto an image using perspective projection results in a significant amount of redundant information, and edge points negatively impact information fusion and judgment. While hyperpoints are downsampled points, they still belong to the image space. Therefore, we calculate the mapping from hyperpoints to pixels using perspective projection. Specifically, given hyperpoint coordinates... Then its pixel coordinates in the corresponding image plane It should meet the following requirements:

[0085]

[0086] In the formula or , These are the camera's intrinsic and extrinsic parameter matrices, respectively. K and T are directly given in datasets such as 3DMatch and KITTI. Using the perspective projection calculation formula above, we can easily obtain the mapping relationship between downsampled superpoints and pixels:

[0087]

[0088]

[0089] Then, the image features are extracted using the torch.nn.functional.grid_sample function. Resampled according to the mapping relationship to have the same arrangement as the point cloud features. If a superpoint is mapped outside the image space, then empty features are filled. Subsequently, we designed the Fusion Module to adaptively fuse point cloud structural features and image semantic features, such as... Figure 2 .

[0090] Resampled image features and point cloud features The input is first processed into the module, where Conv1d is used to extract the point cloud structural features. and image semantic features Sampled to the same dimension Then obtain the overall weight. :

[0091]

[0092] Then, the overall weights are mapped using a linear layer, and the softmax function is used to normalize the weights to the range [0,1]. Finally, the individual weight values ​​are... With their respective characteristics Multiplication completes the fusion process:

[0093]

[0094]

[0095] here To ensure the relevance of the fused features to the point cloud structural features and image semantic features, and to avoid the fused features having a synergistic effect (1+1<1), the uniqueness and geometric distinctiveness of the fused point cloud features are further enhanced. The internal geometric structure of the point cloud is encoded in conjunction with the fused features. Then, based on the fused features, the consistency of the multimodal features of the two point clouds is modeled, ultimately yielding... :

[0096]

[0097]

[0098]

[0099] Select two super points within the point cloud and In the formula k is The number of nearest neighbors, where c is the number of feature channels. for The angles of the surrounding area and k points, and These are the hyperparameters that control Geo sensitivity. More information about the Geo module can be found in Geotransformer.

[0100] In obtaining the blended features of a pair of point cloud superpoints and Next, we first normalize the data, and then we can calculate the Gaussian correlation matrix between the two. ,in ,Right now Figure 3 The SuperPoint Feature Gaussian Correlation part. To avoid fuzzy matching problems caused by single mixed features, we will resample the image features. and Similarly, calculate the Gaussian correlation matrix of the two. ,in ,Right now Figure 3 Image Feature Gaussian Correlation, after performing double normalization on both, can finally be calculated as follows: :

[0101]

[0102] This joint correlation extraction method can suppress some false matches and improve matching accuracy. The resulting Gaussian correlation matrix is ​​then used. We can obtain enough point-to-point correspondences by using the top-k algorithm:

[0103]

[0104] Obtain the super-point set and correspondence. Then, the downsampled point cloud is obtained. and The point-to-point correspondence in P 1 The characteristics are Q 1 The characteristics are Calculate the cost matching matrix :

[0105]

[0106] With learnable parameters Fill the cost matching matrix with a new row and a new column to obtain the soft matching matrix. Calculate the soft matching matrix Then delete the last row and last column that were added earlier to restore it to its original state. , The specific form and M i same, For the matching matrix; As the confidence matrix for selecting dense point pairs, the top-k algorithm is used to select the optimal point pair correspondence:

[0107]

[0108] In the formula, Represents the matching matrix The specific numerical values ​​in the diagram, where x and y represent specific points; the calculated dense point pairs are then merged into a set. : ; It is a set of dense point pair correspondences, serving as the registered point cloud. C contains all the corresponding point relationships predicted by the algorithm.

[0109] Verification of the effectiveness of the method of the present invention:

[0110] We used the indoor real-world scene datasets 3DMatch and 3DLoMatch to evaluate our algorithm. To verify the effectiveness of our invention in geometrically weak scenes, we conducted experiments in the 3DWeakMatch and 3DLoWeakMatch (4.1) scenes, referring to the definition of structured scenes in SLAM

[13] . To verify the effectiveness of our method in outdoor real-world scenes, we also conducted relevant tests on the KITTI dataset.

[0111] Validation of the 3DMatch dataset

[0112] Dataset:

[0113] 3DMatch contains 62 scenes, with 46 used for training, 8 for validation, and 8 for testing. We used the same preprocessed dataset as [previous dataset name] and evaluated it on the 3DMatch and 3DLoMatch benchmarks. 3DMatch and 3DLoMatch are distinguished by the number of points in the point cloud; 3DMatch has a point pair count >30%, while 3DLoMatch has only 10-30%.

[0114] Metrics:

[0115] Similar to the previous method, we mainly use three indicators to evaluate the merits of our method. (1) Inlier Ratio (IR): The proportion of corresponding point pairs inferred by the algorithm whose deviation distance under the action of the true transformation matrix is ​​less than a certain threshold (0.1m). This indicator is mainly used to evaluate the number of correct point pairs that the point cloud matching algorithm can obtain. (2) Feature Matching Recall (FMR): The proportion of point cloud pairs whose IR obtained by inferring between point cloud pairs is greater than a certain threshold (5%). Only point cloud pairs with an IR greater than the threshold can be correctly matched. This indicator is mainly used to evaluate the feature uniqueness obtained by the point cloud matching algorithm and the upper limit of the correct matching. (3) Registration Recall (RR): The proportion of point cloud pairs whose transformation error between the transformation matrix inferred by the algorithm and the true transformation matrix is ​​less than a certain threshold (RMSE<0.2). This indicator is mainly used to evaluate the success rate of the final point cloud matching.

[0116] Comparison to recent state of the arts methods:

[0117] We compared our invention with other recently proposed methods: PerfectMatch

[16] , FCGF

[15] , D3Feat

[17] , Predator

[12] , REGTR

[18] , GeoTransformer[4], and RoReg

[19] . The specific comparison results are shown in Table 1. The results show that our invention improves FMR by about 1% compared with the latest methods, which proves that the features of multimodal fusion can improve the uniqueness of point cloud descriptors and reduce the impact of outliers caused by using only point cloud structural features. We achieved experimental results comparable to existing methods on IR and RR, indicating that our invention can obtain the correct corresponding point pairs and use the RANSAC method to make the two point clouds complete the correct matching. In addition, we also compared our invention with IMFNet, which also uses multimodal fusion. The comparison results are shown in Table 2. It can be seen from the results that their methods deteriorate rapidly when the number of sampling points is small, while our invention has better robustness.

[0118] The overall performance on RR is not as good as Geotransformer, which is based solely on structural information. We speculate that this is because the scene in the overall 3DMatch dataset has parts with rich and weak structural information, and RGB images cannot be mapped to the overall point cloud.

[0119] To further explore the impact of image fusion on point cloud feature descriptors, we found that relying solely on point cloud structural features can lead to registration and matching failures when encountering situations with weak structural information (many planar structures) and similar structural features. Referring to the description of structured scenes in SLAM, we designed a scene classification algorithm based on RANSAC plane segmentation. From the 3DMatch and 3DLoMatch datasets, we extracted 3DWeakMatch and 3DLoWeakMatch based on the number of planes within a scene. First, we calculated the average number of planes in each scene across the eight test sets in 3DMatch, which was approximately 6.884. Therefore, using 7 as the dividing line, all point clouds containing more than 7 planes within a single scene were grouped into 3DWeakMatch. Similarly, 3DLoWeakMatch can be obtained. Based on the processed datasets 3DWeakMatch and 3DLoWeakMatch benchmarks, we conducted experiments with 2500 sampling points, still using FMR, IR, and RR as our evaluation metrics. We also reproduced FCGF, Predator, and GeoTransformer as comparison objects. The specific experimental results are shown in Table 2.

[0120] Table 1

[0121]

[0122] Table 2

[0123]

[0124] The results show that our proposed method of fusing RGB semantic information makes point clouds easier and better registered than methods based solely on structural information when the structural information contained in the point cloud scene is weak. Furthermore, the adaptive weight selection of semantic and structural information can help reduce mismatches during the matching process.

[0125] Ablation studies:

[0126] We conducted ablation experiments to further explore the roles of each module in this invention. To investigate the role of the geometric information module, we also included Geo in the ablation experiments to illustrate the correlation between geometric structural information and RGB semantic information. The results show that, compared with the baseline, FMR was greatly improved when only the FM module was used, indicating that multimodal feature fusion can significantly improve the quality of point cloud feature descriptors; RR was significantly improved when only the Geo module was used, indicating that structural feature information is still very important in point cloud registration tasks; however, with the combined effect of the Geo and FM modules, RR actually decreased, indicating that in some application scenarios, the possibility of fused features being confused increases. To avoid this situation, we designed the IBCM module to reduce its occurrence; after adding the IBCM module, all indicators improved, retaining the characteristic of the FM module in improving the quality of point cloud descriptors while also retaining the advantage of better structural feature matching brought by the Geo module.

[0127] Table 3

[0128]

[0129] Qualitative results:

[0130] We conducted a qualitative comparison of Geotransformer and our method on several cases on the 3DMatch and 3DLoMatch datasets. When Geotransformer fails to match point clouds, our invention can combine the semantic information of the corresponding RGB images of the point clouds to identify geometrically weak parts of the scene (desktop, ground, etc.), find more corresponding points, and successfully match the two point clouds. It is evident that the feature attention mechanism not only focuses on parts with significant geometric changes but also pays more attention to parts with similar RGB semantic information between the two point clouds.

[0131] KITTI

[0132] Dataset:

[0133] KITTI is a 3D point cloud dataset generated by LiDAR, containing 11 scenes. We followed a training method using values ​​0-5, validation using values ​​6-7, and testing using values ​​8-10. Similar to [previous dataset], we used ICP to refine the ground truth poses of the LiDAR point clouds and only used point cloud pairs no larger than 10m for testing.

[0134] Metrics:

[0135] We use three metrics to evaluate the invention: (1) Relative Rotation Error (RRE), the deviation distance between the estimated rotation matrix and the true value; (2) Relative Translation Error (RTE), the Euclidean distance between the estimated translation vector and the true value; and (3) Registration Recall (RR), the proportion of point cloud pairs where both RRE and RTE are below a certain threshold (i.e., RRE < 5° and RTE < 2m).

[0136] Registration results:

[0137] We compared our invention with state-of-the-art RANSAC-based methods: FCGF, D3Feat, SpinNet, Predator, CoFiNet, and GeoTransformer. It's worth noting that the KITTI dataset only provides frontal views of the scene images and their corresponding point clouds, yet our invention still performs comparably to these methods, demonstrating effectiveness even in outdoor scenes. We further compared several methods that do not require RANSAC: FMR, DGR, and GeoTransformer. Following this approach, we also used LGR as our method for solving the transformation matrix, and the results show that our invention outperforms all state-of-the-art methods, as shown in Table 4.

[0138] Table 4

[0139]

[0140] This invention proposes an adaptive multimodal feature fusion method to fuse semantic texture information and point cloud geometric structure information of images, achieving a unique adaptive learning representation of point cloud features. The proposed multi-level joint relevance matching method can effectively suppress mismatched points and help the model find the correct corresponding point pairs. Experiments show that AFMNet can achieve accurate registration in indoor and outdoor, low-overlap point cloud scenes and scenes with weak geometric structures.

[0141] The above specific implementation examples are used to explain and illustrate the present invention, and are only preferred embodiments of the present invention, not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A three-dimensional point cloud registration method, which takes a first point cloud, a first image, a second point cloud, and a second image as inputs. Each point cloud comes from its own point cloud acquisition device. The two point clouds are data representations of the same object from different viewpoints. The first point cloud and the first image are from the same viewpoint, and the second point cloud and the second image are from the same viewpoint. Obtain a first mapping relationship between the first point cloud features and the first image features, and obtain a second mapping relationship between the second point cloud features and the second image features; resample the first mapping relationship and the first image features to form a first resampled feature, and resample the second mapping relationship and the second image features to form a second resampled feature; the expression form of the resampled feature is the same as that of the point cloud features; The resampled features and point cloud features are used as inputs. The resampled feature weights and point cloud weights are obtained through calculation and mapping. The resampled features and resampled feature weights are multiplied to obtain the resampled result. The point cloud features and point cloud weights are multiplied to obtain the point cloud result. The sum of the resampled result and the point cloud result is used as the fusion feature. The first resampled feature and the first point cloud feature, the second resampled feature and the second point cloud feature each have their own fusion features. The first and second fusion features are taken as inputs and calculated using a geometric relation attention mechanism to enhance the first and second fusion features respectively. Calculate the Gaussian correlation matrix for the two resampled features. , Where P' represents the set of source point cloud superpoints, and Q' represents the set of target point cloud superpoints. This represents the first resampling feature. This indicates the second resampling feature. This represents the Gaussian correlation between the i-th point in the first resampled image and the j-th point in the second resampled image; Obtain the Gaussian correlation matrix of the first point cloud and the second point cloud respectively. ,in , Represents the superpoint features of the source point cloud. The superpoint features representing the target point cloud, This represents the Gaussian correlation between the i-th superpoint in the source point cloud and the j-th superpoint in the target point cloud; After normalizing the two Gaussian correlation matrices, perform a dot product to obtain the combined Gaussian correlation matrix. The top-k algorithm is used to obtain the point pair correspondence relationship from the comprehensive Gaussian correlation matrix to achieve feature matching of the first point cloud and the second point cloud. Then, the transformation matrix is ​​estimated, and the data from the two perspectives are registered into a whole point cloud through the transformation matrix.

2. The three-dimensional point cloud registration method as described in claim 1, characterized in that: One of the point clouds mentioned above is called the source point cloud, and the other point cloud is called the target point cloud. The source point cloud is transformed into the coordinate system of the target point cloud to form the overall point cloud; or, the target point cloud is transformed into the coordinate system of the source point cloud to form the overall point cloud; or, both the source point cloud and the target point cloud are transformed into the same world coordinate system to form the overall point cloud.

3. The three-dimensional point cloud registration method as described in claim 1 or 2, characterized in that: Methods for obtaining the mapping relationship between point clouds and images include: downsampling the point cloud to obtain superpoints, performing perspective projection on the superpoints, and calculating the mapping from the superpoints to image pixels; the mapping relationship between superpoints and image pixels should satisfy: ;where [X i ,Y i Z i [u] represents the coordinates of the i-th superpoint; i ,v i ] represents pixel coordinates; K represents the camera's intrinsic parameter matrix, and T represents the camera's extrinsic parameter matrix.

4. The three-dimensional point cloud registration method as described in claim 3, characterized in that: Methods for forming resampled features include: arranging the resampled image features in the same way as the point cloud features; and filling empty features if the point cloud features are mapped outside the image space.

5. The three-dimensional point cloud registration method as described in claim 3, characterized in that: The resampling function is called to resample the semantic features of the image; the resampling function is torch.nn.functional.grid_sample.

6. The three-dimensional point cloud registration method as described in claim 3, characterized in that: The methods for obtaining resampled feature weights and point cloud weights include: convolving the resampled features and point cloud features into the same dimension, and then obtaining the overall weights of the resampled features and point cloud features; mapping the overall weights using a linear layer to divide them into image weights W. img and point cloud weight W PC , .

7. The three-dimensional point cloud registration method as described in claim 6, characterized in that: The overall weights are normalized to the range [0,1]; the softmax function is used for normalization.

8. The three-dimensional point cloud registration method as described in claim 1, characterized in that: Fusion features The method for obtaining it is as follows: , ， ， Select two super points within the point cloud and In the formula k is The number of nearest neighbors, where c is the number of feature channels. for The angles of the surrounding area and k points, and These are hyperparameters that control geosensitivity; D i,j,2x Indicates exceeding the limit and The encoded distance parameter between them, 2x indicates that the calculation is performed in the 2x dimension, D i,j,2x+1 Indicates exceeding the limit and The encoded distance parameter between them, 2x+1 indicates that it is calculated in the 2x+1th dimension; Indicates exceeding the limit and The encoded angle parameter between the k nearest neighbor superpoints, 2x indicates that it is calculated in the 2x dimension; Indicates exceeding the limit and The encoding angle parameter between the k nearest neighbor superpoints, 2x+1 indicates that the calculation is performed in the 2x+1th dimension.

9. The three-dimensional point cloud registration method as described in claim 1, characterized in that: The KPConv architecture is used to downsample the first and second point clouds while extracting point cloud features.

Images