A heterogeneous point cloud registration system and method based on deep learning SR-Net

Abstract

The purpose of the present invention is to provide a heterogeneous point cloud registration model and method based on deep learning SR-Net, which belongs to the technical field of urban area point cloud registration. The SR-Net registration model is based on deep learning technology. Through innovative network structure and learning strategies, it improves tolerance to noise and outliers, optimizes the registration accuracy of partially overlapping areas, and enhances the integrated utilization of global and local structural information. At the same time, to further improve the registration accuracy of the model, an adaptive topk operation is determined and a loss function is designed. The registration method based on this model demonstrates higher accuracy and robustness in heterogeneous point cloud registration tasks, and has important theoretical research and practical application value in the field of urban heterogeneous point cloud registration.

Description

Technical Field

[0001] The present invention belongs to the technical field of urban area point cloud registration, and specifically relates to a heterogeneous point cloud registration system and method based on deep learning SR-Net. Background Art

[0002] Heterogeneous point cloud registration in urban areas is a key technology for 3D spatial data processing, widely used in a variety of fields, including urban planning, infrastructure management, transportation planning, disaster assessment, key building protection, and environmental monitoring. This technology creates a unified, high-precision 3D model by precisely spatially aligning point cloud data collected at different times, from different perspectives, or using different sensors. While urban heterogeneous point cloud registration plays a key role in fusing diverse sensor data to generate 3D models, it also faces numerous challenges. First, due to differences in acquisition mechanisms and capture times, point cloud data collected at different times or under different circumstances may contain noise and outliers, making it difficult to ensure that the same points are located in the same location. Furthermore, it is difficult to generate the same order of 3D points between two acquired point clouds, which can affect the accuracy of the registration process. Second, due to sensor field-of-view limitations, heterogeneous point cloud data are collected from different perspectives, resulting in only partial overlap and incomplete point clouds. This further complicates registration. Finally, the complex and variable structure of urban environments places high demands on the robustness and accuracy of algorithms.

[0003] In the field of heterogeneous point cloud registration, deep learning techniques have been widely used to improve the accuracy and robustness of registration, and end-to-end learning-based registration methods are a very active and common approach. End-to-end learning-based registration methods can directly learn the feature representations required for registration from the raw data and output the registration results in one go. In addition, they have demonstrated superior performance when processing point cloud data with noise, different scales, and distributions. In particular, in cross-source point cloud registration, these methods can effectively process point cloud data captured by different sensors. Deng et al. proposed the RelativeNet method, which achieves fast point cloud alignment by assigning specific directions to key points to eliminate local reference frame calculations, with good generalization performance and speed. The DeepVCP model developed by Lu et al. selects key points through feature weight matching, enhancing the accuracy of registration. The 3DRegNet proposed by Pais et al. can distinguish between internal and external points and accurately estimate the transformation matrix. PointNetLK, which combines neural networks and optimization methods, provides a new approach for point cloud registration through an improved LK algorithm and PointNet model. Huang et al. further improved PointNetLK, reducing its reliance on labels and improving registration accuracy. DeepGMR optimizes transformation solutions by learning the correspondence between points and distribution parameters, while DGR combines convolutional networks, differentiable weighted Procrustes algorithms, and SE(3) optimizers to achieve fine registration. The DCP technology proposed by Wang and Solomon uses DGCNN and Transformer networks to process point cloud features and uses SVD to estimate transformations, demonstrating high noise tolerance. Building on this, PRNet adds a keypoint detection mechanism, optimizing its resistance to low overlap and noise. Lu et al.'s DeepICP method uses PointNet++ feature extraction and performs keypoint detection via a multi-layer perceptron and point weight layer, improving registration accuracy. Yew and Lee's RPMNet combines PPFNet feature extraction with coordinate information, identifying keypoints via a parameter prediction module, enhancing robustness to noise and slightly overlapping data. Ginzburg and Raviv's Deep Weighted Consistency Unsupervised Network utilizes DGCNN to capture global and local features, constructing a probabilistic matching matrix to further improve registration performance. These methods, through deep learning technology, have, to a certain extent, addressed the problems of noise sensitivity, high computational cost, and limited generalization in traditional point cloud registration, demonstrating the significant advantages of deep learning technology in heterogeneous point cloud registration.

[0004] While existing deep learning registration methods are specifically designed and optimized for registration tasks, leveraging the strengths of traditional mathematical theories and deep neural networks in learning complex feature representations to achieve effective registration, they still suffer from several limitations: 1) Sensitivity to noise and outliers: While some models are highly tolerant to noise, point cloud data in urban environments is collected from a diverse range of buildings, roads, vegetation, and various man-made and natural objects. These scenes are not only complex and varied in geometry but also exhibit significant variations in texture and lighting conditions. Therefore, when faced with extreme noise or outliers, deep learning models may produce misleading matches, affecting registration accuracy. 2) Partial overlap: When point cloud data only partially overlap, correct matching becomes more difficult, and deep learning models may struggle to find sufficient corresponding points to accurately estimate the transformation matrix. 3) Insufficient consideration of local structural information: While feature-based registration methods consider local structural information, both global and local structural information are crucial for registration, and thus may still have limitations in certain situations. Summary of the Invention

[0005] In view of the problems existing in the background technology, the purpose of the present invention is to provide a heterogeneous point cloud registration system and method based on deep learning SR-Net. The SR-Net (Super-point Registration Network) registration system is based on deep learning technology. Through innovative network structure and learning strategy, it improves the tolerance to noise and outliers, optimizes the registration accuracy of partially overlapping areas, and enhances the integrated utilization of global and local structural information. At the same time, in order to further improve the registration accuracy of the system, an adaptive topk operation is determined and a loss function is designed. The registration method based on this system has shown higher accuracy and robustness in heterogeneous point cloud registration tasks, and has important theoretical research and practical application value in the field of urban heterogeneous point cloud registration.

[0006] To achieve the above object, the technical solution of the present invention is as follows:

[0007] A heterogeneous point cloud registration system based on deep learning SR-Net, including a pyramid feature extraction module, an aggregate point feature extraction and matching module, a local feature extraction module, a local registration module, and a global registration module;

[0008] The pyramid feature extraction module is used to capture the features of point clouds at different sampling rates to obtain the features of aggregated points and local point features;

[0009] The cluster point feature extraction and matching module extracts the matching relationship between cluster points based on the features of the cluster points; the cluster point feature extraction and matching module is obtained by cascading three identical geometric transformer units and one cluster point matching unit; the geometric transformer unit is used to obtain the fusion features of the cluster points based on the features of the cluster points, and the cluster point matching unit is used to perform correlation matching on the obtained fusion features;

[0010] The geometric transformer unit includes a self-attention feature extraction layer and a cross-attention feature extraction layer; the self-attention feature extraction layer is used to extract the correlation of the geometric space between each cluster point based on the features of the cluster point; the cross-attention feature extraction layer merges information from different point clouds based on the correlation of the geometric space between the cluster points to obtain the fusion feature of the cluster point;

[0011] The local feature extraction module is used to find matching points between two point clouds based on local point features and correlation matching results, and obtain the matching situation of the aggregation points of the source point cloud and the target point cloud to the neighborhood local points, as well as the relationship between the neighborhood points of a pair of aggregation points;

[0012] The local registration module solves the transformation matrix based on the relationship between the neighborhood points of each pair of clustered points;

[0013] The global registration module compares the transformation matrix obtained by solving the relationship between the neighborhood points of all clustered points, selects the transformation matrix that can better satisfy the relationship between the neighborhood points in the global point correspondence, and performs registration based on the transformation matrix.

[0014] Furthermore, the pyramid feature extraction module includes N layers of KPconv feature extraction network layers, each layer of which is used to downsample and integrate features, as well as upsample and reduce the dimension of features on the input data; wherein, the last layer of KPconv feature extraction network layer after downsampling outputs the features of the aggregated points, and the last layer of KPconv feature extraction network layer after upsampling outputs the features of the local points.

[0015] Furthermore, the working process of the self-attention feature extraction layer is as follows:

[0016] For the input feature vector X∈R |X|×d , output eigenvector Y, that is, the correlation of geometric space, the value y in the eigenvector i The calculation formula is,

[0017]

[0018] Among them, d is the characteristic dimension of the input vector, R is the real number field, |X| is the number of cluster points, a i,jis the weight coefficient, i is the i-th row of the eigenvector X, j is the j-th column of the eigenvector X, x j is the feature vector value of the jth column, and V is the value in the attention mechanism;

[0019] Weight coefficient a i,j By the attention score e i,j Calculate the normalized exponential function row by row to get the attention score e i,j The calculation of is shown in formula (2):

[0020]

[0021] Among them, Q is the query in the attention mechanism, K is the key in the attention mechanism, d is the feature dimension of the input vector, C is the projection matrix of the geometric structure embedding, which is used to capture the geometric structure information with transformation invariance inside the point cloud, T represents the transpose of the matrix, r i,j Embedded by geometric structure, embedded by distance and angular embedding Weighted, D is the projection matrix of distance embedding, A is the projection matrix of angle embedding;

[0022] For any two cluster points p i and p j , the distance between them is s i,j =||p i -p j ||2, the distance embedding between these two points As shown in formula (3),

[0023]

[0024] Among them, σ d It is a hyperparameter used to adjust the sensitivity to distance changes, 2k and 2k+1 represent even and odd positions respectively;

[0025] Select the aggregation point p i The k nearest neighbor cluster points form a point set K. For each point p in the point set K l Calculate its value with p i 、p j Angle α i,j,l , α i,j,l =∠(p l -p i ,p j -p i ), p l Different from p i and p j The third aggregation point

[0026] The angle embedding It can be calculated by formula (4),

[0027]

[0028] Among them, σ a is a hyperparameter used to adjust the sensitivity to angle changes.

[0029] Furthermore, the working process of the cross-attention feature extraction layer is as follows:

[0030] The point sets K of the two point clouds output by the self-attention feature extraction layer are defined as point set M and point set N, respectively, and their feature matrices are X 'M and X 'N , that is, the output feature vector Y of the self-attention feature extraction layer, then the cross-attention feature matrix Z of the aggregation point M M , and its calculation method is shown in formula (5):

[0031]

[0032] in, is the cross attention feature matrix Z of the aggregation point M M A row vector in, |N| is the number of points in the point set N, a i,j ′ is composed of the cross attention score e i,j 'Calculate the normalized exponential function row by row to obtain, e i,j The calculation formula of ' is shown in formula (6), where o is the feature dimension of the input vector.

[0033] Furthermore, the working process of the aggregation point matching unit is as follows:

[0034] The final output of the three geometric Transformer units is the fusion features F A and F B , and then normalize the fusion features to the unit sphere to ensure the consistency of the length of the feature vectors, and then calculate their Gaussian correlation matrix Τ∈R |A|×|B| , where the elements of the matrix τ i,j The calculation of is shown in formula (7):

[0035]

[0036] Double normalize the Gaussian correlation matrix T to obtain a new matrix matrix Chinese elements The calculation formula is shown in formula (8):

[0037]

[0038] in, is obtained by calculating the element τ in the Gaussian correlation matrix T. i,j The result of two normalization calculations is to reduce ambiguous matches and effectively eliminate incorrect matches. |A| is the number of points in point cloud A, and |B| is the number of points in point cloud B. represents the sum of the i-th row of the Gaussian correlation matrix T, and k represents the k-th element of the i-th row;

[0039] Finally, from the matrix Select the largest N indices (x i ,y i ), that is, arrange the N indices from large to small and select the first N indices, which correspond to the most likely matching cluster points in the two point clouds, as shown in formula (9):

[0040]

[0041] in, is the fusion feature of point cloud A, is the fusion feature of point cloud B, E∈R N×2 Stores the index information of N pairs of matching cluster points, topk x,y (·) is the adaptive topk operation.

[0042] Furthermore, the specific process of the adaptive topk operation is as follows:

[0043]

[0044] Among them, dist_mat(·) is the distance evaluation function of two features, and δ(·) is the random replication function.

[0045] Furthermore, the working process of the local feature extraction module is as follows:

[0046] For each pair of cluster points ( is an aggregation point of point cloud A, is the aggregate point in the matching point cloud B), first associate it with the local point in the corresponding neighborhood, recorded as and Among them, h is the feature dimension, m is the number of neighborhood points;

[0047] Construct cost matrix cost i ∈R m×m , as shown in formula (11), where Θ is the feature matrix of two local regions,

[0048]

[0049] Calculate the soft assignment matrix Z based on the cost matrix i , to determine how the points around each clustered point align with the points in the corresponding area in the other point cloud, and then from Z i Extract the point correspondence relationship, that is, if a pair of matches is ranked in the top k items of its row and column at the same time, then select the pair of matches, and finally output the matching of the aggregated points of the source point cloud and the target point cloud to the local points in the neighborhood, as shown in formula (12), where and are points from two point clouds A and B respectively;

[0050]

[0051] The present invention also provides a method for performing heterogeneous point cloud registration based on the above-mentioned deep learning SR-Net heterogeneous point cloud registration system, comprising the following steps:

[0052] Step 1. Obtain heterogeneous point set data, preprocess the data, and then divide the preprocessed data into training set and test set;

[0053] Step 2. Pre-train the deep learning SR-Net heterogeneous point cloud registration system based on the existing data set. The trained parameters are used as the initial parameters. Then, the deep learning SR-Net heterogeneous point cloud registration system is trained based on the training set, with the data in the test set as the label. The loss function L during the training process is:

[0054] L=L oc +L p ,

[0055] Among them, L oc is the overlap-aware circular loss for clustered point matching, L p is the point matching loss for matching the neighborhood points of a pair of clustered points,

[0056] is the loss on point cloud A, is the loss on point cloud B, and the calculations for both are the same.

[0057]

[0058] Where Ψ is a set of anchor patches, which contains all superpoints in point cloud A that have at least one overlapping region in point cloud B. and Represents two neighboring points, and are the distances in the feature space of the source point cloud and the target point cloud, Representative neighborhood points and The overlap ratio between and Δ A and Δ n is the interval hyperparameter, and γ is the hyperparameter used to adjust the impact of point matching loss.

[0059]

[0060] N g is the number of randomly sampled cluster points during training, L p,i The corresponding relationship of the aggregation points Point matching loss,

[0061]

[0062] I i and J i Represents the point set of unmatched points, is the aggregation point C i The matching probability between point x and point y, is the aggregation point C i The matching probability of point x and unmatched points, is the matching probability of point y in the target point cloud and the unmatched point in the source point cloud;

[0063] Step 3. The heterogeneous point cloud registration system based on the trained deep learning SR-Net registers the newly collected heterogeneous point cloud data.

[0064] Furthermore, the heterogeneous point cloud data are vehicle-mounted lidar point clouds, drone-mounted lidar point clouds and drone oblique photography reconstruction point clouds.

[0065] Furthermore, the preprocessing in step 1 is a downsampling process.

[0066] Furthermore, the training parameters in step 2 are: Adam optimizer, 80-100 epochs, and weight decay of 10 -6 , the initial learning rate lr = 0.0001, which decays every 4 epochs.

[0067] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0068] The SR-Net system proposed in this paper has superior performance in urban area point cloud registration, with an average RMSE of 1.48; the registration method based on this system shows higher accuracy and robustness in the heterogeneous point cloud registration task, and has important theoretical research and practical application value in the field of urban heterogeneous point cloud registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 This is a schematic diagram of the heterogeneous point cloud registration system based on deep learning SR-Net in the present invention.

[0070] Figure 2 Schematic diagram of the structure of the pyramid feature extraction module in the present invention.

[0071] Figure 3 Schematic diagram of the structure of the aggregation point feature extraction and matching module of the present invention.

[0072] Figure 4 This is an initial position map of heterogeneous point cloud data from three different data sources in the same scene in Example 1 of the present invention.

[0073] Figure 5 This is the heterogeneous point cloud data after manual registration.

[0074] Figure 6 Schematic diagram of segmentation of heterogeneous point cloud data.

[0075] Figure 7 This is a scale comparison chart of the dataset in Example 1 of the present invention.

[0076] Figure 8 This is a comparison diagram of the vehicle point cloud before and after downsampling in Example 1 of the present invention.

[0077] Figure 9 Graph of the training loss function for a system without pre-training technology.

[0078] Figure 10 Graph of the training loss function for a system using pre-training techniques.

[0079] Figure 11 A comparison chart of the RRE indicators on the validation set.

[0080] Figure 12 The following is a comparison chart of the RTE indicators on the validation set. DETAILED DESCRIPTION

[0081] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the implementation methods and drawings.

[0082] A heterogeneous point cloud registration system based on deep learning SR-Net. The structural diagram of the system is as follows: Figure 1 As shown, it includes a pyramid feature extraction module, an aggregate point feature extraction and matching module, a local feature extraction module, a local registration module and a global registration module;

[0083] The pyramid feature extraction module is used to capture the features of point clouds at different sampling rates to obtain the features of aggregated points and local point features. The structural diagram of the pyramid feature extraction module is shown in FIG. Figure 2 As shown in the figure, it includes N layers of KPconv feature extraction network layers, each layer is used to downsample and integrate features of the input data, as well as upsample and reduce feature dimensionality; among them, the last layer of KPconv feature extraction network layer after downsampling outputs the features of the aggregated points, and the last layer of KPconv feature extraction network layer after upsampling outputs the features of the local points.

[0084] The cluster point feature extraction and matching module extracts the matching relationship between cluster points based on the features of the cluster points; the structural diagram of the cluster point feature extraction and matching module is as follows: Figure 3 As shown, it is obtained by cascading three identical geometric Transformer units and an aggregate point matching unit; the geometric Transformer unit is used to obtain the fusion features of the aggregate points based on the features of the aggregate points, and the aggregate point matching unit is used to perform correlation matching on the obtained fusion features;

[0085] The geometric transformer unit includes a self-attention feature extraction layer and a cross-attention feature extraction layer; the self-attention feature extraction layer is used to extract the correlation of the geometric space between each cluster point based on the features of the cluster point; the cross-attention feature extraction layer merges information from different point clouds based on the correlation of the geometric space between the cluster points to obtain the fusion feature of the cluster point;

[0086] The local feature extraction module is used to find matching points between two point clouds based on local point features and correlation matching results, and obtain the matching situation of the aggregation points of the source point cloud and the target point cloud to the neighborhood local points, as well as the relationship between the neighborhood points of a pair of aggregation points;

[0087] The local registration module solves the transformation matrix based on the relationship between the neighborhood points of each pair of clustered points;

[0088] The global registration module compares the transformation matrix obtained by solving the relationship between the neighborhood points of all clustered points, selects the transformation matrix that can better satisfy the relationship between the neighborhood points in the global point correspondence, and performs registration based on the transformation matrix.

[0089] Example 1

[0090] A method for performing heterogeneous point cloud registration based on a deep learning SR-Net heterogeneous point cloud registration system comprises the following steps:

[0091] Step 1. Obtain heterogeneous point set data and preprocess the data. The specific process is as follows:

[0092] Step 1.1. For the same scene, point cloud data is collected using vehicle-mounted lidar, drone-mounted lidar, and drone oblique photography. The initial positions of point cloud data from the three different data sources are shown in the figure below. Figure 4 shown.

[0093] The vehicle-mounted LiDAR point cloud only contains point cloud coordinate information, while the UAV-mounted LiDAR point cloud and the UAV oblique photography reconstruction point cloud also contain color information. The spatial resolution of all three is 5 cm, and the UAV oblique photography reconstruction only contains canopy surface information. The vehicle-mounted LiDAR and UAV-mounted LiDAR have penetrating capabilities and therefore contain some information about the interior of the canopy.

[0094] Step 1.2. Manually align the three collected point clouds from different sources. The method adopted in the present invention is as follows:

[0095] (1) Taking a point cloud (called heterogeneous point cloud A) as a reference, select some matching points with prominent features and clear semantic information in scene A, such as tree trunks, street lights, house corners, etc.

[0096] (2) In the heterogeneous point cloud B, locate the matching points selected in the first step and calculate the transformation matrix between these point pairs;

[0097] (3) In the obtained multiple pairs of rotation matrices, outliers are eliminated, and the average value is calculated, and then the transformation matrix is ​​applied to the heterogeneous point cloud B; finally, the three heterogeneous data sets of the same scene are registered, as shown in the following example: Figure 5 As shown, different colors represent different data sources.

[0098] Next, the scene is subdivided into 17 datasets of mixed vegetation and building areas, such as Figure 6 Finally, the same operation is performed on the datasets of the other two scenarios, and finally 156 pairs of heterogeneous data to be matched are obtained.

[0099] Step 1.3. Downsample the 156 heterogeneous data pairs to be matched.

[0100] According to other public datasets KITTI and 3DMatch datasets, the spatial resolution after downsampling commonly used in the field of deep learning is 0.25m and 0.3m; the data collected in this invention is similar to the KITTI dataset in scale, such as Figure 7 As shown, therefore, this embodiment also sets the downsampling rate to voxel_size=0.3m;

[0101] Comparison of vehicle point clouds before and after downsampling Figure 8As shown in the figure, the white part is the vehicle-mounted point cloud after voxel downsampling (voxel size is 0.3m), and the blue part is the original vehicle-mounted point cloud. As can be seen from the figure, at this downsampling rate, the registration metrics are not only comparable to those before downsampling, but also rich details and visualization effects are retained, and the characteristics of different objects are not lost due to the downsampling process;

[0102] The data obtained by downsampling is divided into two parts, one as a training set and the other as a test set as labels;

[0103] Step 2. Pre-train the deep learning SR-Net heterogeneous point cloud registration system based on the KITTI dataset, and use the trained parameters as the initial parameters.

[0104] No pre-training strategy is used ( Figure 9 ) and used pre-training ( Figure 10 ) Figure 9 and attached Figure 10 As shown in the figure, it can be seen that after the system converges, the loss function of the pre-trained system converges to around 1.7, while the loss function of the non-pre-trained system converges to around 2.3, proving that the pre-training technology has a great effect on improving the system.

[0105] Then, the deep learning SR-Net heterogeneous point cloud registration system is trained based on the training set obtained after downsampling in step 1. The system is trained on KITTI for 80 epochs using the Adam optimizer with a batch size of 1 and a weight decay of 10. -6 , the initial learning rate lr = 0.0001, which decays every 4 epochs; the data in the test set are used as labels during training;

[0106] In addition to the self-collected data, the data in the training set can also use data enhancement methods to randomly rotate and translate the self-collected point cloud and update the transformation matrix to increase the amount of training data;

[0107] On the validation set of each epoch, the system of that epoch is evaluated, such as Figure 11 and Figure 12 As shown in Figure 2, both RRE and RTE show that the pre-trained system (purple) performs better than the non-pre-trained system (blue) in the registration task.

[0108] The loss function L during training is:

[0109] L=L oc +L p ,

[0110] Among them, L oc is the overlap-aware circular loss for clustered point matching, L p is the point matching loss for matching the neighborhood points of a pair of clustered points,

[0111] is the loss on point cloud A, is the loss on point cloud B, and the calculations for both are the same.

[0112]

[0113] Where Ψ is a set of anchor patches, which contains all superpoints in point cloud A that have at least one overlapping region in point cloud B. and Represents two neighboring points, and are the distances in the feature space of the source point cloud and the target point cloud, Representative neighborhood points and The overlap ratio between and Δ A and Δ n is the interval hyperparameter, γ is the hyperparameter used to adjust the impact of point matching loss;

[0114]

[0115] N g is the number of randomly sampled cluster points during training, L p,i The corresponding relationship of the aggregation points Point matching loss,

[0116]

[0117] I i and J i Represents the point set of unmatched points, is the aggregation point C i The matching probability between point x and point y, is the aggregation point C i The matching probability of point x and unmatched points, is the matching probability of point y in the target point cloud and the unmatched point in the source point cloud;

[0118] Step 3. The heterogeneous point cloud registration system based on the trained deep learning SR-Net registers the newly collected heterogeneous point cloud data.

[0119] The registration results are measured using three metrics: relative root mean square error (RRE), relative total error (RTE), and root mean square error (RMSE). Table 1 shows the registration results of the KITTI dataset using the system of the present invention and emerging point cloud registration systems in the prior art, such as GoFiNet and Predator.

[0120] Table 1 Accuracy comparison of different systems on the KITTI dataset

[0121]

[0122] As can be seen from Table 1, on the KITTI dataset, the SR-Net system of the present invention has better registration results. Compared with other existing systems, it has lower registration accuracy error, with RRE and RTE being 0.24 and 6.80 respectively.

[0123] Since calculating the MRSE index requires a certain amount of calculation, which will increase the training time, the MRSE index of the two systems is evaluated only on the test set. On the test set, the comparison of three different systems is shown in Table 2. Kitti_pre_train represents a system that uses the KITTI dataset for pre-training and then trains on the self-collected dataset of the present invention. No_kitti_pre_train represents a dataset that only uses the self-collected dataset of the present invention for training. Only_kitti_train represents a system that is only trained on the KIT TI dataset. Average_a represents the average of the indicators of all data, and average_b represents the average of the indicators after eliminating the data that have not been successfully registered. It can be seen from the bold data in Table 2 that the pre-trained system performs better in the registration task of the vegetation-building mixed scene of the present invention.

[0124] Table 2 Comparison of indicators on the validation set

[0125]

[0126] The above description is only a specific embodiment of the present invention. Any feature disclosed in this specification, unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes; all disclosed features, or all steps in the methods or processes, except for mutually exclusive features and / or steps, can be combined in any way.

Claims

1. A heterogeneous point cloud registration system based on deep learning SR-Net, characterized by: The heterogeneous point cloud registration system based on deep learning SR-Net includes a pyramid feature extraction module, an aggregate point feature extraction and matching module, a local feature extraction module, a local registration module and a global registration module; The pyramid feature extraction module is used to capture the features of point clouds at different sampling rates to obtain the features of aggregated points and local point features. The pyramid feature extraction module includes N layers of KPconv feature extraction network layers, each of which is used to downsample and integrate features, as well as upsample and reduce the dimension of features on the input data. The last KPconv feature extraction network layer after downsampling outputs the features of aggregated points, and the last KPconv feature extraction network layer after upsampling outputs the features of local points. The cluster point feature extraction and matching module extracts the matching relationship between cluster points based on the features of the cluster points; the cluster point feature extraction and matching module is obtained by cascading three identical geometric transformer units and one cluster point matching unit; the geometric transformer unit is used to obtain the fusion features of the cluster points based on the features of the cluster points, and the cluster point matching unit is used to perform correlation matching on the obtained fusion features; The geometric transformer unit includes a self-attention feature extraction layer and a cross-attention feature extraction layer; the self-attention feature extraction layer is used to extract the correlation of the geometric space between each cluster point based on the features of the cluster point; the cross-attention feature extraction layer merges information from different point clouds based on the correlation of the geometric space between the cluster points to obtain the fusion feature of the cluster point; The local feature extraction module is used to find matching points between two point clouds based on local point features and correlation matching results, and obtain the matching situation of the aggregation points of the source point cloud and the target point cloud to the neighborhood local points, as well as the relationship between the neighborhood points of a pair of aggregation points; The local registration module solves the transformation matrix based on the relationship between the neighborhood points of each pair of clustered points; The global registration module compares the transformation matrix obtained by solving the relationship between the neighborhood points of all clustered points, selects the transformation matrix that can better satisfy the relationship between the neighborhood points in the global point correspondence, and performs registration based on the transformation matrix.

2. The heterogeneous point cloud registration system based on deep learning SR-Net according to claim 1, characterized in that: The working process of the self-attention feature extraction layer is as follows: For the input feature vector X∈R |X|×d , output eigenvector Y, that is, the correlation of geometric space, the value y in the eigenvector i The calculation formula is, Among them, d is the characteristic dimension of the input vector, R is the real number field, |X| is the number of cluster points, a i,j is the weight coefficient, i is the i-th row of the eigenvector X, j is the j-th column of the eigenvector X, x j is the feature vector value of the jth column, and V is the value in the attention mechanism; Weight coefficient a i,j By the attention score e i,j Calculate the normalized exponential function row by row to get the attention score e i,j The calculation of is shown in formula (2): Among them, Q is the query in the attention mechanism, K is the key in the attention mechanism, d is the feature dimension of the input vector, C is the projection matrix of the geometric structure embedding, T represents the transpose of the matrix, r i,j Embedded by geometric structure, embedded by distance and angular embedding Weighted, D is the projection matrix of distance embedding, A is the projection matrix of angle embedding; For any two cluster points p i and p j , the distance between them is s i,j =||p i -p j ||2, the distance embedding between these two points As shown in formula (3), Among them, σ d is a hyperparameter, 2k and 2k+1 represent even and odd positions respectively; Select the aggregation point p i The k nearest neighbor cluster points form a point set K. For each point p in the point set K l Calculate its value with p i 、p j Angle α i,j,l , α i,j,l =∠(p l -p i ,p j -p i ), p l Different from p i and p j The third aggregation point The angle embedding It can be calculated by formula (4), Among them, σ a is a hyperparameter.

3. The heterogeneous point cloud registration system based on deep learning SR-Net according to claim 1, characterized in that: The working process of the cross attention feature extraction layer is: The point sets K of the two point clouds output by the self-attention feature extraction layer are defined as point set M and point set N, respectively, and their feature matrices are X 'M and X 'N , that is, the output feature vector Y of the self-attention feature extraction layer, then the cross-attention feature matrix Z of the aggregation point M M , and its calculation method is shown in formula (5): in, is the cross attention feature matrix Z of the aggregation point M M A row vector in, |N| is the number of points in the point set N, a i,j ′ is composed of the cross attention score e i,j 'Calculate the normalized exponential function row by row to obtain, e i,j The calculation formula of ' is shown in formula (6), where o is the feature dimension of the input vector.

4. The heterogeneous point cloud registration system based on deep learning SR-Net according to claim 1, characterized in that The working process of the aggregation point matching unit is as follows: The final output of the three geometric Transformer units is the fusion features F A and F B , and then normalize the fusion features to the unit sphere to ensure the consistency of the length of the feature vectors, and then calculate their Gaussian correlation matrix Τ∈R |A|×|B| , where the elements of the matrix τ i,j The calculation of is shown in formula (7): Double normalize the Gaussian correlation matrix T to obtain a new matrix As shown in formula (8): in, is obtained by calculating the element τ in the Gaussian correlation matrix T. i,j The double normalization calculation is performed, where |A| is the number of points in point cloud A, and |B| is the number of points in point cloud B. represents the sum of the i-th row of the Gaussian correlation matrix T, and k represents the k-th element of the i-th row; Finally, from the matrix Select the largest N indices (x i ,y i ), that is, arrange the N indices from large to small and select the first N indices, which correspond to the most likely matching cluster points in the two point clouds, as shown in formula (9): in, is the fusion feature of point cloud A, is the fusion feature of point cloud B, E∈R N×2 Stores the index information of N pairs of matching cluster points, topk x,y (·) is the adaptive topk operation.

5. The heterogeneous point cloud registration system based on deep learning SR-Net according to claim 4, characterized in that: The specific process of adaptive topk operation is: Among them, dist_mat(·) is the distance evaluation function of two features, and δ(·) is the random replication function.

6. The heterogeneous point cloud registration system based on deep learning SR-Net according to claim 1, characterized in that: The working process of the local feature extraction module is as follows: For each pair of cluster points is an aggregation point of point cloud A, It is the aggregation in the matching point cloud B, which is first associated with the local points in the corresponding neighborhood, recorded as and Among them, h is the feature dimension, m is the number of neighborhood points; Construct cost matrix cost i ∈R m×m , as shown in formula (11), where Θ is the feature matrix of two local regions, Calculate the soft assignment matrix Z based on the cost matrix i , to determine how the points around each clustered point align with the points in the corresponding area in the other point cloud, and then from Z i The point correspondence is extracted from the source point cloud and the target point cloud is finally output as the matching of the aggregated points to the neighborhood local points, as shown in formula (12), where: and are points from two point clouds A and B respectively; 7. A method for performing heterogeneous point cloud registration based on the heterogeneous point cloud registration system based on deep learning SR-Net according to any one of claims 1 to 6, characterized in that: The following steps are involved: Step 1. Obtain heterogeneous point set data, preprocess the data, and then divide the preprocessed data into training set and test set; Step 2. Pre-train the deep learning SR-Net heterogeneous point cloud registration system based on the existing data set. The trained parameters are used as the initial parameters. Then, the deep learning SR-Net heterogeneous point cloud registration system is trained based on the training set, with the data in the test set as the label. The loss function L during the training process is: L=L oc +L p , Among them, L oc is the overlap-aware circular loss for clustered point matching, L p is the point matching loss for matching the neighborhood points of a pair of clustered points, is the loss on point cloud A, is the loss on point cloud B, and the calculations for both are the same. Where Ψ is a set of anchor regions, Ψ contains all superpoints in point cloud A that have at least one overlapping region in point cloud B; and Represents two neighboring points, and are the distances in the feature space of the source point cloud and the target point cloud, Representative neighborhood points and The overlap ratio between and Δ A and Δ n is the interval hyperparameter, γ is the hyperparameter used to adjust the impact of point matching loss; N g is the number of randomly sampled cluster points during training, L p,i The corresponding relationship of the aggregation points Point matching loss, I i and J i Represents the point set of unmatched points, is the aggregation point C i The matching probability between point x and point y, is the aggregation point C i The matching probability of point x and unmatched points, is the matching probability of point y in the target point cloud and the unmatched point in the source point cloud; Step 3. The heterogeneous point cloud registration system based on the trained deep learning SR-Net registers the newly collected heterogeneous point cloud data.

8. The method for heterogeneous point cloud registration according to claim 7, wherein: The heterogeneous point cloud data are vehicle-mounted lidar point clouds, drone-mounted lidar point clouds and drone oblique photography reconstruction point clouds.

9. The method for heterogeneous point cloud registration according to claim 7, wherein: The training parameters in step 2 are: Adam optimizer, 80-100 epochs, and weight decay of 10 -6 , the initial learning rate lr = 0.0001, which decays every 4 epochs.

Images