Multi-source heterogeneous point cloud registration method based on image key points and related device

Abstract

The invention discloses a multi-source heterogeneous point cloud registration method based on image key points and a related device, and relates to the technical field of three-dimensional data processing, and the method comprises the steps: extracting a ground point cloud from target and source point clouds, determining a unit normal vector and a height mean value, correcting the source point clouds through a rotation matrix and a vertical translation vector, and carrying out the registration of the target and source point clouds. The method comprises the following steps of: projecting a target image and a source image, extracting feature key points, screening through descriptor similarity and main direction consistency to obtain initial matching point pairs, calculating a pixel motion vector field through dense optical flow improved by a local binary pattern feature map, selecting same-name matching point pairs according to a displacement threshold value, and combining and de-weighting to obtain a potential image matching point set. According to the method, geometric constraints are established, optimal affine transformation parameters are obtained, point cloud corresponding coordinates are obtained through back projection, horizontal two-dimensional rigid transformation is solved, a coarse registration optimal spatial transformation matrix is calculated in combination with a rotation matrix and a vertical translation vector, and finally multi-source heterogeneous point cloud fine registration is completed through an iterative nearest point algorithm.

Description

Technical Field

[0001] This application relates to the field of three-dimensional data processing technology, and in particular to a multi-source heterogeneous point cloud registration method and related apparatus based on image key points. Background Technology

[0002] In the field of 3D data processing, point cloud registration, as a key technology for unifying point cloud data from different coordinate systems into a single coordinate system, has significant application value in numerous fields such as reverse engineering, 3D reconstruction, and autonomous driving. However, when the point cloud data to be registered originates from different modal sensors and observation platforms (such as fixed-station LiDAR, vehicle-mounted mobile LiDAR, depth cameras, etc.), multi-source heterogeneous point clouds are formed. These point clouds typically exhibit significant differences in point density, noise level, data distribution characteristics, and scale, posing a significant challenge to the registration task. Traditional point cloud registration methods, such as point feature-based registration algorithms, often suffer from low registration efficiency, low accuracy, or even registration failure when dealing with multi-source heterogeneous point clouds due to difficulties in feature extraction, low initial matching accuracy, or insufficient robustness. Especially in the absence of a good initial transformation estimate, directly applying fine registration algorithms such as Iterative Closest Point (ICP) can easily get trapped in local optima, making it difficult to obtain ideal registration results. Therefore, how to achieve efficient, accurate and robust registration for multi-source heterogeneous point clouds is an urgent problem to be solved in the field of current 3D data processing technology. Summary of the Invention

[0003] The purpose of this application is to provide a multi-source heterogeneous point cloud registration method and related apparatus based on image key points. By dividing the registration process into two stages, namely vertical alignment based on ground points and horizontal alignment based on image feature matching, a multi-constraint geometric feature matching strategy is constructed to achieve high-precision registration, thereby improving the accuracy and robustness of point cloud registration on heterogeneous platforms.

[0004] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a multi-source heterogeneous point cloud registration method based on image key points, including: Extract ground point clouds from the target point cloud and the source point cloud respectively. and ground point clouds And determine the ground point clouds respectively. and ground point clouds The unit normal vector and average height; Vertical correction of the source point cloud is performed based on a rotation matrix and a vertical translation matrix; the rotation matrix is ​​based on the ground point cloud. and ground point clouds The unit normal vector is obtained by solving; the vertical translation matrix is ​​based on the ground point cloud. and ground point clouds The difference in the average height was calculated. Feature key points are extracted from the target image and the source image, and initial matching point pairs are selected based on the dual constraints of local similarity of descriptors and consistency of principal direction; the target image and the source image are obtained by projecting the target point cloud and the corrected source point cloud. A dense optical flow method based on local binary pattern feature maps is proposed to calculate pixel-level motion vector fields and select matching point pairs with the same name based on displacement thresholds. The initial matching point pairs and the same-name matching point pairs are combined to remove duplicates, resulting in a potential image matching point set. Based on the potential image matching point set, a geometric consistency constraint is established, and an iterative reweighted least squares optimization framework is adopted to obtain the optimal affine transformation parameters that align the potential image matching point set. Based on the image matching point pairs obtained from the optimal affine transformation parameters, the coordinates of corresponding points in the target point cloud and the source point cloud are obtained by back projection; and based on the coordinate pairs of corresponding points in the target point cloud and the source point cloud, the two-dimensional rigid transformation in the horizontal direction is solved. Based on the rotation matrix, the vertical translation matrix, and the two-dimensional rigid transformation, the optimal spatial transformation matrix for coarse registration of the source point cloud is calculated. Based on the optimal spatial transformation matrix for coarse registration of the source point cloud, after correcting the initial rotation and translation position deviations between the target point cloud and the source point cloud, the iterative nearest point algorithm is used to complete the fine registration between the target point cloud and the source point cloud.

[0005] Secondly, this application provides a multi-source heterogeneous point cloud registration device based on image key points, used to implement the aforementioned multi-source heterogeneous point cloud registration method based on image key points, including: The ground point cloud extraction module is used to extract ground point clouds from the target point cloud and the source point cloud, respectively. and ground point clouds And determine the ground point clouds respectively. and ground point clouds The unit normal vector and average height; The vertical correction module is used to perform vertical correction on the source point cloud based on a rotation matrix and a vertical translation matrix; the rotation matrix is ​​based on the ground point cloud. and ground point clouds The unit normal vector is obtained by solving; the vertical translation matrix is ​​based on the ground point cloud. and ground point clouds The difference in the average height was calculated. The first filtering module is used to extract key feature points from the target image and the source image, and to filter initial matching point pairs based on the dual constraints of local similarity of descriptors and consistency of principal direction; the target image and the source image are obtained by projecting the target point cloud and the corrected source point cloud. The second filtering module is used to calculate pixel-level motion vector fields and filter matching point pairs based on displacement thresholds using a dense optical flow method improved from local binary pattern feature maps. The potential image matching point set determination module is used to perform a union and deduplication of the initial matching point pair and the same-name matching point pair to obtain the potential image matching point set; The constraint module is used to establish geometric consistency constraints based on the potential image matching point set, and to obtain the optimal affine transformation parameters that align the potential image matching point set using an iterative reweighted least squares optimization framework. The back projection module is used to back project the image matching point pairs obtained from the optimal affine transformation parameters to obtain the coordinates of corresponding points in the target point cloud and the source point cloud; and to solve the two-dimensional rigid transformation in the horizontal direction based on the coordinate pairs of corresponding points in the target point cloud and the source point cloud. The registration module is used to calculate the optimal spatial transformation matrix for coarse registration of the source point cloud based on the rotation matrix, the vertical translation matrix, and the two-dimensional rigid transformation; based on the optimal spatial transformation matrix for coarse registration of the source point cloud, after correcting the initial rotation and translation position deviations between the target point cloud and the source point cloud, the fine registration of the target point cloud and the source point cloud is completed by using the iterative nearest point algorithm.

[0006] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-source heterogeneous point cloud registration method based on image key points as described above.

[0007] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-source heterogeneous point cloud registration method based on image key points as described above.

[0008] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application provides a multi-source heterogeneous point cloud registration method and related apparatus based on image key points. By innovatively dividing the registration process into two stages—vertical alignment based on ground point clouds and horizontal alignment based on image feature matching—it effectively solves many problems encountered in target-source point cloud registration. In the vertical alignment stage, the rotation matrix and vertical translation matrix are accurately calculated using the unit normal vector and height mean of the ground point cloud to correct the source point cloud, laying a solid foundation for subsequent horizontal alignment. In the horizontal alignment stage, the projected point cloud is used as an image. Feature key point filtering and matching algorithms, combined with multiple constraints and an optimization framework, are used to obtain the optimal affine transformation parameters and two-dimensional rigid transformation, thereby calculating the optimal spatial transformation matrix for coarse registration of the source point cloud. Finally, the iterative nearest-point algorithm is used to complete the fine registration of the target and source point clouds. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is an application environment diagram of a multi-source heterogeneous point cloud registration method based on image key points in one embodiment of this application; Figure 2 A flowchart illustrating a multi-source heterogeneous point cloud registration method based on image key points provided in an embodiment of this application; Figure 3 A vertical point cloud registration map based on ground points is provided in one embodiment of this application; Figure 4 This application provides a diagram illustrating the point cloud to image conversion and image registration process according to an embodiment of the present application. Figure 5 Figure I shows the qualitative registration result provided in an embodiment of this application; Figure 6 Qualitative registration result provided for an embodiment of this application Figure II ; Figure 7 Qualitative registration result provided for an embodiment of this application Figure III ; Figure 8 Qualitative registration result provided for an embodiment of this application Figure IV ; Figure 9 A schematic diagram of the functional modules of a multi-source heterogeneous point cloud registration device based on image key points provided in an embodiment of this application; Figure 10This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0011] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0012] With the rapid development of fields such as refined 3D urban modeling, autonomous driving navigation, and intelligent transportation planning, how to quickly and effectively acquire high-precision, high-completeness, and high-resolution geospatial big data has become an important research direction. LiDAR (Light Detection and Ranging) scanning, as an active remote sensing technology, has the advantages of all-weather, high-density, and high-precision geospatial data acquisition. In recent years, for the field of digital city construction, common ground-based LiDAR measurement platforms include vehicle-borne mobile laser scanning systems (VMLS) and backpack laser scanning systems (BLS). However, due to inconsistencies in spatial coordinate references, hardware performance, and scanning perspectives among different laser scanning observation systems, heterogeneous point clouds exhibit significant differences in data density, coverage, and sensor noise. Registration faces challenges such as low overlap, non-uniform overlap, geometric heterogeneity, point density variations, and occlusion, severely hindering the development and application of 3D spatiotemporal big data in various fields such as real-world 3D construction, emergency response, and autonomous driving. For example, there are often low-overlap or non-uniform overlap regions between vehicle-mounted and backpack point clouds, leading to a performance degradation of traditional feature description algorithms that rely on high-overlap regions. Secondly, the feature representations of ground and non-ground features in heterogeneous point clouds differ significantly across observation platforms, causing ambiguity in matching effective geometric features. Furthermore, in complex urban scenes, point cloud data may be missing holes due to random occlusion by moving or static objects, resulting in inaccurate registration. Therefore, to overcome the limitations of missing and incomplete 3D information in point clouds collected from a single observation platform, the registration and fusion of vehicle-mounted and backpack point cloud data is crucial. Traditional methods such as ICP (Iterative Closest Point) and its variants are prone to getting trapped in local optima under complex scenes and poor initial pose conditions, while deep learning-based methods rely on training samples or lack transferability, making them difficult to meet practical needs.

[0013] To address these challenges, scholars both domestically and internationally have conducted relevant research, primarily categorized into geometry-based point cloud registration methods and deep learning-based point cloud registration methods. Besl et al. proposed the classic iterative nearest neighbor point cloud registration method, which constructs matching point pairs by progressively searching for nearest neighbors. While simple to operate, this method is prone to getting trapped in local optima in complex scene point clouds. Building upon this, several improved ICP variant point cloud registration methods have been proposed in recent years, achieving some progress in different application scenarios. Some literature extracts line and surface features from the scene to construct similar corresponding relationships, achieving accurate registration of heterogeneous point clouds and improving both registration accuracy and efficiency. Most point cloud registration methods based on geometric features such as points, lines, and surfaces are highly applicable in scenes with rich and regular line and surface elements, but their applicability is greatly affected by the reliability of feature extraction and description, easily leading to feature matching ambiguities when facing weakly structured or densely noisy scene point clouds. In recent years, deep learning-based point cloud registration methods have received widespread attention. Mei et al. proposed an unsupervised deep probabilistic registration framework, UDPReg, for partially overlapping point clouds. This framework combines a Gaussian mixture model posterior probability distribution with a cross-consistency loss network to extract robust local similarity features, achieving good registration performance for low-overlap point clouds. Dan et al. proposed an improved deep learning-based point cloud registration method, GPR, which improves registration accuracy by learning features through a multi-layer propagation module and incorporating neighborhood information. In summary, supervised learning-based point cloud registration methods are highly dependent on the accuracy of training samples, while the transferability of unsupervised learning methods to heterogeneous point clouds from different observation platforms in complex scenarios such as occlusion and strong noise needs further improvement.

[0014] The purpose of this application is to provide a multi-source heterogeneous point cloud registration method and related apparatus based on image key points. By dividing the registration process into two stages—vertical alignment based on ground points and horizontal alignment based on image feature matching—a multi-constraint geometric feature matching strategy is constructed to achieve high-precision registration, aiming to improve the registration accuracy and robustness of the target point cloud and the source point cloud. This application helps to improve the fusion quality of point clouds from heterogeneous platforms in complex urban environments, providing a data foundation and technical support for subsequent refined urban modeling and intelligent perception.

[0015] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] The multi-source heterogeneous point cloud registration method based on image key points provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on another server.

[0017] The terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster composed of multiple servers, or it can be a cloud server.

[0018] In one exemplary embodiment, such as Figure 2 As shown, a multi-source heterogeneous point cloud registration method based on image key points is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps 201 to 208. Wherein: Step 201: Extract ground point clouds from the target point cloud and the source point cloud respectively. and ground point clouds And determine the ground point clouds respectively. and ground point clouds The unit normal vector and average height; Step 202: Perform vertical correction on the source point cloud based on the rotation matrix and the vertical translation matrix; the rotation matrix is ​​based on the ground point cloud. and ground point clouds The unit normal vector is obtained by solving; the vertical translation matrix is ​​based on the ground point cloud. and ground point clouds The difference in the average height was calculated. Step 203: Extract key feature points from the target image and the source image, and select initial matching point pairs based on the dual constraints of local similarity of descriptors and consistency of principal direction; the target image and the source image are obtained by projecting the target point cloud and the corrected source point cloud. Step 204: Based on the improved dense optical flow method of local binary pattern feature map, calculate the pixel-level motion vector field and filter the same-name matching point pairs according to the displacement threshold; Step 205: Perform a union of the initial matching point pairs and the matching point pairs with the same name to remove duplicates, and obtain the potential image matching point set; Step 206: Based on the potential image matching point set, establish geometric consistency constraints, and use an iterative reweighted least squares optimization framework to obtain the optimal affine transformation parameters that align the potential image matching point set. Step 207: Based on the image matching point pairs obtained from the optimal affine transformation parameters, back-project the corresponding point coordinates in the target point cloud and the source point cloud; and based on the corresponding point coordinate pairs in the target point cloud and the source point cloud, solve the two-dimensional rigid transformation in the horizontal direction. Step 208: Based on the rotation matrix, the vertical translation matrix, and the two-dimensional rigid transformation, calculate the optimal spatial transformation matrix for coarse registration of the source point cloud; based on the optimal spatial transformation matrix for coarse registration of the source point cloud, after correcting the initial rotation and translation position deviations between the target point cloud and the source point cloud, use the iterative nearest point algorithm to complete the fine registration of the target point cloud and the source point cloud; the initial rotation and translation position deviations are the deviations caused by factors such as device position, measurement attitude, and loss of positioning signals during the acquisition of the target point cloud and the source point cloud.

[0019] In complex scenarios, due to the significant differences in mounting height and viewing angle between vehicle-mounted LiDAR and backpack LiDAR, the point clouds acquired by both often exhibit large pose deviations in 3D space, especially in the vertical direction. These pose errors severely interfere with subsequent image projection and fine registration of non-ground points. Therefore, it is necessary to pre-perform a coarse registration step based on ground points to achieve vertical alignment. In both vehicle-mounted and backpack point clouds, ground points, as common typical features, possess stable distribution, flat structure, and wide distribution characteristics, making them ideal alignment reference surfaces. Therefore, this application designs a vertical registration method based on ground plane normal alignment.

[0020] In some embodiments, when performing steps 201-211, the specific steps may be as follows: Let the vehicle-mounted point cloud be the target point cloud. Knapsack point cloud as source point cloud Specifically, as follows Figure 3 As shown, (a) target point cloud, (b) target ground point cloud, (c) source point cloud, (d) source ground point cloud, and (e) ground point cloud registration result.

[0021] This embodiment uses ground point cloud extraction methods (e.g., cloth simulation filtering algorithm, morphological filtering, RANSAC fitting, etc.) to extract ground point clouds (denoted as follows). and We approximate these as being distributed on two local planes. Let the unit normal vectors of these two ground planes be: (1) In the formula, They are vectors n T In a three-dimensional coordinate system X , Y , Z Components on the three axes, They are vectors n S In a three-dimensional coordinate system X , Y , Z Components on the three axes.

[0022] The goal of vertical registration is to calculate a rotation matrix. Translation matrix This process aligns the ground normal vector of the source point cloud to the ground normal vector of the target point cloud and corrects the height difference between the source and target point clouds, thereby achieving vertical consistency between the two point clouds. The rotation matrix is ​​involved. This involves solving the minimum rotation problem between two normal vectors, which can be obtained using Rodrigues' rotation formula: (2) In the formula, for identity matrix Let be the axis of rotation, and let the angle between the normal vectors be... , For the unit axis of rotation, It is a cross product matrix in antisymmetric matrix form.

[0023] Furthermore, the average value can represent the overall trend of ground elevation, and it is well-suited to representing the ground reference elevation in relatively flat scenarios such as urban roads. The vertical translation matrix can be defined as the difference between the two mean elevation values: (3) In the formula, For the first Target point cloud ground point Coordinate values For the first A rotated source cloud ground point Coordinate values and The total number of ground points, and the final vertical translation matrix. .

[0024] Combining rotation and translation, the overall transformation of the source point cloud is as follows: (4) This transformation not only aligns the normals of the two sets of ground points, but also ensures that they are in... Basic alignment in direction ensures that subsequent image projections of non-ground points are within the same vertical reference frame.

[0025] Then, using the vehicle-mounted point cloud as the target point cloud Knapsack point cloud as source point cloud The core idea is to project non-terrestrial point clouds into two-dimensional images, establish an affine model under structural constraints, and achieve rigid registration between two structural images through robust geometric optimization, thereby achieving high-precision horizontal alignment of heterogeneous point clouds. As a crucial prerequisite for point cloud to image conversion, to eliminate the negative impact of unstable terrain features such as low vegetation, pedestrians, and vehicles on the image matching process, this embodiment uses ground point clouds as the basis, taking point clouds higher than 2m above the Digital Terrain Model (DTM) as the effective point set, and performing voxel downsampling on high-density point clouds to reduce data volume and improve processing efficiency. Subsequently, to ensure image projection... The imaging process has a unified spatial reference, and this application consistently uses the minimum coordinate system of the target point cloud (vehicle-mounted) x , y The origin of the image coordinate system is used as the reference point. A horizontal grid is established based on the spatial measurement range of the point cloud. Each point is projected onto the two-dimensional image plane. A top-view orthographic projection is used to vertically project all data points onto the grid. Columns and rows of pixels corresponding to points with valid coordinates are marked in black, and those without are marked in white, resulting in the final binary image (see...). Figure 4 (a) in Figure 4 (d) in the middle.

[0026] To find a sufficiently robust spatial transformation relationship for image registration, the effectiveness of the initial matching point pairs between the target and source images is crucial. For example... Figure 4 As shown in (e), feature keypoints are extracted from the two images respectively (keypoint extraction methods include Scale-Invariant Feature Transform (SIFT) keypoint detectors and ISS methods, etc.). For each source image, keypoints are extracted... Find nearest neighbor candidate points in the target image It satisfies the following dual constraints: (1) Local similarity of descriptors: The main idea behind the Lowe's ratio test is to filter out more reliable matches by comparing the distance ratio between the nearest and second nearest neighbors of each feature point during feature point matching. Only when the nearest neighbor is significantly smaller than the second nearest neighbor can the match be considered reliable, thus improving the accuracy of feature matching. Following the Lowe's ratio test, incorrect matching point pairs that significantly deviate from the normal distance are quickly eliminated. (5) In the formula, and Representing the target image and The two feature keypoints with the smallest and second smallest keypoint distances, This is the ratio value, typically taken as 0.7.

[0027] (2) Main direction consistency constraint (gradient direction): When performing feature point matching between the target image and the source image, if the corresponding keypoints originate from the same real-world region (such as building edges), their principal directions should be approximately the same. Based on this, the gradient magnitude is calculated for all pixels in the neighborhood of the keypoint. and direction All direction values ​​are accumulated in a Gaussian weighted manner to a 36- bin In the direction histogram (each bin (10°), the direction of the maximum peak value in the histogram is the main direction of the key point. The specific calculation formula is: (6) (7) In the formula, For pixel coordinates, These represent the horizontal and vertical gradients, respectively, characterizing the gradient of this pixel. Changes in intensity in a direction.

[0028] Therefore, the condition for image matching point pairs to satisfy the orientation consistency constraint is set as follows: (8) When both the local similarity of the descriptor and the consistency of the principal direction are satisfied, this point pair Only then are they considered potential valid matches.

[0029] Considering the randomness of the distribution of repetitive components in large outdoor scenes, candidate matching relationships obtained solely based on principal direction consistency constraints in complex backgrounds still have uncertainties, and numerous symmetrical structures may cause some corresponding point pairs to shift positions. Therefore, to further supplement candidate matching points and provide as many robust candidate matching relationships as possible, this embodiment proposes a dense optical flow motion estimation method based on Local Binary Pattern (LBP) feature maps. This method calculates the displacement of each pixel to construct a robust corresponding point matching strategy.

[0030] Optical flow assumes that pixel grayscale remains constant between two adjacent frames and calculates the pixel's motion vector field by incorporating smoothing conditions within the pixel's neighborhood. Since both the source and target images are binarized, in this embodiment, the constraint based on brightness gradients in the traditional optical flow method is extended to constraints based on Local Binary Pattern (LBP) feature maps. Specifically, this embodiment optimizes the error during motion prediction of candidate matching points based on the following steps: First, this embodiment starts from the source image. and target image Extracting LBP feature maps from given pixels and its neighborhood (In this embodiment, 8 neighboring points are selected), and the LBP value of this pixel is: (9) In the formula, It is the center pixel grayscale value at that location The radius of the neighborhood points, The angle of the neighboring points (uniformly distributed over 360°). It is a sign function, when hour ,otherwise ,Right now Compare center pixels The brightness value.

[0031] Based on this, this embodiment calculates a binary pattern for each pixel, representing the texture information surrounding that pixel. This texture information is more stable than brightness changes. Therefore, the source image can be obtained. and target image LBP feature map: (10) Subsequently, this embodiment defines the optical flow fields of the source image and the target image as follows: (11) In the formula, Location in the source image The light flow at that location, These represent vector displacements in the horizontal and vertical directions, respectively.

[0032] A joint Farneback dense optical flow estimation method is used to calculate pixel changes between two images to estimate the motion field. This is achieved by constructing a pyramid image and using local multinomial fitting to describe the local changes in the image. It is assumed that the change in each pixel in the image follows a multinomial model, i.e.: (12) In the formula, The polynomial coefficients are estimated using LBP feature information from local image patches.

[0033] The optical flow constraint equation plays a crucial role in the optical flow estimation at each image pixel. It is based on the assumption that the brightness of objects in an image remains constant across consecutive frames. This assumption leads to the optical flow constraint equation: (13) In the formula, and These represent the images in Gradient in direction and time.

[0034] Based on the approximate solution of the polynomial and the gradient of the image, the Farneback algorithm estimates the error function of each pixel. : (14) Finally, a threshold is set based on the optical flow vector displacement. (Note that in this embodiment, the threshold) As an adaptive parameter, slightly lower than the median of all optical flow displacements (because the median distance accurately reflects the central tendency of the effective plane offset distance), mismatch points are eliminated: (15) In the formula, This indicates a valid same-name correspondence in the image matching. , They represent the first i The x and y coordinates of each feature point in the source image; , They represent the first j The actual matching coordinates of each feature point in the target image.

[0035] This embodiment, by replacing the traditional grayscale image optical flow calculation method, demonstrates more robust accuracy and stability in same-name matching selection for dense optical flow calculation based on LBP features in binary images. LBP feature maps provide more stable local texture information than grayscale images, making optical flow calculation more accurate and robust in scenes with varying illumination and uneven texture. Using the Farneback algorithm, this embodiment can efficiently estimate the optical flow field based on LBP features, capturing the motion characteristics of keypoint neighborhoods while reducing the impact of noise and discontinuous changes. Furthermore, error filtering eliminates invalid matching points, resulting in accurate motion estimation.

[0036] The final image matching point pairs are the union of the principal direction consistency constraint and the LBP dense optical flow calculation results (the same set of matching point pairs is automatically deduplicated), which provides the potential correspondence containing the most in-place points with the same name to the greatest extent.

[0037] When performing precise image registration using the IRLS optimization framework, the specific steps can be as follows: Based on obtaining potential matching point pairs, the core objective is to estimate the optimal affine transformation. This makes the source feature points Align the target image feature points as much as possible after the affine transformation. .

[0038] To eliminate false matches, filtering can be based on the geometric consistency between matching pairs. Assume... For two matching pairs, the relative displacement difference between them should be as consistent as possible: (16) Subsequently, to further suppress the impact of mismatches caused by discrete noise or occlusion on image registration accuracy, an Iteratively Reweighted Least Squares (IRLS) optimization framework was designed, combined with the Huber robust kernel function, to stably estimate the parameters in the affine transformation model, thereby improving the accuracy and robustness of image registration. The overall objective function is designed as follows: (17) In the formula, Here is the Huber cost function. For affine transformation residuals .

[0039] Furthermore, the Huber function combines the stability of L2 (squared error) with the robustness of L1 (absolute error) with large errors: (18) In the formula, the parameters This represents the key hyperparameters in the kernel function, which are estimated based on the absolute deviation of the median in this embodiment. The Euclidean residuals of all point pairs are calculated using the initial matching. , = , The empirical threshold is set to 1.5 (close to the standard deviation).

[0040] For the target image and the source image, let the initial affine transformation matrix be... For the first In the next iteration, the residuals of the matching point pairs between images are calculated as follows: (19) The weights are updated using the Huber kernel function: (20) Solving the weighted least squares problem, i.e., minimizing the sum of squared weighted errors, achieves... The iterative update process is used to obtain the most robust image keypoint matching relationship under the optimal affine transformation, see... Figure 4 (f)-(g): (twenty one) In the formula, To minimize the parameters, A The linear transformation part of the affine transformation matrix. t It is a translation vector.

[0041] In the process of calculating the optimal spatial transformation parameters, the image matching point pairs obtained from the optimal affine transformation are back-projected to obtain the coordinates of corresponding points in the target point cloud and the source point cloud, which are respectively represented as target point sets. Source point set Solving for the objective of the two-dimensional rigid transformation is That is, the total squared error is minimized: (twenty two) In the formula, It is a two-dimensional orthogonal rotation matrix in the horizontal direction. This is the horizontal translation vector.

[0042] The surrounding area obtained from the vertical correction process Rotation parameters of direction and elevation direction translation parameters Joint image registration yields results around The transformation parameters of the orientation ultimately yield the optimal spatial transformation matrix of the source point cloud as follows: (twenty three) After the source point cloud and target point cloud were corrected by the above spatial transformation matrix, they were basically aligned. However, in order to obtain a more accurate registration result, the stable and easy-to-operate ICP algorithm was selected to perform fine registration of the point cloud, so as to achieve a high degree of fusion of the heterogeneous point clouds from the vehicle and backpack.

[0043] The method proposed in this embodiment follows an incremental registration process from coarse to fine. Therefore, the point cloud alignment results of coarse and fine registration in the incremental registration experiment are shown respectively. Based on this, to more clearly demonstrate the reliability of the image matching-based point cloud registration method proposed in this embodiment, the local registration details of ground features at different registration stages are simultaneously magnified. The qualitative registration results are as follows: Figures 5-8As shown, the proposed method successfully achieved accurate registration of four datasets. Among the four heterogeneous point cloud datasets, (a) represents the target vehicle-mounted point cloud, (b) the source backpack point cloud, (c)-(d) the original positional deviations and local details of the vehicle-mounted and backpack point clouds, (e)-(f) the coarse registration results and local details, and (g)-(h) the fine registration results and local details. After registration, the source point cloud with original positional deviations is basically aligned with the target point cloud. Although gaps still exist between corresponding ground features after coarse registration, the alignment of the corresponding objects is improved after registration using the ICP fine registration algorithm, and the ground feature point clouds achieve high overlap. Furthermore, the local details of the registration results of multiple datasets show that the incremental registration process of the proposed method exhibits robust registration performance. For example, Figures 5-8 In (f)-(g), the edges of typical ground features such as streetlights, traffic sign poles, and building facades were accurately overlaid, and the high-precision registration of vehicle-mounted and backpack point clouds accurately restored the original outer contour morphology information of the ground features.

[0044] Based on the same inventive concept, this application also provides an image key-based multi-source heterogeneous point cloud registration device for implementing the above-mentioned image key-based multi-source heterogeneous point cloud registration method. The solution provided by this device is similar to the implementation described in the above method. Therefore, the specific limitations of one or more image key-based multi-source heterogeneous point cloud registration device embodiments provided below can be found in the limitations of the image key-based multi-source heterogeneous point cloud registration method described above, and will not be repeated here.

[0045] In one exemplary embodiment, such as Figure 9 As shown, a multi-source heterogeneous point cloud registration device based on image key points is provided, comprising: Ground point cloud extraction module 901 is used to extract ground point clouds from the target point cloud and the source point cloud, respectively. and ground point clouds And determine the ground point clouds respectively. and ground point clouds The unit normal vector and average height; Vertical correction module 902 is used to perform vertical correction on the source point cloud based on a rotation matrix and a vertical translation matrix; the rotation matrix is ​​based on the ground point cloud. and ground point clouds The unit normal vector is obtained by solving; the vertical translation matrix is ​​based on the ground point cloud. and ground point clouds The difference in the average height was calculated. The first filtering module 903 is used to extract feature key points from the target image and the source image, and to filter initial matching point pairs based on the dual constraints of local similarity of descriptors and consistency of principal direction; the target image and the source image are obtained by projecting the target point cloud and the corrected source point cloud. The second filtering module 904 is used to calculate the pixel-level motion vector field and filter the same-name matching point pairs according to the displacement threshold of the dense optical flow method based on the improved local binary pattern feature map. The potential image matching point set determination module 905 is used to perform a union and deduplication of the initial matching point pair and the same-name matching point pair to obtain the potential image matching point set; The constraint module 906 is used to establish geometric consistency constraints based on the potential image matching point set, and to obtain the optimal affine transformation parameters that align the potential image matching point set using an iterative reweighted least squares optimization framework. The back projection module 907 is used to back project the image matching point pairs obtained from the optimal affine transformation parameters to obtain the coordinates of corresponding points in the target point cloud and the source point cloud; and to solve the two-dimensional rigid transformation in the horizontal direction based on the coordinate pairs of corresponding points in the target point cloud and the source point cloud. The registration module 908 is used to calculate the optimal spatial transformation matrix for coarse registration of the source point cloud based on the rotation matrix, the vertical translation matrix and the two-dimensional rigid transformation; based on the optimal spatial transformation matrix for coarse registration of the source point cloud, after correcting the initial rotation and translation position deviations between the target point cloud and the source point cloud, the fine registration of the target point cloud and the source point cloud is completed by using the iterative nearest point algorithm.

[0046] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 10 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores multi-source heterogeneous point cloud registration data based on image key points. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a multi-source heterogeneous point cloud registration method based on image key points.

[0047] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0048] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0049] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0050] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0051] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0052] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0053] In summary, this application has the following technical effects: 1) Significantly improved registration accuracy, specifically with an average rotation error of <0.0005 rad, an average translation error of <0.065 m, and an average point-by-point error of <0.06 m. Key technical aspects contributing to this advantage include two-stage step-by-step registration (vertical + horizontal), principal direction consistency constraints, LBP-improved dense optical flow matching, and a coarse-to-fine progressive registration process. The core reason for this is that: a precise registration reference is established through step-by-step alignment, while multiple constraint mechanisms effectively eliminate erroneous matching points; the LBP improvement method further enhances the reliability of matching; and the coarse-to-fine progressive registration process achieves fine optimization of the registration results, ultimately achieving high-precision registration.

[0054] 2) The method provided in this application is more robust and can effectively overcome adverse conditions such as low overlap, missing occlusion, geometric heterogeneity, and point density variations. This is due to its multi-constraint geometric feature matching strategy, LBP-improved dense optical flow method, and two-stage registration. The core reason is that the multi-constraint mechanism can compensate for the lack of effective information, while the LBP-improved method can resist the interference caused by point density variations and geometric heterogeneity. At the same time, a stable benchmark is established through ground point alignment, thereby significantly improving adaptability in complex scenes.

[0055] 3) The method provided in this application has higher registration efficiency and can efficiently process large point clouds. Its key technical points include voxel downsampling, coarse-fine progressive process, and principal direction consistency constraint. The core reason is that voxel downsampling can effectively reduce the amount of data, reducing the burden on subsequent processing; the coarse registration stage can quickly converge to the optimal solution, thereby reducing the number of iterations in the fine registration stage; at the same time, the principal direction consistency constraint reduces the computational load by filtering effective data, significantly improving the overall efficiency while ensuring registration accuracy.

[0056] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0057] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for registering multi-source heterogeneous point clouds based on image key points, characterized in that, include: Extract ground point clouds from the target point cloud and the source point cloud respectively. and ground point clouds And determine the ground point clouds respectively. and ground point clouds The unit normal vector and average height; Vertical correction of the source point cloud is performed based on a rotation matrix and a vertical translation matrix; the rotation matrix is ​​based on the ground point cloud. and ground point clouds The unit normal vector is obtained by solving; the vertical translation matrix is ​​based on the ground point cloud. and ground point clouds The difference in the average height was calculated. Feature key points are extracted from the target image and the source image, and initial matching point pairs are selected based on the dual constraints of local similarity of descriptors and consistency of principal direction; the target image and the source image are obtained by projecting the target point cloud and the corrected source point cloud. A dense optical flow method based on local binary pattern feature maps is proposed to calculate pixel-level motion vector fields and select matching point pairs with the same name based on displacement thresholds. The initial matching point pairs and the same-name matching point pairs are combined to remove duplicates, resulting in a potential image matching point set. Based on the potential image matching point set, a geometric consistency constraint is established, and an iterative reweighted least squares optimization framework is adopted to obtain the optimal affine transformation parameters that align the potential image matching point set. Based on the image matching point pairs obtained from the optimal affine transformation parameters, the coordinates of corresponding points in the target point cloud and the source point cloud are obtained by back projection; and based on the coordinate pairs of corresponding points in the target point cloud and the source point cloud, the two-dimensional rigid transformation in the horizontal direction is solved. Based on the rotation matrix, the vertical translation matrix, and the two-dimensional rigid transformation, the optimal spatial transformation matrix for coarse registration of the source point cloud is calculated. Based on the optimal spatial transformation matrix for coarse registration of the source point cloud, after correcting the initial rotation and translation position deviations between the target point cloud and the source point cloud, the iterative nearest point algorithm is used to complete the fine registration between the target point cloud and the source point cloud.

2. The multi-source heterogeneous point cloud registration method based on image key points according to claim 1, characterized in that, The dual constraint conditions of local similarity of descriptors and consistency of principal direction specifically include: The formula for the local similarity of the descriptors is: ； The formula for the consistency of the main direction is as follows: ； in, and In the target image, respectively The two feature keypoints with the smallest and second smallest keypoint distances, The ratio value. For the source image, the first i The main direction angle of each key point For the target image, the first j The main direction angle of each key point.

3. The multi-source heterogeneous point cloud registration method based on image key points according to claim 1, characterized in that, Based on the coordinate pairs of corresponding points in the target point cloud and the source point cloud, the two-dimensional rigid transformation in the horizontal direction is solved, specifically including: The corresponding point coordinate pairs in the target point cloud and the source point cloud are represented as target point sets, respectively. Source point set ; According to the formula Solve for the two-dimensional rigid transformation in the horizontal direction; where, It is a two-dimensional orthogonal rotation matrix. It is a translation vector.

4. The multi-source heterogeneous point cloud registration method based on image key points according to claim 1, characterized in that, The local binary pattern feature map is extracted from the target image and the source image; A dense optical flow method based on local binary pattern feature maps is proposed to calculate pixel-level motion vector fields and filter matching point pairs based on displacement thresholds. Specifically, this includes: According to the formula Determine the LBP value of the pixel; where, It is a pixel grayscale value at that location The radius of the neighborhood points, For the angle of the neighboring point, It is a sign function, when hour ,otherwise ; Determine the optical flow fields of the source and target images. In the formula, Location in the source image The flow of light at that location, These represent vector displacements in the horizontal and vertical directions, respectively. based on Farneback Dense optical flow estimation methods calculate pixel changes between source and target images and compute pixel-level motion vector fields; Set threshold based on optical flow vector displacement By eliminating incorrect matches, we obtain pairs of matches with the same name.

5. The multi-source heterogeneous point cloud registration method based on image key points according to claim 4, characterized in that, Set threshold based on optical flow vector displacement Eliminating false matches specifically includes: According to the formula Eliminate false matches; In the formula, This indicates a valid same-name correspondence in the image matching. , They represent the first i The x and y coordinates of each feature point in the source image; , They represent the first j The actual matching coordinates of each feature point in the target image.

6. The multi-source heterogeneous point cloud registration method based on image key points according to claim 1, characterized in that, Based on the potential image matching point set, geometric consistency constraints are established, and an iterative reweighted least squares optimization framework is used to obtain the optimal affine transformation parameters that align the potential image matching point set, specifically including: Based on the potential image matching point set, geometric consistency constraints are established, and an iterative reweighted least squares optimization framework incorporating the Huber robust kernel function is used to obtain the optimal affine transformation parameters that align the potential image matching point set. The iterative reweighted least squares optimization framework is used to suppress the impact of mismatches caused by discrete noise or occlusion on image registration accuracy. The formula for the overall objective function incorporating the Huber robust kernel equation is as follows: In the formula, Here is the Huber cost function. For affine transformation residuals , To minimize the parameters, A The linear transformation part of the affine transformation matrix. t It is a translation vector.

7. The multi-source heterogeneous point cloud registration method based on image key points according to claim 1, characterized in that, The formula for the optimal spatial transformation matrix for coarse registration of the source point cloud is as follows: ； In the formula, This is a rotation matrix in the vertical direction. This is the vertical translation matrix. It is a two-dimensional orthogonal rotation matrix in the horizontal direction. It is the horizontal translation vector. 、 、 To register the 3D coordinates of all points in the source point cloud, 、 、 Register the 3D coordinates of all points in the source point cloud.

8. A multi-source heterogeneous point cloud registration device based on image key points, used to implement the multi-source heterogeneous point cloud registration method based on image key points as described in any one of claims 1-7, characterized in that, include: The ground point cloud extraction module is used to extract ground point clouds from the target point cloud and the source point cloud, respectively. and ground point clouds And determine the ground point clouds respectively. and ground point clouds The unit normal vector and average height; The vertical correction module is used to perform vertical correction on the source point cloud based on a rotation matrix and a vertical translation matrix; the rotation matrix is ​​based on the ground point cloud. and ground point clouds The unit normal vector is obtained by solving; the vertical translation matrix is ​​based on the ground point cloud. and ground point clouds The difference in the average height was calculated. The first filtering module is used to extract key feature points from the target image and the source image, and to filter initial matching point pairs based on the dual constraints of local similarity of descriptors and consistency of principal direction; the target image and the source image are obtained by projecting the target point cloud and the corrected source point cloud. The second filtering module is used to calculate pixel-level motion vector fields and filter matching point pairs based on displacement thresholds using a dense optical flow method improved from local binary pattern feature maps. The potential image matching point set determination module is used to perform a union and deduplication of the initial matching point pair and the same-name matching point pair to obtain the potential image matching point set; The constraint module is used to establish geometric consistency constraints based on the potential image matching point set, and to obtain the optimal affine transformation parameters that align the potential image matching point set using an iterative reweighted least squares optimization framework. The back projection module is used to back-project the image matching point pairs obtained from the optimal affine transformation parameters to obtain the coordinates of corresponding points in the target point cloud and the source point cloud. Based on the coordinate pairs of corresponding points in the target point cloud and the source point cloud, the two-dimensional rigid transformation in the horizontal direction is solved; The registration module is used to calculate the optimal spatial transformation matrix for coarse registration of the source point cloud based on the rotation matrix, the vertical translation matrix, and the two-dimensional rigid transformation; based on the optimal spatial transformation matrix for coarse registration of the source point cloud, after correcting the initial rotation and translation position deviations between the target point cloud and the source point cloud, the fine registration of the target point cloud and the source point cloud is completed by using the iterative nearest point algorithm.

9. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement a multi-source heterogeneous point cloud registration method based on image key points according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-source heterogeneous point cloud registration method based on image key points as described in any one of claims 1-7.

Images