A method and system for adaptive registration of oblique photography and laser point cloud

Abstract

This application relates to the fields of surveying and remote sensing technology, and discloses an adaptive registration method and system for oblique photogrammetry and laser point clouds. It aims to solve the problems of low registration efficiency, insufficient accuracy, and poor robustness caused by existing technologies relying on manual control points or iterative nearest-point algorithms. The method includes: acquiring oblique photogrammetry images and laser point cloud data; extracting multi-scale image structural features and point cloud geometric and semantic features respectively; constructing a cross-modal feature matching candidate set; selecting the optimal initial pose through random sampling consistency; and performing nonlinear optimization based on dual constraints of projection consistency and surface continuity to generate the final registration transformation matrix. This application achieves high-precision, fully automatic, end-to-end registration under control point-free conditions, significantly improving the efficiency and accuracy of 3D modeling in large-scale complex scenes.

Description

Technical Field

[0001] This invention belongs to the field of surveying and remote sensing technology, specifically relating to an adaptive registration method and system for oblique photography and laser point clouds. Background Technology

[0002] In the field of 3D geographic information acquisition, oblique photogrammetry and LiDAR scanning are two core technologies. Oblique photogrammetry can efficiently acquire high-resolution texture information of the Earth's surface, while LiDAR can directly and accurately capture 3D geometric structures. Fusing the data from both can generate realistic 3D models with both high-precision geometry and lifelike texture, which is crucial for fields such as smart cities and fine mapping. The prerequisite for achieving this fusion is completing the spatial registration between the two technologies.

[0003] Existing registration methods are mainly divided into two categories. One category relies on manually established control points in the field. While this method offers high accuracy, it is time-consuming and labor-intensive, difficult to implement in complex or hard-to-reach areas, and has a low degree of automation. The other category employs automatic registration algorithms, such as the iterative nearest-point algorithm. However, due to the fundamental differences in data modality between oblique photogrammetry data (usually represented as a 3D surface model) and laser point clouds, traditional algorithms often get stuck in local optima when handling such cross-modal registration because it is difficult to reliably establish feature correspondences. They are also sensitive to initial positions and lack robustness and accuracy in large scenes with missing textures and complex structures.

[0004] Therefore, current technology cannot achieve high-precision and robust fully automatic registration of oblique photography and laser point clouds in various complex scenarios without human intervention, which restricts the large-scale and efficient application of 3D reality modeling technology. Summary of the Invention

[0005] This invention provides an adaptive registration method and system for oblique photogrammetry and laser point cloud data, aiming to solve the technical problems of low registration efficiency, insufficient accuracy, poor robustness, and difficulty in achieving automated registration of large-scale scenes caused by reliance on manual control points or iterative nearest-point algorithms in existing technologies. This invention achieves high-precision, fully automatic, end-to-end registration of oblique photogrammetry images and laser point cloud data without prior control information by constructing a multi-level feature extraction and cross-modal semantic alignment mechanism, combined with a geometrically constrained global optimization strategy.

[0006] According to one aspect of the present invention, an adaptive registration method for oblique photography and laser point clouds is provided, comprising: Obtain oblique photogrammetry image dataset and laser point cloud dataset; Structural features are extracted from the oblique photogrammetry image dataset to obtain an image structural feature map, and geometric semantic features are extracted from the laser point cloud dataset to obtain a point cloud geometric semantic feature body. Based on the image structural feature map and the point cloud geometric semantic feature body, line-edge matching and plane-plane matching are performed to construct a cross-modal feature matching candidate set; Based on the cross-modal feature matching candidate set, a robust pose estimation algorithm is used, wherein the randomly sampled minimum sample set contains at least one plane-to-plane matching pair to select the optimal initial pose. Starting from the optimal initial pose, a nonlinear optimization based on dual constraints of projection consistency and surface continuity is performed to minimize the objective function, which is a weighted sum including reprojection error term and normal vector alignment error term, and the final registration transformation matrix is ​​obtained by solving. The registration transformation matrix is ​​applied to the laser point cloud dataset to achieve spatial alignment with the oblique photogrammetric image dataset.

[0007] Furthermore, structural features are extracted from the oblique photogrammetry image dataset to obtain image structural feature maps, including: Construct a multi-scale image pyramid of the oblique photographic images; Edge detection and line segment extraction are performed at each scale to obtain multi-scale line features; Superpixel segmentation is performed on the original resolution image, and plane fitting is performed on each superpixel region in the gray value space to obtain superpixel planar features; The image structure feature map is composed of the multi-scale linear features and the superpixel planar features.

[0008] Furthermore, geometric semantic features are extracted from the laser point cloud dataset to obtain the point cloud geometric semantic feature body, including: Calculate the local neighborhood normal vector and curvature value of each point in the point cloud; Based on the curvature value, the points are semantically classified, and the classification includes at least planar points, edge points, and corner points; For points classified as planar points, a region growing algorithm is applied to aggregate them into continuous planar entities; The point cloud geometric semantic feature body includes the semantic classification information of points and the planar entity formed by aggregation.

[0009] Furthermore, based on the image structural feature map and point cloud geometric semantic feature volume, line-edge matching and plane-plane matching are performed, including: For line-edge matching: the angle between the direction vector of the image line in three-dimensional space and the direction vector of the point cloud edge line segment is calculated as the direction deviation, and the average distance from the point on the point cloud edge line segment to the image three-dimensional line is calculated as the position deviation; when the direction deviation is less than the first angle threshold and the position deviation is less than the first distance threshold, it is determined to be a valid matching pair; For plane-to-plane matching: calculate the angle between the normal vectors of the image plane and the point cloud plane entities, as well as the distance offset of the center point along the normal direction; calculate a joint metric based on the angle between the normal vectors and the distance offset; when the joint metric is less than a preset threshold, it is determined to be a valid matching pair; The first angle threshold is 15 degrees, and the first distance threshold is 10 centimeters.

[0010] Furthermore, a robust pose estimation algorithm based on random sampling consensus is employed, including: Multiple random samplings are performed from the cross-modal feature matching candidate set, with each sampling extracting at least three non-collinear feature matching pairs and at least one plane-to-plane matching pair to form the minimum sample set; For each of the minimum sample sets, an initial rigid body transformation matrix is ​​calculated by solving the Procrassa problem; Each of the initial rigid body transformation matrices is applied to the entire cross-modal feature matching candidate set, and the number of interior points that satisfy the preset tolerance is counted. The initial rigid body transformation matrix with the largest number of interior points is used as the optimal initial pose output.

[0011] Furthermore, nonlinear optimization based on the dual constraints of projection consistency and surface continuity includes: Define the objective function ,in, : indicates the first feature selected from the point cloud features A three-dimensional point, : Indicates the rigid body transformation to be optimized. ,in It is a rotation matrix. It is a translation vector. : Represents the perspective projection function; : Represents a point The corresponding image features in the first Two-dimensional pixel coordinates on a tilted photographic image : The total number of point pairs involved in the reprojection error calculation; The objective function is iteratively minimized in the Lie algebra space using the Levenberg-Marquardt algorithm, with the optimization variable being the Lie algebra parameter corresponding to the optimal initial pose.

[0012] The weighting coefficient α is set to 1.0 and β is set to 0.5.

[0013] Furthermore, the oblique photogrammetry image dataset and the laser point cloud dataset were obtained, including: Acquire oblique photographic images with initial exterior orientation elements, captured by a multi-view camera array; Acquire laser point clouds with spatial resolution down to the centimeter level and a point density of no less than 50 points per square meter; A unified time synchronization system ensures that the timestamps of the oblique photographic images are aligned with those of the laser point cloud.

[0014] According to another aspect of the present invention, an adaptive registration system for oblique photography and laser point clouds is provided, comprising: The data acquisition module is used to acquire and store oblique photogrammetry image datasets and laser point cloud datasets; The image feature extraction module is used to extract structural features from oblique photogrammetric image datasets and generate image structural feature maps. The point cloud feature extraction module is used to extract geometric and semantic features from the laser point cloud dataset and generate a point cloud geometric and semantic feature body. A cross-modal matching module is used to construct a cross-modal feature matching candidate set based on the image structural feature map and the point cloud geometric semantic feature volume; The initial pose estimation module is used to select the optimal initial pose based on the cross-modal feature matching candidate set and employs a random sampling consensus algorithm. Its random sampling strategy forces the inclusion of at least one plane-to-plane matching pair. A fine-grained registration optimization module is used to perform nonlinear optimization based on dual constraints of projection consistency and surface continuity, starting from the optimal initial pose, and outputs the final registration transformation matrix; and The spatial alignment execution module is used to apply the registration transformation matrix to the laser point cloud dataset and output the spatially aligned point cloud data.

[0015] In summary, this application includes the following beneficial technical effects: This invention eliminates the reliance on manually set control points and overcomes the shortcomings of traditional iterative nearest-point algorithms, which are prone to getting trapped in local optima during cross-modal data registration. By constructing a cross-modal matching mechanism between image structural features and point cloud geometric and semantic features, a semantic link between oblique photogrammetry and laser point clouds is effectively established. Multi-scale feature extraction ensures feature stability under different resolutions and complexities. Initial pose estimation based on random sampling consistency significantly improves the robustness of registration, obtaining reliable initial solutions even in the presence of significant noise in feature matching. Nonlinear optimization with dual constraints of projection consistency and surface continuity constrains the registration process from both geometric and visual dimensions, greatly improving the final registration accuracy. Experiments show that this invention can achieve sub-pixel-level registration accuracy in various complex scenes such as cities, mountains, and forests, meeting the mapping requirements of 1:500 and even higher scales, and supporting fully automatic, end-to-end processing of large-scale scenes, significantly improving the efficiency and quality of real-world 3D modeling. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the architecture of the oblique photogrammetry and laser point cloud adaptive registration method; Figure 2 A schematic diagram illustrating the principle framework of cross-modal feature matching and initial pose estimation; Figure 3 This is a logical framework diagram for registration optimization based on the dual constraints of projection consistency and surface continuity. Detailed Implementation

[0017] This invention provides an adaptive registration method and system for oblique photogrammetry and laser point clouds, aiming to solve the technical problems of low registration efficiency, insufficient accuracy, poor robustness, and difficulty in achieving automated registration of large-scale scenes caused by reliance on manual control points or iterative nearest-point algorithms in existing technologies. The specific implementation of the method will be described in detail below in conjunction with the technical solution of this invention.

[0018] The first aspect is the adaptive registration method for oblique photography and laser point clouds disclosed in this application, which includes the following steps: S1, Obtain the oblique photogrammetry image dataset and the laser point cloud dataset; S2 extracts multi-scale edge and planar structure features from the oblique photogrammetry dataset to generate an image structure feature map; performs normal vector estimation and local curvature analysis on the laser point cloud dataset to generate a point cloud geometric semantic feature body. S3, based on image structural feature maps and point cloud geometric semantic feature volumes, constructs a cross-modal feature matching candidate set; S4. The candidate set is matched with cross-modal features to establish the initial pose hypothesis set, and the optimal initial pose is selected by a robust estimator based on random sampling consistency. S5, starting from the optimal initial pose, performs a nonlinear optimization process based on dual constraints of projection consistency and surface continuity to generate the final registration transformation matrix; S6 applies the registration transformation matrix to the laser point cloud dataset to achieve spatial alignment with the oblique photogrammetric image dataset.

[0019] The first step, S1, is the data preparation stage of the entire registration process. Its core purpose is to acquire and prepare two heterogeneous but complementary remote sensing data sources—oblique photogrammetry and laser point clouds—ensuring their spatiotemporal and logical correlation. This lays a reliable data foundation for subsequent fully automated, high-precision cross-modal registration. This step clarifies the specific data sources, key technical indicators, synchronization methods, and storage specifications to ensure the quality and consistency of the input data. Specifically, it includes the following ordered sub-steps: Step S101: Obtain the oblique photogrammetry image dataset, which is acquired by a multi-view camera array mounted on a flight platform. The dataset must contain images from no fewer than five different viewpoints. Each image is accompanied by initial exterior orientation elements calculated by the fusion of the inertial navigation system and the global positioning system. These elements accurately provide the camera's three-dimensional position coordinates in space and its attitude angle at the moment of the image capture.

[0020] Step S102: Obtain the laser point cloud dataset, which is obtained by scanning the target area using an airborne or ground-based lidar system. The point cloud data must meet centimeter-level spatial resolution and have a point density of no less than 50 points per square meter. Each point cloud data point must contain accurate three-dimensional spatial coordinate information and its corresponding reflection intensity value.

[0021] The above steps S101 and S102 complete the independent acquisition of two types of core raw data, providing high-resolution surface texture information and high-precision three-dimensional geometric information respectively.

[0022] Step S103: Implement spatiotemporal data synchronization. During data acquisition, the capture of oblique photogrammetric images and the scanning of laser point clouds are coordinated through a unified timing system to ensure that each data item is recorded with the same reference timestamp. This measure ensures strict alignment of the two types of data in the time dimension, providing a prerequisite for subsequently establishing the spatiotemporal correspondence of cross-modal features.

[0023] Step S104: Complete data encapsulation and logical association. The acquired oblique photogrammetric image data is stored in its uncompressed raw format to avoid loss of texture details due to compression encoding. The laser point cloud data is encapsulated and managed using the standard LAS format commonly used in the surveying and mapping field. Finally, using the unique task identifier generated by this data acquisition task, all image files and point cloud files are logically associated to form a complete and traceable input dataset.

[0024] In summary, step S1 defines a complete, clear, and operable set of data input specifications, from data source and accuracy metrics to synchronization mechanisms and storage formats. This ensures that subsequent processing flows obtain high-quality, aligned, and clearly correlated raw data, making it the first crucial step in achieving adaptive registration without control points. Based on this, the system will then enter the core stage of feature extraction and matching.

[0025] Step S2 is a crucial step in achieving control-point-free adaptive registration. Specifically, it transforms the original oblique photogrammetric imagery and laser point cloud data into intermediate feature representations with clear geometric and physical meanings. By performing multi-scale structural analysis on the imagery and local geometric semantic parsing on the point cloud, this step establishes a unified dialogue foundation for subsequent cross-modal feature matching.

[0026] This process aims to extract robust, structured features from 2D images that can be used for 3D spatial comparison, mainly including prominent edge lines and homogeneous texture planes.

[0027] Step S201: For each input oblique photographic image, a Gaussian filter is first applied for smoothing to suppress noise. The two-dimensional kernel function of the Gaussian filter is controlled by the standard deviation σ parameter; in this embodiment, σ = 0.5. The smoothed image is downsampled at a fixed ratio to construct an image pyramid containing three scales. Let the original image be layer 0; then the three generated scale levels are: layer 1 (scale factor of 1, i.e., original resolution), layer 2 (scale factor of 1 / 2), and layer 3 (scale factor of 1 / 4).

[0028] Step S202: Perform Canny edge detection independently on the three scale images generated in step S201. For each scale image, first use the Sobel operator to calculate the gradient magnitude of each pixel. With direction .in, , , and These are the gradient components in the horizontal and vertical directions, respectively. The high threshold of the Canny operator. Set as the gradient magnitude of all pixels in the image at this scale. 95th percentile; low threshold Set as A standard Hough transform is applied to the detected edge pixels to extract line segments. Only line segments longer than 30 pixels are retained, and adjacent collinear short line segments with directional deviations within 5 degrees are connected to form longer line segments. Finally, the features of each straight line are... From its starting point in the original image coordinate system ,end Unit direction vector and its corresponding scale level Joint description.

[0029] Step S203: On the original resolution image, a simple linear iterative clustering algorithm is used for superpixel segmentation. This algorithm jointly clusters pixels in the CIELAB color space and the XY coordinate space. In this embodiment, the weighting factor for color space distance... Set to 10, the weighting factor for spatial distance. Set to 20, target superpixel count Based on the total number of pixels in the image Dynamically determined, the calculation formula is as follows The algorithm iterates until convergence, dividing the image into... A visually continuous, homogeneous superpixel region.

[0030] Step S204: For each superpixel region obtained in step S203 Calculate the column coordinates of all pixels within it. row coordinates and the normalized grayscale value Consider as a three-dimensional point A plane is fitted using the least squares method, and its equation is: in, Let be the plane parameters. The optimal solution is obtained by solving the normal equations formed by all pixels within this superpixel. The fitted plane is in Unit normal vector in space The calculation formula is: Simultaneously, record the geometric center coordinates of the superpixel region. and its average gray value .

[0031] Steps S201 to S204 systematically transform the image from a pixel array into a set of geometric features. A multi-scale strategy ensures the stability of features at different resolutions, while superpixel planar fitting maps local texture information into surface elements with three-dimensional geometric interpretation. All extracted linear and planar features together constitute an "image structure feature atlas".

[0032] This process aims to analyze the local 3D geometric characteristics of laser point clouds, perform semantic classification of points, and aggregate them to form a higher-level geometric structure, providing a geometric basis for alignment with image features.

[0033] Step S205: For any point in the point cloud First, it is necessary to determine its local neighborhood range. Then, calculate the average point spacing of the entire point cloud. The method involves randomly sampling at least 100 points, calculating the Euclidean distance from each point to its nearest neighbor, and then taking the arithmetic mean of these distances. Then, using the points... Construct a radius of centered at . A spherical neighborhood, where the radius is Determined by the following formula: Collect all points within the spherical neighborhood to form a local point set. Suppose it contains Calculate the centroid of this local point set. : Next, the local point set is calculated. covariance matrix : For covariance matrix Eigenvalue decomposition yields three eigenvalues. (satisfy ) and its corresponding unit eigenvector The corresponding smallest eigenvalue eigenvectors Initially defined as a point normal vector To ensure consistency in the direction of the normal vector (e.g., pointing outwards from the scanner), a global reference direction is preset. (like ),like Then let .

[0034] Step S206: Based on the feature values ​​obtained in step S205, calculate the points. Gaussian curvature and mean curvature The calculation formula is as follows: in, It is a point Eigenvalues ​​of the local neighborhood covariance matrix. Semantic classification of points is based on curvature values, according to the following rules: like and Then the point Marked as "planar point".

[0035] like and Then the point Marked as "corner point".

[0036] If the above conditions are not met, then the point will be... Marked as "edge point".

[0037] Step S207: For all points marked as "planar points", execute the region growing algorithm to aggregate them into continuous planar patches. The specific process is as follows: 1. Create an array of access markers, initially all planar points are unvisited.

[0038] 2. Among the unvisited planar points, find the one with the minimum... The point with the value is used as the seed point.

[0039] 3. Initialize an empty set to store the points of the current planar cluster, and add the seed point.

[0040] 4. Traverse the spatial neighborhood (radius) of each point in the current planar cluster. For unvisited planar points within the neighborhood Check two conditions: a) Its normal vector Average normal vector of the current plane cluster a) The included angle is less than 10 degrees; b) Point The distance to the current plane cluster fitting plane is less than 5 centimeters.

[0041] 5. If the conditions are met, then set the point... Add it to the current cluster and mark it as visited.

[0042] 6. Repeat steps 4-5 until the current cluster can no longer grow.

[0043] 7. Output the current planar cluster as a "planar entity", recording the set of points it contains and the plane equation refitted by least squares. and geometric center .

[0044] 8. Return to step 2 and find the next seed point until all plane points have been visited.

[0045] In summary, step S2, through parallel dual-channel processing, extracts the texture information of the image and the geometric information of the point cloud into intermediate feature representations centered on "straight lines," "planes," and "surface types." This method of mapping different modal data to a common geometric semantic space is the core innovation and foundation for achieving subsequent high-precision, control-point-free cross-modal registration in this application.

[0046] Regarding step S3: constructing a cross-modal feature matching candidate set, step S3 is a bridging step in this method, aiming to establish a direct geometric correspondence between oblique photogrammetric image features and laser point cloud features. By utilizing the image structure feature map and point cloud geometric semantic feature volume generated in step S2, this step searches for feature pairs satisfying specific geometric consistency constraints between the two types of heterogeneous data, thereby constructing a reliable matching candidate set and providing a data foundation for subsequent solution of spatial transformation parameters. This process mainly revolves around matching two types of geometric features: linear features and planar features.

[0047] Step S301: For each salient line segment in the image structure feature map, it needs to be transformed from its 2D image coordinate system to a 3D world coordinate system unified with the laser point cloud. This transformation depends on the initial exterior orientation elements of the image to which the line segment belongs. Specifically, using the image's interior orientation elements and the initial exterior orientation elements, the pixel coordinates of the two endpoints of the line segment are back-projected into two 3D spatial rays through collinearity equations. These two rays form a plane in 3D space, and the line segment corresponds to the intersection of this plane with a virtual plane parallel to the image plane that passes through a point on the object side.

[0048] To simplify the calculation, its three-dimensional direction vector can be taken as the main constraint. Finally, the expression of the image line in three-dimensional space is obtained, and its unit direction vector is denoted as... Furthermore, a three-dimensional point can be derived from its back-projected ray plane equation. Use it as a reference point for the line.

[0049] Step S302: Extract all points marked as "edge points" from the point cloud geometric semantic feature volume. To compare with straight lines in the image, these discrete edge points need to be connected into continuous line segments. The connection rule can be based on the scan line order of the point cloud or on spatial proximity. Specifically, for each edge point, search for neighboring points that are also edge points among its K nearest neighbors. If the spatial distance between two points is less than a threshold... Furthermore, the consistency between the direction of the line connecting the two points and the direction of the local edge is higher than the threshold. Then connect the two points.

[0050] Repeat this process to form a series of three-dimensional polyline segments composed of edge points. For each approximately straight polyline segment, use its starting point... and the end point Define a three-dimensional line segment with a unit direction vector as follows: .

[0051] Step S303: For the three-dimensional line obtained in step S301 and the point cloud edge segment obtained in step S302, calculate the directional deviation between them. with positional deviation .

[0052] Directional deviation Defined as the absolute value of the angle between the direction vectors of two lines: in, and These are the unit direction vectors for the image lines and the point cloud line segments, respectively. This represents the dot product of vectors.

[0053] Positional deviation Defined as the average Euclidean distance from all points on the edge segment of the point cloud to the 3D straight line in the image. Let the point cloud segment consist of a set of points... Composition, then distance For point To the direction is and past the reference point The distance to the straight line is calculated using the following formula: in Let represent the cross product of vectors. Then the average positional deviation is: Step S304: Set the orientation deviation threshold and position deviation threshold If a pair of image lines and point cloud edge segments satisfy... and If they form a valid line-edge matching pair, they are added to the initial matching candidate set.

[0054] Step S305: For each planar region in the image structure feature map, its features have been expressed as a planar equation and its three-dimensional geometric interpretation in the image grayscale space. It needs to be transformed to the same three-dimensional spatial coordinate system as the laser point cloud. This transformation is based on the initial exterior orientation elements of the image. Specifically, the geometric center coordinates of the superpixels are... and its average gray value Treating it as a three-dimensional point, and combining the interior and exterior orientation elements of the image, the three-dimensional object coordinates corresponding to this center point are calculated using the principles of forward intersection or spatial backward intersection. .

[0055] Meanwhile, the normal vector of the image plane in the grayscale space It also requires the image's attitude angle rotation matrix. Transform to object space to obtain the plane normal vector in three-dimensional space. Therefore, the image plane in the point cloud coordinate system can be determined by its three-dimensional center point. and three-dimensional unit normal vector definition.

[0056] Step S306: For an image plane obtained in step S305 and an aggregated planar entity extracted from the point cloud geometric semantic feature volume, calculate two metrics: the angle between the normal vectors. Distance offset from the center point along the normal direction .

[0057] Angle between normal vectors The calculation formula is: in, It is the unit normal vector of the point cloud planar entity.

[0058] Distance offset The calculation formula is: in, These are the coordinates of the three-dimensional center point of the point cloud planar entity.

[0059] To comprehensively evaluate the similarity of two planes, a joint metric is defined. : in, It is the weighting coefficient for the angle term, with a value of 1 radian per degree, used to harmonize the angular and distance dimensions; This is the weighting coefficient for the distance term, with a value of 1.

[0060] Step S307: Set the joint metric threshold If a pair of image planes and point cloud planes satisfy... If they are found to form a valid plane-to-plane matching pair, they are added to the initial matching candidate set.

[0061] Step S308: Combine all line-edge matching pairs selected in step S304 and all plane-plane matching pairs selected in step S307 to form the final cross-modal feature matching candidate set. Each match in the candidate set All of them clearly record the unique identifiers of image features, the unique identifiers of point cloud features, the matching type, and the calculated geometric error index.

[0062] In summary, step S3 systematically establishes the correspondence between the image and the point cloud based on line and planar elements through two rigorously defined geometric feature matching processes. This process fully utilizes the high-level features extracted in step S2 and uses preset, quantified geometric thresholds to screen the reliability of the matches, thereby generating an initial matching set rich in semantic information and with high geometric consistency. This candidate matching set directly serves as the input for robustly estimating the initial pose based on the random sampling consensus algorithm in subsequent step S4. It is a crucial link in the entire registration process, and its quality directly affects the accuracy and success rate of the final registration.

[0063] Regarding step S4: Robust initial pose estimation based on RANSAC, the core objective of step S4 is to robustly estimate an optimal initial rigid body transformation from a cross-modal feature matching candidate set that may contain noise and outliers. This is a crucial step in the entire registration process to ensure that subsequent optimization converges to the correct solution. This step employs a random sampling consensus algorithm framework, through extensive random sampling, model solving, and consensus verification, to select the spatial transformation compatible with the most matching features, thereby providing a high-quality and stable starting point for subsequent nonlinear optimization.

[0064] Step S401: Set the total number of random samplings Set a distance tolerance threshold for evaluating interior points. Rice. Initialize an empty collection. Used to store all generated initial pose assumptions and initialize the optimal pose variables. and the corresponding maximum number of interior points .

[0065] Step S402: From the cross-modal feature matching candidate set generated in step S3 In this process, a set of matching pairs is randomly selected. To ensure that the transformation of a three-dimensional rigid body with six degrees of freedom can be solved, this set of matching pairs must contain at least three pairs of non-collinear feature matches. Furthermore, to improve the stability of the solution, this set of matching pairs must contain at least one plane-plane matching pair. The selected set of matching pairs constitutes a minimal sample set. .

[0066] Step S403: For the minimum sample set extracted in step S402 It is necessary to solve for a rigid body transformation matrix. This ensures that the transformation best aligns these matched features. Specifically, from each matching pair, the position of the image features in the three-dimensional coordinate system of the image is extracted. and the position of the corresponding point cloud features in the point cloud coordinate system The location here can be the midpoint of the line segment for linear features, and the center point for planar features.

[0067] The goal is to find a rotation matrix. and a translation vector This minimizes the overall distance between the transformed image feature points and their corresponding point cloud feature points. This constitutes a Procrustean problem, mathematically expressed as: in, It is a 3x3 orthogonal rotation matrix. It is a 3x1 translation vector.

[0068] This problem can be solved using the singular value decomposition method through the following steps: 1. Calculate the image feature point set separately. and point cloud feature point set center of mass and .

[0069] 2. Calculate the centroid coordinates: , .

[0070] 3. Calculate the matrix .

[0071] 4. For the matrix Perform singular value decomposition: .

[0072] 5. The solution to the rotation matrix is: To ensure the determinant of the rotation matrix is ​​+1 (excluding mirror images), if Then Recalculate after inverting the last column of the matrix. .

[0073] 6. The solution for the translation vector is: .

[0074] The final rigid body transformation matrix This is an initial pose assumption.

[0075] Step S404: Calculate the transformation matrix obtained in step S403. Applied to the entire matching candidate set All image feature locations. For each matching pair Calculate its image feature points after Transformed coordinates Then calculate the feature points of the point cloud corresponding to that point. Euclidean distance between If the distance Less than the preset tolerance threshold Then the matching pair is determined. For the current transformation An interior point. Count the total number of interior points in all matched pairs. .

[0076] Step S405: Compare the current hypothesis Number of interior points Compared to the historical maximum number of interior points .if Then the current hypothesis is updated to the optimal hypothesis, that is, let and update At the same time, the current assumptions will be... and the number of its interior points Stored as a record in the hypothesis set .

[0077] Step S406: Repeat steps S402 to S405 until the preset total number of random samples is reached. .go through After the iteration, suppose the set It contains all generated pose assumptions and their interior point evaluation results, while This stores the transformation matrix with the largest number of interior points.

[0078] Step S407: Apply the optimal pose assumption obtained in step S406. This serves as the output of the selected optimal initial pose. This pose represents a rigid body transformation consistent with the most cross-modal feature matches, exhibiting good robustness and global consistency.

[0079] In summary, step S4, through systematic random sampling, model solving, and interior point verification, robustly estimates the optimal initial spatial transformation from noisy matching data. This process effectively overcomes the sensitivity of traditional methods to initial values, providing a crucial starting point close to the global optimum for the refined nonlinear optimization in step S5, thus ensuring the accuracy and reliability of the final registration result.

[0080] Regarding step S5: Nonlinear Refined Registration Based on Dual Constraints, step S5 is the final optimization stage of this method. Its core objective is to perform high-precision nonlinear iterative optimization based on the robust initial pose provided in step S4 by establishing a comprehensive objective function that integrates visual projection and geometric continuity information. This step aims to simultaneously minimize the reprojection error of the point cloud onto the image and maximize the consistency between the point cloud and the image surface normal vectors, thereby solving for the optimal rigid body transformation matrix that can achieve sub-pixel-level alignment. This is the decisive step in improving the final accuracy of the entire registration scheme.

[0081] Step S501: This step defines a comprehensive objective function E to quantify the quality of the rigid body transformation matrix T to be determined. Function E consists of a weighted sum of two terms, corresponding to the projection consistency constraint and the surface continuity constraint, respectively.

[0082] The specific mathematical form of the objective function E is as follows: The following is a clear definition of each term in the formula: First item: Reprojection error term: : indicates the first feature selected from the point cloud features Three-dimensional points, which are typically derived from the feature points successfully matched in step S3 or their neighbors. : Indicates the rigid body transformation to be optimized. ,in It is a rotation matrix. It is a translation vector. : Represents the perspective projection function, which projects points in three-dimensional space onto a two-dimensional image plane. This function is determined by the camera's intrinsic parameter matrix. Definition, specifically The first two components. : Represents a point The corresponding image features in the first Two-dimensional pixel coordinates on a tilted photographic image. : The total number of point pairs involved in the reprojection error calculation. The weighting coefficient of the first term is used to balance the magnitude and importance of the two errors. In this embodiment, it is set to a value of [value missing]. .

[0083] Second term: Normal vector alignment error term : Represents the first extracted from the geometric semantic feature volume of the point cloud. The unit normal vector of a planar entity. : Rotation matrix (transformation) in the current iteration (The rotating part). : Represents a planar entity in a point cloud The three-dimensional unit normal vector of the matching image plane region (which has been converted to object space in step S3). Calculate the absolute value of the cosine of the angle between the transformed point cloud normal vector and the image normal vector. When the two directions are completely aligned, this value is 1, and the error is 0; when they are perpendicular, it is 0, and the error is 1. : The total number of plane matching pairs involved in the normal vector alignment calculation. The weighting coefficient for the second term, in this embodiment, is taken as: Objective function The smaller the value, the better the current transformation. The better the point cloud and the image are aligned in terms of projection position and surface geometry, the better.

[0084] Step S502: Initialize the parameters required by the nonlinear optimizer. This includes setting the optimization variables, i.e., the rigid body transformation. , denoted as in its Lie algebra space 6-dimensional vector on This representation is beneficial for derivative and update calculations. Optimized initial values. Set as the optimal initial pose output in step S4 The corresponding Lie algebra.

[0085] Set the termination condition for iterative optimization: maximum number of iterations. and the convergence threshold of the objective function value change. When the number of iterations reaches Or the absolute change in the objective function value between two adjacent iterations is less than When the time comes, optimization stops.

[0086] Steps S501 and S502 together complete the preparation work for fine-grained optimization, clearly defining the target to be minimized as well as the starting point and stopping rules for optimization.

[0087] Step S503: Use the Levenberg-Marquardt algorithm to evaluate the objective function. Perform iterative minimization. This is a process of solving a linear equation in each iteration to update the parameters.

[0088] In each iteration In the middle, perform the following operations: 1. Calculate the Jacobian matrix and gradient: Calculate the current parameters. Objective function Regarding Lie algebra parameters Jacobian matrix Simultaneously, the gradient vector is calculated. ,in It is a residual vector composed of all reprojection residuals and normal vector aligned residuals.

[0089] 2. Constructing the incremental equation: Constructing a linear equation to solve for the parameter increment: in, It is an approximation of the Hessian matrix of the objective function at the current parameters; It is the damping factor, used to control the update step size; It is an identity matrix.

[0090] 3. Solve and test updates: Solve the above linear equations to obtain candidate parameter increments. Calculate the trial parameters. ,in This represents the update operation on Lie algebras.

[0091] 4. Evaluate and decide on updates: Calculate the objective function value under the trial parameters. .

[0092] if This indicates that the update is valid. Then accept the update: Let And reduce the damping factor, for example, by making This is so that we can make bolder iterations in the future.

[0093] if This indicates that the update may be invalid. Therefore, reject the update: And increase the damping factor, for example, by making This is to allow for more conservative iterations in the future.

[0094] Step S504: Check if the iteration meets the termination condition set in step S502. If the current iteration number... Less than And the change in the objective function value Greater than or equal to If the condition is met, return to step S503 and continue with the next iteration. .

[0095] Step S505: When the iterative process terminates due to meeting the convergence condition or reaching the maximum number of iterations, the final iteration parameters are... Convert back to its corresponding rigid body transformation matrix form Specifically, Lie algebras are mapped through exponential mapping. Mapping to special Euclidean group This yields the final rotation matrix. Translation vector Thus, the registration transformation matrix is ​​formed. .

[0096] In summary, step S5 transforms the cross-modal registration problem into a standard nonlinear least-squares optimization problem through the objective function. This objective function simultaneously considers the visual projection accuracy of feature points and the consistency of surface geometry, and is solved efficiently using the mature Levenberg-Marquardt algorithm. This step fully utilizes the reliable matching and initial values ​​provided in the preceding steps, ultimately outputting a high-precision spatial transformation matrix that achieves optimal geometric and visual alignment between the oblique photogrammetric image and the laser point cloud, providing the final solution for completing the entire registration process.

[0097] Step S6: Performing Spatial Alignment. Step S6 is the final execution step of this method. It applies the optimal spatial transformation relationship obtained in the previous steps to the original laser point cloud data, thereby achieving accurate alignment with the oblique photogrammetry image dataset in a unified coordinate system. This step is the output of the entire technical process. It does not involve complex feature extraction or optimization calculations, but rather transforms the theoretical calculation results into standardized spatial data that can be directly used for subsequent 3D applications.

[0098] Step S601: Receive and load the final registration transformation matrix output by the nonlinear fine-tuning registration process in step S5. This matrix is ​​a 4x4 homogeneous transformation matrix, with the following form: in, It is a 3x3 orthogonal rotation matrix, representing the optimal rotation components; It is a 3x1 translation vector representing the optimal translation components. This matrix encapsulates all the spatial transformation parameters required to transform the laser point cloud from its original coordinate system to one aligned with the oblique photogrammetric image coordinate system.

[0099] Step S602: Traverse every 3D point in the laser point cloud dataset For each point, the transformation matrix obtained in step S601 is used. Calculate its coordinates in the new coordinate system Coordinate transformation is accomplished through matrix multiplication, and the specific calculation formula is as follows: Or, equivalently, expressed in homogeneous coordinates as: This operation will click The coordinates are rigidly transformed from the original lidar coordinate system to the target coordinate system aligned with the oblique photogrammetry image coordinate system.

[0100] Step S603: Transform all the points obtained The original oblique photogrammetry image dataset (containing image files and their exterior orientation elements) is reorganized to form a new, registered laser point cloud dataset. Simultaneously, the original dataset is logically associated with and stored with this new point cloud dataset using a unified coordinate system reference. Finally, a spatially precisely aligned oblique photogrammetry-laser point cloud fusion dataset is output.

[0101] In summary, step S6 constitutes a clear and executable data output process. It receives the core result of the complex calculations preceding the registration transformation matrix and applies it to the original data through direct geometric transformations, ultimately producing a standard spatial data product that can directly serve downstream applications such as realistic 3D modeling, high-precision surveying, texture mapping, and change detection. Thus, the fully automated, control-point-free registration process described in this invention, from data preparation, feature extraction, matching estimation to optimized alignment, is complete.

[0102] The implementation of the above method completely eliminates the reliance on manual control points and establishes a reliable association between images and point clouds through multi-level feature extraction and cross-modal semantic alignment. Initial pose estimation based on random sampling consistency effectively suppresses the influence of noisy matching, ensuring robustness. Dual-constraint nonlinear optimization improves registration accuracy from both geometric and visual dimensions, ultimately achieving sub-pixel-level alignment.

[0103] On the other hand, the present invention also provides an adaptive registration system for oblique photography and laser point clouds, which includes a data acquisition module, an image feature extraction module, a point cloud feature extraction module, a cross-modal matching module, an initial pose estimation module, a fine registration optimization module, and a spatial alignment execution module.

[0104] The data acquisition module is configured to receive and store oblique photogrammetry datasets and laser point cloud datasets synchronously collected by the flight platform, ensuring data timestamp alignment and metadata integrity.

[0105] The image feature extraction module is configured to perform multi-scale edge detection and superpixel plane fitting on oblique photogrammetric image datasets, generating image structure feature maps containing line segments and planar regions. This module integrates a Gaussian pyramid construction unit, a Canny edge detection unit, a Hough transform line extraction unit, a superpixel segmentation unit, and a plane fitting unit.

[0106] The point cloud feature extraction module is configured to perform neighborhood construction, principal component analysis, normal vector calculation, and curvature classification on the laser point cloud dataset to generate a point cloud geometric semantic feature body. This module includes a sphere neighborhood search unit, a covariance matrix calculation unit, an eigenvalue decomposition unit, and a curvature classification unit.

[0107] The cross-modal matching module is configured to receive image structural feature maps and point cloud geometric semantic feature volumes, perform line-edge and plane-plane cross-modal matching, and output a cross-modal feature matching candidate set. This module integrates a 3D projection unit, an orientation and position deviation calculation unit, a normal vector angle and distance offset calculation unit, and a matching filtering unit.

[0108] The initial pose estimation module is configured to randomly sample from a cross-modal feature matching candidate set, solve for the initial pose assumption, and filter the optimal initial pose by interior point counting. This module includes a random sampling unit, a rigid body transformation solution unit, and an interior point evaluation unit.

[0109] The fine registration optimization module is configured to use the optimal initial pose as the initial value, perform nonlinear optimization based on dual constraints of projection consistency and surface continuity, and output the final registration transformation matrix. This module integrates an objective function construction unit, a Jacobian matrix calculation unit, and a Levenberg-Marquardt optimization engine.

[0110] The spatial alignment execution module is configured to apply the registration transformation matrix to the original laser point cloud dataset and output spatially aligned point cloud data. This module includes a point cloud coordinate transformation unit and a data output interface.

[0111] The modules communicate with each other through standardized data interfaces, and the data flow strictly follows the sequence of method steps to ensure the continuity and traceability of the processing. The system as a whole supports multi-threaded parallel processing to meet the data throughput requirements of large-scale scenarios.

[0112] In summary, this embodiment discloses in detail the specific implementation details of the method and system of the present invention, covering the complete technical chain from data input, feature extraction, cross-modal matching, initial pose estimation, fine optimization to final alignment, fully meeting the requirements of the Patent Law for full disclosure, and providing solid technical support for the scope of protection of the claims.

[0113] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects.

[0114] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. An adaptive registration method for oblique photography and laser point clouds, characterized in that, include: Obtain oblique photogrammetry image dataset and laser point cloud dataset; Structural features are extracted from the oblique photogrammetry image dataset to obtain an image structural feature map, and geometric semantic features are extracted from the laser point cloud dataset to obtain a point cloud geometric semantic feature body. Based on the image structural feature map and the point cloud geometric semantic feature body, line-edge matching and plane-plane matching are performed to construct a cross-modal feature matching candidate set; Based on the cross-modal feature matching candidate set, a robust pose estimation algorithm is used, wherein the randomly sampled minimum sample set contains at least one plane-to-plane matching pair to select the optimal initial pose. Starting from the optimal initial pose, a nonlinear optimization based on dual constraints of projection consistency and surface continuity is performed to minimize the objective function, which is a weighted sum including reprojection error term and normal vector alignment error term, and the final registration transformation matrix is ​​obtained by solving. The registration transformation matrix is ​​applied to the laser point cloud dataset to achieve spatial alignment with the oblique photogrammetric image dataset.

2. The method according to claim 1, characterized in that, Structural features were extracted from the oblique photogrammetry image dataset to obtain image structural feature maps, including: Construct a multi-scale image pyramid of the oblique photographic images; Edge detection and line segment extraction are performed at each scale to obtain multi-scale line features; Superpixel segmentation is performed on the original resolution image, and plane fitting is performed on each superpixel region in the gray value space to obtain superpixel plane features, i.e., the image plane. The image structure feature map is composed of the multi-scale linear features and the superpixel planar features.

3. The method according to claim 1 or 2, characterized in that, Geometric semantic features are extracted from the laser point cloud dataset to obtain the point cloud geometric semantic feature body, including: Calculate the local neighborhood normal vector and curvature value of each point in the point cloud; Based on the curvature value, the points are semantically classified, and the classification includes at least planar points, edge points, and corner points; For points classified as planar points, a region growing algorithm is applied to aggregate them into continuous planar entities; The point cloud geometric semantic feature body includes the semantic classification information of points and the planar entity formed by aggregation.

4. The method according to claim 3, characterized in that, Based on the image structural feature map and point cloud geometric semantic feature volume, line-edge matching and plane-plane matching are performed, including: For line-edge matching: the angle between the direction vector of the image line in three-dimensional space and the direction vector of the point cloud edge line segment is calculated as the direction deviation, and the average distance from the point on the point cloud edge line segment to the image three-dimensional line is calculated as the position deviation; when the direction deviation is less than the first angle threshold and the position deviation is less than the first distance threshold, it is determined to be a valid matching pair; For plane-to-plane matching: calculate the angle between the normal vectors of the image plane and the point cloud plane entities, as well as the distance offset of the center point along the normal direction; calculate a joint metric based on the angle between the normal vectors and the distance offset; when the joint metric is less than a preset threshold, it is determined to be a valid matching pair; The first angle threshold is 15 degrees, and the first distance threshold is 10 centimeters.

5. The method according to claim 1, characterized in that, Robust pose estimation is performed using a random sampling consensus algorithm, including: Multiple random samplings are performed from the cross-modal feature matching candidate set, with each sampling extracting at least three non-collinear feature matching pairs and at least one plane-to-plane matching pair to form the minimum sample set; For each of the minimum sample sets, an initial rigid body transformation matrix is ​​calculated by solving the Procrassa problem; Each of the initial rigid body transformation matrices is applied to the entire cross-modal feature matching candidate set, and the number of interior points that satisfy the preset tolerance is counted. The initial rigid body transformation matrix with the largest number of interior points is used as the optimal initial pose output.

6. The method according to claim 1, characterized in that, Nonlinear optimization based on dual constraints of projection consistency and surface continuity includes: Define the objective function ,in, : indicates the first feature selected from the point cloud features A three-dimensional point, : Indicates the rigid body transformation to be optimized. ,in It is a rotation matrix. It is a translation vector. : Represents the perspective projection function; : Represents a point The corresponding image features in the first Two-dimensional pixel coordinates on a tilted photographic image : The total number of point pairs involved in the reprojection error calculation; The objective function is iteratively minimized in the Lie algebra space using the Levenberg-Marquardt algorithm, with the optimization variable being the Lie algebra parameter corresponding to the optimal initial pose. The weighting coefficient α is set to 1.0 and β is set to 0.

5.

7. The method according to claim 1, characterized in that, Obtain the oblique photogrammetry image dataset and the laser point cloud dataset, including: Acquire oblique photographic images with initial exterior orientation elements, captured by a multi-view camera array; Acquire laser point clouds with spatial resolution down to the centimeter level and a point density of no less than 50 points per square meter; A unified time synchronization system ensures that the timestamps of the oblique photographic images are aligned with those of the laser point cloud.

8. An adaptive registration system for oblique photography and laser point clouds, characterized in that, The system for implementing the method of any one of claims 1 to 7 comprises: The data acquisition module is used to acquire and store oblique photogrammetry image datasets and laser point cloud datasets; The image feature extraction module is used to extract structural features from oblique photogrammetric image datasets and generate image structural feature maps. The point cloud feature extraction module is used to extract geometric and semantic features from the laser point cloud dataset and generate a point cloud geometric and semantic feature body. A cross-modal matching module is used to construct a cross-modal feature matching candidate set based on the image structural feature map and the point cloud geometric semantic feature volume; The initial pose estimation module is used to select the optimal initial pose based on the cross-modal feature matching candidate set using a random sampling consensus algorithm. Its random sampling strategy forces the inclusion of at least one plane-to-plane matching pair. A fine-grained registration optimization module is used to perform nonlinear optimization based on dual constraints of projection consistency and surface continuity, starting from the optimal initial pose, and outputs the final registration transformation matrix; and The spatial alignment execution module is used to apply the registration transformation matrix to the laser point cloud dataset and output the spatially aligned point cloud data.

Images