Urban aerial image and LiDAR point cloud registration method using multi-element features

Abstract

The application discloses a kind of urban aerial image and LiDAR point cloud registration method using multiple features, belong to photogrammetry and remote sensing field, including: image pre-processing generates sparse feature point cloud and three-dimensional feature line, adopts deep learning method to extract image building roof profile, and utilizes RANSAC algorithm and PCA statistical analysis method to extract the feature line and feature surface of LiDAR point cloud;By the ICL algorithm based on line and surface features, the sparse feature point cloud generated by image and LiDAR point cloud are globally roughly registered;Joint LiDAR point cloud point, line, surface multiple control information constraint is carried out image area network adjustment solution, and local distortion of image area network is inhibited;Global rough registration and local distortion optimization are iterated, until the change amount of image orientation error meets the preset requirement.The application is by joint point, line, surface feature constitutes multiple constraint, and using adaptive weight strategy, can preferably solve the high-precision robust registration problem of image and LiDAR point cloud in urban scene.

Description

Technical Field

[0001] This invention belongs to the field of photogrammetry and computer vision, specifically relating to a method for registering urban aerial images with LiDAR point clouds using multi-feature features. The key technology of this invention is the high-precision automatic registration of aerial images with LiDAR point clouds that combines multi-feature control information of urban scene points, lines, and surfaces. Background Technology

[0002] With the accelerating pace of urbanization, the demand for detailed 3D spatial information of cities is growing rapidly. Aerial photogrammetry, as a crucial traditional means of acquiring 3D spatial information, has long relied heavily on high-precision field control points to achieve accurate geometric orientation. However, the high cost, long cycle, and low level of automation of field control point acquisition have become major bottlenecks restricting the efficiency and intelligent development of aerial image processing.

[0003] The rapid development and widespread application of LiDAR technology have provided new opportunities for high-precision geospatial information in cities. LiDAR can directly acquire high-precision, high-density 3D point cloud data, with spatial positioning accuracy typically within 10 centimeters, effectively replacing traditional field control points as a source of control information. However, due to differences in sensor modes and acquisition time, there are significant modal differences and spatial deviations between aerial imagery and LiDAR point clouds. Furthermore, complex urban scenes are rich in geometric features, and while traditional single-type feature registration methods (such as ICP and ICL) can be used in the coarse registration stage, their stability and robustness are insufficient in complex scenes, especially in the effective utilization of surface features.

[0004] Existing LiDAR point cloud-controlled aerial photogrammetry methods mostly focus on point feature or preliminary line feature registration, neglecting the more stable surface features in urban environments, especially the matching of building roof outline features. Furthermore, current technologies do not adequately utilize artificial intelligence, particularly deep learning methods, in feature extraction and registration, resulting in low automation and efficiency in the feature extraction process. Therefore, how to integrate advanced artificial intelligence methods to mine and effectively utilize line and surface features, tailored to the characteristics of complex urban scenes, to achieve precise, efficient, and robust automatic registration of aerial imagery and LiDAR point clouds has become a crucial problem that urgently needs to be solved. Summary of the Invention

[0005] To address the issues of insufficient geometric registration accuracy and poor robustness of single features in existing technologies for urban aerial imagery and LiDAR point clouds, this invention aims to provide a high-precision registration method for urban aerial imagery and LiDAR point clouds that integrates multi-feature and artificial intelligence technologies, specifically designed for complex urban scenes. The method emphasizes the fine matching process of line and surface features. First, deep learning and statistical analysis methods are used to extract rich point, line, and surface features from the imagery and LiDAR point clouds, respectively. Then, a global coarse registration of the imagery point cloud and LiDAR point cloud is achieved through an Iterative Nearest Neighbor (ICL) algorithm based on line and surface constraints. Furthermore, multi-feature features of the urban scene are combined, and local distortion is eliminated through regional network adjustment, thereby significantly improving the registration accuracy and robustness.

[0006] This invention is mainly achieved through the following technical solutions:

[0007] A method for registering urban aerial imagery with LiDAR point clouds using multiple features includes the following steps:

[0008] Step 1: Preprocess the image to generate sparse feature point cloud and image 3D feature lines, and use an improved deep learning model to extract the building roof outline information of the image.

[0009] Step 2: Line and surface feature information are extracted from the LiDAR point cloud using the random sampling consensus algorithm and the statistical feature analysis method of principal component analysis.

[0010] Step 3: Using the improved ICL algorithm, global coarse registration is performed by using the correspondence between the image 3D feature lines and the LiDAR 3D feature lines, and by adding the projection constraints between the image roof outline and the LiDAR building plane.

[0011] Step 4: Based on Step 3, using the point, line, and surface features extracted from the urban scene as joint control information, an extended energy equation is constructed to perform image area network adjustment optimization, correcting local deformation caused by camera distortion and uneven image connectivity.

[0012] Step 5: Iterate through steps 3 and 4 multiple times until the change in image positioning error meets the preset requirements.

[0013] Optionally, in step 1, the preprocessing of the image to generate a sparse feature point cloud and a three-dimensional feature line of the image includes: using the positioning and orientation information obtained by the POS system to perform preliminary calculation of the internal and external orientation elements of the image, and combining the feature matching algorithm to generate a sparse feature point cloud and a three-dimensional feature line of the image, wherein the sparse feature point cloud is generated by the matching results of the same feature points between images, and the three-dimensional feature line is obtained by matching the same line segments between images and performing three-dimensional reconstruction.

[0014] The method of extracting building roof outline information from images using an improved deep learning model includes: using a deep convolutional neural network model based on U-Net, in which the encoder uses ResNet-50 to replace the traditional convolutional module to enhance feature expression capabilities, and the decoder introduces an attention mechanism to focus on building edge details and reduce background interference.

[0015] The computational process of the attention mechanism is represented as follows:

[0016]

[0017] in, Indicates attention weights, For the high-level features transmitted by the encoder, The low-level features transmitted by the decoder. , For learnable convolutional kernel weights, For bias, Use the Sigmoid activation function;

[0018] The binary cross-entropy loss function is used during training:

[0019]

[0020] in, For real labels, The predicted label output by the network. Given the total number of samples, by minimizing this loss function, the network model can effectively learn and stably output high-precision building outlines.

[0021] Optionally, in step 2, the statistical feature analysis method using random sampling consensus algorithm and principal component analysis to extract line and surface feature information from the LiDAR point cloud includes:

[0022] The RANSAC algorithm is used to initially fit the linear structure in point cloud data. Specifically, this involves: randomly selecting a minimum number of points from the LiDAR point cloud to fit an initial line segment equation; then, defining a distance threshold, considering all points less than the threshold as support interiors of the model; iteratively executing this process until the number of interiors reaches its maximum, thereby determining the optimal linear model. The specific three-dimensional line equation is expressed as follows:

[0023]

[0024] In the formula, Let be the coordinates of a point on the line, and let the direction vector be . ;

[0025] After obtaining the optimal parameters of the line model using the RANSAC algorithm, the PCA method is used to further analyze the spatial distribution characteristics of the interior point set and determine the most stable 3D feature line.

[0026]

[0027] In the formula, Represents the coordinates of points inside the line feature. Denotes the centroid of an interior point. Number of interior points; matrix The principal eigenvector corresponds to the largest eigenvalue, and its direction is the most stable direction of the line segment.

[0028] Three non-collinear points are randomly selected to construct an initial plane equation. The plane equation is then optimized iteratively to ensure that more points satisfy the condition that their distance from the plane is less than a given threshold. The expression for the plane equation is:

[0029]

[0030] in, Let be the normal vector of the plane. For constant terms;

[0031] In each iteration, the perpendicular distance from the point to the plane is calculated:

[0032]

[0033] in, Let be the coordinates of any point in the LiDAR point cloud. The goal of iterative optimization is to minimize the negative value of the number of distance points within the threshold, so as to obtain the optimal planar model with the most interior points.

[0034] Optionally, in step 3, the improved ICL algorithm is used to perform global coarse registration by using the correspondence between the image's three-dimensional feature lines and the LiDAR's three-dimensional feature lines, and by adding projection constraints between the image's roof outline and the LiDAR building plane. This includes:

[0035] The three-dimensional feature lines of the image extracted in step 1 are matched with the three-dimensional feature lines of LiDAR extracted in step 2 to establish an initial correspondence between the two sets of data. Then, the rigid transformation parameters are solved iteratively using the ICL algorithm. The core objective function of the improved ICL algorithm is to minimize the matching error between the three-dimensional feature lines.

[0036] The specific formula is described below:

[0037] Define the image feature line set as The set of LiDAR feature lines is { The feature lines of the two sets have a preliminary correspondence, and the optimization objective function of the ICL algorithm is expressed as:

[0038]

[0039] In the formula, and These represent the three-dimensional feature lines corresponding to the image and LiDAR, respectively. Represents the rotation matrix. Represents the translation vector. Indicates the number of corresponding feature line pairs;

[0040] The optimal solution is obtained by iteratively solving the above equations. and Rigid registration of feature lines between the image and the LiDAR point cloud was completed.

[0041] Image contour features The planar features of buildings corresponding to LiDAR The transformed matrix obtained so far is projected onto the spatial plane containing the image contour, forming the projected contour line. This leads to the surface profile projection error constraint function:

[0042]

[0043] in, This indicates the number of matching pairs between the image contour and the LiDAR surface features.

[0044] Furthermore, by combining the aforementioned line and surface constraints and integrating line and surface feature constraints into a unified optimization framework, the final overall objective function is expressed as:

[0045]

[0046] In the formula, the weighting factor It is used to adjust the relative strength of surface feature constraints and line feature constraints.

[0047] Optionally, in step 4, the step of using the point, line, and area features extracted from the urban scene as joint control information to construct an extended energy equation for image area network adjustment optimization includes the following specific implementation of point feature constraints:

[0048] The principle of closest point-to-surface distance is used to select the corresponding "control point" in the LiDAR point cloud for each sparse feature point in the image. For any sparse feature point... Search for the LiDAR point cloud within a spherical neighborhood of radius r, and record the set of nearest neighbor point clouds as follows: , For the first The number of LiDAR point clouds within the neighborhood of each sparse feature point; based on the point cloud set The space plane is obtained by fitting all points within the space using the least squares method. , with sparse feature points to plane The perpendicular point is used as the corresponding "control point" in the LiDAR point cloud; geometric control constraints of point features. It can be represented as follows

[0049]

[0050] In the formula, For prior weights, This is a function for calculating the distance between a point and a surface in space. The average ground resolution of the image is used to... Normalized to pixel units.

[0051] Furthermore, the specific implementation of the line feature constraint in step 4 includes:

[0052] The principle of closest point-to-line distance is used to select the corresponding "control point" in the LiDAR point cloud for each sparse feature point in the image; let the three-dimensional feature lines of the image and LiDAR point cloud extracted in steps 1 and 2 be respectively... and After projecting them onto the horizontal plane, they are respectively denoted as and , The point on is denoted as (t=1,2,…,n), where n is The number of points on the image feature line; the planar projection points on the image feature line. to the straight line The perpendicular point is used as the corresponding "control point" in the LiDAR point cloud;

[0053] Geometric control constraints of line features It is expressed as follows:

[0054]

[0055] In the formula, It is a planar projection point on the image feature line. To LiDAR projection line The shortest Euclidean distance, It is the average ground resolution of the image. It is the first Weights corresponding to the line features.

[0056] Furthermore, the specific implementation of the surface feature constraints in step 4 includes:

[0057] Let the images extracted in steps 1 and 2 and the building roofs in the LiDAR point cloud be respectively... and ,Will After being projected onto the horizontal plane, it is denoted as For a plane The outline edge is marked with a positioning point at regular intervals of a certain number of pixels. (s=1,2,…,u), where u is the number of positioning points on the contour edge; with positioning points to plane The nearest edge contour line The perpendicular point is used as the corresponding "control point" in the LiDAR point cloud for this positioning point;

[0058] Geometric control constraints of surface features Represented as:

[0059]

[0060] In the formula, This represents the image contour points within the horizontal plane. to the edge segment of the LiDAR projection profile The shortest Euclidean distance, It is the average ground resolution of the image. It is the first Weights of individual facet features.

[0061] Furthermore, in step 4, the energy equation for the classic regional network adjustment is extended to construct the energy equation shown in equation (13):

[0062]

[0063] In the formula, The image point residual term defined for classical regional network adjustment. For the first Orientation parameters of the image, For its in the Image point observation on Zhang's image, The projection function defined for the collinearity condition equation; Geometric constraints are formed by combining point, line, and surface feature information.

[0064] Furthermore, appropriate prior weights are assigned to the geometric constraints formed by the joint point, line, and surface feature information to maximize the registration accuracy. Specific implementations include:

[0065] For point and surface feature constraints, a fitted plane is used. and The local coplanarity of the LiDAR point set at time is adaptively used to determine the prior weights of the LiDAR control information, i.e., in equations (10) and (12). and The spatial plane fitting error is used as an indicator of coplanarity.

[0066] For linear feature constraints, use the fitted line The local collinearity of the LiDAR point set at time is adaptively determined to determine the prior weights of the LiDAR control information, i.e., in equation (11). The linear fitting error is used as an indicator of collinearity.

[0067] When the fitting error of the spatial plane and line is small, a larger prior weight is assigned to the corresponding LiDAR control information; otherwise, a smaller prior weight is assigned.

[0068] The present invention has the following advantages: (1) By combining point, line and surface features to form multi-dimensional constraints, the problem of high-precision and robust registration between images and LiDAR point clouds can be solved, thereby realizing high-precision geometric orientation of aerial images of urban scenes under the condition of no field control points; (2) Adaptive weighting strategy can solve the problem of inconsistent accuracy and reliability of multi-dimensional control information, thereby maximizing the registration accuracy. Attached Figure Description

[0069] Figure 1 This is a flowchart of the method of the present invention;

[0070] Figure 2 This is a schematic diagram of the point feature constraint of the present invention;

[0071] Figure 3 This is a schematic diagram of the line feature constraint of the present invention;

[0072] Figure 4 This is a schematic diagram of the surface feature constraint of the present invention;

[0073] Figure 5 These are schematic diagrams illustrating different types of sparse image feature points according to the present invention;

[0074] Figure 6 This is a schematic diagram showing the registration of aerial images with LiDAR point clouds before and after the present invention. In the figure:

[0075] (a) Typical region 1 before registration, (b) Typical region 1 after registration;

[0076] (c) Typical region 2 before registration; (d) Typical region 2 after registration;

[0077] (e) Typical region 3 before registration, (f) Typical region 3 after registration. Detailed Implementation

[0078] The technical solution of the present invention will be further described in detail below through embodiments and in conjunction with the accompanying drawings.

[0079] This invention provides a method for registering urban aerial imagery with LiDAR point clouds using multi-dimensional features. First, it employs artificial intelligence and statistical analysis methods to extract rich geometric features (points, lines, and surfaces) from both aerial imagery and LiDAR point clouds. Second, it uses an ICL algorithm based on line and surface features to achieve efficient coarse registration between the aerial imagery feature point cloud and the LiDAR point cloud. Then, it constructs an extended regional network adjustment energy equation using the extracted multi-dimensional features, and eliminates local distortion by combining point, line, and surface control information, ultimately achieving high-precision and robust registration between the imagery and the point cloud. The overall technical approach is attached. Figure 1 As shown, its core process includes the following steps:

[0080] Step 1: Preprocess the image to generate sparse feature point cloud and three-dimensional feature lines of the image, and use an improved deep learning model to extract the roof outline information of buildings in the image.

[0081] In step 1, the present invention first employs POS-assisted aerial triangulation technology to perform geometric preprocessing on the original aerial imagery. This process utilizes the positioning and attitude information acquired by the POS system to perform preliminary calculations of the image's interior and exterior orientation elements, and combines this with a feature matching algorithm to generate a sparse feature point cloud and three-dimensional feature lines from the imagery. The sparse feature point cloud is generated from the matching results of corresponding feature points between images, while the three-dimensional feature lines are obtained through matching corresponding line segments between images and performing three-dimensional reconstruction. This provides the initial geometric basis for the subsequent registration process.

[0082] Specifically, for any two-dimensional feature points on an image Its corresponding three-dimensional spatial coordinates The collinearity equations of classical photogrammetry are solved to establish the geometric correspondence between two-dimensional points in the image and three-dimensional points in the object, so as to obtain the preliminary three-dimensional geometric information of the image.

[0083] Secondly, to accurately extract the unique rooftop contour features of buildings in urban scenes, this invention utilizes deep learning methods for image semantic segmentation. Specifically, it employs a deep convolutional neural network model based on an improved U-Net. In the network structure, the encoder uses ResNet-50 to replace the traditional convolutional module to enhance feature representation capabilities, and the decoder introduces an attention gate to focus on building edge details and reduce background interference, thereby effectively improving the automatic extraction accuracy of urban building contours.

[0084] During the forward propagation of the network, the feature map The computational process of spatial attention mechanism can be represented as follows:

[0085]

[0086] in, Indicates attention weights, For the high-level features transmitted by the encoder, The low-level features transmitted by the decoder. , For learnable convolutional kernel weights, For bias, This is the Sigmoid activation function. The attention mechanism significantly enhances the feature response of the target region (building roof outline) and weakens the features of non-target regions through this formula, thus significantly improving the contour extraction performance of the network.

[0087] The data required for training the network comes from the publicly available standard dataset ISPRS and the labeled dataset built by our research team. The binary cross-entropy (BCE) loss function is used during training.

[0088]

[0089] in, For real labels, The predicted label output by the network. Given the total number of samples, by minimizing this loss function, the network model can effectively learn and stably output high-precision building outlines.

[0090] Step 2: The statistical feature analysis methods of Random Sampling Consensus Algorithm (RANSAC) and Principal Component Analysis (PCA) are used to automatically extract line and surface feature information from the LiDAR point cloud;

[0091] In step 2, in order to achieve high-precision automated registration between the image and the LiDAR point cloud, this invention uses the RANSAC algorithm and PCA analysis method to automatically and efficiently extract line features and surface features (especially building planar features) from the LiDAR point cloud data.

[0092] First, during the extraction of 3D feature lines, the RANSAC algorithm is used to initially fit the linear structure in the point cloud data. Specifically, this involves randomly selecting a minimum number of points from the LiDAR point cloud to fit an initial line segment equation. Then, by defining a distance threshold (e.g., 15 cm if the image ground resolution is 10 cm), all points with a distance less than the threshold are considered support inliers of the model. This process is iteratively executed until the number of inliers reaches its maximum, thus determining the optimal linear model. The specific 3D line equation is expressed as:

[0093]

[0094] In the formula, Let be the coordinates of a point on the line, and let the direction vector be . After optimizing the line model using the RANSAC algorithm, the spatial distribution characteristics of the interior point set are further analyzed using the PCA method to determine the most stable three-dimensional feature line. Specifically, the eigenvalues ​​and eigenvectors of the covariance matrix of the interior point set are solved using PCA:

[0095]

[0096] In the formula, Represents the coordinates of points inside the line feature. Denotes the centroid of an interior point. The number of interior points. Matrix The principal eigenvector corresponds to the largest eigenvalue, and its direction is the most stable direction of the line segment.

[0097] Secondly, this invention employs a similar RANSAC strategy to extract planar features from LiDAR point clouds, particularly for salient planar regions such as rooftops of urban buildings. Specifically, three non-collinear points are randomly selected to construct an initial plane equation, which is then iteratively optimized to ensure that more points satisfy the condition that their distance from the plane is less than a given threshold. The expression for the plane equation is:

[0098]

[0099] in, Let be the normal vector of the plane. This is a constant term. In each iteration, the perpendicular distance from the point to the plane is calculated:

[0100]

[0101] in, Let be the coordinates of any point in the LiDAR point cloud. The goal of iterative optimization is to minimize the negative value of the number of distance points within a certain threshold, thereby obtaining the optimal planar model with the most interior points. The planar parameters are thus determined, and a large number of stable planar features, such as building rooftops, in the LiDAR point cloud are automatically and efficiently extracted.

[0102] Step 3: The improved ICL algorithm is adopted, which utilizes the correspondence between the three-dimensional feature lines of the image and the three-dimensional feature lines of LiDAR, and adds the projection constraint between the roof outline of the image and the LiDAR building plane to achieve fast and robust global coarse registration.

[0103] In step 3, this invention employs an improved ICL algorithm, which determines the optimal rigid transformation matrix between the two sets of data by matching the 3D feature lines of the point cloud with the 3D feature lines of the image. Specifically, firstly, the 3D feature lines of the image extracted in step 1 are matched with the 3D feature lines of LiDAR extracted in step 2 to establish an initial correspondence between the two sets of data. Then, the ICL algorithm is used to iteratively solve for the rigid transformation parameters (rotation matrix). Translation vector The core objective function of the improved ICL algorithm is to minimize the matching error between 3D feature lines, and the specific formula is described as follows:

[0104] Define the image feature line set as The set of LiDAR feature lines is { The feature lines of the two sets have a preliminary correspondence, and the optimization objective function of the ICL algorithm is expressed as:

[0105]

[0106] In the formula, and These represent the three-dimensional feature lines corresponding to the image and LiDAR, respectively. Represents the rotation matrix. Represents the translation vector. This represents the number of corresponding line feature pairs. The optimal solution is obtained by iteratively solving the above equation. and This completes the rigid registration of feature lines between the image and the LiDAR point cloud.

[0107] Secondly, this invention further introduces surface feature constraints to improve the stability of the registration results. Specifically, it utilizes the image building contour features extracted based on deep learning in step 1, combined with the LiDAR building planar features extracted in step 2, to perform contour projection constraint matching. The specific implementation method is as follows:

[0108] Image contour features The planar features of buildings corresponding to LiDAR The transformed matrix obtained so far is projected onto the spatial plane containing the image contour, forming the projected contour line. This leads to the surface profile projection error constraint function:

[0109]

[0110] in, This represents the number of matching pairs between the image contour and the LiDAR surface features. This formula further eliminates registration bias caused by modal differences and systematic errors, improving the algorithm's adaptability and stability in complex urban environments.

[0111] Combining the above-mentioned line and surface constraints, this invention integrates line and surface feature constraints into a unified optimization framework, and the final overall objective function is expressed as:

[0112]

[0113] In the formula, the weighting factor It is used to adjust the relative strength of surface feature constraints and feature line constraints to ensure that the coarse registration process can obtain stable and robust results in different urban scenarios.

[0114] Step 4: Using the point, line, and surface features extracted from the urban scene as joint control information, an extended energy equation is constructed to perform image area network adjustment optimization, correcting local deformation caused by camera distortion and uneven image connectivity.

[0115] In step 4, the image region network is adjusted by combining the multi-dimensional control information of point, line and surface features of the urban scene, and the camera distortion parameters, image interior orientation elements and exterior orientation elements are calculated. This suppresses the local deformation and distortion of the image region network caused by factors such as camera distortion, inconsistent image connectivity strength and the resulting error accumulation, and obtains more reliable registration results.

[0116] The process of point feature constraint is as follows: (see attached image) Figure 2 As shown, the principle of point-to-plane proximity is used to select the corresponding "control point" in the LiDAR point cloud for each sparse feature point in the image. For any sparse feature point... Search for the LiDAR point cloud within a spherical neighborhood of radius r, and record the set of nearest neighbor point clouds as follows: , For the first The number of LiDAR point clouds within the neighborhood of each sparse feature point. Based on the point cloud set. The space plane is obtained by fitting all points within the space using the least squares method. , with sparse feature points to plane The perpendicular point is used as the corresponding "control point" in the LiDAR point cloud.

[0117] Unlike field control points, sparse feature points in imagery do not have a strict physical correspondence with LiDAR control information. Therefore, they can only be used as geometric constraints, not as control points in the regional network adjustment solution. Geometric control constraints of point features. It can be represented as follows:

[0118]

[0119] In the formula, For prior weights, This is a function for calculating the distance between a point and a surface in space. The average ground resolution of the image is used to... Normalized to pixel units.

[0120] The process of line feature constraint is as follows: (see attached) Figure 3 As shown, the point-to-line nearest neighbor principle is used to select the corresponding "control point" in the LiDAR point cloud for each sparse feature point in the image. Let the three-dimensional feature lines of the image and LiDAR point cloud extracted in steps 1 and 2 be respectively... and After projecting them onto the horizontal plane (to solve planar registration), they are respectively denoted as... and , The point on is denoted as (t=1,2,…,n), where n is The number of points on the image feature line. (This refers to the planar projection points on the image feature line.) to the straight line The perpendicular point is used as the corresponding "control point" in the LiDAR point cloud.

[0121] Similarly, there is no strict physical correspondence between the 3D feature lines of the image and the feature lines of the LiDAR point cloud; the line feature control information can only be used as a geometric constraint. Geometric control constraints of line features. It can be represented as follows:

[0122]

[0123] In equation (11), It is a planar projection point on the image feature line. To LiDAR projection line The shortest Euclidean distance, It is the average ground resolution of the image (used for distance normalization), while Then it is the first Weights corresponding to the line features.

[0124] The process of constraining surface features is as follows: (see attached image) Figure 4 As shown, let the images extracted in steps 1 and 2 and the building roofs in the LiDAR point cloud be respectively... and ,Will After being projected onto the horizontal plane, it is denoted as For a plane The outline edge is marked with a positioning point at regular intervals of a certain number of pixels. (s=1,2,…,u), where u is the number of positioning points on the contour edge (including corner points). Positioning points... to plane The nearest edge contour line The perpendicular point is used as the corresponding "control point" in the LiDAR point cloud.

[0125] Similarly, there is no strict physical correspondence between the roof feature surfaces of image buildings and the feature surfaces of LiDAR point clouds; the feature control information can only be used as geometric constraints. Geometric control constraints of feature surfaces. It can be represented as follows:

[0126]

[0127] In equation (12), This represents the image contour points within the horizontal plane. to the edge segment of the LiDAR projection profile The shortest Euclidean distance, It is the average ground resolution of the image. Then it is the first Weights of individual facet features.

[0128] The energy equation for the classic regional network adjustment is extended by adding control constraints on the joint point, line and surface features, and the energy equation shown in equation (13) is constructed.

[0129]

[0130] In the formula, The image point residual term defined for classical regional network adjustment. For the first Orientation parameters of the image, For its in the Image point observation on Zhang's image, The projection function defined for the collinearity condition equation; Geometric constraints are formed by combining point, line, and surface feature information.

[0131] Sparse feature points in images originate from various ground features, such as buildings, ground, and vegetation, resulting in complex real-world scenes. The neighboring LiDAR point sets possess different local characteristics. Different errors exist when using LiDAR point clouds for line fitting and plane fitting, leading to varying accuracy of LiDAR control information. Therefore, it is necessary to assign appropriate prior weights to the LiDAR control information to maximize registration accuracy.

[0132] For point and surface feature constraints, a fitted plane is used. and The local coplanarity of the LiDAR point set at time is adaptively used to determine the prior weights of the LiDAR control information, i.e., in equations (10) and (12). and The spatial plane fitting error is used as a metric for coplanarity. For line feature constraints, the fitted line is used... The local collinearity of the LiDAR point set at time is adaptively determined to determine the prior weights of the LiDAR control information, i.e., in equation (11). The linear fitting error is used as an indicator of collinearity. (See attached image.) Figure 5 As shown, when the fitting error of the spatial plane and line is small, the corresponding LiDAR control information is assigned a larger prior weight; conversely, a smaller prior weight is assigned. In the adjustment calculation, based on the theory of posterior variance component estimation, the posterior weights of various control information are automatically estimated. At the same time, control observations that differ significantly from the prior weights are eliminated to ensure the accuracy and consistency of the regional network adjustment results.

[0133] Step 5: Repeat steps 3 and 4 iteratively until the change in image positioning error meets the set requirements (less than 0.001 pixels), and finally achieve high-precision and robust registration.

[0134] In step 5, by iteratively implementing steps 3 and 4, the registration parameters are gradually optimized until the registration accuracy stabilizes within the set threshold range, i.e., the change in image positioning error is less than 0.001 pixels, thus ensuring high accuracy and robustness of the registration.

[0135] Example 1

[0136] A specific area was selected as an example, and aerial orthophotos with a resolution of 0.05 meters and LiDAR point cloud data of 27.065 points / square meter were acquired. The final results using the registration method of this invention are shown in the attached figure. Figure 6 As shown.

[0137] Specifically designed for complex urban scenarios, this invention can highlight the fine matching process of line and surface features, and achieve high-precision, highly automated and robust registration of urban images and LiDAR data without the need for field control points.

[0138] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A method for registering urban aerial images and LiDAR point clouds using multi-element features, characterized in that, The method comprises the following steps: Step 1, pre-processing the image to generate sparse feature point cloud and image three-dimensional feature line, and using an improved deep learning model to extract the building roof contour information of the image; Step 2, using a random sample consensus algorithm and a principal component analysis statistical feature analysis method to extract line feature and surface feature information from the LiDAR point cloud; Step 3, using an improved ICL algorithm, using the correspondence relationship between the image three-dimensional feature line and the LiDAR three-dimensional feature line, and adding the projection constraint of the image roof contour and the LiDAR building plane, to perform global coarse registration; Step 4, based on step 3, using the point, line and surface multi-feature extracted from the urban scene as joint control information, constructing an extended energy equation to perform image block adjustment optimization, and correcting the local deformation caused by camera distortion and uneven image connection strength; Step 5, iteratively implementing steps 3 and 4 multiple times until the change amount of the image positioning error meets the preset requirement; In step 1, the image is pre-processed to generate sparse feature point cloud and image three-dimensional feature line, which comprises: obtaining the positioning and orientation information through the POS system to realize the preliminary calculation of the image interior and exterior orientation elements, and combining the feature matching algorithm to generate sparse feature point cloud and image three-dimensional feature line, wherein the sparse feature point cloud is generated by matching the same feature points between images, and the three-dimensional feature line is obtained by matching the same line segments between images and performing three-dimensional reconstruction; The improved deep learning model is used to extract the building roof contour information of the image, which comprises: using a deep convolutional neural network model based on the improved U-Net, replacing the traditional convolution module with ResNet-50 in the encoder to enhance the feature expression ability, and introducing an attention mechanism in the decoder to focus on the building edge details and reduce background interference; The attention mechanism calculation process is represented as: wherein, denotes an attention weight, is a high-level feature passed to the encoder, is a low-level feature passed to the decoder, , is a learnable convolution kernel weight, is a bias, is a Sigmoid activation function; A binary cross-entropy loss function is used in the training process: wherein, is the true label, is the predicted label output by the network, is the total number of samples, by minimizing the loss function, the network model effectively learns and stabilizes the output of high-precision building contours; In step 4, the point, line and surface multi-feature extracted from the urban scene is used as joint control information to construct an extended energy equation for image block adjustment optimization, wherein the specific implementation of the point feature constraint comprises: The principle of closest point-to-surface distance is used to select the corresponding "control point" in the LiDAR point cloud for each sparse feature point in the image. For any sparse feature point... Search for the LiDAR point cloud within a spherical neighborhood of radius r, and record the set of nearest neighbor point clouds as follows: , For the first The number of LiDAR point clouds within the neighborhood of each sparse feature point; based on the point cloud set The space plane is obtained by fitting all points within the space using the least squares method. , with sparse feature points to plane The perpendicular point is used as the corresponding "control point" in the LiDAR point cloud; geometric control constraints of point features. It is expressed as follows wherein is a priori weight, is a spatial point-plane distance calculation function, is the average ground resolution of the imagery, used to normalize to image units, is the average ground resolution of the imagery, used to normalize to image units; The specific implementation of the line feature constraint in step 4 comprises: The principle of closest point-to-line distance is used to select the corresponding "control point" in the LiDAR point cloud for each sparse feature point in the image; let the three-dimensional feature lines of the image and LiDAR point cloud extracted in steps 1 and 2 be respectively... and After projecting them onto the horizontal plane, they are respectively denoted as and , The point on is denoted as (t=1,2,…,n), where n is The number of points on the image feature line; the planar projection points on the image feature line. to the straight line The perpendicular point is used as the corresponding "control point" in the LiDAR point cloud; Geometric control constraints for line features is represented as follows: In the formula, is the shortest Euclidean distance from the plane projection point on the image feature line to the LiDAR projection line, is the shortest Euclidean distance from the plane projection point on the image feature line to the LiDAR projection line, is the shortest Euclidean distance from the plane projection point on the image feature line to the LiDAR projection line, is the average ground resolution of the image, is the weight corresponding to the i-th line feature, is the weight corresponding to the i-th line feature; The specific implementation of the surface feature constraint in step 4 comprises: Let the images extracted in steps 1 and 2 and the building roofs in the LiDAR point cloud be respectively... and ,Will After being projected onto the horizontal plane, it is denoted as For a plane The outline edge is marked with a positioning point at regular intervals of a certain number of pixels. Where u is the number of positioning points on the contour edge; with positioning points to plane The nearest edge contour line The perpendicular point is used as the corresponding "control point" in the LiDAR point cloud for this positioning point; Geometric control constraints of facial features is represented as: In the formula, This represents the image contour points within the horizontal plane. to the edge segment of the LiDAR projection profile The shortest Euclidean distance, It is the average ground resolution of the image. It is the first Weights of individual surface features; Appropriate prior weights are assigned to the geometric constraints formed by the joint point, line, and surface feature information to maximize registration accuracy. Specific implementation includes: For point and plane feature constraints, the local coplanarity of LiDAR point set is utilized to adaptively determine the prior weight of LiDAR control information, i.e., in equations (10) and (12) and The spatial plane fitting error is taken as an index to measure the coplanarity. and The spatial plane fitting error is taken as an index to measure the coplanarity. For line feature constraint, the local collinearity of LiDAR point set at the time of fitting line determines the prior weight of LiDAR control information, i.e., in equation (11) The straight line fitting error is taken as an index to measure collinearity. When the fitting error of the spatial plane and the line is small, the corresponding LiDAR control information is given a larger priori weight; otherwise, a smaller priori weight is given.

2. The method of claim 1, wherein, In step 2, the random sample consensus algorithm and the principal component analysis statistical feature analysis method are used to extract line feature and surface feature information from the LiDAR point cloud, which comprises: The RANSAC algorithm is used to preliminarily fit the linear structure in the point cloud data, which comprises: randomly selecting a minimum number of point sets from the LiDAR point cloud, fitting to obtain an initial line segment equation, defining a distance threshold, regarding all points with a distance less than the threshold as inliers of the model, and iteratively executing this process until the number of inliers reaches the maximum, thereby determining the optimal linear model, and the specific three-dimensional straight line equation is expressed as: In the formula, is a coordinate of a point on the line, and the direction vector is ; After the RANSAC algorithm optimizes the best parameters of the line model, the PCA method is used to further analyze the spatial distribution characteristics of the inlier set to determine the most stable three-dimensional feature line: wherein, represents the line feature inner point coordinate, represents the centroid of the inner point, is the number of inner points; matrix The principal eigenvector of corresponds to the maximum eigenvalue, and the direction is the most stable direction of the line segment. Randomly select three non-collinear points to construct an initial plane equation, and then optimize the plane equation through iteration to make more points satisfy the condition that the distance from the plane is less than a given threshold. The expression of the plane equation is: wherein is a normal vector to the plane, is a constant term; The vertical distance from the point to the plane is calculated in each iteration: wherein, For any point coordinate in LiDAR point cloud, the target of iterative optimization is to minimize the negative value of the number of points satisfying the distance within the threshold to obtain the optimal plane model with the most inliers.

3. The method of claim 1, wherein the method further comprises: In step 3, the improved ICL algorithm is used to establish the correspondence between the image three-dimensional feature line and the LiDAR three-dimensional feature line, and the projection constraint of the image roof contour and the LiDAR building plane is added to perform global coarse registration, including: The image three-dimensional feature line extracted in step 1 is matched with the LiDAR three-dimensional feature line extracted in step 2 to establish the initial correspondence between the two sets of data. Then, the ICL algorithm is used to iteratively solve the rigid transformation parameters. The core objective function of the improved ICL algorithm is to minimize the matching error between the three-dimensional feature lines, The specific formula is described as follows: Definition of image feature line set is , LiDAR feature line set is , the two sets of feature lines exist a preliminary corresponding relationship, the optimization objective function of ICL algorithm is represented as: wherein, and respectively represent three-dimensional feature lines corresponding to the image and LiDAR, represents a rotation matrix, represents a translation vector, represents the number of corresponding line feature pairs; The optimal solution is obtained by iteratively solving the above equations With , complete the rigid registration of line features between images and LiDAR point clouds; Image contour feature Building plan feature corresponding to LiDAR Projecting to the space plane where the image contour lies through the current obtained transformation matrix, forming a projected contour line Thus, a face profile projection error constraint function is formed: wherein, represents the number of matching pairs of image contours and LiDAR surface features.

4. The method of claim 3, wherein the method further comprises: By integrating the line and surface constraints, the line and surface feature constraints are integrated into a unified optimization framework. The final overall objective function is represented as: where the weight factor is used to adjust the relative strength of the face feature constraint and the line feature constraint.

5. The method of claim 1, wherein, In step 4, the energy equation of the classical block adjustment is extended to construct the energy equation as shown in equation (13): In the formula, The image point residual term defined for classical regional network adjustment. For the first Orientation parameters of the image, For its in the Image point observation on Zhang's image, The projection function defined for the collinearity condition equation; Geometric constraints are formed by combining point, line, and surface feature information.

Images