LiDAR assistance-based unmanned aerial vehicle live-action three-dimensional modeling method

Abstract

The invention discloses an unmanned aerial vehicle live-action three-dimensional modeling method based on LiDAR assistance, and the method is characterized in that the method comprises the following steps: (1) data collection: employing an IMU + GNSS integrated navigation system to collect the pose data of an unmanned aerial vehicle in real time, carrying out the filtering, intensity value normalization and down-sampling processing of a LiDAR point cloud, and obtaining the pose data of the unmanned aerial vehicle; performing multi-view image feature point extraction and feature point descriptor calculation and matching on the image data; (2) adjusting a conversion matrix through a dynamic weight calculation model; and (3) performing iterative optimization by adopting a nonlinear least square method, converting a registration problem into nonlinear optimization, obtaining an initial conversion matrix from a dynamic weight stage, setting Lie algebra parameterized expression, determining an optimization boundary condition, establishing an iterative optimization process, and gradually and finely adjusting a coordinate conversion relation to enable an overall matching error to be minimum and ensure that a registration error is less than or equal to 0.5 cm. The method has the beneficial effects that the offset of the coordinate system is corrected in real time through the dynamic weight registration algorithm, and the registration error precision is improved by more than three times.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of engineering surveying, and in particular to a LiDAR-assisted unmanned aerial vehicle (UAV) real scene three-dimensional modeling method. BACKGROUND

[0002] The UAV real scene three-dimensional modeling technology has wide application prospects in smart cities, digital twins, engineering surveying, etc., and is mainly dependent on photogrammetry methods to realize three-dimensional scene reconstruction. Photogrammetry technology collects multi-angle aerial images, and generates a three-dimensional model using feature point matching and triangulation principles. However, in practical applications, there are significant bottlenecks that affect the accuracy, completeness, and efficiency of the model. In texture-rich scenes such as vegetation-covered areas and urban green belts, photogrammetry methods can easily lead to point cloud matching errors due to excessive surface details. This can cause geometric distortion, such as jagged edges or volume calculation deviations in vegetation area models, which cannot meet the sub-meter accuracy requirements. Traditional solutions attempt to alleviate this by increasing image resolution, but this increases computational complexity and exacerbates data noise, ultimately affecting model reliability. In addition, the traditional optimization strategies used in existing technologies are not only inefficient but also increase system complexity. For example, high overlap rates lead to data redundancy, exacerbating storage and computing burdens; manual retakes rely on experience and are prone to human error. The above defects make existing technical means unable to adapt to large-scale, high-dynamic scenes, limiting the promotion of three-dimensional models in digital twin applications.

[0003] Chinese patent CN120125760A discloses a high-precision airport area three-dimensional reconstruction method based on oblique photography. The method ensures the completeness of building textures by fusing multi-view data. It mainly uses a texture mapping algorithm to perform texture mapping on the initial three-dimensional point cloud model to obtain a three-dimensional point cloud model, fuses multiple texture sources, and reduces color inconsistencies caused by differences in lighting conditions through lighting correction to obtain fused texture data. The method optimizes the three-dimensional point cloud data with high precision, but does not solve the problem of point cloud matching errors caused by excessive surface details. SUMMARY

[0004] The present application solves the technical problem of existing UAV real scene three-dimensional modeling technology, which causes point cloud matching errors due to excessive surface details, resulting in geometric distortion and inability to meet sub-meter accuracy requirements. Therefore, a LiDAR-assisted UAV real scene three-dimensional modeling method is provided.

[0005] The technical solution of the present application is as follows: a LiDAR-assisted UAV real scene three-dimensional modeling method, comprising the following steps: (1) Data acquisition: real-time acquisition of unmanned aerial vehicle pose data by using IMU+GNSS integrated navigation system, the sampling frequency is set to 100Hz, filtering, intensity value normalization and down-sampling processing are performed on LiDAR point cloud, multi-view image feature point extraction and feature point descriptor calculation and matching are performed on image data; (2) adjusting the conversion matrix through a dynamic weight calculation model: based on the above two groups of data, the conversion matrix is adjusted through a dynamic weight calculation model to sequentially perform coarse registration and fine registration, according to the feature reliability of different regions of the scene, weighted least squares is used to solve, the weight distribution is re-evaluated after each iteration, and the weight distribution of LiDAR point cloud and image matching is adjusted; (3) iterative optimization by using nonlinear least squares method, the registration problem is converted into nonlinear optimization, the initial conversion matrix is obtained from the dynamic weight stage and the Lie algebra parameterization expression is set, the optimization boundary condition is determined, the iterative optimization process is established, the coordinate conversion relationship is gradually adjusted, the overall matching error is minimized, and the registration error is ensured to be less than or equal to 0.5 cm.

[0006] The conversion matrix in step (2) in the above scheme is a matrix used to describe the transformation relationship between coordinate systems in three-dimensional space, which is used for LiDAR and image coordinate registration.

[0007] The coarse registration in step (2) in the above scheme is to select significant feature points and estimate the initial conversion matrix using the 4-point method.

[0008] The fine registration in step (2) in the above scheme is to establish a preliminary correspondence relationship by using KD tree search and calculate the re-projection error of each pair of matching points.

[0009] The feature reliability of different regions in step (2) in the above scheme includes: reflection intensity reliability, geometric shape reliability and image texture reliability.

[0010] The adjustment of the weight distribution of LiDAR point cloud and image matching in step (2) in the above scheme includes: the geometric shape reliability is assigned a weight of 60% in the initial stage, the weight proportion of the three types of reliability is adjusted according to the error iteration, the image texture reliability weight is increased by 10% in the convergence stage to fully utilize the texture features, and the solution result accuracy quality is ensured based on residual statistical analysis.

[0011] The optimization boundary condition in step (3) in the above scheme is the maximum rotation / translation amount.

[0012] The iterative optimization process in step (3) in the above scheme includes the following steps: 1, establishing an error calculation layer, 2, calculating the Jacobian matrix, 3, establishing an incremental solution layer, and 4, performing adaptive adjustment.

[0013] The beneficial effect of this invention is that it corrects the coordinate system offset in real time through dynamic weight registration algorithm, and combines LiDAR skeleton mesh (±2 cm accuracy) with image super-resolution refinement to stabilize and compress the plane error to 1.5 cm. The measured error of this scheme is ≤1.2 cm under level 6 wind disturbance, which is more than 3 times more accurate than the traditional method. Detailed Implementation

[0014] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0015] This invention addresses the problem of LiDAR and image coordinate system drift, and the implementation steps are as follows: First, the UAV pose data is collected in real time using an IMU+GNSS integrated navigation system (POS accuracy 0.02°), with the sampling frequency set to 100 Hz. Secondly, the LiDAR point cloud is filtered, its intensity values ​​are normalized, and downsampled. Multi-view image feature points are extracted, and feature point descriptors are calculated and matched. Based on these two sets of data, the transformation matrix is ​​adjusted using a dynamic weight calculation model. The transformation matrix describes the transformation relationship between coordinate systems in three-dimensional space and is mainly used for LiDAR and image coordinate registration. Coarse registration is performed sequentially: significant feature points are selected, and the initial transformation matrix is ​​estimated using the 4-point method. Fine registration is performed: a preliminary correspondence is established using KD-tree search, and the reprojection error of each pair of matching points is calculated. Based on the reliability of features in different areas of the scene, including reflection intensity reliability, geometric reliability, and image texture reliability, weighted least squares is used for solving the problem. After each iteration, the weight distribution is re-evaluated, and the weight allocation when matching the LiDAR point cloud with the image is intelligently adjusted. Initially, the geometric reliability weight is assigned 60%. The weight ratio of the three reliability categories is adjusted iteratively based on the error. During the convergence phase, the image texture reliability weight is appropriately increased by 10% to fully utilize texture features. Residual statistical analysis is used to ensure the accuracy and quality of the solution results. At a flight speed of 10... At a speed of m / s, the weights are updated every 0.1 seconds. Finally, a nonlinear least squares iterative optimization method is adopted, with ≥5 iterations, transforming the registration problem into a nonlinear optimization. The initial transformation matrix is ​​obtained from the dynamic weight stage, and the Lie algebra parameterization is set. The optimization boundary condition is determined as the maximum rotation / translation. The iterative optimization process is established, specifically including the following steps: 1. Establishing an error calculation layer; 2. Performing Jacobian matrix calculation; 3. Establishing an incremental solution layer; 4. Performing adaptive adjustment, by gradually fine-tuning the coordinate transformation relationship, to minimize the overall matching error and ensure that the registration error is ≤0.5 cm.

[0016] The implementation steps include the following: S1. Data Acquisition and Preprocessing The specific process of S1 is as follows: S1-1. Real-time acquisition of UAV attitude data is performed using an IMU+GNSS integrated navigation system (POS accuracy 0.02°), with a sampling frequency of 100 Hz. The forward overlap rate is set to 85%, and the lateral overlap rate is set to 70%.

[0017] S1-2. Real-time acquisition of LiDAR point cloud and image data; filtering, intensity value normalization, and downsampling of the LiDAR point cloud; multi-view image feature point extraction and feature point descriptor calculation and matching of the image data. Attitude angle information provided by IMU / GNSS is timestamped to the point cloud acquisition time and converted into a rotation matrix for initial coordinate transformation.

[0018] S2. Initialize the transformation matrix Based on the data obtained in S1-2, significant feature points are selected, and a rigid body transformation matrix (i.e., transformation matrix) is constructed using the 4-point method. The LiDAR point cloud coordinate system is transformed to the camera coordinate system, and the initial transformation matrix is ​​obtained through pose combination calculation.

[0019] The transformation matrix shown below is a 4x4 matrix used to describe the transformation relationship between coordinate systems in three-dimensional space.

[0020]

[0021] in: Rotation component: a 3x3 submatrix (R) used to describe three-dimensional rotation (pitch, roll, yaw). Translation component: a 3*1 vector (T) used to describe three-axis displacement; Perspective transformation: The last row remains (0,0,0,1), in Euclidean space; 4-point method: 1: Select 4 sets of non-coplanar corresponding points from the point cloud and feature image respectively; 2: Construct an overdetermined system of equations and find the least squares solution through SVD decomposition; 3: Verify the rough overlap between the transformed point cloud and the image; S3. Adjust the transformation matrix using a dynamic weight calculation model. The specific process of S3 is as follows: S3-1. Construct multi-feature weights and adopt an adaptive fusion mechanism of intensity-geometry-texture triple features: Reflection intensity weight w I By using Gaussian and function quantization to ensure consistency between point cloud reflection intensity and image grayscale, the influence of this feature can be reduced when there are significant differences in sensor reflection characteristics.

[0022] Geometric structure weight w G The Hausdorff distance is calculated based on local surface fitting of point clouds, and a geometric similarity coefficient is generated. Regions with gentle curvature changes are given higher weights.

[0023] Texture feature weight w T The accuracy of SIFT feature matching (number of inliers / number of matching pairs) is determined by removing mismatched points after RANSAC filtering.

[0024] Final dynamic weight allocation: The final weights are obtained by normalizing three terms to ensure that the sum of the weights is 1.

[0025] At a flight speed of 10 m / s, the weights are updated every 0.1 seconds.

[0026] S3-2, Weighted Fine Registration Stage under Dynamic Weights.

[0027] A preliminary correspondence is established using KD-tree search. The reprojection error of each pair of matching points is calculated. The nearest neighbor is searched for each target point, and the search radius is 3-5 times the initial reprojection error (pixels) under the initial transformation matrix obtained by S2.

[0028] Based on the initialization transformation matrix obtained in S2 and the weight determination method in S3-1, an improved ICP algorithm combined with Lie algebra optimization is adopted: 1. The initialization transformation matrix is ​​parameterized as a Lie algebra; 2. A weighted objective function is constructed. , Where I is the reflection intensity difference term, G is the point-to-surface distance term, T is the reprojection error term, and w is the initial stage geometric shape confidence weight. G Assign a value of 60%. 3. Solve iteratively using the Levenberg-Marquardt algorithm: 1) Calculate the Jacobian matrix, 2) Establish an incremental solution layer, 3) Perform adaptive adjustment. If the number of iterations is ≥5, appropriately increase the image texture confidence weight w during the convergence phase. T 10% is used to fully utilize texture features to obtain the optimal transformation matrix. 4. Error control mechanism: 1) Reprojection error verification: The registered LiDAR point cloud is projected onto the image plane, the pixel-level error with the corresponding pixel is calculated, and outliers exceeding the threshold are removed. 2) Iterative convergence conditions: Reference parameter update amount, objective function change rate, and the maximum number of iterations are set. By gradually fine-tuning the coordinate transformation relationship, the overall matching error is minimized, ensuring that the registration error is ≤0.5 cm.

[0029] In terms of planar accuracy, traditional photogrammetry methods are limited by sensor drift and matching errors, with planar errors in complex scenes such as dense urban building clusters typically ranging from 3 to 5 centimeters. This invention uses a dynamic weighted registration algorithm to correct coordinate system offsets in real time, combined with a LiDAR skeleton mesh with ±2 cm accuracy and image super-resolution refinement, to stably compress the planar error to 1.5 cm. This accuracy was verified in the Hangzhou Olympic Sports Center project with 200 ground control points (GCPs), achieving an error ≤0.5 cm under high-speed flight conditions of 10 m / s, a 6-fold improvement over traditional methods. Under interference from a Force 6 gale with a wind speed of 13 m / s, the IMU+GNSS integrated navigation system with a POS accuracy of 0.02° ensures modeling accuracy fluctuations ≤1.2 cm, improving anti-interference capability by 300%. The LiDAR mentioned above refers to lidar radar.

[0030] In this specification, the reference to the term "embodiment" means that a specific feature or characteristic described in connection with that embodiment is included in at least one embodiment of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment. Furthermore, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments.

[0031] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A method for real-world 3D modeling of unmanned aerial vehicles (UAVs) based on LiDAR assistance, characterized in that, Includes the following steps: (1) Data acquisition: The UAV pose data is acquired in real time using the IMU+GNSS integrated navigation system. The sampling frequency is set to 100Hz. The LiDAR point cloud is filtered, the intensity value is normalized and downsampled. The image data is extracted from multi-view image feature points and the feature point descriptor is calculated and matched. (2) Adjust the transformation matrix through dynamic weight calculation model: Based on the above two sets of data, the transformation matrix is ​​adjusted through dynamic weight calculation model to perform coarse registration and fine registration in sequence. According to the feature reliability of different areas of the scene, the weighted least squares solution is adopted. After each iteration, the weight distribution is re-evaluated and the weight allocation when matching LiDAR point cloud with image is adjusted. (3) Iterative optimization using nonlinear least squares method is adopted to transform the registration problem into nonlinear optimization. The initial transformation matrix is ​​obtained from the dynamic weight stage and the Lie algebra parameterization expression is set. The optimization boundary conditions are determined and the iterative optimization process is established. By gradually fine-tuning the coordinate transformation relationship, the overall matching error is minimized and the registration error is ensured to be ≤0.5 cm.

2. The method for real-world 3D modeling of UAVs based on LiDAR assistance as described in claim 1, characterized in that, The transformation matrix in step (2) is a matrix used to describe the transformation relationship between coordinate systems in three-dimensional space, and is used for LiDAR and image coordinate registration.

3. The LiDAR-assisted UAV real-scene 3D modeling method as described in claim 1, characterized in that, The coarse registration in step (2) involves selecting significant feature points and using the 4-point method to estimate the initial transformation matrix.

4. The method for real-world 3D modeling of UAVs based on LiDAR assistance as described in claim 1, characterized in that, The fine registration in step (2) involves using a KD tree search to establish a preliminary correspondence and calculating the reprojection error of each pair of matching points.

5. The method for real-world 3D modeling of UAVs based on LiDAR assistance as described in claim 1, characterized in that, The reliability of features in different regions in step (2) includes: reliability of reflection intensity, reliability of geometry, and reliability of image texture.

6. The method for real-world 3D modeling of UAVs based on LiDAR assistance as described in claim 1, characterized in that, The weight allocation for adjusting the LiDAR point cloud and image matching in step (2) includes: initially assigning a geometric confidence weight of 60%, adjusting the weight ratio of the three confidence categories according to error iteration, increasing the image texture confidence weight by 10% in the convergence phase to fully utilize texture features, and ensuring the accuracy and quality of the solution results based on residual statistical analysis.

7. The method for real-world 3D modeling of UAVs based on LiDAR assistance as described in claim 1, characterized in that, The optimized boundary condition in step (3) is the maximum rotation / translation.

8. The method for real-world 3D modeling of UAVs based on LiDAR assistance as described in claim 1, characterized in that, The iterative optimization process in step (3) includes the following steps:

1. Establishing an error calculation layer; 2. Performing Jacobian matrix calculation; 3. Establishing an incremental solution layer; 4. Performing adaptive adjustment.

Images