An unmanned aerial vehicle image image control-free mapping method

Abstract

The application discloses a kind of global unmanned plane free image control image mapping methods based on multi-source data fusion processing, comprising: preliminary preparation and parameter calibration, three-dimensional calibration field grid parameter calibration and three-level time synchronization scheme are used to realize multi-sensor high-precision synchronization;Data acquisition, synchronously acquire image, laser Lidar, GNSS, IMU and other multi-source data;Data preprocessing, including image distortion correction, PPK / IMU multi-sensor fusion POS solution and POS eccentricity correction;Three-dimensional reconstruction and mapping, through multiscale fusion uniform color, SIFT feature matching, laser point cloud auxiliary free image control aerial triangulation encryption, SGM dense matching and point cloud fusion, finally generate DSM and TDOM true color orthophoto.The application overcomes the defects of low precision, weak texture area failure, obvious error accumulation and other defects in the prior art in large survey area surveying and mapping, realizes high-precision, high-robustness global unmanned plane free image control mapping.

Description

Technical Field

[0001] This invention belongs to the field of UAV photogrammetry and mapping technology, and in particular relates to a UAV image mapping method without image control. Background Technology

[0002] Traditional UAV mapping methods require the deployment of ground control points (GCPs), which have significant drawbacks, such as high workload, high cost, terrain limitations, and unsuitability for surveying dangerous or remote areas. Current GCP-free mapping technology has entered a mature application stage of "centimeter-level positioning + intelligent processing + multi-source fusion." Its core relies on high-precision RTK or PPK POS, tight coupling of GNSS / INS, and aerial triangulation adjustment optimization. Horizontal accuracy generally meets the requirements of 1:500 topographic maps, while elevation accuracy remains a key area for optimization. Multi-source data fusion is the main development direction. Existing PPK-based GCP-free methods are prone to cumulative errors in large-area surveys, leading to decreased mapping accuracy. Current technologies also suffer from insufficient correction of distortion in non-measurement camera images, resulting in low edge matching accuracy. Furthermore, they rely on single-sensor data and are prone to failure in occluded / weak texture areas. Summary of the Invention

[0003] In view of this, the present invention aims to overcome the shortcomings of existing UAV image mapping methods without image control in large survey areas, such as low accuracy, insufficient correction of distortion of non-measurement camera images, easy failure in weak texture areas, and significant error accumulation. The present invention provides a UAV image mapping method without image control that is highly accurate, robust, and applicable to complex scenarios.

[0004] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0005] In a first aspect, the present invention provides a method for image processing without image control from unmanned aerial vehicles (UAVs), comprising the following steps:

[0006] Step 1: Preliminary preparation and parameter calibration, including:

[0007] The camera intrinsic parameters were calibrated using a three-dimensional calibration field grid intrinsic parameter calibration method.

[0008] Time synchronization calibration of GNSS, laser, IMU, and camera;

[0009] Perform POS eccentricity correction preprocessing for IMU pose;

[0010] Flight route planning is conducted based on the survey area and terrain features.

[0011] Step 2: Data acquisition. Simultaneously acquire airborne aerial digital imagery, airborne aerial LiDAR data, GNSS raw data, IMU data, and POS data, and record the GNSS timestamp of the exposure time.

[0012] Step 3: Data preprocessing, including:

[0013] Based on the three-dimensional calibration field grid calibration scheme, the parameters of the camera intra-field grid distortion correction model are calculated, and pixel-by-pixel distortion correction and image enhancement are performed on the image.

[0014] A POS solution scheme that integrates PPK and IMU multi-sensor fusion is adopted. By fusing GNSS and IMU data through extended Kalman filtering, high-precision position, velocity, and attitude information are obtained.

[0015] POS eccentricity correction is performed based on IMU pose and GNSS antenna position to calculate accurate photo pose information;

[0016] Step 4: 3D reconstruction and core mapping workflow, including:

[0017] Multi-scale fusion and color balancing processing of the image;

[0018] The SIFT feature matching algorithm is used for feature point extraction and matching, and the RANSAC method is used to remove mismatched points to obtain the correct matching pairs.

[0019] A method for image-free aerial triple densification based on laser point cloud, GNSS, and IMU assistance achieves accurate registration between imagery and laser point cloud through projection matching, point-area correlation, and hierarchical beam adjustment optimization.

[0020] A dense point cloud is generated using semi-global matching (SGM) and then fused with a laser point cloud.

[0021] True-color orthophotos of DSM and TDOM are generated based on fused point clouds.

[0022] Furthermore, step 1, specifically the time synchronization calibration of GNSS with the laser, IMU, and camera, includes:

[0023] At the hardware level, the airborne time synchronization unit outputs a trigger pulse signal of a uniform frequency to control the GNSS, IMU, camera and laser to start data acquisition synchronously.

[0024] The reference anchoring layer uses the second pulse signal output by GNSS as the absolute time reference and records the corresponding UTC timestamp;

[0025] At the software level, the deviation between the timestamp of each sensor data frame and the PPS anchoring time is extracted, and the time deviation model is fitted using the least squares method to correct the original data timestamp.

[0026] Furthermore, in step 3, the grid distortion correction model is constructed by establishing a mapping relationship between the observed coordinates and the corrected ideal coordinates using high-order polynomials. It includes interior orientation elements and three types of distortion parameters, and its mathematical expression is:

[0027]

[0028] in, For the observation coordinates, As the distortion center, For normalized coordinates, , , Radial distance, Radial distortion factor ,in Radial distortion coefficient; and For radial distortion, The eccentricity distortion coefficient, For affine / nonorthogonal coefficients, These are the coordinates of the image point after distortion correction.

[0029] Furthermore, step 3, POS eccentricity correction, includes:

[0030] Convert the latitude, longitude, and altitude coordinates of the moment the photo was taken to geocentric coordinates;

[0031] Convert geocentric coordinates to local horizontal coordinates;

[0032] Convert local horizontal coordinates to body coordinates;

[0033] Perform coordinate correction in the body coordinate system;

[0034] The corrected coordinates are then converted back to geodetic latitude and longitude in reverse order.

[0035] Furthermore, in step 4, the multi-scale fusion color homogenization includes the following steps:

[0036] The acquired photos were downsampled by a ratio of 1:8 to obtain photos with reduced resolution.

[0037] Based on the down-resolution photo, combined with pose information and the average elevation of the survey area, digital differential correction is used to perform absolute orientation.

[0038] After orientation, the images from each shooting station are processed using multi-scale fusion technology to perform multi-scale weighted fusion of pixel values ​​in overlapping areas, resulting in a color-balanced, scaled orthophoto of the survey area.

[0039] Based on the geographical range of the current photo, the corresponding sub-image is extracted from the orthophoto and upsampled at a ratio of 1:8. The upsampled image is used as a reference image, and the original image is color-matched using histogram matching to obtain a pre-processed photo with consistent color.

[0040] Furthermore, in step 4, the image-free spatial three-dimensional encryption method includes the following steps:

[0041] Generate initial 3D points with scale based on LiDAR point cloud projection;

[0042] Incremental PnP and triangulation methods are used for image registration and 3D point expansion.

[0043] Establish point-to-surface relationships between visual 3D points and LiDAR point cloud planes;

[0044] Layered execution of incremental BA, batch BA and global BA, jointly optimizing visual reprojection error and LiDAR point-to-area distance error;

[0045] Output the registered camera pose and scaled 3D point set.

[0046] Furthermore, the objective function for joint optimization is:

[0047] ;

[0048] ;

[0049] in, R represents the coordinates in the 3D point camera coordinate system. i Let t be the rotation matrix of the camera. i Let X be the translation vector of the camera. k The coordinates of the visual 3D point in the world coordinate system. For 3D point trajectory features, w is the LiDAR point weight, and n q Let l be the normal vector of the LiDAR plane. q For the corresponding LiDAR points.

[0050] Furthermore, in step 4, point cloud fusion includes the following steps:

[0051] Calculate the survey area range and generate voxel blocks for the survey area range;

[0052] The registered laser point cloud and the visually matched point cloud are mapped into voxels respectively;

[0053] The point cloud within the voxel is sorted and clustered according to the Z coordinate value, and the mean of the middle position point set is taken as the fused point cloud.

[0054] Secondly, the present invention provides an electronic device, including a processor and a memory communicatively connected to the processor and used to store executable instructions of the processor, wherein the processor is used to execute the above-described method for image-controlled mapping of UAV images.

[0055] Thirdly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned method for image capture without image control for unmanned aerial vehicles.

[0056] Compared with existing technologies, the UAV imagery-based image processing method described in this invention has the following advantages:

[0057] 1. No ground-based image control is required, significantly reducing fieldwork workload and improving operational efficiency;

[0058] 2. Multi-source data fusion solves the problems of low accuracy and error accumulation in traditional methods;

[0059] 3. Laser point cloud assistance improves the robustness of image generation in weak texture areas and avoids failure;

[0060] 4. System-level calibration + automated process, adaptable to large-scale testing areas, balancing accuracy and mass production requirements. Attached Figure Description

[0061] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0062] Figure 1 This is a schematic diagram of an image-control-free mapping method for UAV images according to an embodiment of the present invention. Detailed Implementation

[0063] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0064] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0065] Example 1:

[0066] refer to Figure 1 This invention provides a method for image processing without image control for UAV images, specifically including the following:

[0067] Step 1: Preliminary Preparation and Parameter Calibration

[0068] Data acquisition equipment: Use a multi-rotor or fixed-wing UAV equipped with a GNSS antenna (supporting RTK / PPK), an IMU inertial measurement unit, and a full-frame aerial camera;

[0069] Parameter calibration:

[0070] A three-dimensional calibration field grid intrinsic parameter calibration method is adopted to accurately calibrate the camera intrinsic parameters, obtaining complete intrinsic parameter information including focal length, principal point coordinates, and distortion parameters. For time synchronization calibration of GNSS, laser, IMU, and camera, a three-level synchronization scheme of "unified hardware triggering + GNSS PPS reference anchoring + software secondary correction" is adopted. The core logic is to use GNSS time as the absolute reference, achieve synchronization of the four sensors' data acquisition start through hardware triggering, and then eliminate time drift and trigger delay through software correction. Specifically: at the hardware level, a unified frequency trigger pulse signal is output through the UAV's onboard high-precision time synchronization unit, which is connected to the trigger interfaces of the GNSS receiver, IMU inertial measurement unit, camera, and laser, respectively, controlling the four sensors to synchronously start data acquisition under the same trigger signal, ensuring the consistency of the time reference of the original data acquisition; at the reference anchoring level, the pulse-of-seconds (PPS) signal output by GNSS is used as the absolute time reference, recording the timestamp (UTC time) corresponding to each PPS signal, and using it as the anchoring reference for the four sensor timestamps; at the software level, the timestamps of each sensor data frame are extracted and compared with GNSS data. The deviation value of PPS anchoring time was determined by fitting the time deviation model of each sensor (including initial deviation and drift coefficient) using the least squares method. The original data timestamps of IMU, camera, and laser were corrected respectively. At the same time, the delay difference between the camera exposure time, laser emission time and trigger signal was calibrated to achieve time synchronization of the four sensors. Simultaneously, preprocessing preparation based on IMU pose POS eccentricity correction was performed. The specific parameter ranges are as follows: IMU sampling frequency 200-500Hz, laser emission frequency 100-200kHz, time synchronization error ≤1ms, intrinsic parameter calibration distortion error ≤0.01 pixels, and PPS signal synchronization accuracy ≤0.1μs.

[0071] Flight route planning: Based on the flight route design strategy (such as overlap rate and flight altitude) based on the survey area range and terrain features, the forward overlap rate is 70%-80%, the lateral overlap rate is 60%-70%, and the flight altitude is set to 100-300m according to the resolution requirements of the survey area.

[0072] Step 2: Data Collection

[0073] Data types acquired and recorded: airborne aerial digital imagery, airborne aerial laser LiDAR data, GNSS raw data (pseudorange, carrier phase), IMU data (attitude angle, acceleration), and POS data (position, heading angle);

[0074] Acquisition control conditions: During image acquisition, the GNSS timestamp of the exposure time is recorded simultaneously to ensure accurate matching of the timestamps of the image and the positioning data.

[0075] Step 3: Data Preprocessing

[0076] Camera intrinsic parameter calibration and grid distortion correction model parameter calculation employ a 3D calibration field grid calibration scheme. Uniformly distributed grid control points (grid spacing 0.5m) are deployed as calibration targets within a pre-defined 3D calibration field. Target images are captured from 20 different angles, and the pixel coordinates (distortion coordinates) and 3D world coordinates of the target grid points are extracted. Based on the pinhole camera model and the image grid distortion correction model, the least squares method is used to jointly calculate the camera intrinsic parameter matrix K, distortion correction model parameters, and exterior orientation elements. The calculated camera intrinsic parameter matrix K and grid distortion correction model parameters are k1, k2, k3, k4, k5, k6, p1, p2, b1, and b2. These parameters are then substituted into the grid distortion correction model to perform pixel-by-pixel correction on the original aerial images. Contrast stretching is used to enhance the grayscale of the images, and Gaussian denoising is used to remove noise.

[0077] The essence of the grid distortion correction model is to establish a mapping relationship between the observed coordinates and the corrected ideal coordinates through high-order polynomials. It includes interior orientation elements and three types of distortion parameters, and the specific mathematical expression is as follows:

[0078]

[0079] in, These are the observed coordinates, i.e., the actual pixel coordinates detected in the image (pixel coordinate system). The distortion center, or principal point offset, is usually close to the center of the image and serves as the reference point for distortion correction. For normalized coordinates, , Eliminate the influence of principal point offset; Radial distance, That is, the distance from the normalized coordinates to the distortion center; radial distortion factor. ,in Radial distortion coefficient; Radial distortion term: and It mainly corrects the scaling deviation between the image center and the edge, and solves the problems of barrel distortion (edges expand outward) or pincushion distortion (edges shrink inward); It is the eccentricity distortion coefficient, used to correct tangential offset caused by the lens optical center not coinciding with the image plane center or the lens not being parallel to the image plane; These are affine / non-orthogonal coefficients, used to correct for inconsistent pixel ratios and non-perpendicular coordinate axes caused by sensor manufacturing or installation errors (these have a relatively small impact in real-world scenarios and are usually set to 0). These are the coordinates of the image point after distortion correction.

[0080] The parameters of the grid distortion correction model need to be solved through camera calibration experiments. The core method is to minimize the reprojection error using bundle adjustment. The specific process is as follows:

[0081] S1, Data Acquisition

[0082] The camera to be calibrated was used to capture images of a 3D calibration field with high-precision control points, obtaining 20 sets of calibration images from different shooting angles and distances. Simultaneously, the world coordinates of each control point in the 3D calibration field were recorded (which can be obtained using a high-precision total station).

[0083] S2, Feature Extraction

[0084] For each acquired calibration image, the observed pixel coordinates of the control point target center are extracted using a combination of feature detection algorithms and manual point selection. .

[0085] S3. Bundle adjustment solution parameters

[0086] An error equation is constructed with the objective of minimizing the reprojection error (the sum of squares of the differences between the corrected pixel coordinates and the ideal coordinates projected from the world coordinates onto the image), and the interior orientation elements and distortion parameters are solved. The core form of the error equation is:

[0087]

[0088] in This is a projection function that maps the corrected pixel coordinates to the world coordinate system. Here are the world coordinates of the control points. The error equations are solved using the least squares method to obtain the final model parameters.

[0089] S4, Accuracy Verification

[0090] Select some control points that were not included in the calibration as check points, substitute their observed coordinates into the distortion correction formula, and calculate the corrected reprojection error. If the root mean square error (RMS) is less than a preset threshold (e.g., 0.5 pixels), the parameter calibration is effective; otherwise, data needs to be re-acquired or the feature extraction process needs to be optimized until the parameters converge and stabilize.

[0091] Positioning data preprocessing: A PPK and IMU multi-sensor fusion POS solution is adopted. The core fusion logic is to use GNSS to correct the cumulative drift of the IMU and use the IMU to fill the positioning gap when the GNSS signal is interrupted. The following are the detailed implementation steps:

[0092] (1) Sensor preprocessing

[0093] IMU preprocessing: Elimination of zero bias, scale factor, and installation errors (using a six-sided calibration method). Data filtering: Removal of high-frequency noise using a low-pass filter. Time synchronization: Alignment with the GNSS timestamp (via hardware triggering).

[0094] GNSS preprocessing: Solve satellite ephemeris, calculate satellite position and velocity; extract pseudorange and pseudorange rate observations, and remove gross errors;

[0095] (2) Inertial navigation solution

[0096] IMU outputs angular velocity Given the force f, we obtain the position, velocity, and attitude (PVA) through integration. The core formula is:

[0097] Pose update (quaternion method, avoiding gimbal lock):

[0098]

[0099] in For IMU gyroscope zero bias, The pose quaternion before the update, For the updated pose quaternion.

[0100] Speed ​​updates:

[0101]

[0102] in This is the rotation matrix from the body coordinate system to the navigation coordinate system. To add zero bias to the table, The gravity vector The Earth's rotational angular velocity in the navigation system. Angular velocity of the navigation system relative to the Earth system For the velocity components in the navigation system, This is the derivative of velocity with respect to time in the navigation system.

[0103] Location update:

[0104] ;

[0105] For the velocity components in the navigation system, This is the derivative of position with respect to time;

[0106] Location is usually expressed in geodetic latitude and longitude (BLH) or ECEF coordinates.

[0107] (3) Extended Kalman Filter (EKF) Fusion

[0108] EKF is a core tool for handling nonlinear systems, consisting of a prediction step and an update step.

[0109] State vector design (core): Navigation error + IMU error are selected as state variables. Typical state vector:

[0110]

[0111] in, These are position, velocity, and attitude errors, respectively. To add zero bias error to the table; This refers to the zero-bias error of the gyroscope.

[0112] Prediction Step: Based on IMU inertial calculations, the system state equations are used to predict the state and covariance at the next time step.

[0113]

[0114] in, It is the optimal estimate from the previous time step (k-1 time step). The state transition matrix at the previous time step (k-1 time step), Estimating the prior state at time k

[0115]

[0116] in Let's consider the covariance matrix of the previous time step (k-1). This is the state transition matrix for the previous time step (k-1). This is the transpose of the state transition matrix from the previous time step (k-1). The process noise covariance matrix. Let be the prior covariance matrix at the current time (time k).

[0117] Update step: Correct the predicted values ​​with GNSS pseudorange / pseudorange rate observations to obtain the optimal estimate:

[0118] Calculate the observation residuals:

[0119] ;

[0120] Where z k To observe the residuals, These are pseudorange observations. These are IMU observations.

[0121] Calculate the Kalman gain:

[0122]

[0123] Update state and covariance:

[0124]

[0125]

[0126] in, Let k be the prior state estimate. This is the posterior state estimate at time k. Let k be the state covariance matrix at time k. For Kalman gain, For the observation matrix, To observe the noise covariance, I is the identity matrix. This is the transpose of the observation matrix.

[0127] S4. Error Correction and Output

[0128] The error state estimated by EKF is fed back into the IMU inertial calculation results to eliminate accumulated errors.

[0129] Calibration position:

[0130]

[0131] in, This is the corrected position. The position calculated by the inertial navigation system. This represents the cumulative position error.

[0132] Correction speed:

[0133] ;

[0134] in, This is the corrected position. The position calculated by the inertial navigation system. This represents the cumulative speed error.

[0135] Correcting attitude:

[0136]

[0137] in, This is the corrected posture. For the attitude calculated by the inertial navigation system, This is for accumulating attitude error.

[0138] Output the final PVA results (latitude, longitude, altitude, velocity, attitude angle).

[0139] Based on the data fusion results, a POS eccentricity correction step is performed using IMU pose, image capture time, and GNSS antenna position information to calculate the accurate image pose information. The POS eccentricity correction steps are as follows:

[0140] (1) The position of the GNSS antenna at the moment of taking the picture The formula for converting latitude, longitude, and altitude (L, B, H) to geocentric coordinates (X, Y, Z) is:

[0141]

[0142] is the radius of curvature of the prime vertical;

[0143] a is the semi-major axis of the ellipsoid, ;

[0144] b is the semi-minor axis of the ellipsoid, ;

[0145] e is the first eccentricity, ;

[0146] (2)Geocentric coordinates to local horizontal coordinates, two key parameters are required for the conversion:

[0147] The geocentric coordinate system coordinates of the local origin :

[0148] The position of the GNSS antenna phase center of the point to be converted in the geocentric coordinate system:

[0149] The geodetic latitude of the origin , longitude (which needs to be obtained in advance through ECEF→BLH iterative calculation)

[0150] First, calculate the geocentric coordinate difference of the point to be converted relative to the origin ( , , ), ( , , ) is denoted as the relative coordinate in the geocentric coordinate system, and the formula is as follows:

[0151]

[0152] Then construct the rotation matrix , the rotation matrix of the ENU coordinate system is obtained by first rotating around the Z-axis by , and then rotating around the X-axis by . The finally simplified rotation matrix is:

[0153]

[0154] The relationship between the local horizontal coordinates ( , , ) and the relative coordinate in the geocentric coordinate system ( , , ) is:

[0155]

[0156] The expanded component formula:

[0157]

[0158] The transformation from the local horizontal plane coordinate system (ENU) to the body coordinate system essentially involves rotating the ENU coordinate system in ZYX order to obtain a coordinate system aligned with the body. The transformation process consists of three steps, including the overall rotation matrix. It is a right multiplication of a three-step rotation matrix, with the rotation order being: first yaw ( → Then tilt up and down ( → Final roll ( ).

[0159] Yaw angle rotation matrix (Rotation around ENU-Z axis) )

[0160]

[0161] Pitch angle rotation matrix (Rotation around the rotated Y-axis) )

[0162]

[0163] Roll angle rotation matrix (Rotation around the rotated X-axis) )

[0164]

[0165] The total rotation matrix from ENU to the body coordinate system is the product of three rotation matrices:

[0166]

[0167] After unfolding, the final rotation matrix is ​​obtained:

[0168]

[0169] (3) Coordinates in the local horizontal coordinate system converted to the body coordinate system

[0170] Let the vector in the ENU coordinate system be The vector in the body coordinate system is The conversion formula is:

[0171]

[0172] Expand into component form:

[0173]

[0174] (4) In the body coordinate system, perform coordinate correction processing, as follows:

[0175]

[0176] in,( , , (X) represents the eccentricity-corrected coordinates in the body coordinate system. l ,Y l Z l () represents the arm offset from the camera projection center to the GNSS antenna phase center in the body coordinate system.

[0177] After eccentricity correction, the corrected coordinates need to be transformed back from the body frame to geodetic latitude and longitude (BLH). This is essentially a multi-step reverse process: first, the body frame is transformed back to the local horizontal plane (ENU), then back to the geocentric coordinate system (ECEF), and finally, geodetic latitude and longitude (BLH) is calculated from ECEF.

[0178] (5) Transform the body coordinate system to the local horizontal coordinate system

[0179] In positive conversion Therefore, the inverse transformation is: (A rotation matrix is ​​an orthogonal matrix, and its inverse is its transpose.)

[0180] Inverse rotation matrix (after transpose):

[0181]

[0182] Component formula:

[0183]

[0184] (6) Transformation from local horizontal coordinate system to geocentric coordinate system

[0185] ENU is relative to the origin. The local coordinates are thus inversely transformed into:

[0186]

[0187] Inverse rotation matrix (ENU→ECEF):

[0188]

[0189] Component formula:

[0190]

[0191] (7) Transformation from geocentric coordinate system (ECEF) to geographic coordinate system (BLH) (requires iterative solution)

[0192] Since the transformation has no closed-form solution, it needs to be solved using Newton's iteration method, including:

[0193] Initialization: Calculate geocentric latitude As the initial value of latitude ;

[0194] Iterative calculation of the radius of curvature of the zonal loop ;

[0195] Corrected latitude: ;

[0196] Repeat steps S6-S7 until... The radius (approximately 0.0000057°) meets the accuracy requirements;

[0197] Final solution:

[0198] longitude: ;

[0199] The height of the earth: .

[0200] Step 4: 3D Reconstruction and Core Mapping Process

[0201] Image preprocessing and feature matching: First, aerial images are color-matched by improving multi-scale fusion color-matching technology to eliminate brightness differences under different exposure conditions; then, the conventional SIFT feature matching algorithm is used for feature point extraction and matching; finally, the RANSAC method is used to remove mismatched points and obtain correct matching pairs.

[0202] The main steps of multi-scale fusion color equalization technology are:

[0203] S1. Downsample the acquired photos by a ratio of 1:8 to obtain photos with reduced resolution;

[0204] S2. Based on the down-resolution photograph, combined with pose information and the average elevation of the survey area, absolute orientation is performed using digital differential correction;

[0205] S3. After orientation, the images from each shooting station are processed using multi-scale fusion technology to perform multi-scale weighted fusion of the pixel values ​​in the overlapping areas, resulting in a color-balanced, scaled orthophoto of the survey area.

[0206] S4. Based on the geographical range of the current photo, extract the corresponding sub-image from the orthophoto and upsample it by 1:8. Use the upsampled image as the reference image and use the histogram matching method to perform color uniform processing on the original image to obtain a preprocessed photo with consistent color.

[0207] Image-free aerial triangulation encryption: This method employs a self-calibration scheme using laser point cloud, GNSS, and IMU-assisted aerial triangulation. It uses POS data fused from multiple sensors and corrected for eccentricity as initial exterior orientation elements, combined with feature matching and connection point matching pairs, to construct a regional network adjustment model. The algorithm retains the classic SfM workflow framework while adding LiDAR point cloud constraints as its core design concept. Based on the original SfM visual reconstruction algorithm, it achieves accurate registration between the image and the LiDAR point cloud through key steps such as projection matching points, point-to-area association, and layered beam adjustment (BA), providing reliable data support for subsequent data fusion processing. Its technical route can be divided into three main modules: preprocessing, core workflow, and optimized output. Each stage is closely connected and has a clear division of labor, as detailed below:

[0208] The core of the algorithm is "visual reconstruction + LiDAR point cloud constraint dual-drive". The overall process inherits the incremental SfM logic, while adding a constraint process related to LiDAR laser point clouds. The two are deeply integrated through "point-area association" and "dual error optimization". The overall process is as follows:

[0209] (1) LiDAR point cloud preprocessing: plane extraction and normal vector calculation: the laser LiDAR point cloud is divided into voxels (fixed size of 5cm), and the plane is fitted by the neighborhood points to calculate the unit normal vector of the plane to which each LiDAR point belongs. , Representing three-dimensional real space), forming "LiDAR point-plane" correlation pairs ( This provides a foundation for subsequent calculations of the "point-to-surface distance error".

[0210] (2) Point cloud filtering: Statistical filtering is used to remove outliers (such as isolated noise points) and retain structured planar points such as ground points and wall points to improve the robustness of subsequent matching;

[0211] (3) Feature extraction: Using classic SIFT visual features, 2D feature points are extracted from the input image. , (where i represents the image index and j represents the feature point index in the two-dimensional real space), generating feature descriptors;

[0212] (4) Feature matching: By using descriptor similarity matching, we identify the same feature points between different images and determine the overlapping areas of the scene. We use the RANSAC algorithm to remove false matches and retain inliers to ensure the reliability of the matching.

[0213] (5) Initial camera attitude settings

[0214] Pose input: Set the pose of the PPK / IMU fused image as the initial image ( The initial pose prior (rotation matrix) +Translation vector The initial pose determines the "initial viewpoint" of the LiDAR point cloud projection, providing a benchmark for subsequent generation of scaled initial 3D points;

[0215] (6) The key steps of vision and LiDAR fusion are achieved through four steps: “projection matching points, LiDAR point cloud association, incremental reconstruction, and joint optimization”, so as to realize the constraint fusion of visual 3D points and LiDAR point cloud plane and solve the problem of high-precision registration.

[0216] (61) Initialization: LiDAR point cloud projection generates initial 3D points with scale. The core of initialization is to use LiDAR point cloud to provide "real scale" for visual reconstruction, avoiding the scale blur problem of monocular SfM and the orientation error problem caused by initial POS error. Projection principle: using the initial image Based on the pose, construct a quadrangular pyramid (height is an adjustable parameter) with the camera center and the four corners of the image. Traverse the LiDAR point cloud voxels within the pyramid, transform the LiDAR points from the world coordinate system to the camera coordinate system, and then project them onto the image plane (imaging model):

[0217] Transform from world coordinate system to camera coordinate system:

[0218]

[0219] Converting camera coordinates to image coordinates (pixel coordinates)

[0220]

[0221] in, For world coordinates LiDAR points, Lidar point camera coordinate system coordinates ( [0], [1], [2]) is an abbreviation. For camera focal length, These are the coordinates of the principal point of the image.

[0222] Initial 3D point generation: If the pixel coordinates of the projected LiDAR points ( ) and image 2D feature points ( The LiDAR point is considered as the "depth reference" for the feature points, and the 3D coordinates of the feature points in the camera coordinate system are inferred by combining the camera intrinsic parameters; then, it is transformed to the LiDAR point cloud world coordinate system to form an initial 3D point set with real scale. ).

[0223] (62) Incremental Image Registration: PnP+triangulation expands the 3D model. This step inherits the original incremental SfM logic. The core is to gradually add new images to expand the 3D reconstruction range, while providing more visual 3D points for subsequent LiDAR point cloud constraints. Image Selection: From the images to be registered, select the image with the most "common visible points and the most uniform distribution" with the current 3D model. To ensure registration stability. PnP registration: using corresponding 3D points in the current 3D model and the image. The 2D feature points are obtained by optimizing the solution of the image using the EPnP (Exterior Point Avoidance) algorithm. posture ( Register it to an existing model. Triangulation: Triangulate the newly registered image. The untriangulated feature points are combined with the pose of the registered image and triangulated using the Direct Linear Transform (DLT) algorithm to generate new 3D points. The sampling area is adjusted using RANSAC recursive sampling to remove erroneous 3D points caused by noise, ensuring the reliability of the reconstruction.

[0224] (63) Point-to-plane association: Establishing constraints between visual 3D points and the LiDAR point cloud plane. Point-to-plane association is the "constraint core" of the algorithm. It binds visual 3D points to the LiDAR plane through two methods, providing LiDAR point cloud constraints for subsequent double-error optimization. For each visual 3D point ( Find the corresponding LiDAR point cloud plane, so that Minimize the distance to this plane to ensure that the 3D point scale is consistent with the LiDAR point cloud scale;

[0225] (64) Layered beam adjustment optimization (BA): This step is the "accuracy core". By minimizing the "visual reprojection error + LiDAR point-to-surface distance error", it optimizes the camera pose and 3D point position, and eliminates cumulative drift and scale deviation.

[0226] (641) Incremental BA: Local error correction

[0227] This is performed after triangulation of each new image is complete. It only applies to images with new data. Optimize closely related images (local image sets) that "share a large number of features" to avoid excessive global computation. Variable and constant settings: Variables: pose of related images, 3D points that do not reach the "trajectory feature threshold" (few trajectory features, low accuracy, needs adjustment); Constants: pose of other images, 3D points with sufficient trajectory features, LiDAR point cloud plane (fixed scale reference). Error calculation: visual reprojection error (… ): Ensure consistency between 3D points and image features; LiDAR point-to-surface distance error ( ): Ensure that the 3D point scale is consistent with LiDAR;

[0228]

[0229] in, R i Let t be the rotation matrix of the camera. i Let X be the translation vector of the camera. k The coordinates of the visual 3D point in the world coordinate system. The 3D coordinate vector in the camera coordinate system ( () is an abbreviation of ) ) represents the 2D projection trajectory feature points of 3D points, and w represents the LiDAR point weights (projection points). Nearest neighbor Ground point (Set different values ​​respectively) The normal vector of the LiDAR plane. For the corresponding LiDAR points.

[0230] Optimization goal: The optimal camera pose and 3D point position are solved by the Gauss-Newton method.

[0231] (642) Batch BA: Local region consistency optimization, executed after registration of a certain number of images (e.g., every 10 images), to resolve "local association error between early images and new images". Using new images... The camera center is used as the center of a sphere, and a sphere with a fixed radius is drawn. All images within the sphere are included in the optimization (images within the same region have related poses). Variables and constants are set as follows: Variables: pose of images within the sphere, observable 3D points within the sphere; Constants: pose of images outside the sphere, 3D points outside the sphere, LiDAR point cloud plane. Point-to-surface association: The "nearest neighbor search method" (point-to-surface association) is used to rematch the variable 3D points with the LiDAR point cloud plane to ensure the effectiveness of local optimization constraints. The purpose of this type of optimization is to reduce pose drift within the same region and improve the consistency of local reconstruction.

[0232] (643) Global BA: Improves the accuracy of the entire model. Executed after incremental reconstruction, it performs final optimization of the entire 3D model. It considers the pose of all registered images and all visual 3D points. Point-to-surface association: Matches all 3D points to the LiDAR point cloud plane using the nearest neighbor search method to ensure global scale uniformity. Eliminates global accumulated errors, ensuring the entire reconstruction result is perfectly aligned with the scale and position of the LiDAR point cloud.

[0233] (65) Registration result output, camera pose: Output the precise position and pose of all images ( It can be directly used for image localization; Scaled 3D points: Outputs a visual 3D point set consistent with the LiDAR scale, supporting subsequent visual matching point and LiDAR point cloud registration and fusion; Registration accuracy verification: Verify the reliability of the results through "left-right consistency detection", "reprojection error analysis" and "point-surface distance convergence".

[0234] Reconstruction and Mapping: The SGM semi-global dense point cloud generation algorithm is used to generate high-density, high-precision dense point clouds based on accurate exterior orientation elements, and then fused with the laser point cloud. The fusion steps are as follows:

[0235] Calculate the survey area range, generate voxel blocks for the survey area range, and set the voxel size to the resolution of the survey area.

[0236] The registered laser point cloud and the visually matched point cloud are mapped onto the voxels of the test area, respectively.

[0237] The point cloud within the voxel is sorted and clustered according to the Z coordinate value, and the mean of the middle position point set is taken as the fused point cloud.

[0238] Based on the fused point cloud, GPU-accelerated DSM generation technology is introduced to improve DSM construction efficiency; based on the DSM and the color-balanced image, TDOM seamless fusion and stitching technology is used to generate true-color digital true orthophotos.

[0239] Example 2:

[0240] An electronic device includes a processor and a memory communicatively connected to the processor and used to store processor-executable instructions, the processor being used to execute the above-described method for image-controlled mapping of unmanned aerial vehicles (UAVs).

[0241] Example 3:

[0242] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described method for image capture without image control for unmanned aerial vehicles (UAVs).

[0243] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in this disclosure all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0244] It should be noted that personal information collected from users should be used for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. Furthermore, such collection / sharing should only be conducted after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign an agreement / authorization that includes authorization of relevant user information before the user uses the function. In addition, any necessary steps must be taken to protect and safeguard access to such personal information data and ensure that others with access to personal information data comply with their privacy and procedures.

[0245] This disclosure is intended to provide implementation schemes for users to selectively prevent the use or access to their personal information data. Specifically, this disclosure is intended to provide hardware and / or software to prevent or block access to such personal information data. Once personal information data is no longer needed, risks can be minimized by restricting data collection and deleting data. Furthermore, where applicable, such personal information is de-identified to protect user privacy.

[0246] The acquisition, transmission, storage, use, and processing of data in this disclosed technical solution all comply with the relevant provisions of national laws and regulations.

[0247] It should be noted that in the embodiments disclosed herein, certain software, components, models, and other existing solutions in the industry may be mentioned. These should be considered as exemplary and are intended only to illustrate the feasibility of implementing the technical solution of this application. However, they do not mean that the applicant has used or necessarily used such solutions.

[0248] In the foregoing descriptions of the embodiments, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0249] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0250] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain.

[0251] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0252] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0253] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0254] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0255] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present disclosure.

Claims

1. A method for image generation without image control from unmanned aerial vehicle (UAV) images, characterized in that: Includes the following steps: Step 1: Preliminary preparation and parameter calibration, including: The camera intrinsic parameters were calibrated using a three-dimensional calibration field grid intrinsic parameter calibration method. Time synchronization calibration of GNSS, laser, IMU, and camera; Perform POS eccentricity correction preprocessing for IMU pose; Flight route planning is conducted based on the survey area and terrain features. Step 2: Data acquisition. Simultaneously acquire airborne aerial digital imagery, airborne aerial LiDAR data, GNSS raw data, IMU data, and POS data, and record the GNSS timestamp of the exposure time. Step 3: Data preprocessing, including: Based on the three-dimensional calibration field grid calibration scheme, the parameters of the camera intra-field grid distortion correction model are calculated, and pixel-by-pixel distortion correction and image enhancement are performed on the image. A POS solution scheme that integrates PPK and IMU multi-sensor data is adopted. By fusing GNSS and IMU data through extended Kalman filtering, high-precision position, velocity, and attitude information are obtained. POS eccentricity correction is performed based on IMU pose and GNSS antenna position to calculate accurate photo pose information; Step 4: 3D reconstruction and core mapping process, including: Multi-scale fusion and color balancing processing of the image; The SIFT feature matching algorithm is used for feature point extraction and matching, and the RANSAC method is used to remove mismatched points to obtain the correct matching pairs. A method for image-free aerial triple densification based on laser point cloud, GNSS, and IMU assistance achieves accurate registration between imagery and laser point cloud through projection matching, point-area correlation, and hierarchical beam adjustment optimization. A dense point cloud is generated using semi-global matching (SGM) and then fused with a laser point cloud. True-color orthophotos of DSM and TDOM are generated based on fused point clouds.

2. The method for image processing without image control for UAV images according to claim 1, characterized in that: Step 1, specifically the time synchronization calibration of GNSS with the laser, IMU, and camera, includes: At the hardware level, the airborne time synchronization unit outputs a trigger pulse signal of a uniform frequency to control the GNSS, IMU, camera and laser to start data acquisition synchronously. The reference anchoring layer uses the second pulse signal output by GNSS as the absolute time reference and records the corresponding UTC timestamp; At the software level, the deviation between the timestamp of each sensor data frame and the PPS anchoring time is extracted, and the time deviation model is fitted using the least squares method to correct the original data timestamp.

3. The method for image processing without image control for UAV images according to claim 1, characterized in that: In step 3, the grid distortion correction model is constructed by establishing a mapping relationship between the observed coordinates and the corrected ideal coordinates using high-order polynomials. It includes interior orientation elements and three types of distortion parameters, and its mathematical expression is: ； in, For the observation coordinates, As the distortion center, For normalized coordinates, , , Radial distance, Radial distortion factor ,in Radial distortion coefficient; and For radial distortion, The eccentricity distortion coefficient, For affine / nonorthogonal coefficients, These are the coordinates of the image point after distortion correction.

4. The method for image processing without image control for UAV images according to claim 1, characterized in that: Step 3, POS eccentricity correction, includes: Convert the latitude, longitude, and altitude coordinates of the moment the photo was taken to geocentric coordinates; Convert geocentric coordinates to local horizontal coordinates; Convert local horizontal coordinates to body coordinates; Perform coordinate correction in the body coordinate system; The corrected coordinates are then converted back to latitude and longitude in reverse order.

5. The method for image processing without image control for UAV images according to claim 1, characterized in that: Step 4, multi-scale color fusion and homogenization, includes the following steps: The acquired photos were downsampled by a ratio of 1:8 to obtain photos with reduced resolution. Based on the down-resolution photo, combined with pose information and the average elevation of the survey area, digital differential correction is used to perform absolute orientation. After orientation, the images from each shooting station are processed using multi-scale fusion technology to perform multi-scale weighted fusion of pixel values ​​in overlapping areas, resulting in a color-balanced, scaled orthophoto of the survey area. Based on the geographical range of the current photo, the corresponding sub-image is extracted from the orthophoto and upsampled at a ratio of 1:

8. The upsampled image is used as a reference image, and the original image is color-matched using histogram matching to obtain a pre-processed photo with consistent color.

6. The method for image processing without image control for UAV images according to claim 1, characterized in that: Step 4 of the image-free spatial three-dimensional encryption method includes the following steps: Initial 3D points with scale are generated based on LiDAR point cloud projection; Incremental PnP and triangulation methods are used for image registration and 3D point expansion. Establish point-to-surface relationships between visual 3D points and LiDAR point cloud planes; Layered execution of incremental BA, batch BA and global BA, jointly optimizing visual reprojection error and LiDAR point-to-area distance error; Output the registered camera pose and scaled 3D point set.

7. The method for image processing without image control for UAV images according to claim 6, characterized in that: The objective function for joint optimization is: ； ； in, R represents the coordinates in the 3D point camera coordinate system. i Let t be the rotation matrix of the camera. i Let X be the translation vector of the camera. k The coordinates of the points in the visual 3D world coordinate system. For 3D point trajectory features, w is the LiDAR point weight, and n q Let l be the normal vector of the LiDAR plane. q For the corresponding LiDAR points.

8. The method for image processing without image control for UAV images according to claim 1, characterized in that: Step 4, point cloud fusion, includes the following steps: Calculate the survey area range and generate voxel blocks for the survey area range; The registered laser point cloud and the visually matched point cloud are mapped into voxels respectively; The point cloud within the voxel is sorted and clustered according to the Z coordinate value, and the mean of the middle position point set is taken as the fused point cloud.

9. An electronic device comprising a processor and a memory communicatively connected to the processor and used for storing processor-executable instructions, characterized in that: The processor is used to execute the method described in any one of claims 1-8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the method described in any one of claims 1-8.

Images