An unmanned aerial vehicle automatic positioning method and system based on deep learning image registration

Abstract

The application discloses an unmanned aerial vehicle automatic positioning method and system based on deep learning image registration, and belongs to the field of remote sensing science and technology. The method comprises the following steps: acquiring unmanned aerial vehicle images and satellite base maps with geographic reference information, and rotating and normalizing the unmanned aerial vehicle images by using an airborne heading angle; constructing a multi-scale pyramid on the satellite base map, extracting a candidate region by sliding window, and performing geometric consistency scoring based on deep features to screen out an optimal matching region and realize coarse positioning; extracting and matching feature points in the optimal region by using a deep sparse feature matching network, solving a homography matrix through robust estimation, and obtaining an accurate pixel position; for unmanned aerial vehicle sequence images, a search window is predicted based on a motion model, and confidence evaluation is performed to realize time sequence position inheritance and optimization; and the pixel position is solved into WGS84 latitude and longitude coordinates by using an affine transformation matrix. The application significantly improves the positioning robustness and engineering practicability in a complex scene.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing science and technology, specifically relating to an automatic localization method and system for unmanned aerial vehicles (UAVs) based on deep learning image registration. Background Technology

[0002] Due to their mobility, low cost, and high resolution, unmanned aerial vehicles (UAVs) have become an important tool for remote sensing observation and geographic information acquisition. To integrate UAV imagery into a geographic information system (GIS), precise spatial registration between the imagery and the geographic reference map is required to determine its accurate location in the absolute geographic coordinate system.

[0003] Traditional UAV georeferencing relies primarily on external positioning devices or ground control points. The former uses onboard GPS / IMU to directly acquire location information, but consumer-grade devices suffer from error accumulation and drift issues, resulting in positioning accuracy typically only at the level of a few meters. The latter involves manually mapping ground control points for geometric correction, which can achieve higher accuracy, but the deployment and measurement process is time-consuming and labor-intensive, making it difficult to implement in disaster areas or complex terrain.

[0004] Content-based automatic registration techniques have become a research hotspot due to their low cost and high automation potential. These methods are mainly divided into three categories: feature matching-based methods, region correlation-based methods, and deep learning-based methods.

[0005] Feature-matching methods such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up Robust Features), and ORB (Oriented Fast and Rotated BRIEF) achieve registration by extracting local features and estimating geometric transformation parameters. However, in cross-view registration of UAV and satellite imagery, there are significant differences in resolution and scale, as well as obvious rotation and viewpoint deviations. Traditional manually designed features are difficult to extract common descriptors, and the accuracy and stability are insufficient to meet engineering requirements.

[0006] Region-based methods, such as NCC (Normalized Cross-Correlation) and mutual information, use sliding window search and similarity metrics for matching. However, these methods are computationally complex, sensitive to rotation and lighting, lack semantic robustness, and struggle to handle complex situations involving changes in both scale and orientation.

[0007] In recent years, deep learning methods such as SuperPoint, SuperGlue, and LoFTR have been introduced into feature extraction and matching tasks, overcoming to some extent the limitations of lighting variations and weakly textured scenes. However, many problems still exist in the practical application of UAV-satellite imagery: large scene differences lead to a decrease in model generalization ability; there is a lack of modeling for arbitrary heading angles and scale differences of UAVs; for video sequences, existing algorithms use frame-by-frame independent matching, which cannot utilize temporal correlation, resulting in computational redundancy and trajectory jitter; in addition, most algorithms only output pixel coordinates without combining them with geographic projection parameters, making it difficult to directly serve GIS applications.

[0008] In conclusion, current technologies have not yet achieved truly automated, high-precision, and real-time UAV geolocation. Summary of the Invention

[0009] To address the aforementioned technical issues, this invention provides an automatic UAV localization method and system based on deep learning image registration. It eliminates directional differences through heading-aware rotation normalization, achieves coarse-to-fine registration by combining multi-scale pyramid coarse localization with deep feature fine matching, and utilizes a temporal position inheritance mechanism to track and optimize video sequences, ultimately calculating and outputting geographic coordinates.

[0010] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0011] An automatic localization method for unmanned aerial vehicles (UAVs) based on deep learning image registration, the method comprising:

[0012] Step 1: Acquire the original UAV image and the satellite reference base map image as a geographic benchmark, and perform preprocessing to align the orientation of the original UAV image and the satellite reference base map image.

[0013] Step 2: Construct a multi-scale pyramid on the preprocessed satellite reference base map image, extract the candidate region set through sliding window, calculate the geometric consistency score between each candidate region and the preprocessed UAV original image, and select the best matching candidate region with the highest score.

[0014] Step 3: Within the optimal matching candidate region, a deep sparse feature matching network is used to extract and match feature points. The homography matrix is ​​solved through robust estimation to obtain the precise pixel position of the preprocessed UAV original image in the preprocessed satellite reference base map image.

[0015] Step 4: Using the affine transformation matrix of the preprocessed satellite reference base map image, convert the precise pixel position into projection plane coordinates, and then inversely project it to the WGS84 coordinate system to output the latitude and longitude of the current geographical location of the UAV.

[0016] Furthermore, the preprocessing in step 1 includes: converting the satellite reference base map image to the WGS84 / UTM projection coordinate system and calculating the projection zone number; downsampling the satellite reference base map image to a spatial resolution similar to the original UAV image using bilinear interpolation; performing radiometric calibration and Z-score normalization on the original UAV image and the satellite reference base map image; constructing a rotation matrix using the airborne heading angle to inversely rotate the original UAV image, and aligning it with the true north direction of the satellite reference base map image through interpolation resampling.

[0017] Furthermore, step 2 includes:

[0018] A multi-level image pyramid is constructed on the preprocessed satellite reference base map. At each level, a sliding window is used to extract candidate regions with a fixed stride and window size. A deep convolutional network with shared weights is used to extract the depth features of each candidate region and the preprocessed UAV original image. An initial matching pair set is established through nearest neighbor search. The local homography matrix is ​​estimated using the initial matching pairs. The number of matching points with reprojection errors less than a preset threshold is calculated as the geometric consistency score. The candidate region with the highest score is selected as the optimal matching candidate region.

[0019] Furthermore, the deep sparse feature matching network in step 3 includes two subnetworks: feature extraction and feature matching. The feature extraction subnetwork uses a pre-trained point feature detector to map the image to a high-dimensional feature space, extracting the key point set and L2-normalized descriptors. The feature matching subnetwork uses a multilayer perceptron to process the nonlinear relationships between features, and generates an initial matching set through a bidirectional mutual nearest neighbor strategy and similarity threshold filtering. The RANSAC algorithm is used to perform robust homography estimation on the initial matching set, and interior points are filtered by iteratively sampling the minimum subset. Finally, a nonlinear least squares problem is constructed to optimize and solve the homography matrix.

[0020] Furthermore, the method also includes: for subsequent frames in the UAV image sequence, the current frame search center is initialized as the superposition of the previous frame position and the motion velocity vector based on the constant velocity motion model, and the feature matching search range is limited to a local window around the prediction center; the matching confidence of the current frame is evaluated according to the inlier rate calculation formula; if the inlier rate is lower than a preset threshold, tracking failure is determined and a failure counter is started; when multiple consecutive frames fail, global relocalization is triggered, and the process switches to step 2 to perform a global search again.

[0021] Furthermore, the formula for calculating the inlier rate is: the inlier rate equals the number of inliers that satisfy the reprojection error being less than a preset threshold divided by the total number of matching points; the reprojection error is the Euclidean distance between the matching point and the target point after homography matrix transformation.

[0022] Furthermore, in step 4, the parameters of the affine transformation matrix are directly parsed from the georeferenced information attached to the satellite reference base map image. The affine transformation matrix includes a first parameter and a fifth parameter representing spatial resolution, a second parameter and a fourth parameter representing rotation, and a zeroth parameter and a third parameter representing translation offset. The precise pixel coordinates are converted into projection plane coordinates through matrix multiplication. Then, according to the projection coordinate system type defined in the base map metadata, the corresponding inverse projection operator is dynamically called to convert the plane projection coordinates into longitude and latitude in the WGS84 coordinate system.

[0023] On the other hand, the present invention provides an automatic localization system for unmanned aerial vehicles based on deep learning image registration, comprising:

[0024] The preprocessing module is used to acquire the original UAV image and the satellite reference base map image as a geographic benchmark and perform preprocessing to align the orientation of the original UAV image and the satellite reference base map image.

[0025] The filtering module is used to construct a multi-scale pyramid on the preprocessed satellite reference base map image, extract the candidate region set through sliding window, calculate the geometric consistency score between each candidate region and the preprocessed UAV original image, and filter out the best matching candidate region with the highest score.

[0026] The solution module is used to extract and match feature points in the optimal matching candidate region using a deep sparse feature matching network, solve the homography matrix through robust estimation, and obtain the precise pixel position of the preprocessed UAV original image in the preprocessed satellite reference base map image.

[0027] The location output module is used to convert the precise pixel position into projection plane coordinates using the affine transformation matrix of the preprocessed satellite reference base map image, and then inversely project it to the WGS84 coordinate system to output the latitude and longitude of the current geographical location of the UAV.

[0028] Thirdly, the present invention provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method for automatic localization of unmanned aerial vehicles based on deep learning image registration.

[0029] Fourthly, the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enable the processor to implement the aforementioned method for automatic localization of unmanned aerial vehicles based on deep learning image registration.

[0030] The beneficial effects of this invention are as follows:

[0031] Highly efficient large-scale search and localization capability: To address the problem that existing algorithms struggle to efficiently locate data in ultra-large-scale reference images, this invention designs a multi-level pyramid image generation and adaptive candidate region selection mechanism. By guiding the local fine matching of low-level features through global coarse localization of top-level features, the complexity of the search space is significantly reduced, realizing the transformation from "global traversal" to "local optimization," and greatly improving the speed and stability of large-scale image registration.

[0032] Highly robust deep feature matching: Compared with the shortcomings of traditional handmade features which are easily affected by changes in lighting and viewpoint, a learnable deep feature extraction and matching network is introduced to learn high-dimensional semantic features with invariance to lighting, viewpoint and scale from end to end. This enables the automatic optimization of feature point alignment parameters, ensuring matching stability in complex field environments and reducing reliance on human intervention.

[0033] Real-time temporal tracking optimization: For UAV video sequences, an innovative sequence position inheritance and motion constraint model is proposed. The position information of historical frames is used to predict the search window of the current frame, which significantly reduces the search space of feature matching from the global to the local region. An interior point rate confidence evaluation mechanism is introduced to greatly reduce computational redundancy while ensuring accuracy and effectively suppress matching trajectory jitter.

[0034] Engineered real-time deployment capability: To meet the real-time requirements of UAV online navigation, the algorithm is deployed on a portable AI development board. Through system-level optimizations such as model quantization and multi-threaded pipeline asynchronous processing, the system achieves real-time registration and geographic coordinate output of UAV images during flight, significantly improving the system's response speed and field availability.

[0035] Complete geolocation closed loop: Establish a full-link mapping from "UAV image pixels" to "base map pixels" to "projected plane coordinates" and finally output "WGS84 latitude and longitude", so as to output high-precision absolute geolocation results through visual observation only in the absence of GNSS environment, which can directly serve GIS applications and UAV navigation systems. Attached Figure Description

[0036] Figure 1 This is a flowchart of an automatic localization method for unmanned aerial vehicles based on deep learning image registration according to the present invention.

[0037] Figure 2 This is a schematic diagram of the point feature extraction and matching network structure based on deep learning in this invention;

[0038] Figure 3 This is a schematic diagram of the TOP-5 candidate regions generated in large-scale remote sensing images according to the present invention;

[0039] Figure 4This is a schematic diagram of the point matching results based on the optimal candidate region according to the present invention;

[0040] Figure 5 This is a schematic diagram showing the geographic location of the UAV imagery in large-scale remote sensing imagery and the output coordinate log file.

[0041] Figure 6 This is a diagram illustrating the effect of implementing the present invention. Detailed Implementation

[0042] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0043] like Figure 1 As shown, this invention provides an automatic UAV localization method based on deep learning image registration. It establishes a pyramid-shaped multi-scale search mechanism that requires no manual intervention. The method constructs a multi-resolution pyramid on a reference image and quantitatively scores candidate regions by calculating a matching score function, thereby automatically selecting the region with the highest matching probability. This mechanism divides the registration process into three levels: "candidate region generation—region scoring—fine matching," achieving adaptive search from coarse to fine, effectively reducing computational complexity, and significantly improving the stability and speed of large-scale image registration. Specifically, the method includes:

[0044] Step 1, Data Acquisition and Preprocessing: Acquire the raw UAV image sequence and the satellite reference base map image as a geographic benchmark, and perform preprocessing to align the orientation of the raw UAV image and the satellite reference base map image; wherein,

[0045] The input data used in this method mainly includes: the original image sequence acquired by the UAV's onboard terminal, the reference base map image used as a geographic benchmark, the metadata and extrinsic parameters corresponding to the images, and auxiliary geographic information data. The specific definitions are as follows:

[0046] Raw UAV imagery: High-resolution image sequences acquired by optical sensors mounted on multi-rotor or fixed-wing UAVs. Data formats include, but are not limited to, JPEG, TIFF, or H.264 / H.265 video stream frames. The images have high spatial resolution (GSD ground sampling distance of 0.05m-0.5m), and due to changes in flight attitude, there are varying degrees of overlap and viewpoint deviation between image frames.

[0047] Reference base map imagery: Large-scale orthorectified imagery (satellite optical or aerial imagery) containing georeferenced information (GeoTransform, projected WKT). Large-scale orthorectified imagery (DOM) covering the mission area is typically derived from satellite remote sensing (such as the Gaofen series) or high-altitude aerial photography.

[0048] Image metadata and extrinsic parameters: The reference base map image must have accurate geographic coordinate reference information (such as GeoTIFF format, containing six GeoTransform parameters or RPC rational polynomial coefficients), and the coordinate system is preferably WGS84 latitude and longitude or UTM projected coordinate system. For raw UAV imagery, coarse extrinsic parameters (initial longitude, latitude, and heading angle) can optionally be provided. These parameters are used to provide initial prior estimates for registration.

[0049] Auxiliary geographic information data: Optionally provided, including Digital Elevation Model (DEM / DSM) or Land Cover Mask. In the algorithm, the auxiliary data is mapped to a feature extraction network to generate an attention mask, which automatically suppresses feature responses from low-texture, easily confused areas such as water bodies or moving vehicles, thereby improving the robustness of matching.

[0050] To eliminate distribution differences among multi-source images caused by variations in sensor, coordinate system definition, and shooting angle, and to ensure the numerical stability and spatial consistency of the matching, this invention performs the following preprocessing steps before feature extraction:

[0051] (1) Coordinate and pixel scale are unified

[0052] Transform the reference base map image to the coordinate system required for the work (preferably WGS84 / UTM projection). Read the affine transformation matrix of the reference base map image and calculate the projection zone number. If a projection coordinate system is selected, it needs to be projected according to the central longitude to determine its corresponding UTM projection zone number.

[0053] (1)

[0054] in, Represents the longitude value of the image center point. This indicates the floor function. This represents the UTM projection zone number. Furthermore, based on the estimated ground resolution of the original UAV imagery, the reference base map imagery is downsampled to a spatial resolution similar to the original UAV imagery using bilinear interpolation, thus constructing a matching space at the same scale.

[0055] (2) Radiation intensity calibration and normalization

[0056] To suppress sensor noise and adapt to the input distribution requirements of deep learning models, the pixel values ​​DN of the original UAV images are first linearly radiometrically calibrated and converted into physical quantities:

[0057] (2)

[0058] in, For spectral radiance, The gain coefficient of the sensor. This is the bias coefficient. Subsequently, for... Perform Z-Score standardization:

[0059] (3)

[0060] in, and These are the pixel mean and standard deviation of the current image, respectively. The input is a standardized tensor for the depth feature extraction network. The UAV imagery, after radiation intensity calibration and normalization, further enters the heading-aware rotation normalization step.

[0061] (3) Heading perception rotation normalization

[0062] To address the drastic rotational differences caused by arbitrary UAV flight direction, a rotation matrix is ​​constructed using the heading angle θ (defined as the angle between the UAV's nose and true north) obtained from an onboard compass or IMU. The normalized UAV imagery is then inversely rotated to align with true north, achieving orientation alignment with the satellite reference base imagery. The coordinate mapping relationship is as follows:

[0063] (4)

[0064] in, Here, u and v represent the image coordinates of a pixel in the original UAV image, where u and v represent the horizontal and vertical coordinates of the pixel in the image coordinate system, respectively. These are the pixel coordinates after heading correction. In practice, the rotated pixel values ​​are resampled using bilinear interpolation or bicubic interpolation, and zero-padding is applied to the blank areas created by the rotation. The UAV image after rotation normalization and resampling is used as the image to be registered in the subsequent coarse localization and fine matching steps.

[0065] Step 2, Pyramid Candidate Region Generation and Coarse Localization: Construct a multi-scale pyramid on the preprocessed satellite reference base map image, extract the candidate region set through sliding window, use a deep feature extraction network to calculate the geometric consistency score between each candidate region and the preprocessed UAV original image, and select the best matching candidate region with the highest score as the coarse localization result.

[0066] To address the issues of scale differences between UAV imagery and large-scale reference base maps, as well as the excessively large search space, this invention proposes a sliding window search and geometric consistency verification strategy based on a multi-scale pyramid.

[0067] Let the preprocessed satellite reference base map image be (The superscript indicates the length, width, and number of channels; the same applies below.) The UAV image to be registered (i.e., the pre-processed original UAV image) is... The specific implementation steps are as follows:

[0068] (1) Multi-scale pyramid construction and slice generation

[0069] First, the preprocessed satellite reference base map image Cropping is performed to construct an image pyramid with L levels. The scale of the l-th layer is 1 / 2l of the original image. Then, at the l-th pyramid level... The above uses a sliding window with step size S and window size t to extract candidate regions. :

[0070] (5)

[0071] in, The pixel coordinates of the top-left corner of the candidate region window in the hierarchical image. To intercept the first layer at layer l Candidate regions.

[0072] (2) Geometric consistency scoring based on deep features

[0073] The first one intercepted at the lth layer Candidate regions With the drone image to be registered A deep convolutional network with shared weights is used to extract features, and then an initial set of matching pairs is established through nearest neighbor search. ,in They are respectively and The coordinates of the feature points are given, k is the index of the matching pair, and N is the total number of matching pairs in the initial matching pair set. Without loss of generality, for the candidate region set... The aforementioned calculations are performed for each candidate region.

[0074] To robustly evaluate matching quality, the initial local homography matrix is ​​estimated using the initial set of matched pairs. And calculate the geometric consistency score. The scoring formula is defined as follows:

[0075] (6)

[0076] in, This is a confidence index function, which takes the value of 1 when the condition in parentheses is met, and 0 otherwise. Representing the homogeneous coordinates of a point, is the reprojection error threshold, k is the matching pair index, and N is the total number of initial matching pairs corresponding to the current candidate region.

[0077] (3) Global optimal candidate strategy

[0078] Essentially, this process involves finding the sub-region that maximizes geometric confidence. The algorithm can retain the top-K candidate regions with the highest scores as initial values ​​for refining the search, but in this embodiment, the focus is on the globally optimal candidate region. :

[0079] (7)

[0080] in, This is considered the optimal candidate region that achieves the highest score. This refers to the approximate geographic location area deemed the best match for the UAV imagery to be registered. This optimal matching candidate area... This will serve as the target for the next stage of fine-grained registration, thereby transforming the global search problem into a local optimization problem and significantly reducing computational complexity. For example... Figure 3 As shown in the figure, the top 5 candidate regions obtained by screening large-scale remote sensing images according to the method of the present invention are illustrated. The green boxes indicate candidate regions with higher scores.

[0081] Step 3, Fine registration and robust geometric estimation: Within the optimal matching candidate region, a deep sparse feature matching network is used to extract and match feature points. The homography matrix is ​​solved through robust estimation to obtain the precise pixel position of the preprocessed UAV original image in the preprocessed satellite reference base map image.

[0082] To establish drone imagery to be registered Matching candidate regions To address the geometric mapping relationship between features, this invention proposes a computational process based on sparse feature matching. This process mainly includes three stages: feature extraction, similarity measurement, and geometric verification based on RANSAC (Random Sample Consensus).

[0083] (1) Sparse feature extraction and similarity measurement

[0084] Using pre-trained point feature detectors This maps the image to a high-dimensional feature space. For the optimal matching candidate region... and drone images to be registered They are mapped to a high-dimensional feature space to extract keypoint sets and descriptors:

[0085] (8)

[0086] (9)

[0087] Where Nr and Nu represent the optimal matching candidate region and the number of keypoints detected in the UAV image to be registered, respectively; D represents the channel dimension of the feature descriptor; and ki and kj represent the indices of the keypoints in the two images, respectively. They represent the optimal matching candidate regions respectively. The coordinates and descriptor of the ki-th key point, These represent the drone images to be registered. The coordinates of the kj-th keypoint and its corresponding descriptor are given. The descriptor is L2 normalized to ensure that the feature vectors are distributed on the unit hypersphere.

[0088] To establish the correspondence between features, we define the matching matrix as M, and use feature descriptors to match the set of key points:

[0089] (10)

[0090] in, To control the temperature parameters for similarity determination, The · sign represents the activation function, acting as a non-linear mapping and normalization mechanism. The MLP(·) is a multilayer perceptron that handles the non-linear relationships between features and learns more abstract feature representations. M is the matching matrix. This represents the optimal matching candidate region. The m-th key point descriptor With the drone image to be registered The nth key point descriptor The matching score between the two images is given, where m and n are the indices of key points in the two images, respectively.

[0091] To eliminate erroneous matches, this invention employs a combination of a bidirectional nearest neighbor strategy and threshold filtering to generate an initial matching set. :

[0092] (11)

[0093] in, The similarity threshold; Indicates the optimal matching candidate region The coordinates of the m-th key point in the middle. Indicates the drone image to be registered The coordinates of the nth keypoint are given; m and n are the keypoint indices in the two images, respectively. This constraint ensures that feature point pairs are the best match for each other in the other's image, and the similarity is higher than the confidence threshold, thus significantly reducing the "many-to-one" false match rate. Figure 4 The figure shows the point matching results based on the optimal candidate region according to the present invention.

[0094] (2) Robust homography estimation based on RANSAC

[0095] like Figure 2 As shown, due to the initial matching set Mismatches caused by disparity or repetitive textures still inevitably exist, and direct regression of geometric parameters will lead to solution shifts. Therefore, this invention employs the RANSAC algorithm for robust estimation. This process consists of two stages:

[0096] Phase 1: Hypothesis Validation and Interior Point Selection. Iteratively sample the minimum subset (4 pairs of points) to calculate the homography matrix hypothesis, count the number of interior points that satisfy the projection error threshold, and select the model with the most interior points as the initial solution.

[0097] Phase Two: Nonlinear Refinement. A nonlinear least squares problem is constructed based on the selected points. The global optimum is found through iterative optimization by minimizing the robust reprojection error. The final homography matrix is ​​defined as:

[0098] (12)

[0099] In this formula, This represents the final homography matrix. This represents the homogeneous coordinates of the matching points in the UAV image to be registered. This is a perspective division function used to transform homogeneous coordinates. Map back to Euclidean space. As a robust kernel function, compared to the standard squared error, the kernel function can effectively reduce the weight of residual large error points in the calculation process, ensuring the numerical stability of the geometric solution in complex scenarios.

[0100] Without loss of generality, for subsequent UAV image frames, the present invention performs sequence position inheritance and time consistency constraints: based on the precise pixel position and constant velocity motion model, the local search window of the current frame is predicted, fine registration is performed only within the local search window, and global relocalization is triggered based on the matching confidence.

[0101] Considering drone image sequences It exhibits significant temporal continuity, with geographical location changes between adjacent frames typically being smooth and limited. To avoid computationally expensive global searches in each frame, this invention proposes an adaptive sequence position inheritance mechanism based on motion priors.

[0102] (1) Motion prediction and search window initialization

[0103] Let the center of the target region determined after successful matching in the previous frame (time t-1) be... Based on the constant velocity motion model, the search center of the current frame (time t) is determined. Initialize to the sum of the previous position and the motion increment:

[0104] (13)

[0105] in, This represents the translational velocity vector estimated based on historical trajectories. This strategy expands the search scope for feature matching from the entire preprocessed satellite reference base map image. Significantly scaled to local window centered .because The search space complexity of feature matching is from Significantly reduced to This greatly improves the real-time performance of the algorithm.

[0106] (2) Confidence assessment and failure recovery based on internal point ratio

[0107] However, local search relies on the accuracy of motion prediction. To prevent tracking drift caused by violent maneuvers, missing textures, or prediction errors, this invention introduces the interior point rate. As a confidence index, a reprojection error threshold is set. With confidence threshold The following feedback strategy was designed:

[0108] Tracking successful ): Determine if the current frame match is reliable. Accept the current solution result and update the position. And refresh the velocity vector It is used for prediction of the next frame.

[0109] Tracking failure ( If the current frame match is deemed unreliable, the system discards the motion estimate for that frame while maintaining the velocity vector from the previous frame. Simultaneously, a failure counter is activated. If multiple consecutive frames fail to match, the algorithm automatically triggers a relocation mechanism, switching back to global search mode and relocating within the entire domain to ensure system robustness over extended periods.

[0110] Among them, the interior point rate The calculation method is as follows:

[0111] (14)

[0112] in, This represents the total number of matching points. To ensure that the number of interior points whose reprojection error is less than a preset threshold.

[0113] Step 4: Geographic coordinate inverse calculation and closed-loop output; Using the affine transformation matrix of the preprocessed satellite reference base map image, the precise pixel position is converted into projection plane coordinates, and then inversely projected to the WGS84 coordinate system to output the latitude and longitude of the current geographical location of the UAV.

[0114] The UAV's current field of view center was determined in the preprocessed satellite reference image using the aforementioned feature matching and sequence tracking algorithms. Precise pixel coordinates Finally, the system's ultimate goal is to restore its absolute geographical location in the real world.

[0115] Preprocessed satellite reference base map image (i.e., baseline base map) It pre-includes geo-reference information, and the mapping relationship between its pixel coordinate system and the projected map coordinate system (such as UTM projection) is determined by the affine transformation matrix. describe:

[0116] (15)

[0117] in, , Characterizes spatial resolution (pixels per meter). , Characterizing the rotation term, , For translation offset. In the implementation, the matrix The parameters are directly parsed from the georeferenced information attached to the base map, ensuring strict alignment of the mapping relationship. Based on this, the UAV's current projected map coordinates... It can be solved directly by matrix multiplication:

[0118] (16)

[0119] To ensure compatibility with General Navigation Satellite Systems (GNSS), an inverse map projection function is further introduced. This module dynamically calls the corresponding inverse projection operator based on the projection coordinate system type (such as UTM zone) defined in the base map metadata, converting the planar projection coordinates into longitude in the geodetic coordinate system. with latitude :

[0120] (17)

[0121] This completes the end-to-end mapping from "UAV image pixels" to "reference map pixels," then to "projected plane coordinates," and finally to "WGS84 latitude and longitude." This mechanism enables the system to output high-precision absolute geolocation results solely through visual observation in GNSS-free environments. Figure 5 As shown, this is the geographical location of UAV imagery in large-scale remote sensing imagery. The red box in the left imagery represents the projected location of the UAV imagery, and the right imagery is the output coordinate log file.

[0122] To meet the stringent requirements of low latency and low power consumption for online navigation of unmanned aerial vehicles (UAVs), this invention deploys the proposed positioning system on an embedded edge computing platform (e.g., a computing device based on an ARM+GPU architecture). Through hardware and software co-design, system-level optimizations were performed at the following three levels:

[0123] (1) Lightweight Model and Mixed-Precision Inference: For the computationally intensive feature extraction module, a mixed-precision quantization strategy of half-precision (FP16) and INT8 was adopted. Specifically, the TensorRT inference engine was used to perform sensitivity analysis on the network layers, retaining FP16 precision for high-level semantic features and using INT8 for low-level texture feature layers. This significantly reduced memory usage and inference latency while maintaining feature discriminative power.

[0124] (2) Multi-threaded asynchronous pipeline processing: Image acquisition, pyramid candidate region generation, feature matching calculation, geographic coordinate inverse calculation, and log write-back are decoupled into independent parallel threads. For candidate block evaluation, a GPU batch processing mechanism is adopted to maximize the hardware's parallel throughput and effectively mask the latency caused by data transmission.

[0125] (3) Search pruning based on temporal prior: Combining the continuity of UAV flight, a sequence position inheritance mechanism is proposed. The system uses a velocity model to predict the center of the current frame based on the position of the previous frame, strictly limiting the search space for feature matching to a local region. Global search and relocalization is only triggered when registration fails for a long time.

[0126] To verify the effectiveness of this mechanism, a comparative experiment was further conducted. The experiment used 100 consecutive UAV images as the test objects, testing the average processing time under two methods: "search pruning based on temporal prior" and "no mechanism, frame-by-frame independent global search". The results show that with search pruning based on temporal prior, the average processing time per image is 15.50 s; without this mechanism, the average processing time per image is 54.1589 s. Compared to the frame-by-frame global search method, the average processing time is reduced by approximately 71.4%, significantly improving the processing efficiency in continuous UAV localization tasks. Figure 6 The center dot represents the projection of the UAV position on the reference large image of several key frames in a continuous flight sequence. The positioning center of adjacent frames shows a continuous and smooth change trend along the flight direction, indicating that it is reasonable and effective to use the positioning result of the previous frame as the local search prior of the current frame.

[0127] On the other hand, the present invention provides an automatic localization system for unmanned aerial vehicles (UAVs) based on deep learning image registration, which includes modules capable of implementing the steps of the aforementioned method, specifically including:

[0128] The preprocessing module is used to acquire the original UAV image and the satellite reference base map image as a geographic benchmark and perform preprocessing to align the orientation of the original UAV image and the satellite reference base map image.

[0129] The filtering module is used to construct a multi-scale pyramid on the preprocessed satellite reference base map image, extract the candidate region set through sliding window, calculate the geometric consistency score between each candidate region and the preprocessed UAV original image, and filter out the best matching candidate region with the highest score.

[0130] The solution module is used to extract and match feature points in the optimal matching candidate region using a deep sparse feature matching network, solve the homography matrix through robust estimation, and obtain the precise pixel position of the preprocessed UAV original image in the preprocessed satellite reference base map image.

[0131] The location output module is used to convert the precise pixel position into projection plane coordinates using the affine transformation matrix of the preprocessed satellite reference base map image, and then inversely project it to the WGS84 coordinate system to output the latitude and longitude of the current geographical location of the UAV.

[0132] Thirdly, the present invention provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method for automatic localization of unmanned aerial vehicles based on deep learning image registration.

[0133] Fourthly, the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enable the processor to implement the aforementioned method for automatic localization of unmanned aerial vehicles based on deep learning image registration.

[0134] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An automatic localization method for unmanned aerial vehicles (UAVs) based on deep learning image registration, characterized in that, The method includes: Step 1: Acquire the original UAV image and the satellite reference base map image as a geographic benchmark, and perform preprocessing to align the orientation of the original UAV image and the satellite reference base map image. Step 2: Construct a multi-scale pyramid on the preprocessed satellite reference base map image, extract the candidate region set through sliding window, calculate the geometric consistency score between each candidate region and the preprocessed UAV original image, and select the best matching candidate region with the highest score. Step 3: Within the optimal matching candidate region, a deep sparse feature matching network is used to extract and match feature points. The homography matrix is ​​solved through robust estimation to obtain the precise pixel position of the preprocessed UAV original image in the preprocessed satellite reference base map image. Step 4: Using the affine transformation matrix of the preprocessed satellite reference base map image, convert the precise pixel position into projection plane coordinates, and then inversely project it to the WGS84 coordinate system to output the latitude and longitude of the current geographical location of the UAV.

2. The automatic localization method for unmanned aerial vehicles based on deep learning image registration according to claim 1, characterized in that, The preprocessing in step 1 includes: converting the satellite reference base map image to the WGS84 / UTM projection coordinate system and calculating the projection zone number; downsampling the satellite reference base map image to a spatial resolution similar to the original UAV image using bilinear interpolation; performing radiometric calibration and Z-score normalization on the original UAV image and the satellite reference base map image; constructing a rotation matrix using the airborne heading angle to inversely rotate the original UAV image, and aligning it with the true north direction of the satellite reference base map image through interpolation resampling.

3. The automatic localization method for unmanned aerial vehicles based on deep learning image registration according to claim 1, characterized in that, Step 2 includes: A multi-level image pyramid is constructed on the preprocessed satellite reference base map. At each level, a sliding window is used to extract candidate regions with a fixed stride and window size. A deep convolutional network with shared weights is used to extract the depth features of each candidate region and the preprocessed UAV original image. An initial matching pair set is established through nearest neighbor search. The local homography matrix is ​​estimated using the initial matching pairs. The number of matching points with reprojection errors less than a preset threshold is calculated as the geometric consistency score. The candidate region with the highest score is selected as the optimal matching candidate region.

4. The automatic localization method for unmanned aerial vehicles based on deep learning image registration according to claim 1, characterized in that, The deep sparse feature matching network in step 3 includes two subnetworks: feature extraction and feature matching. The feature extraction subnetwork uses a pre-trained point feature detector to map the image to a high-dimensional feature space and extracts the key point set and L2-normalized descriptors. The feature matching subnetwork uses a multilayer perceptron to process the nonlinear relationship between features and generates an initial matching set through a bidirectional mutual nearest neighbor strategy and similarity threshold filtering. The RANSAC algorithm is used to perform robust homography estimation on the initial matching set, and interior points are filtered by iteratively sampling the minimum subset. Finally, a nonlinear least squares problem is constructed to optimize and solve the homography matrix.

5. The automatic localization method for unmanned aerial vehicles based on deep learning image registration according to claim 1, characterized in that, The method further includes: for subsequent frames in the UAV image sequence, the current frame search center is initialized as the superposition of the previous frame position and the motion velocity vector based on the constant velocity motion model, and the feature matching search range is limited to a local window around the prediction center; the matching confidence of the current frame is evaluated according to the inlier rate calculation formula; if the inlier rate is lower than a preset threshold, the tracking is determined to be failed and the failure counter is started; when multiple consecutive frames fail, global relocalization is triggered, and the process switches to step 2 to perform a global search again.

6. The automatic localization method for unmanned aerial vehicles based on deep learning image registration according to claim 5, characterized in that, The formula for calculating the inlier rate is: the inlier rate is equal to the number of inliers that satisfy the reprojection error being less than a preset threshold divided by the total number of matching points; the reprojection error is the Euclidean distance between the matching point and the target point after homography matrix transformation.

7. The automatic localization method for unmanned aerial vehicles based on deep learning image registration according to claim 1, characterized in that, In step 4, the parameters of the affine transformation matrix are directly parsed from the georeferenced information attached to the satellite reference base map image. The affine transformation matrix includes the first and fifth parameters representing spatial resolution, the second and fourth parameters representing rotation, and the zeroth and third parameters representing translation offset. The precise pixel coordinates are converted into projection plane coordinates through matrix multiplication. Then, according to the projection coordinate system type defined in the base map metadata, the corresponding inverse projection operator is dynamically called to convert the plane projection coordinates into longitude and latitude in the WGS84 coordinate system.

8. An automatic localization system for unmanned aerial vehicles (UAVs) based on deep learning image registration, characterized in that, include: The preprocessing module is used to acquire the original UAV image and the satellite reference base map image as a geographic benchmark and perform preprocessing to align the orientation of the original UAV image and the satellite reference base map image. The filtering module is used to construct a multi-scale pyramid on the preprocessed satellite reference base map image, extract the candidate region set through sliding window, calculate the geometric consistency score between each candidate region and the preprocessed UAV original image, and filter out the best matching candidate region with the highest score. The solution module is used to extract and match feature points in the optimal matching candidate region using a deep sparse feature matching network, solve the homography matrix through robust estimation, and obtain the precise pixel position of the preprocessed UAV original image in the preprocessed satellite reference base map image. The location output module is used to convert the precise pixel position into projection plane coordinates using the affine transformation matrix of the preprocessed satellite reference base map image, and then inversely project it to the WGS84 coordinate system to output the latitude and longitude of the current geographical location of the UAV.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When one or more programs are executed by the one or more processors, the one or more processors implement the UAV automatic localization method based on deep learning image registration as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores executable instructions that, when executed by a processor, enable the processor to implement the UAV automatic localization method based on deep learning image registration as described in any one of claims 1-7.

Images