A method for quickly geolocating and splicing images in a sparse scene

Abstract

The present application provides a kind of image fast geographic positioning splicing method under sparse scene, it is related to image splicing technical field;First, the original image sequence and carrier satellite positioning and inertial navigation data of aerial camera are obtained and preprocessed to ground shooting;The transformation matrix set and feature matching set between adjacent frames of image sequence are calculated, and factor graph is constructed;Based on factor graph, the objective function is established and optimized to obtain a group of optimal camera pose sequence;According to the optimized camera pose, the preprocessed image in each batch is projected onto a unified plane, and the batch splicing image is generated, and the satellite base map corresponding to the batch area is obtained;The transformation model between splicing image and satellite base map is established, and the splicing image is converted to satellite base map coordinate system according to the transformation model, so as to draw the converted splicing image on satellite base map, realize geographic positioning correction.The present application can solve the problem of few feature points and low quality matching in sparse scene.

Description

Technical Field

[0001] This invention relates to the field of image stitching technology, specifically a method for rapid geolocation and stitching of images in sparse scenes. Background Technology

[0002] Aerial photography, with its high efficiency and flexibility, has been widely applied in surveying, agricultural monitoring, and disaster assessment. However, in sparsely textured environments such as deserts, grasslands, and oceans, aerial image stitching and geolocation still face significant technical challenges. These environments lack salient features, making it difficult for traditional feature-point-based matching methods to accurately extract and associate image information. Simultaneously, positioning sensors such as GPS / IMUs are prone to accumulating errors during long-term flights, failing to provide absolutely accurate initial poses for the images. Combined with the propagation of small errors during image stitching, this leads to noticeable distortion and misalignment in the stitched results. Furthermore, sparse areas often make it difficult to establish or identify ground control points (GCPs), resulting in decreased geolocation accuracy. These issues collectively limit the reliability and practicality of aerial image data processing in vast, sparse environments.

[0003] Currently, common aerial image stitching and geolocation methods mainly fall into the following categories: First, feature-point-based algorithms (such as SIFT and SURF) rely on local image features for matching, which perform well in textured scenes but significantly reduce matching accuracy in feature-sparse scenes; second, optical flow-based methods are suitable for estimating small displacements between consecutive frames, but have limited adaptability to large viewpoint changes or rapid motion; third, optimization methods based on bundle adjustment can optimize camera parameters and 3D structures through global adjustment, but have high computational overhead and are sensitive to initial values; and fourth, deep learning-based methods utilize neural networks to achieve feature extraction and registration, which can handle complex scenes, but still require a large amount of labeled data, and their generalization ability is limited, especially in unfamiliar environments.

[0004] Existing methods generally suffer from three prominent problems when dealing with sparse scenarios: First, insufficient accuracy and robustness, with missing features leading to increased matching failure rates and significant error accumulation, resulting in distortion and misalignment of the stitched results; second, low computational efficiency, especially global optimization methods and deep learning models, which consume a large amount of computational resources and cannot meet the requirements of real-time or near-real-time processing; and third, limited applicability, with most methods relying on favorable initial conditions or specific environmental assumptions, resulting in a significant decrease in geolocation accuracy and system stability in the absence of ground control points or prior geographical information. Therefore, current technologies have not yet effectively solved the problem of balancing high accuracy, high efficiency, and strong adaptability in sparse scenarios. Summary of the Invention

[0005] To address the technical problems existing in the prior art, this invention proposes a method for fast geolocation and stitching of images in sparse scenes, comprising the following steps:

[0006] Step 1: Obtain the raw image sequence of the ground taken by the aerial camera. and the positioning data output by the aircraft's inertial navigation system or GNSS positioning system at the corresponding time. The original image sequence is preprocessed, and the location data is optimized and filtered to obtain the preprocessed image sequence. And optimized location data ;

[0007] Step 2: Utilize the preprocessed image sequence And optimized location data Calculate image sequence The set of transformation matrices between adjacent image frames in and feature matching set ;

[0008] Step 3: Based on feature matching set and the set of transformation matrices A factor graph is constructed; the variable nodes in the factor graph are the camera poses corresponding to each image frame time, and the edges in the factor graph include the transformation relationships and feature matching between adjacent image frames and between non-adjacent image frames after loop closure verification; an objective function is established based on the factor graph and optimized to obtain a set of optimal camera pose sequences that minimize the overall error of all factors; during the optimization process, batch management is performed based on the cumulative error index, and the image sequences are processed accordingly. The image sequences are divided into several batches, and optimization solutions are obtained for each batch.

[0009] Step 4: Project the preprocessed images from each batch onto a unified plane according to the optimized camera pose to generate the stitched image for that batch, and obtain the satellite base map of the corresponding area for that batch; establish a transformation model between the stitched image and the satellite base map, and transform the stitched image to the satellite base map coordinate system according to the transformation model, and then draw the transformed stitched image on the satellite base map to achieve geolocation correction.

[0010] Furthermore, step 2 includes the following steps:

[0011] Step 2.1: Process the image sequence Feature points are extracted from each frame of the image to obtain the coordinates and feature descriptors of each feature point in each frame. Then, feature point matching pairs between adjacent image frames are obtained by matching the feature descriptors.

[0012] Step 2.2: Construct a motion model that conforms to physical laws using the aircraft's own motion priors. Using this motion model as a causal constraint, select feature point matching pairs that conform to the aircraft's motion laws from the feature point matching pairs between adjacent image frames, forming a feature matching set between adjacent image frames. ;

[0013] Step 2.3: Utilize the feature matching set between adjacent image frames The homography matrix between adjacent image frames is calculated and used as the transformation matrix to form a set of transformation matrices between adjacent image frames. .

[0014] Furthermore, in step 2.1, the image is evenly divided into several grids, and the number of feature points extracted in each grid is counted. If the number of feature points in a certain grid is lower than a set threshold, it is determined to be a feature sparse grid. For feature sparse grids, a feature detector is used to supplement feature extraction, and a descriptor is calculated for the newly extracted focus.

[0015] Furthermore, in step 2.2, the process of selecting matching pairs that conform to the motion law of the carrier from the feature point matching pairs between adjacent image frames is as follows:

[0016] Step 2.2.1: From the optimized positioning data Extract the carrier parameter variable set Construct the expected motion model As a causal model; the expected motion model From the set of parameter variables It was calculated;

[0017] Step 2.2.2: Based on the established expected motion model The consistency score between each feature point matching pair and the expected motion model in adjacent image frames is calculated, and feature point matching pairs are filtered according to the consistency score, retaining feature point matching pairs that conform to the motion law of the carrier aircraft.

[0018] Furthermore, in step 2.2.1, the expected motion model From the set of parameter variables The derived epipolar geometric model or a priori homography model is represented.

[0019] Furthermore, in step 3, the image sequence is processed using the bag-of-words model method. Similarity retrieval is performed on non-adjacent image frames to identify a potential closed-loop frame set. For the closed-loop frame set For each pair of closed-loop frames, closed-loop verification is performed, and the closed-loop constraints of the closed-loop frames that pass the closed-loop verification are added to the factor graph; the closed-loop constraints are the transformation matrix and feature point matching pair set between the two image frames of each pair of closed-loop frames; the closed-loop verification is to perform causal inference-assisted verification on the two image frames of each pair of closed-loop frames with the motion model as the causal constraint.

[0020] Furthermore, in step 3, the objective function is constructed based on the factor graph as follows:

[0021]

[0022] The objective function is then solved to obtain the optimized camera pose sequence. , For the optimized first The camera pose corresponding to the frame image; where For a set of adjacent image frames, For the closed-loop frame set, Let the reprojection error function be... For variable nodes Corresponding image frames and variable nodes The reprojection error between corresponding image frames. For variable nodes and The covariance matrix between them This is a robust kernel function.

[0023] Furthermore, in step 3, for the image sequence Starting from the first frame, calculate the cumulative error index.

[0024]

[0025] in This indicates the number of adjacent image frames used in calculating the cumulative error index; when When the batch ends, a batch termination check is triggered. The set error threshold is used to obtain the current batch. Record the ending frame of the current batch as the starting frame of the next batch, and finally process the image sequence. Divided into One batch, , of which batch , For the first batch The starting frame in For the first batch The terminating frame in the text.

[0026] Furthermore, in step 4, for adjacent batches... and Overlapping areas Perform feature matching to obtain a set of matching point pairs in the overlapping region. And the local seam transformation model is estimated. Using the local seam transformation model For overlapping regions Make adjustments to achieve a smooth transition between batches.

[0027] Furthermore, in step 1, the preprocessing of the original image sequence includes image distortion correction and exposure equalization; the optimization filtering of the positioning data adopts Kalman filtering or complementary filtering methods.

[0028] Beneficial effects:

[0029] The present invention has the following effects:

[0030] 1. This invention adopts a feature point matching and filtering method based on causal reasoning, which can solve the problems of scarce feature points and low-quality matching in sparse scenes. By using prior knowledge of the carrier parameters, feature points that conform to the laws of physical motion are effectively filtered, achieving a stable and reliable image stitching effect, and maintaining good performance even in scenes with extremely sparse textures.

[0031] 2. This invention adopts a batch management strategy based on error monitoring combined with factor graph optimization, which can solve the problem of stitching distortion caused by accumulated errors in traditional methods, realize large-scale high-precision image stitching, and avoid infinite accumulation of errors through batch management.

[0032] 3. This invention adopts a multi-level fusion method of satellite base map and aerial camera image, which can solve the problem of insufficient geolocation reference in sparse scenes and achieve high-precision geolocation correction.

[0033] 4. This invention adopts a batch-based progressive stitching and geographic correction process, which can monitor the stitching quality in real time and promptly register the batch stitching results with the satellite base map, effectively avoiding large-scale deformation caused by long-term stitching.

[0034] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0035] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0036] Figure 1This is a flowchart of the method of the present invention. Detailed Implementation

[0037] The embodiments of the present invention are described in detail below. These embodiments are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0038] This embodiment proposes a fast geolocation and stitching method for sparse scenes, such as deserts, grasslands, and oceans, which includes the following steps:

[0039] Step 1: Obtain the raw image sequence of the ground taken by the aerial camera. and the positioning data output by the onboard GPS / IMU at the corresponding time. The original image sequence is preprocessed, and the location data is optimized and filtered to obtain the preprocessed image sequence. and optimized GPS / IMU data .

[0040] In this embodiment, the input for step 1 is the original image sequence taken by the aerial camera of the ground. and the positioning data output by the carrier's GPS / IMU at the corresponding time. subscript Indicates the first That moment.

[0041] The processing procedure is as follows:

[0042] Image distortion correction:

[0043] Image distortion correction primarily targets geometrical distortions in images caused by inherent defects in camera lenses (such as fisheye effect and pincushion / barrel distortion). It restores the image to an ideal perspective projection that conforms to the pinhole camera model, laying the foundation for subsequent accurate feature matching and 3D reconstruction. For the original image sequence... Each image in the dataset is processed using a camera intrinsic parameter matrix. and distortion coefficient Image distortion correction is performed on the original image to obtain the corrected image sequence. ,in, , , The radial distortion coefficient is... , This represents the tangential distortion coefficient. Radial distortion, caused by the camera lens shape, results in straight lines bending at the image edges. , , These correspond to the first, second, and third order radial distortion coefficients, respectively, while tangential distortion is caused by lens manufacturing and installation errors (the lens is not parallel to the imaging plane).

[0044] Exposure equalization:

[0045] To eliminate the differences in brightness and contrast between sequential images caused by factors such as shooting time, angle, and cloud cover, which affect the performance of subsequent feature point extraction and feature matching algorithms and lead to a decrease in the number of feature points or an increase in mismatches, we use a histogram matching method to adjust the exposure of the images, thereby reducing the differences in brightness and contrast between different images caused by changes in lighting.

[0046] GPS / IMU data processing:

[0047] To remove noise from the raw GPS / IMU data and provide accurate external initial values ​​for subsequent image stitching and geolocation, Kalman filtering or complementary filtering is used to filter the noise in the GPS / IMU data and transform it into a unified geographic coordinate system.

[0048] The final output of this step is the preprocessed image sequence. and optimized GPS / IMU data .

[0049] Step 2: Utilize the preprocessed image sequence and optimized GPS / IMU data Calculate image sequence The set of transformation matrices between adjacent image frames in and feature matching set .

[0050] In this embodiment, the input for step 2 is the preprocessed image sequence. and optimized GPS / IMU data .

[0051] The processing procedure is as follows:

[0052] Step 2.1: Use the SuperPoint deep learning model to process the image sequence. Feature point extraction is performed on each frame of the image, and a feature point set is extracted from each frame of the image. ,in For the first The feature point set on the frame image For the first The first frame of the image Each feature point describes a vector. , , For the first The first frame of the image The coordinates of the feature points For the first The first frame of the image The feature descriptors of each feature point; based on the feature descriptors of each feature point in adjacent image frames, several feature point matching pairs are obtained through standard descriptor matching.

[0053] Since standard descriptor matching is computationally inefficient compared to brute-force matching or approximate nearest neighbor search across the entire image, this embodiment further utilizes GPS data to assist in feature point matching pair filtering. That is, based on the location prior information provided by GPS data, the feature search range is narrowed from the entire image to a small image window, thereby greatly reducing the amount of matching computation and completely eliminating "cross-region" erroneous matching.

[0054] Furthermore, since this invention targets sparse scenes, in order to ensure that feature points are evenly distributed in the image and avoid being overly concentrated in textured areas (such as grass and trees), while textured areas (such as roads, water surfaces, and smooth walls) have few features, leading to inaccurate alignment of these areas during subsequent stitching or reconstruction, this embodiment also employs an adaptive region feature enhancement method. That is, the image is evenly divided into several grids, and the number of feature points extracted by SuperPoint in each grid is counted. If the number of feature points in a certain grid is lower than a set threshold, it is determined to be a feature sparse grid. For feature sparse grids, traditional feature detectors (such as FAST corner detectors or Shi-Tomasi corner detectors) are used to supplement feature extraction, and descriptors such as ORB descriptors or BRIEF descriptors are calculated for the newly extracted focal points.

[0055] Step 2.2: Causal reasoning-assisted feature point matching pair filtering;

[0056] Step 2.1 obtains several feature point matching pairs between adjacent image frames through standard descriptor matching. However, these feature point matching pairs contain a large number of mismatches that do not conform to physical laws. Therefore, this step utilizes the aircraft's own motion prior (i.e., optimized GPS / IMU data). We construct a motion model that conforms to physical laws, and use this model as a "causal constraint" to select highly reliable matching pairs that conform to the motion laws of the aircraft from the feature point matching pairs of adjacent image frames, thereby reducing the proportion of mismatches.

[0057] Step 2.2.1: Since the motion of the carrier aircraft can be approximated as rigid motion determined by its attitude and velocity changes, the optimized GPS / IMU data of the carrier aircraft... Extract the carrier parameter variable set Construct the expected motion model As a causal model; where the set of parameter variables Including the Frame and the The frame corresponds to the heading angle, pitch angle, velocity, and the aircraft's position in the world coordinate system at that moment; while the expected motion model... It can be determined by the set of parameter variables. The derived epipolar geometric model or a priori homography model is specifically represented as follows:

[0058] Based on the parameter variable set The camera coordinate system is calculated from the first... Frame to the Relative rotation matrix of frames and relative translation direction :

[0059]

[0060]

[0061] in and Divided into the following categories: Frame and the The rotation matrix from the world coordinate system to the camera coordinate system at the corresponding moment in the frame image. for The inverse matrix, and The order is number 1 Frame and the The position vector of the camera in the world coordinate system at the corresponding moment of the frame image; , , and All through parameter variable set get.

[0062] Based on the camera coordinate system, from the first Frame to the Relative rotation matrix of frames and relative translation direction Two commonly used models of expected motion can be constructed: epipolar geometry model or prior homography model.

[0063] Epipolar geometry model:

[0064] According to the relative rotation matrix and relative translation direction Establish the prior essence matrix :

[0065]

[0066] in for The antisymmetric matrix;

[0067] For feature point set A feature point in In the feature point set The matching point in the equation theoretically falls on the polar line. nearby:

[0068]

[0069] in This indicates transpose.

[0070] Prior homography model:

[0071] According to the formula

[0072]

[0073] Calculate the prior homography matrix ,in For the camera intrinsic parameter matrix, The ground normal vector, The scaling factor is provided by the prior homography matrix from the feature point set. to feature point set A rough pixel-level mapping relationship.

[0074] For feature point sets A feature point in In the feature point set The matching point is located at the estimated point. Within a small neighborhood nearby.

[0075] Step 2.2.2: Based on the established expected motion model The consistency score between each feature point matching pair and the expected motion model in adjacent image frames is calculated, and feature point matching pairs are selected based on the consistency score, retaining high-reliability feature point matching pairs that conform to the motion law of the carrier aircraft.

[0076] For feature point matching pairs If the expected motion model adopts an epipolar geometry model, then the consistency score is taken from the feature points. arrive Corresponding polar lines The consistency score is calculated by taking the inverse of the distance; the smaller the distance, the higher the consistency score. If the expected motion model adopts a prior homography model, the consistency score is calculated based on the feature points. to the estimated point The smaller the distance, the higher the consistency score.

[0077] Step 2.2 yields a feature matching set consisting of the filtered feature point matching pairs from adjacent image frames. ,in For the first Frame image and the first A set of feature point matching pairs between frame images.

[0078] Step 2.3: Utilize the first Frame image and the first Set of feature point matching pairs between frame images The RANSAC (RandomSampleConsensus) algorithm was used to calculate the first... Frame image and the first Homography matrix between frames As a transformation matrix, it is used to describe the first... Frame image and the first Geometric transformation relationships of frame images.

[0079] The final output of this step is the feature matching set. and the set of transformation matrices .

[0080] Step 3: Based on feature matching set and the set of transformation matrices A factor graph is constructed, where variable nodes represent the camera pose at each image frame time. Edges in the factor graph include transformation relationships and feature matching between adjacent image frames and between non-adjacent image frames after loop closure verification. An objective function is established based on the factor graph and optimized to obtain a set of optimal camera pose sequences that minimize the overall error of all factors. During the optimization process, cumulative error is calculated, and batch management is implemented to optimize the image sequences. The process is divided into several stitching batches for subsequent stitching and geolocation. This step primarily addresses the inherent cumulative error problem in visual odometry and uses batch management to ensure computational feasibility and stability in large-scale scenarios.

[0081] In this embodiment, the input of step 3 is a feature matching set. and the set of transformation matrices .

[0082] The processing procedure is as follows:

[0083] Step 3.1: Construct the factor graph:

[0084] Image sequence The camera pose corresponding to each image frame is used as a variable node in the factor graph, resulting in a set of variable nodes. , where variable nodes ,in Divided into the first The rotation matrix from the world coordinate system to the camera coordinate system at the corresponding moment in the frame image. For the first The position vector of the camera in the world coordinate system at the corresponding moment of the frame image.

[0085] The set of edges of a factor graph Includes the set of edges between adjacent image frames. ,in That is, the edges of the factor graph consist of the transformation matrix between two image frames and the set of feature point matching pairs.

[0086] Step 3.2: Process the image sequence using the Bag-of-Words (BoW) model. Similarity retrieval is performed on non-adjacent image frames to identify a potential closed-loop frame set. For the closed-loop frame set For each pair of closed-loop frames, closed-loop verification is performed, and the closed-loop constraints of the closed-loop frames that pass the closed-loop verification are added to the factor graph; the closed-loop constraints are the transformation matrix and feature point matching pair set between the two image frames of each pair of closed-loop frames; the closed-loop verification is to perform causal inference-assisted verification on the two image frames of each pair of closed-loop frames with the motion model as the causal constraint.

[0087] Step 3.3: Construct the objective function based on the factor graph

[0088]

[0089] The objective function is then solved to obtain the optimized camera pose sequence. , For the optimized first The camera pose corresponding to the frame image; where For a set of adjacent image frames, For the closed-loop frame set, Let the reprojection error function be... For variable nodes Corresponding image frames and variable nodes The reprojection error between corresponding image frames. For variable nodes and The covariance matrix between them For robust kernel functions, the Huber kernel or Cauchy kernel is used to reduce the impact of mismatches on the optimization process. When a certain error... When the value is large (potentially due to an incorrect closed loop), the kernel function will weaken its influence to prevent it from "biasing" the entire optimization result.

[0090] In this embodiment, the Levenberg-Marquardt algorithm is used to optimize the objective function.

[0091] During the optimization process, a hierarchical optimization strategy is adopted: a coarse-to-fine multi-level optimization method is used, first performing global optimization on a subset of keyframes, and then gradually refining to all frames to improve optimization efficiency; that is, not all frames participate in optimization, but representative keyframes are selected from the image sequence, for example, keyframes are created only when the motion amplitude exceeds a certain threshold or a new region is seen, and optimization is initially performed only on the factor map composed of keyframes, which greatly reduces the number of variables and improves computational efficiency; after the key map optimization converges, the poses of all image frames are added for joint optimization.

[0092] When the image sequence is too long or the image scene is too large, the optimization problem becomes too large and difficult to solve. Therefore, this embodiment proposes batch management based on the cumulative error index to process the image sequence. The problem is divided into several batches, and the image sequences of each batch are optimized and solved separately.

[0093] For image sequences Starting from the first frame, calculate the cumulative error index.

[0094]

[0095] in This indicates the number of adjacent image frames used in calculating the cumulative error index; when When the batch ends, a batch termination check is triggered. The set error threshold is used to obtain the current batch. Record the ending frame of the current batch as the starting frame of the next batch to ensure a smooth transition between batches. This process generates the image sequence. Divided into One batch, , of which batch , For the first batch The starting frame in For the first batch The terminating frame in the text.

[0096] The final output of this step is the optimized camera pose sequence. and batch division .

[0097] Step 4: Project the preprocessed images from each batch onto a unified plane according to the optimized camera pose to generate the stitched image for that batch, and obtain the satellite base map of the corresponding area for that batch; establish a transformation model between the stitched image and the satellite base map, and transform the stitched image to the satellite base map coordinate system according to the transformation model, so as to draw the transformed stitched image on the satellite base map and realize geolocation correction.

[0098] In this embodiment, the input for step 4 is the optimized camera pose sequence. Preprocessed image sequence and batch division .

[0099] The processing procedure is as follows:

[0100] Step 4.1: Divide according to batch The preprocessed image sequence Distributed into various batches; for the first batch Based on the optimized camera pose corresponding to each image frame, the image frames are projected onto a unified plane to generate the first image frame. The stitched images of each batch .

[0101] Step 4.2: Based on the mosaic images of each batch, obtain the geographic boundaries of the mosaic images, and obtain the satellite base map of the corresponding area according to the geographic boundaries. .

[0102] Step 4.3: For each batch, perform feature matching on the stitched image and satellite base image of that batch, from coarse to fine, level by level using a multi-resolution pyramid structure; for example, for the first batch... Each batch, through a multi-resolution pyramid structure, performs feature matching from coarse to fine level to build a stitched image. Compared with satellite base map Set of matching points Based on the set of matching point pairs The transformation model is estimated. The transformation model can be either the thin plate spline (TPS) model or the piecewise affine transformation model to accommodate different types of geometric deformation.

[0103] Step 4.4: Transform the mosaic image to the satellite base map coordinate system according to the transformation model, and then draw the transformed mosaic image on the satellite base map to achieve geolocation correction; for example, for the first... Each batch will have a stitched image. By transforming the model Transform to the satellite base map coordinate system, and then on the satellite base map. The transformed mosaic image is plotted on the top, resulting in a geo-corrected mosaic image. This enables geolocation correction.

[0104] During the splicing process, this embodiment also takes into account the inter-batch seam treatment, the specific process of which is as follows:

[0105] For adjacent batches and Overlapping areas Perform feature matching to obtain a set of matching point pairs in the overlapping region. And the local seam transformation model is estimated. Using the local seam transformation model For overlapping regions Make adjustments to achieve a smooth transition between batches.

[0106] The final output of this step is the geo-location calibrated mosaic image sets for each batch. Geographic reference information sets of each batch of mosaic images And a global stitching index, wherein the geographic reference information includes coordinate system, top-left corner coordinates, pixel size, etc., while the global stitching index records the positional relationship of each batch on the satellite base map.

[0107] In this embodiment, the above method is implemented using a parallel computing framework:

[0108] (1) The entire processing flow is decomposed into parallel sub-tasks, including: image preprocessing task, feature extraction and matching task, factor graph construction and optimization task, and splicing and fusion task.

[0109] (2) Based on the task characteristics and dependencies, design a multi-level parallel computing framework, including:

[0110] Data-level parallelism: Parallel processing of image preprocessing and feature extraction tasks that have no dependencies;

[0111] Task-level parallelism: Parallel processing of independent subtasks such as loop closure detection in factor graph construction;

[0112] Parallel pipeline: Establish a processing pipeline of "preprocessing → feature extraction → local stitching → global optimization → fusion";

[0113] (3) To address the computational characteristics of different processing stages, a CPU-GPU hybrid computing architecture is adopted:

[0114] GPU acceleration: designed for computationally intensive tasks such as feature extraction, matching, and image transformation;

[0115] CPU processing: For complex logical tasks such as factor graph construction and optimization;

[0116] (4) Memory management strategy: Design a dynamic memory management mechanism to reduce memory usage during large-scale image processing through precise control of data flow.

[0117] Experimental comparison:

[0118] The table below shows a performance comparison between the method of this invention and existing typical methods on sparse scene datasets:

[0119]

[0120] Experimental results show that the method of this invention outperforms existing algorithms in key indicators such as stitching accuracy, geolocation accuracy, processing speed, loop closure detection accuracy, and feature matching success rate. In particular, the feature point selection method based on causal reasoning in this invention improves the feature matching success rate by 14.8 percentage points compared to simply using SuperPoint, which fully demonstrates the significant advantages of the causal reasoning selection technique proposed in this invention for feature matching in sparse scenes. Compared with the traditional SIFT method, this invention improves the feature matching success rate by 33.5 percentage points, showing a clear advantage in sparse feature scenarios.

[0121] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention without departing from the principles and spirit of the present invention.

Claims

1. A method for fast geolocation and stitching of images in sparse scenes, characterized in that: Includes the following steps: Step 1: Obtain the raw image sequence of the ground taken by the aerial camera. and the positioning data output by the aircraft's inertial navigation system or GNSS positioning system at the corresponding time. The original image sequence is preprocessed, and the location data is optimized and filtered to obtain the preprocessed image sequence. And optimized location data ; Step 2: Utilize the preprocessed image sequence And optimized location data Calculate image sequence The set of transformation matrices between adjacent image frames in and feature matching set ; Step 3: Based on feature matching set and the set of transformation matrices A factor graph is constructed; the variable nodes in the factor graph are the camera poses corresponding to each image frame time, and the edges in the factor graph include the transformation relationships and feature matching between adjacent image frames and between non-adjacent image frames that have undergone loop closure verification; an objective function is established based on the factor graph and optimized to obtain a set of optimal camera pose sequences, so that the overall error of all factors is minimized; During the optimization process, batch management is implemented based on the cumulative error index, and the image sequences are... The image sequences are divided into several batches, and optimization solutions are obtained for each batch. Step 4: Project the preprocessed images from each batch onto a unified plane according to the optimized camera pose to generate the stitched image for that batch, and obtain the satellite base map of the corresponding area for that batch; establish a transformation model between the stitched image and the satellite base map, and transform the stitched image to the satellite base map coordinate system according to the transformation model, and then draw the transformed stitched image on the satellite base map to achieve geolocation correction.

2. The method for fast geolocation and stitching of images in sparse scenes according to claim 1, characterized in that: Step 2 includes the following steps: Step 2.1: Process the image sequence Feature points are extracted from each frame of the image to obtain the coordinates and feature descriptors of each feature point in each frame. Then, feature point matching pairs between adjacent image frames are obtained by matching the feature descriptors. Step 2.2: Construct a motion model that conforms to physical laws using the aircraft's own motion priors. Using this motion model as a causal constraint, select feature point matching pairs that conform to the aircraft's motion laws from the feature point matching pairs between adjacent image frames, forming a feature matching set between adjacent image frames. ; Step 2.3: Utilize the feature matching set between adjacent image frames The homography matrix between adjacent image frames is calculated and used as the transformation matrix to form a set of transformation matrices between adjacent image frames. .

3. The method for fast geolocation and stitching of images in sparse scenes according to claim 2, characterized in that: In step 2.1, the image is evenly divided into several grids, and the number of feature points extracted in each grid is counted. If the number of feature points in a certain grid is lower than a set threshold, it is determined to be a sparse feature grid. For sparse feature grids, a feature detector is used to supplement feature extraction, and a descriptor is calculated for the newly extracted focus.

4. The method for fast geolocation and stitching of images in sparse scenes according to claim 2, characterized in that: In step 2.2, the process of selecting matching pairs that conform to the motion law of the carrier from the feature point matching pairs between adjacent image frames is as follows: Step 2.2.1: From the optimized positioning data Extract the carrier parameter variable set Construct the expected motion model As a causal model; the expected motion model From the set of parameter variables It was calculated; Step 2.2.2: Based on the established expected motion model The consistency score between each feature point matching pair and the expected motion model in adjacent image frames is calculated, and feature point matching pairs are filtered according to the consistency score, retaining feature point matching pairs that conform to the motion law of the carrier aircraft.

5. The method for fast geolocation and stitching of images in sparse scenes according to claim 4, characterized in that: In step 2.2.1, the expected motion model From the set of parameter variables The derived epipolar geometric model or a priori homography model is represented.

6. The method for fast geolocation and stitching of images in sparse scenes according to claim 2, characterized in that: In step 3, the image sequence is processed using the bag-of-words model. Similarity retrieval is performed on non-adjacent image frames to identify a potential closed-loop frame set. For the closed-loop frame set For each pair of closed-loop frames, closed-loop verification is performed, and the closed-loop constraints of the closed-loop frames that pass the closed-loop verification are added to the factor graph; the closed-loop constraints are the transformation matrix and feature point matching pair set between the two image frames of each pair of closed-loop frames; the closed-loop verification is to perform causal inference-assisted verification on the two image frames of each pair of closed-loop frames with the motion model as the causal constraint.

7. The method for fast geolocation and stitching of images in sparse scenes according to claim 2, characterized in that: In step 3, the objective function is constructed based on the factor graph as follows: The objective function is then solved to obtain the optimized camera pose sequence. , For the optimized first The camera pose corresponding to the frame image; where For a set of adjacent image frames, For the closed-loop frame set, Let the reprojection error function be... For variable nodes Corresponding image frames and variable nodes The reprojection error between corresponding image frames. For variable nodes and The covariance matrix between them This is a robust kernel function.

8. The method for fast geolocation and stitching of images in sparse scenes according to claim 7, characterized in that: In step 3, for the image sequence Starting from the first frame, calculate the cumulative error index. in This indicates the number of adjacent image frames used in calculating the cumulative error index; when When the batch ends, a batch termination check is triggered. The set error threshold is used to obtain the current batch. Record the ending frame of the current batch as the starting frame of the next batch, and finally convert the image sequence... Divided into One batch, , of which batch , For the first batch The starting frame in For the first batch The terminating frame in the text.

9. The method for fast geolocation and stitching of images in sparse scenes according to claim 8, characterized in that: In step 4, for adjacent batches and Overlapping areas Perform feature matching to obtain a set of matching point pairs in the overlapping region. And the local seam transformation model is estimated. Using the local seam transformation model For overlapping regions Make adjustments to achieve a smooth transition between batches.

10. The method for fast geolocation and stitching of images in sparse scenes according to claim 1, characterized in that: In step 1, the preprocessing of the original image sequence includes image distortion correction and exposure equalization; the optimization filtering of the positioning data adopts Kalman filtering or complementary filtering methods.

Images