Unmanned aerial vehicle low overlap remote sensing image stitching method

Abstract

The application discloses a UAV low-overlap remote sensing image splicing method, which comprises the following steps: acquiring UAV remote sensing images and their attitude information in the order of collection time sequence, and constructing candidate image pairs based on the time sequence adjacency relation and global image features; performing splicing feasibility evaluation, overlapping area reliability evaluation and effective splicing relation screening on the candidate image pairs; selecting a fast processing path or an enhanced processing path according to the splicing feasibility probability and the area proportion of the region of interest, obtaining a local alignment result and its uncertainty map; performing adaptive fusion based on the content difference, the uncertainty and the edge information, determining the edge weight according to the splicing feasibility, the region quality and the average uncertainty, performing global transformation consistency optimization and risk propagation processing, and generating a splicing result. The application can improve the stability and continuity of image splicing in the low-overlap and local mismatch scene.

Description

Technical Field

[0001] This invention relates to the field of UAV remote sensing image processing technology, specifically to a method for stitching low-overlapping UAV remote sensing images, which can be applied to remote sensing image registration and fusion in scenarios such as low-altitude mapping, regional aerial photography, inspection and monitoring, disaster assessment, and geographic information updating. Background Technology

[0002] With the development of applications such as low-altitude mapping, agricultural and forestry surveys, road inspections, disaster assessments, and geographic information updates, UAV remote sensing images are widely used to acquire spatial information about surface targets, topography, and engineering areas. During flight, UAVs typically acquire multiple remote sensing images from local perspectives. To create remote sensing images with wider coverage and more complete spatial representation, these multiple images need to be registered, stitched together, and fused.

[0003] Existing UAV remote sensing image stitching methods typically include image registration methods based on local feature matching, stitching methods based on global transform estimation, and image matching methods based on learning models. These methods can achieve good stitching results when the image overlap area is sufficient, the ground texture is clear, and the acquisition posture is stable.

[0004] However, during continuous aerial photography by low-altitude drones, factors such as flight speed, shooting interval, attitude disturbance, terrain undulation, local occlusion, texture repetition, and flight path planning errors can result in only a small effective overlap between adjacent or near-adjacent images. For low-overlapping image pairs, directly performing conventional feature matching and geometric transformation estimation can easily lead to problems such as insufficient number of matchable features, increased mismatch rate, and difficulty in correcting local misalignments, thus affecting the stability of image stitching.

[0005] Therefore, it is necessary to propose a stitching method suitable for low-overlap remote sensing images from UAVs. In cases where reliable positioning and attitude information is lacking or the effective overlap area is small, this method should be able to reasonably construct candidate stitching relationships, identify truly reliable overlapping areas, and perform global consistency optimization and risk control fusion based on the reliability of local alignment, thereby improving the efficiency, stability, and image quality of UAV remote sensing image stitching. Summary of the Invention

[0006] The purpose of this invention is to provide a method for stitching low-overlapping UAV remote sensing images based on temporal adjacency prior and uncertainty propagation, in order to solve the problems of low candidate image pair screening efficiency, large amount of computation for invalid matching, difficulty in locating the real overlapping area, unstable local geometric alignment, and easy accumulation and propagation of local errors in the continuous stitching process under low overlap conditions in existing UAV remote sensing image stitching methods.

[0007] The technical solution adopted in this invention is a method for stitching low-overlap remote sensing images from unmanned aerial vehicles (UAVs), comprising: Acquire multiple remote sensing images continuously collected by the UAV, and construct a sequence of remote sensing images to be stitched together according to the acquisition order of each image; Based on the temporal adjacency relationship and image content similarity relationship between the remote sensing images in the remote sensing image sequence to be stitched, a candidate image pair set is constructed; For candidate image pairs in the candidate image pair set, their overlap confidence information is determined, and a set of effective stitching relationships is obtained by filtering based on the overlap confidence information. The corresponding geometric alignment processing method is determined based on the overlap confidence level of the candidate image pairs, and the local alignment result between the image pairs is obtained. Spatial alignment uncertainty information is constructed based on the local alignment results. Then, the global transformation relationship between multiple remote sensing images is optimized based on the information and the set of effective stitching relationships. Finally, image fusion is performed based on the optimized global transformation relationship to generate a stitched remote sensing image.

[0008] As one preferred embodiment, the acquisition order is determined based on at least one of the acquisition time information and image sequence information of each remote sensing image, and an image index relationship is established for each remote sensing image according to the acquisition order.

[0009] As one preferred embodiment, the construction of the candidate image pair set includes: Remote sensing images whose image sequence difference does not exceed a preset adjacent window threshold, or whose acquisition time interval does not exceed a preset time threshold, are identified as local temporal adjacent images. Global visual representation features of each remote sensing image are extracted, and non-local candidate images are supplemented from the local temporally adjacent images based on the similarity between the global visual representation features. A set of candidate image pairs is constructed based on the local temporal adjacent images and the non-local candidate images.

[0010] As one preferred embodiment, the overlap confidence information includes the overlap confidence distribution and / or the quality of the region of interest; The overlap confidence distribution is determined based on the correlation of local visual features between two remote sensing images in a candidate image pair. The quality of the region of interest is determined based on at least one of the following: area ratio, connectivity, and spatial distribution.

[0011] As one preferred embodiment, the method for filtering overlapping reliable information includes: When the overlap confidence level of a candidate image pair is lower than a preset overlap confidence threshold, subsequent geometric alignment of the candidate image pair is stopped. When the overlap confidence level of a candidate image pair meets the preset overlap confidence condition, the candidate image pair is added to the set of valid stitching relationships.

[0012] As one preferred embodiment, the geometric alignment process includes: For image pairs whose overlap confidence level meets the preset fast processing conditions, local feature matching relationships are established in the corresponding regions of interest, and the geometric transformation relationship between the image pairs is estimated based on the local feature matching relationships. For image pairs that have been added to the set of effective stitching relationships but do not meet the preset fast processing threshold, coarse geometric alignment is performed first, and then a local mesh deformation model or a dense displacement field model is used to correct the local residual misalignment after coarse geometric alignment.

[0013] As one preferred embodiment, the spatial alignment uncertainty information is determined based on at least two of the following: feature matching confidence, matching inlier ratio, reprojection error, local photometric residual, local structural residual, and local deformation gradient.

[0014] As one preferred embodiment, the optimization of the global transformation relationship includes: A global pose graph is constructed using each remote sensing image as a node and the image pairs in the set of effective stitching relationships as edges. The edge weights in the global pose graph are determined based on the overlap confidence level of the image pairs, the quality of the region of interest, and the spatial alignment uncertainty information. Based on the aforementioned edge weights, the transformation relationships of each remote sensing image in the global coordinate system are jointly optimized.

[0015] As one preferred approach, abnormal splicing relationships are determined based on the optimization results of global transformation relationships, and at least one of the following processes is performed on the abnormal splicing relationships: realignment, candidate relationship replacement, or abnormal relationship elimination.

[0016] As one preferred approach, when performing image fusion based on the optimized global transformation relationship, the contribution of high uncertainty regions to the image fusion process is reduced based on the spatial alignment uncertainty information.

[0017] The advantages and beneficial effects of this invention are as follows: This invention constructs local candidate relationships by utilizing the temporal adjacency prior between images continuously acquired by UAVs, and supplements non-local candidate relationships with global visual representations. Furthermore, it filters out low-confidence candidate image pairs through overlapping confidence region perception and performs hierarchical geometric alignment for image pairs of different qualities. After completing local alignment, it constructs a spatial alignment uncertainty distribution based on image content differences, local structural differences, and feature matching states, and propagates this uncertainty to the global pose graph optimization and image fusion processes. This reduces the impact of erroneous stitching relationships and high-risk regions on the final stitching result, improving the efficiency, stability, and image quality of low-overlap remote sensing image stitching. Detailed Implementation

[0018] The present invention will be further described below with reference to specific embodiments. It should be understood that the following embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention. Where there is no conflict, the technical features in the following embodiments can be combined with each other.

[0019] Example 1 This embodiment provides a method for stitching low-overlapping remote sensing images from unmanned aerial vehicles (UAVs). This method is applicable to remote sensing images continuously acquired by UAVs in scenarios such as low-altitude mapping, regional aerial photography, road inspection, disaster assessment, and geographic information updates. It is particularly suitable for situations where the effective overlap area between adjacent images is small, ground textures are repetitive, there are many local occlusions, or the positioning and attitude information is incomplete. The method first establishes a remote sensing image sequence using the image acquisition order, and then constructs a candidate image pair set by combining temporal adjacency relationships and image content similarity relationships. This eliminates the need for subsequent overlapping reliable area perception, geometric alignment, and global fusion to perform pairwise complete matching on all images, thereby reducing invalid matching calculations and mitigating the risk of missing effective stitching relationships due to relying solely on the acquisition order.

[0020] In one embodiment, this method can be executed by a UAV ground station, a remote sensing data processing platform, an edge computing device, or a cloud server. The multiple remote sensing images continuously acquired by the UAV can be visible light images or low-altitude aerial images that have undergone radiometric correction, noise reduction, size standardization, or channel normalization. For image files generated by the UAV mission system, camera system, or data archiving system, the image file name or image metadata may include at least one of the following: acquisition time field, image sequence number field, flight path number field, or camera number field. This embodiment preferentially uses the acquisition time field or image sequence number field to determine the image acquisition order; when some images lack complete positioning and attitude information, the image sequence can also be constructed at least based on the image sequence number or the order field in the file name.

[0021] Specifically, this embodiment acquires multiple remote sensing images continuously collected by a UAV during a single mapping flight path or inspection mission, and sorts these images according to their acquisition time or image sequence number. If an image has both acquisition time and image sequence number, the acquisition time can be used as the primary sorting criterion, and the image sequence number as a secondary sorting criterion at the same time granularity. If the acquisition time field is missing or abnormal, the sorting order can be determined based on the image sequence number field. Through this processing, the system can establish the temporal organizational relationship between images without relying on complete geographic coordinates and attitude angle data, providing an index foundation for subsequent candidate relationship construction.

[0022] After completing image denoising, normalization, and size unification processing, the remote sensing image sequence acquired by the UAV is... Represented as: in, This indicates the total number of images acquired by the UAV during a single mapping route or a continuous data acquisition mission. This indicates the number after being sorted according to the collection order. A remote sensing image. In one embodiment, the remote sensing image can be represented as: Where H and W represent the height and width of the image, respectively, and C represents the number of image channels. For visible light RGB images, C can be 3; for preprocessed grayscale images or multi-channel remote sensing images, C can also be determined according to the actual image type.

[0023] To further preserve the image acquisition status, this embodiment can also perform image processing for each image. Record associated information provided by the UAV flight control system, positioning and attitude determination unit, or image file metadata. , denoted as: in, Representing an image The collection timestamp, This represents the two-dimensional geographical location at the time of image acquisition. Indicates flight altitude. This indicates the drone's attitude angle information. The aforementioned associated information is not required to be complete in every image; when some fields are missing, the acquired fields can be retained, and the sortability of the image sequence can be ensured by using the acquisition timestamp, image sequence number, or filename order. This forms an image-associated information sequence. : This image-related information sequence This is used to describe each remote sensing image and its corresponding acquisition status, so that subsequent candidate relationship construction can not only supplement and filter based on image content features, but also be constrained by the acquisition order and available spatial proximity information.

[0024] To transform the continuously acquired image sequence into a structured input suitable for subsequent relationship modeling, this embodiment constructs an image index set according to the image acquisition order: Among them, set Each index in the database corresponds to a remote sensing image. In other words, index Corresponding image ,index Corresponding image Using the aforementioned indexing relationships, subsequent candidate relationships can be first expressed in the index space and then mapped to actual image pairs. This process facilitates unified management of candidate edges, similarity, overlap confidence, geometric transformations, and node relationships in the subsequent global pose graph.

[0025] Furthermore, after completing the construction of the remote sensing image sequence, this embodiment constructs a candidate image pair set based on the temporal adjacency relationship and image content similarity relationship between the remote sensing images in the sequence to be stitched. This candidate relationship construction process includes two parts: local temporal adjacency candidate relationship construction and non-local adjacency candidate relationship supplementation. The former utilizes the inherent prior knowledge of the acquisition order of continuously acquired images by the UAV to quickly obtain adjacent or near-neighbor image pairs; the latter utilizes the global visual representation of the images to supplement non-adjacent stitchable image pairs that may be caused by flight path reversals, detours, abnormal sorting, or repeated ground features.

[0026] For any remote sensing image in the sequence to be stitched, images whose sequence number difference does not exceed a preset adjacency window threshold, or whose acquisition time interval does not exceed a preset time threshold, can be identified as locally temporally adjacent images. Specifically, images are defined as follows: Local temporal adjacency index set for: in, This indicates the preset adjacent window threshold. This represents a preset time threshold. The preset adjacency window threshold and the preset time threshold can be determined based on the UAV's flight speed, shooting interval, forward overlap rate, lateral overlap rate, or mission flight path type. For example, in aerial photography missions with short image acquisition intervals, several adjacent frames can be selected as local temporally adjacent images; in missions with unstable acquisition intervals, the local candidate range can be determined by combining timestamp differences.

[0027] Based on the aforementioned set of local temporal adjacency indices, a set of local temporal adjacency candidate relations is defined. for: in,( ) indicates that by the first Image and the first Candidate index pairs are formed from the images. These candidate index pairs can be stored as an ordered relationship for subsequent determination of the reference image and the image to be aligned; in implementations where orientation is not required, only the image itself can be retained. < Index pairs are used to reduce duplicate relationships. By building When there are a large number of images, the system can prioritize image pairs that are adjacent or close to each other in the acquisition order, reducing the computational overhead of full pairwise matching.

[0028] To supplement potential effective stitching relationships beyond local temporal adjacency relationships, this embodiment further extracts global visual representation features from each remote sensing image. Specifically, a global feature extraction network based on a visual Transformer can be used to encode the remote sensing images. First, the images are... Divided into sizes The image patches are then expanded into a vector sequence: in: , This represents the number of image blocks. In one embodiment, when the image height or width is not divisible by P, the image boundaries can be padded or cropped to make the image block division satisfy the above expression. This is achieved through a linear mapping matrix. : Image patch vectors are mapped to token representations and then superimposed with a position encoding matrix. : Get the input token sequence : in, This represents the feature dimension of the token. The input consists of a global coding network composed of L layers of Transformer encoders. To obtain intermediate features Represented as: To obtain the global representation vector for each image, intermediate features Perform average pooling on the token dimension and project the header. Mapping to a fixed dimension ,get: in, Indicates the first A global visual representation feature vector for a remote sensing image. This global visual representation describes the overall scene information of the image, such as terrain features, road orientation, farmland patch distribution, building roof textures, or other low-altitude remote sensing scene features. Using this feature, even if two images are not adjacent in the acquisition order, it is possible to determine whether they might be stitched together based on the similarity of their content.

[0029] Based on image global feature vector With images global feature vector Define the global feature similarity between images for: in, Represents the 2-norm ( ), Representing an image With images The degree of similarity in the global feature space. The higher the similarity, the closer the two images are in terms of overall scene content, and the more likely they are to be stitched together.

[0030] To complement the local temporal adjacency candidate relationships, this embodiment supplements the local temporal adjacency images with non-local candidate images. Specifically, for each image... Excluding itself and the local temporal adjacency index set Then, based on global feature similarity, the images with higher similarity are selected from the remaining images. For each image, construct a set of nonlocal adjacency candidate relationships: in, This indicates that the top [number] items are selected based on global feature similarity from high to low. Each image index. By supplementing non-local candidate images in addition to local temporally adjacent images, this embodiment can handle the problem of missing candidate relationships in cases such as flight path reversal, local detours, image sequence breaks, repeated acquisitions, or abnormal sorting of some images.

[0031] Therefore, by merging local temporal adjacency candidate relations and non-local adjacency candidate relations, a set of candidate index pairs is obtained. : in, The elements in are candidate index pairs To correspond with the description of the candidate image pair set, this embodiment further maps the candidate index pair set to the actual candidate image pair set. : in, Represents the set of candidate index pairs The set of candidate image pairs obtained by mapping.

[0032] In one embodiment, to enable subsequent overlapping reliable region perception priority processing of image pairs that are more likely to meet the stitching conditions, candidate index pairs can also be targeted. Construct a splicing feasibility assessment vector: Among them, symbols This represents the vector concatenation operation. This means taking the absolute value of each element of the input vector. This represents element-wise multiplication. The concatenated feasibility assessment vector is input into a feasibility discrimination network composed of fully connected layers. And mapped to the feasibility probability of splicing via the Sigmoid function. : in, Indicates candidate image pairs The probability of stitching feasibility at the global visual representation level. This stitching feasibility probability can serve as a ranking or pre-screening criterion for subsequent overlapping reliable region perception; in one embodiment, a stitching feasibility threshold can also be set. The following subset of candidate feasibility samples were obtained: in, This represents the subset of candidate relations retained after initial global feasibility screening. Subsequent processes such as overlapping reliable region detection, effective stitching relation selection, and geometric alignment are all based on this. As an entry set; that is, candidate relations excluded by the splicing feasibility threshold will no longer participate in subsequent geometric alignment. If in a specific task only... Using it as a sorting criterion without performing hard filtering allows = And according to the sorting results, they enter the subsequent overlapping reliable region sensing process in sequence.

[0033] Furthermore, after obtaining the candidate image pair set, this embodiment determines whether there are truly overlapping regions between the candidate image pairs that can be used for stitching. Since UAV low-altitude remote sensing images may only have limited overlapping regions at image edges, road extension directions, building sections, or farmland boundaries under low-overlap conditions, directly performing local feature matching across the entire image area can easily lead to mismatches due to repetitive textures, occlusions, and similar features in non-overlapping areas. Therefore, this embodiment generates an overlap confidence distribution for the candidate image pairs and determines the region of interest based on this distribution, ensuring that subsequent geometric alignment focuses on image areas that may actually overlap.

[0034] Specifically, this refers to the subset of candidate relations retained after the initial global feasibility screening. Candidate index pairs in The corresponding candidate image pairs are For this candidate image pair, a shared convolutional neural network encoder is used to extract local visual feature maps from the two images: in, This represents a convolutional feature extraction network. , and These represent the height and width of the local visual feature map, respectively. This represents the number of local visual feature channels. The convolutional feature extraction network can adopt a shared parameter structure so that the local features of the two candidate images are in the same feature space, which facilitates subsequent correlation calculation.

[0035] To estimate the image Different positions and images This embodiment constructs a correlation body based on the local visual feature maps of two images to determine the potential correspondences between different locations. : in, Representing an image Location at the feature map scale Representing an image Location at the feature map scale and These represent the local visual feature vectors at corresponding locations. The aforementioned correlation vectors can be used to describe the image. Each local location in the image The degree of correlation between features at different local locations.

[0036] Furthermore, regarding the correlation in The image is obtained by taking the maximum value in each dimension. Initial matching confidence at each feature map location: in, Representing an image Feature map location In the image The confidence level is used to find potential corresponding locations. This initial matching confidence level is used to reflect whether the corresponding local location is likely to belong to the overlapping area between the two images.

[0037] To reduce the impact of isolated high-response points and repetitive textures on overlapping region determination, this embodiment inputs the initial matching confidence into the overlapping confidence decoding network to obtain the final overlapping confidence distribution: in, This indicates an overlap confidence decoding network. Indicates the location of the feature map The overlap confidence level is determined by the degree of confidence of candidate image pairs in potential overlapping regions. The overlap confidence decoding network can employ a convolutional decoding structure, or it can combine neighborhood smoothing, upsampling, or multi-scale fusion methods to output a spatially continuous overlap confidence distribution.

[0038] According to the aforementioned overlap confidence distribution Set an overlap confidence threshold This yields the set of regions of interest at the feature map scale: in, This represents the potential overlapping region of candidate image pairs at the feature map scale. When Not lower than the overlap confidence threshold When the feature map location is below the overlap confidence threshold, the location is determined as the region of interest; when it is below the overlap confidence threshold, the location is not considered as the focus region for subsequent geometric alignment.

[0039] When the resolution of the overlap confidence distribution is lower than that of the original image, this embodiment maps the region of interest at the feature map scale back to the original image coordinate system based on the ratio between the feature map size and the original image size, thus obtaining the region of interest at the original image scale. : in, This represents a scale mapping function used to map data based on the original image size. With feature map size According to the scale ratio therebetween, the feature map coordinates are mapped to original image coordinates. In one embodiment, proper outward expansion may be performed on the boundary of the mapped region to retain potential matching information at the edge of the region of interest.

[0040] Further, for the subsequent local image patches, content difference maps, spatial alignment uncertainty maps, stitching weight maps, and summation regions involved in the training loss, this embodiment maps the region of interest at the feature map scale to the local image patch coordinate system, which is denoted as . where, represents the scale mapping relationship from the feature map coordinates to the local image patch coordinates, represents the set of regions participating in local alignment, uncertainty modeling and image fusion at the local image patch scale. Whenever it involves , , , or the position , it can be understood as the position in .

[0041] Further, the area ratio of the overlapping region is defined as: where, represents the number of positions contained in the region of interest at the feature map scale, hw represents the total number of positions of the feature map, is used to characterize the proportion of the potential overlapping region of the candidate image pair in the entire feature map. A larger area ratio generally indicates that the overlapping region available for stitching between the two images is more sufficient; an excessively small area ratio indicates that the candidate image pair may lack a stable geometric alignment basis.

[0042] After determining the region of interest, this embodiment determines the overlap confidence level of the candidate image pair according to at least one of the area ratio, connectivity and spatial distribution state of the region of interest. Specifically, when the area ratio of the region of interest is lower than a preset area threshold, or the region of interest is in an excessively discrete state, or the region of interest does not form a continuous region available for stable matching, it can be determined that the candidate image pair does not meet the preset overlap confidence condition, and the subsequent geometric alignment on the candidate image pair is stopped; when the area ratio, connectivity or spatial distribution state of the region of interest meets the preset overlap confidence condition, the candidate image pair is added to the effective stitching relationship set.

[0043] In one embodiment, the effective stitching relationship set may be expressed as: E v ={(i,j)∈E s | ≥τa and Satisfy the preset spatial distribution conditions} Where, τ a E represents the preset area ratio threshold. v Indicating in the candidate feasible subset E s Based on this, a set of valid stitching relationships is retained after overlapping reliable region perception and candidate relationship filtering. The preset spatial distribution conditions may include regions of interest that are continuously distributed, do not exhibit excessive dispersion, or are located at the image edge, in the flight direction, or other locations that conform to the continuous acquisition patterns of the UAV, but are not limited to these.

[0044] Furthermore, to ensure that valid image pairs of different qualities enter different geometric alignment processing paths, this embodiment uses the probability of stitching feasibility to determine the appropriate path. Area ratio of overlapping regions Define path selection variables: in, This indicates a preset fast processing threshold, and ; This indicates the preset area ratio threshold. This indicates that the image meets the conditions for fast processing and will subsequently enter the fast stitching path; This indicates that although the image pair meets the basic stitching conditions, its stitching feasibility probability or overlapping area ratio is insufficient to support direct and rapid alignment, and it subsequently enters the enhanced stitching path. Through this path selection method, image pairs with relatively clear overlapping areas and high global feasibility can undergo lighter geometric alignment processing; image pairs with small overlapping areas, high risk of local misalignment, or insufficient matching stability will undergo coarse geometric alignment and local deformation correction in subsequent steps.

[0045] Through the above processing, this embodiment further filters the candidate image pair set into a set of valid stitching relationships, and determines the region of interest and stitching processing path for each valid image pair, so that subsequent geometric alignment is concentrated in the regions that may actually overlap, reducing the interference of non-overlapping regions, repeated textures and local occlusion on the matching results.

[0046] Furthermore, in obtaining a set of valid splicing relationships Area of ​​Interest and path selection variables Subsequently, this embodiment performs geometric alignment processing on each image pair in the effective stitching relationship set. Since UAV low-altitude remote sensing images are easily affected by changes in flight attitude, viewing angle, terrain undulations, building height differences, and local occlusion during acquisition, relying solely on global matching of the entire image can easily introduce mismatches in low-overlap areas. Therefore, this embodiment establishes a local geometric correspondence between two images within the region of interest (ROI), and selects either fast geometric alignment or enhanced geometric alignment processing based on the overlap reliability of the image pair and the quality of the ROI.

[0047] Specifically, for the set of valid splicing relationships A set of image pairs (i,j) from the image With images Local image patches are cropped from the region of interest and denoted as follows: ,in, Represents reference image The local image patch corresponding to the region of interest. Indicates the image to be aligned A local image patch corresponding to the region of interest. In one embodiment, the region of interest can be composed of... Alternatively, boundary expansion can be performed after mapping back to the original image coordinate system to preserve potential matching information at the edges of the region of interest. By cropping local image patches within the region of interest, interference from non-overlapping areas, repetitive texture areas, and irrelevant background areas on subsequent geometric alignment can be reduced.

[0048] Subsequently, a convolutional feature extraction network was used to extract local features from the two local image patches, resulting in local feature maps: in, This represents a convolutional feature extraction network. , and These represent the height and width of the feature map of the region of interest, respectively. This represents the feature channel dimension. The convolutional feature extraction network can share parameters with the local visual feature extraction network, or it can be set separately according to the geometric alignment task, as long as it can output feature maps for establishing local correspondences.

[0049] For image pairs that meet preset fast processing conditions, this embodiment establishes local feature matching relationships within the region of interest and estimates the geometric transformation relationships between the image pairs based on these local feature matching relationships. Specifically, this can be done in the local feature map. and The local feature correlation response is calculated between the pairs, and the feature locations that are the nearest neighbors and whose correlation response is not lower than a preset matching threshold are determined as the initial matching relationship. Matching points located in low-confidence regions are then eliminated based on the aforementioned overlap confidence distribution. Furthermore, inlier filtering or weighted homography estimation can be used to obtain the inlier matching set from the initial matching relationship, and the coarse geometric transformation matrix between image pairs can be estimated accordingly. .

[0050] In one embodiment, when the number of stable extractable local feature points within the region of interest is small, pooling can also be performed on the local feature map to obtain a local description vector. and The two are then concatenated and input into a geometric regression network. Output image pairs The coarse geometric transformation parameters are then used. In this case, the local feature matching relationship can be used to verify or correct the coarse geometric transformation matrix output by the geometric regression network, thereby reducing the risk of geometric estimation instability caused by feature sparsity in low-overlap regions.

[0051] in, , used to represent the eight free parameters of the homography matrix. Further, the coarse geometric transformation parameters are reduced to a coarse geometric transformation matrix: in, Representing vectors The Each component. The coarse geometric transformation matrix is ​​used to describe the image. Relative to image The geometric mapping relationship is used to resample local regions of the image to be aligned to the local coordinate system of the reference image.

[0052] set up Representing an image The two-dimensional pixel coordinates in the image are denoted in homogeneous form as follows: Then the image In the image The coarse alignment result in the coordinate system is expressed as follows: in, This represents the homogeneous normalized projection operator, used to map three-dimensional homogeneous coordinates to two-dimensional pixel coordinates. This process first eliminates the main viewpoint and scale differences between image pairs, allowing subsequent local deformation corrections to be based on a more stable coarse alignment.

[0053] When path selection variables When this condition is met, it indicates that the probability of stitching the image pair and the ratio of the overlapping area satisfy the preset fast processing conditions. In this case, this embodiment can directly use the coarse alignment result as the local alignment result: in, Representing an image Align to image Local alignment results in the coordinate system. This fast processing method is suitable for image pairs with well-defined regions of interest, high overlap ratios, and good matching stability, and can reduce unnecessary local deformation calculations.

[0054] When path selection variables This indicates that while the image pair meets the basic stitching conditions, the overlapping area is small, the local misalignment is significant, or the matching stability is insufficient. In this case, this embodiment further performs local deformation correction based on coarse geometric alignment. Specifically, it references local image blocks. Results of coarse alignment Input local deformation feature encoder respectively The corresponding alignment feature map is obtained: in, , This represents the channel dimension of the local deformation features. The two sets of features are concatenated along the channel dimension and then input into the displacement decoding network. Generate a two-dimensional displacement field: in, This is used to represent the local displacement correction amount at different locations within the region of interest. The displacement decoding network can adopt a U-Net structure, a local mesh deformation model, or a dense displacement field model, as long as it can correct the local residual misalignment after coarse geometric alignment.

[0055] Under the enhanced stitching path, a two-dimensional displacement field is applied to the coarse alignment result to obtain the finely aligned local alignment result: in, Indicates position The displacement correction amount at the location. Through this processing, residual misalignment caused by terrain undulation, building height differences, low-altitude flight attitude disturbances, or local parallax can be compensated, resulting in a more stable local alignment of the effective image pair within the region of interest.

[0056] In one embodiment, to avoid excessive distortion introduced by local deformation correction, constraints can be placed on the amplitude, smoothness, or local gradient changes of the two-dimensional displacement field; alternatively, displacement correction can be performed only within the region of interest or in a localized area extending beyond the region of interest, without applying dense deformation to the entire image. This approach improves the local alignment stability of low-overlapping regions while reducing invalid computations in non-overlapping regions or regions far from the stitching boundary.

[0057] Through the above processing, this embodiment determines the corresponding geometric alignment processing method for each image pair in the effective stitching relationship set based on its overlap confidence level and region of interest quality: for image pairs that meet the preset fast processing conditions, fast geometric alignment is performed within the region of interest; for image pairs that meet the basic stitching conditions but not the preset fast processing conditions, coarse geometric alignment is performed first, and then local deformation correction is used to compensate for the local residual misalignment after coarse geometric alignment. This yields the local alignment results between image pairs, providing input for subsequent spatial alignment uncertainty modeling and global consistency optimization.

[0058] In this embodiment, Using the aforementioned backsampling mapping relationship, it is used to represent the data from the reference image. Coordinate system to image to be aligned The geometric correspondence of the coordinate system, so as to visualize the image. Resampling to image In the coordinate system.

[0059] Furthermore, to enable the local alignment results to be used in subsequent risk perception fusion, this embodiment constructs a spatial alignment uncertainty map based on the differences in image content and local structure after fine alignment. Specifically, at the local image patch scale, a photometric residual map is defined. ): in, Indicates position The pixel content differences between the two images after fine alignment are further defined. : in, Represents the image gradient operator. Used to characterize location The degree of inconsistency in local structures such as road edges, building outlines, and field boundaries.

[0060] Photometric residual map With structural residual diagram The data is concatenated along the channel dimension and then input into an uncertainty estimation network. The spatial alignment uncertainty map is obtained as follows: in, This indicates concatenation along the channel dimension. This represents the Sigmoid activation function. , Represents the location at the local image patch scale Spatial alignment uncertainty at the location. The larger the value, the higher the risk of ghosting, misalignment, or structural breakage, even though the geometric alignment at that location has been completed. The smaller the value, the more reliable the local alignment result at that location. In one embodiment, feature matching confidence, the proportion of matched inliers, reprojection error, or local deformation gradient can also be used as additional input channels, along with the photometric residual map and the structural residual map, to generate a spatial alignment uncertainty map.

[0061] Through the above processing, the local alignment result includes not only the image content after geometric transformation, but also a spatial alignment uncertainty map used to characterize the alignment reliability of different spatial locations, thus providing a basis for subsequent stitching weight generation, global edge weight adjustment, and weight reduction fusion of high uncertainty regions.

[0062] After modeling the uncertainties in geometric and spatial alignment, it is necessary to further determine which image's content should be retained more at different locations within the overlapping region. In low-overlap remote sensing images, repetitive textures, local occlusion, parallax residues, and structural edge misalignments can easily cause ghosting, misalignment, or abrupt seams in simple average fusion. Therefore, this embodiment incorporates the differences in aligned content, spatial alignment uncertainty, and the structural continuity of the reference image into the stitching weight generation process, enabling the fusion process to adaptively adjust the contribution ratio of the two images according to the reliability of different spatial locations.

[0063] Specifically, first, define the content difference diagram: in, Indicates position The degree of pixel difference between two images after fine alignment. Indicates from reference image The local image patch obtained by cropping the region of interest. Representing an image The image was resampled after coarse geometric alignment and necessary local deformation correction. Fine alignment results in coordinate system express Norm.

[0064] Further define the structural edge intensity map of the reference image: in, Represents the image gradient operator. Used to characterize location The structural edge strength at the location. By introducing a structural edge strength map, the subsequent weight generation process can be aware of structural information such as road edges, building outlines, and field boundaries in the reference image, reducing the risk of misalignment when the stitching boundary passes through a strong structural edge.

[0065] Content Difference Map Spatial alignment uncertainty diagram and structural edge strength diagram The input is concatenated along the channel dimension and fed into a weight generation network composed of convolutional decoders. The resulting stitched weighted graph is: in, This indicates concatenation along the channel dimension. This represents the Sigmoid activation function. , Indicates position Reference image retained Content weighting. When spatial alignment uncertainty is plotted. Content Difference Diagram or structural edge strength diagram When the scale of the local image patch is inconsistent with the scale of the region of interest, the corresponding image can be interpolated or resampled according to the scale mapping relationship between the region of interest and the original image coordinates, so that it can participate in weight generation in the same local coordinate system.

[0066] Through the aforementioned weight generation method, the system can reduce the fusion contribution of unreliable image content in locations with high uncertainty, significant local differences, or strong structural edges, while increasing the retention rate of corresponding image content in locations with reliable alignment and structural continuity. Therefore, the stitching weights are not solely determined by pixel differences, but rather adjust the fusion contribution by incorporating spatial alignment uncertainty information, thus providing local support for reducing the contribution of high-uncertainty regions in subsequent global fusion.

[0067] Based on weight graph The image content within the overlapping areas is weighted and fused to obtain a local stitching result. : in, This represents the local fusion result of image pair (i,j) within the region of interest. When When the value is close to 1, the reference image is mainly preserved at this position. Content; when When the value is close to 0, this position primarily retains the finely aligned image. The content. For For larger locations, the weight generation process reduces the contribution of the corresponding image content in the fusion; for locations with high overlap confidence, continuous local structure, and small alignment differences, the weight generation process increases their retention rate in the local stitching result.

[0068] In one embodiment, the partial stitching result Corresponding image pair index (i,j), region of interest Probability of splicing Overlapping area ratio and spatial alignment uncertainty diagram The associated data is saved for subsequent global consistency optimization and risk control fusion. Through this association, subsequent processing can identify the reliability of different local stitching relationships and reduce the contribution of high-uncertainty regions during global fusion, thus preventing local erroneous stitching relationships from being amplified during the stitching of multiple images.

[0069] To ensure the spatial continuity of the weight map and avoid fragmentation, abrupt changes, or local jitter at the stitching boundaries, smoothing can be performed after the weight map is generated, or a smoothing constraint can be applied to the weight map during the model training phase. This smoothing process does not change the aforementioned weighted fusion relationship; it is only used to keep the weight changes of adjacent pixel positions continuous, thereby improving the visual coherence of the local stitching results.

[0070] This step enables adaptive fusion of image content within overlapping areas based on local alignment reliability. It prioritizes avoiding high-uncertainty and strongly structurally abrupt regions at the stitching boundary, reducing ghosting, misalignment, and distortion discontinuities in low-overlap UAV remote sensing image stitching. This provides stable local fusion results for subsequent global consistency optimization and the generation of the final stitched remote sensing image. This step reduces the impact of local alignment errors on the stitching result, providing local fusion results for subsequent global consistency optimization.

[0071] Specifically, for each image Define the transformation matrix to the global coordinate system: in, Representing an image The pose relationship to the global stitching plane. For each set of valid candidate images... The coarse geometric matrix obtained by local alignment A consistency relationship is satisfied between the pose and the global pose: in, Using the aforementioned backsampling mapping relationship, it is used to describe the data from the reference image. Coordinate system to image to be aligned Geometric correspondence of coordinate systems. Through this consistency relationship, the geometric constraints of local image pairs can be uniformly expressed in the global pose graph, so that the local alignment results between different image pairs can jointly constrain the position of each image in the global stitching plane.

[0072] To give more reliable image pairs a greater impact in global optimization, edge weights are introduced: in, Representing image pairs The probability of feasibility of splicing; The quality parameter representing the region of interest can be derived from the aforementioned overlapping region area ratio. It can also be determined by combining the area ratio of the overlapping area with the connectivity of the region of interest, the proportion of the largest connected region, the spatial distribution status, and whether it conforms to the continuous data collection pattern of the UAV. This represents the average uncertainty. The average uncertainty is defined as: in, Indicates position Spatial alignment uncertainty at the location, Represents the region of interest at the feature map scale. The set of regions mapped to the local image patch scale. Therefore, image pairs with high stitching feasibility, good region of interest quality, and low average uncertainty have larger edge weights in the global pose graph; conversely, image pairs with lower reliability have a correspondingly reduced impact on the global optimization result.

[0073] Further construct a global consistency optimization objective: in, This represents the Frobenius norm, which is the square root of the sum of the squares of all corresponding elements of a matrix; its square is used to measure matrix differences. Through this optimization process, the local geometric relationships between remote sensing images are uniformly constrained to the global coordinate system, which can reduce the impact of local errors in a single image on subsequent continuous stitching results and improve the geometric consistency of the overall stitched image.

[0074] To describe the potential impact of the current local stitching relationship on the overall result, this embodiment further constructs a stitching risk index. First, the local alignment residual is defined: in, Representing image pairs Average content difference at the local image patch scale The location of the aforementioned content differences is shown in the diagram. The degree of pixel difference at that location Indicates the region of interest The set of regions mapped to the local image patch scale. Based on this, the feasibility of stitching, the quality of the region of interest, uncertainty, and local residuals are unified and integrated into a risk indicator: in, , , , It is a non-negative weighting coefficient used to balance the impact of different risk sources on risk indicators. The larger the value, the higher the risk that the image will adversely affect the overall stitching result.

[0075] When the risk indicator exceeds the preset threshold At that time, the splicing self-recovery control mechanism is triggered. Specifically, for conditions that meet: The image is used to adjust the execution strategy. The strategy adjustment includes: lowering the overlap confidence threshold and recalculating the region of interest to expand the potential stitchable region; setting the path selection flag to an enhanced path state, i.e., letting: Then re-perform coarse geometric alignment and local deformation correction; if the risk still does not decrease after re-alignment, then select from the candidate image pair set. New adjacent image pairs are added, and the process of overlapping reliable region perception and candidate relation selection is re-entered. For abnormal stitching relations that still cannot meet the requirements of reliable stitching, candidate relation replacement or abnormal relation elimination can be performed to prevent unreliable edges from continuing to participate in global pose optimization.

[0076] After completing the global consistency optimization, based on the optimized global transformation relationship Each remote sensing image is projected onto the global mosaic plane and combined with the local mosaic results. Spatial alignment uncertainty diagram and splicing weighted graph Risk control fusion is implemented. For regions with high spatial alignment uncertainty, their contribution to the final image fusion process is reduced; for regions with high overlap confidence, small local residuals, and structural continuity, their retention rate in the final mosaic image is increased. Thus, the final fusion process can utilize local uncertainty information to suppress the impact of high-risk regions on the final mapping result. Through the above global consistency optimization and risk control fusion, the final mosaic remote sensing image is generated.

[0077] Example 2: Training Phase and Model Update In an optional embodiment, to enable the global representation network, splicing feasibility assessment network, overlap credibility prediction network, geometric alignment network, uncertainty modeling network, and splicing weight generation network to converge collaboratively, a joint loss function can be constructed during the training phase to jointly optimize geometric alignment accuracy, uncertainty modeling, and splicing weight continuity.

[0078] First, define the alignment consistency loss: in, This represents a robust penalty function, such as Huber or Charbonnier, used to reduce the impact of local outlier pixels on the training process. This represents a local image patch of the reference image within the region of interest; This represents the finely aligned image, which has been resampled to the reference image coordinate system after coarse geometric alignment and necessary local deformation correction. This represents the overlapping reliable region corresponding to image pair (i,j). This loss is used to constrain the fine alignment result to be as close as possible to the reference image within the overlapping region, enabling the geometric alignment network and the local deformation correction network to learn alignment relationships applicable to low-overlap regions.

[0079] Secondly, to ensure that the spatial alignment uncertainty plot has a clear error indication function, a normalized error plot is defined: in, This represents the image pair (i,j) at position Differences in content, Representing overlapping reliable regions Pixel position within, To prevent extremely small constants with a denominator of zero, this normalization process maps local differences between different image pairs to a relatively uniform scale, facilitating the uncertainty estimation network's learning of the correspondence between alignment error and uncertainty.

[0080] Based on the normalized error map, an uncertainty calibration loss is constructed: in, Indicates position The spatial alignment uncertainty is addressed by using this loss function to constrain the spatial alignment uncertainty map to remain consistent with the actual local alignment error. This ensures that the uncertainty map reflects the alignment reliability at different locations within the overlapping region and provides a stable reference for subsequent global edge weight determination and image fusion.

[0081] Furthermore, in order to make the spliced ​​weight map To maintain spatial continuity and reduce splicing boundary breakage and jitter caused by abrupt changes in local weights, a weight smoothing loss is defined: in, Represents the image gradient operator. Indicates position The stitching weights at each location are used to constrain the weight changes of adjacent spatial locations, ensuring good continuity of the weight map within overlapping areas, thereby improving the visual coherence of the local fusion results.

[0082] Combining the above loss terms, the overall training objective is obtained: in, and The loss weight coefficients are used to balance the relationship between geometric alignment accuracy, uncertainty modeling accuracy, and stitching weight continuity. Through iterative optimization of the above overall training objective, the global representation network, stitching feasibility assessment network, overlap confidence prediction network, geometric alignment network, uncertainty modeling network, and stitching weight generation network can gradually obtain model parameters suitable for UAV low-overlap remote sensing image stitching.

[0083] In one embodiment, the training phase can use UAV remote sensing images with known adjacency relationships or those that have already been registered and calibrated as training samples, or it can use a sample sequence consisting of continuously acquired images from the same flight path for weakly supervised or self-supervised training. During training, the network parameters can be jointly updated based on the image acquisition order, local overlapping areas, alignment residuals, and the continuity of the fusion results. For datasets formed by different flight altitudes, different ground cover types, or different shooting intervals, the model parameters can be updated separately or trained together on a unified training set to improve the model's adaptability to low-altitude mapping, inspection and monitoring, and regional aerial photography scenarios.

[0084] It should be noted that the training phase and model updates are not mandatory steps for every image stitching run. Once stable model parameters are obtained, the trained model can be directly invoked to perform remote sensing image sequence construction, candidate relation construction, overlapping reliable region perception, hierarchical selective geometric alignment, local alignment uncertainty modeling, local fusion, and global consistency optimization and risk control fusion. Through this training method, the coordination between network modules can be improved, enabling the final stitched remote sensing image to maintain good geometric consistency and visual continuity even under conditions of low overlap, repetitive textures, and local misalignment.

[0085] The present invention has been described above by way of example. It should be noted that any simple modifications, alterations or other equivalent substitutions that can be made by those skilled in the art without creative effort without departing from the core of the present invention fall within the protection scope of the present invention.

Claims

1. A method for stitching low-overlapping remote sensing images from unmanned aerial vehicles (UAVs), characterized in that, include: Acquire multiple remote sensing images continuously collected by the UAV, and construct a sequence of remote sensing images to be stitched together according to the acquisition order of each image; Based on the temporal adjacency relationship and image content similarity relationship between the remote sensing images in the remote sensing image sequence to be stitched, a candidate image pair set is constructed; the image content similarity relationship is determined by extracting the global visual representation features of each remote sensing image and then determining the similarity between the global visual representation features. For candidate image pairs in the candidate image pair set, their overlap confidence information is determined, and an effective stitching relationship set is obtained by filtering based on the overlap confidence information. The corresponding geometric alignment processing method is determined based on the overlap confidence level of the candidate image pairs, and the local alignment result between the image pairs is obtained. The overlap confidence information includes overlap confidence distribution and / or region of interest quality. The overlap confidence distribution is determined based on the correlation of local visual features of the two remote sensing images in the candidate image pair. The region of interest quality is determined based on at least one of the area ratio, connectivity, and spatial distribution state of the region of interest. The geometric alignment processing method includes: for image pairs whose overlap confidence level meets preset fast processing conditions, establishing a local feature matching relationship within the corresponding region of interest, and estimating the geometric transformation relationship between the image pairs based on the local feature matching relationship; for image pairs that have been added to the effective stitching relationship set but do not meet the preset fast processing conditions, coarse geometric alignment is first performed, and then a local mesh deformation model or a dense displacement field model is used to correct the local residual misalignment after coarse geometric alignment. Spatial alignment uncertainty information is constructed based on the local alignment results. Then, the global transformation relationship between multiple remote sensing images is optimized based on the spatial alignment uncertainty information and the effective stitching relationship set. Image fusion is then performed based on the optimized global transformation relationship to generate a stitched remote sensing image. The spatial alignment uncertainty information is determined based on at least two of the following: feature matching confidence, matching inlier ratio, reprojection error, local photometric residual, local structural residual, and local deformation gradient. The optimization of the global transformation relationship includes: constructing a global pose graph using each remote sensing image as a node and image pairs in the effective stitching relationship set as edges; determining the edge weights in the global pose graph based on the overlap confidence of the image pairs, the quality of the region of interest, and the spatial alignment uncertainty information; jointly optimizing the transformation relationship of each remote sensing image in the global coordinate system based on the edge weights; and reducing the contribution of high uncertainty regions in the image fusion process based on the spatial alignment uncertainty information when performing image fusion based on the optimized global transformation relationship.

2. The method for stitching low-overlapping remote sensing images from unmanned aerial vehicles according to claim 1, characterized in that, The acquisition order is determined based on at least one of the acquisition time information and image sequence information of each remote sensing image, and an image index relationship is established for each remote sensing image according to the acquisition order.

3. The method for stitching low-overlapping remote sensing images from unmanned aerial vehicles according to claim 1, characterized in that, The construction of the candidate image pair set includes: Remote sensing images whose image sequence difference does not exceed a preset adjacent window threshold, or whose acquisition time interval does not exceed a preset time threshold, are identified as local temporal adjacent images. Global visual representation features of each remote sensing image are extracted, and non-local candidate images are supplemented from the local temporally adjacent images based on the similarity between the global visual representation features. A set of candidate image pairs is constructed based on the local temporal adjacent images and the non-local candidate images.

4. The method for stitching low-overlapping remote sensing images from unmanned aerial vehicles according to claim 1, characterized in that, The method for filtering overlapping reliable information includes: When the overlap confidence level of a candidate image pair is lower than a preset overlap confidence threshold, subsequent geometric alignment of the candidate image pair is stopped. When the overlap confidence level of a candidate image pair meets the preset overlap confidence condition, the candidate image pair is added to the set of valid stitching relationships.

5. The method for stitching low-overlapping remote sensing images from unmanned aerial vehicles according to claim 1, characterized in that, Based on the optimization results of the global transformation relationship, abnormal splicing relationships are determined, and at least one of the following processes is performed on the abnormal splicing relationships: realignment, candidate relationship replacement, or abnormal relationship elimination.

Images