An airborne visible light image automatic splicing method

Abstract

The application discloses an airborne visible light image automatic splicing method, and relates to the field of unmanned aerial vehicle image processing. The method comprises the following steps: acquiring multiple images collected by an unmanned aerial vehicle during flight and corresponding spatial position information and attitude information; determining a registrable image pair based on spatial position relations and image overlapping information between the images, and constructing a connection graph atlas with the images as nodes and the registrable image pair as edges; detecting whether spatial connection is broken due to image loss in the atlas, and if so, inserting a virtual node and constructing a virtual image containing an edge region and a transition region; further calculating geometric transformation parameters between the images, and resampling all image contents to a unified coordinate system to perform pixel-level fusion processing in the image overlapping region. The application solves the problem of discontinuous unmanned aerial vehicle image splicing caused by the existence of no-fly zones, privacy protection zones and other reasons.

Description

Technical Field

[0001] This invention relates to the field of image processing for unmanned aerial vehicles (UAVs), and more specifically, to an automatic image stitching method for airborne visible light. Background Technology

[0002] With the widespread application of drone aerial photography technology, airborne visible light imagery is extensively used in surveying, agricultural inspection, urban modeling, and emergency rescue. Typically, drones continuously acquire ground images during flight using their onboard cameras, and then stitch these images together to generate a high-resolution panoramic or orthophoto map of the covered area. To achieve accurate stitching, spatial overlap, feature matching, and geometric transformation models between images are often employed.

[0003] However, in actual flight missions, due to the following reasons, there are often serious image gaps that affect the continuity and integrity of image stitching. For example, some areas (such as airports, military restricted areas, power facilities, etc.) are clearly designated as no-fly zones for drones, and drones cannot collect images in these areas. In urban environments, some residential areas, private buildings, or sensitive areas are not allowed to be photographed or need to avoid detailed presentation due to privacy protection requirements.

[0004] The above reasons can cause holes, breaks, or misalignments in the stitched images. In particular, when images in key connecting areas are missing, the overall stitching structure will be interrupted, affecting the integrity and accuracy of the final image. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide an automatic airborne visible light image stitching method to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] An automatic image stitching method for airborne visible light images includes the following steps:

[0008] Acquire multiple images captured by the UAV during flight, along with their corresponding spatial position and attitude information;

[0009] Based on the spatial positional relationship between images and image overlap information, registerable image pairs are determined and a connection graph is constructed. The connection graph uses images as nodes and registerable image pairs as edges to describe the spatial topological relationship between images.

[0010] Detect whether there is a spatial connection break in the connection map due to image missingness. If there is a spatial connection break, insert a virtual node at the break position and construct a corresponding virtual image for the virtual node. The virtual image includes the edge region of adjacent images and the smooth transition region between the edge regions.

[0011] For any two connected image nodes in the connection graph, the geometric transformation parameters between the images are calculated based on their spatial position and orientation information.

[0012] Using any image other than the virtual image as a reference image, based on the geometric transformation parameters and the connection relationships in the connection map, all image contents are resampled to a unified coordinate system, and pixel-level fusion processing is performed in the overlapping areas of the images.

[0013] In some embodiments, the step of determining the registerable image pairs includes:

[0014] Calculate the spatial relationship between any two images and determine whether the spatial distance between their shooting positions is less than a preset distance threshold.

[0015] In image pairs that meet the spatial position and orientation constraints, image content is further extracted, and the overlap index of the image edge regions is calculated. When the overlap exceeds a set threshold, the image pair is determined to be a registrable image pair.

[0016] In some embodiments, the step of detecting whether there are spatial connectivity breaks in the connectivity map due to image missingness includes:

[0017] Based on the spatial location information of image nodes and the relative positional relationship between the images corresponding to the connecting edges, a spatial geometric connectivity graph corresponding to the atlas is constructed.

[0018] A connectivity analysis is performed on the geometrically connected graph. If there are two image nodes that should be adjacent but have not established a connection edge through an image pair, then it is determined that there is a connection break between the two image nodes due to the missing image, and this is confirmed by manual judgment.

[0019] In some embodiments, the steps of constructing a virtual node include:

[0020] Based on the spatial location and orientation information of adjacent image nodes with spatial connection breaks in the connection graph, the intermediate position and average orientation between them are calculated as the spatial attributes of the virtual node.

[0021] Using a virtual node as the center, construct edges connecting it to the two adjacent image nodes before and after it, and add them to the connection graph to restore the graph connectivity.

[0022] In some embodiments, the construction of the virtual image includes:

[0023] Extract the edge region image content of adjacent images on both sides of the fracture location, and project the edge images onto the same intermediate coordinate system based on the spatial position and pose information of the two images;

[0024] In the intermediate coordinate system, an interpolation filling operation is performed on the blank area between the two edge images to generate a virtual image containing the edge images and the intermediate smooth transition area, which is used to replace the missing image and participate in the stitching process.

[0025] In some embodiments, the step of calculating the geometric transformation parameters between any two connected image nodes in the connection map includes:

[0026] Based on the spatial location and pose information of each image, the viewpoint transformation relationship between adjacent images is initially estimated, and an initial geometric transformation model is generated;

[0027] Based on the initial geometric transformation model, local image features are extracted from the two images, and corresponding point pairs in the images are obtained using a feature matching algorithm;

[0028] Based on the corresponding point pairs in the images, the initial geometric transformation model is corrected by an optimization method that minimizes the projection error, resulting in the final geometric transformation parameters between the images.

[0029] In some embodiments, the method further includes: calculating the corresponding image confidence score based on the attitude change rate and acceleration jitter amplitude of the UAV during the acquisition of each image;

[0030] The image with the highest confidence score is selected as the reference image, and its image coordinate system is defined as a unified coordinate system.

[0031] In some embodiments, the method further includes: prioritizing the fusion of high-confidence images based on the ranking results of the image confidence scores.

[0032] In some embodiments, the method further includes: in the output stitched image, performing attribute marking on the spatial region corresponding to the virtual image.

[0033] In some embodiments, the method further includes: the attribute includes a no-fly zone or a privacy-blocking zone.

[0034] The advantages of this invention compared to existing technologies lie in its provision of an automatic airborne visible light image stitching method. Addressing the common image loss issues encountered by UAVs during flight, particularly in areas uncollectible due to no-fly zones and privacy protection zones, this invention proposes a stitching strategy combining graph-based modeling and virtual compensation. By constructing a connectivity graph with images as nodes and registered image pairs as edges, and automatically inserting virtual nodes and corresponding virtual images at breaks in the graph, the spatial connectivity of the image sequence is effectively restored. The generation of virtual images not only considers the geometric relationships and edge content between images but also supports attribute labeling of the filled areas, such as no-fly zones, privacy-occluded areas, or uncollectible areas, providing semantic guidance for subsequent image fusion and display, and preventing misunderstandings and misuse. Furthermore, this invention introduces a confidence scoring mechanism based on attitude stability, prioritizing images with high stability as references during the stitching process, thus improving resampling accuracy and fusion quality. This invention supports adaptive stitching processing in complex scenarios, significantly improving the stitching integrity, visual consistency, and engineering adaptability of UAV images. It is particularly suitable for application scenarios with large-area field of view limitations, such as urban environments, mountainous areas, and infrastructure inspections, and has good practicality and promotional value. Attached Figure Description

[0035] Figure 1 This is the overall flowchart of the method of the present invention;

[0036] Figure 2 This is a schematic diagram of the virtual image generation of the present invention. Detailed Implementation

[0037] The specific embodiments of the present invention will now be described with reference to the accompanying drawings.

[0038] This invention discloses an automatic visible light image stitching method for airborne applications, aiming to solve the problem of stitching difficulties caused by image gaps during UAV flight, especially in areas where image acquisition is impossible, such as no-fly zones and privacy protection zones. Through techniques such as atlas modeling, virtual compensation, and geometric transformations, this method achieves efficient stitching of multiple images.

[0039] like Figure 1 As shown, the method of the present invention specifically includes the following steps:

[0040] Acquire multiple images captured by the UAV during flight, along with their corresponding spatial position and attitude information;

[0041] Based on the spatial positional relationship between images and image overlap information, registerable image pairs are determined and a connection graph is constructed. The connection graph uses images as nodes and registerable image pairs as edges to describe the spatial topological relationship between images.

[0042] Detect whether there is a spatial connection break in the connection map due to image missingness. If there is a spatial connection break, insert a virtual node at the break position and construct a corresponding virtual image for the virtual node. The virtual image includes the edge region of adjacent images and the transition region between the edge regions.

[0043] For any two connected image nodes in the connection graph, the geometric transformation parameters between the images are calculated based on their spatial position and orientation information.

[0044] Using any image other than the virtual image as a reference image, based on the geometric transformation parameters and the connection relationships in the connection map, all image contents are resampled to a unified coordinate system, and pixel-level fusion processing is performed in the overlapping areas of the images.

[0045] More specifically, in the implementation of this invention, the drone acquires multiple images during flight using an onboard visible light camera. Each image not only contains pixel content but also records its corresponding spatial location and attitude information. Spatial location information typically includes latitude, longitude, and altitude, which can be obtained through the drone's GPS module or inertial navigation system (INS), with an accuracy generally within the meter range. Attitude information includes yaw, pitch, and roll angles, provided by the onboard inertial measurement unit, with angular accuracy typically around 0.1 degrees. This information provides the foundation for subsequent image registration and stitching. For example, when flying over a field, the drone may take multiple images along a straight path at a fixed altitude, such as 100 meters, with the shooting position and camera orientation of each image precisely recorded.

[0046] To clarify the spatial relationships between multiple images and lay the foundation for image stitching, a connectivity graph needs to be constructed. This graph uses images as nodes and registered image pairs as edges, reflecting the spatial topology between the images.

[0047] When determining which images can be registered, the spatial relationship between the two images is first analyzed. The Euclidean distance between their shooting locations is calculated, and if the distance is less than a preset distance threshold (e.g., 50 meters), the two images are considered to be spatially close enough.

[0048] Image pairs that meet the above conditions still need further verification of content overlap. Edge regions of the two images are extracted, and an overlap index is calculated. Overlap can be assessed using feature matching methods, such as SIFT (Scale Invariant Feature Transform) to extract keypoints and calculate the matching point pair ratio, or pixel-level similarity, such as normalized cross-correlation coefficient, to evaluate the similarity of overlapping regions. If the overlap exceeds a set threshold, for example, 10% (which can be adjusted according to image resolution and application requirements), the two images are determined to be a registrable image pair. For example, if two images respectively capture adjacent areas of farmland, and their overlapping areas contain the same landmark, such as a tree, a registration relationship can be confirmed.

[0049] Based on the identified registerable image pairs, a connectivity graph is constructed. Each node in the graph represents an image, and each edge indicates a registration relationship between two images. This structure is similar to an undirected graph and can be stored using an adjacency matrix or adjacency list. For example, suppose a drone captured five images I1 to I5, where I1 is registered with I2, I2 with I3, I3 with I4, and I4 with I5. A path from I1 to I5 will then be formed in the graph. This topological structure provides a basis for subsequent detection of missing regions and image stitching.

[0050] Because no-fly zones or certain private areas prohibit photography, image sequences may contain gaps, causing breaks in the connectivity map. Detecting and repairing these breaks is a crucial step in ensuring the integrity of the stitching.

[0051] To identify structural breaks caused by image gaps during image stitching, this invention introduces a mechanism for constructing and analyzing the connectivity of spatial geometric connectivity graphs. Specifically:

[0052] Traverse all image node pairs and filter out node pairs whose spatial distance is less than the adjacency threshold, and record them as the adjacency candidate set.

[0053] For each pair of candidate nodes, perform a connectivity search in the graph structure, such as using depth-first search, breadth-first search, or Dijkstra's path detection algorithm, to determine whether there is a connecting path between the node pairs consisting of registered edges.

[0054] If a candidate node pair should be spatially adjacent but lacks a path connection in the graph structure and there are no registration nodes between them, it is identified as a potential image breakpoint. This is then combined with manual assessment to determine whether the location is indeed an image-missing region, such as a no-fly zone.

[0055] In practical applications, if the area covered by the photograph is not particularly large, the attributes of the missing region in the image can be determined manually. Specifically, users can identify no-fly zones, privacy protection zones, or other restricted areas within the flight area based on aviation management documents, local regulations, or mission planning constraints, and determine the geographical location of these areas and their positional relationship with other photographs already taken, thereby determining the location of the image break.

[0056] Upon detecting a break, a virtual node is inserted at the break location to restore connectivity. Assuming the break occurs between I2 and I4, the spatial midpoints of I2 and I4 are calculated, including average latitude, longitude, and altitude, as well as average attitude information such as yaw, pitch, and roll angles. These are used as the spatial attributes of the virtual node V3. Then, connecting edges are added from V3 to I2 and from V3 to I4, centered on V3, updating the graph structure. The addition of the virtual node reconnects the graph, providing a bridge for subsequent stitching.

[0057] like Figure 2 As shown, virtual nodes require corresponding virtual images to participate in the stitching. The edge regions of I2 and I4 are extracted, such as the right edge of I2 and the left edge of I4. Then, based on their spatial position and pose information, these edges are projected into an intermediate coordinate system. This intermediate coordinate system can be calculated through interpolation, for example, by taking the intermediate state of the camera poses of I2 and I4. Next, the blank areas between the two edges are filled. The filling method can be linear interpolation, i.e., generating a transition region based on the color gradient of the edge pixels; or higher-order spline interpolation can be used to obtain a more natural visual effect. For example, if the edge of I2 is a green farmland and the edge of I4 is a yellow wheat field, the virtual image will generate a smooth transition region from green to yellow as a replacement for the missing image.

[0058] For any two connected nodes in the graph, including real and virtual images, it is necessary to calculate the geometric transformation parameters between them in order to achieve image alignment.

[0059] Based on the spatial location and pose information of each image, the perspective transformation relationship between adjacent images is initially estimated. For example, if the camera yaw angle of I1 is 0 degrees and that of I2 is 5 degrees, an affine transformation involving rotation and translation can be assumed. The initial transformation model can be a two-dimensional affine transformation, including rotation, scaling, and translation, or a perspective transformation, taking into account camera perspective distortion.

[0060] Based on the initial model, local feature points of the image are extracted, for example, by using the SURF algorithm to detect keypoints and generate descriptors. Then, the feature points are matched using the RANSAC algorithm, and reliable corresponding point pairs are obtained after removing outliers. Based on these point pairs, the initial model is optimized using the least squares method to minimize the projection error. For example, if a point (x1, y1) in I1 should be mapped to (x2, y2) in I2, the optimization objective is to minimize the sum of squared projection errors of all matched points. The final transformation parameters can accurately describe the geometric relationship between the two images.

[0061] Using an image outside the virtual image as a reference image, all images are resampled to a unified coordinate system, and pixel fusion in overlapping areas is processed.

[0062] To ensure stitching quality, a confidence score is calculated based on the drone's status at the time of image acquisition. The scoring formula can be as follows:

[0063] ;

[0064] Among them, the attitude change rate is the rate of change of the UAV's orientation, reflecting the stability of the camera's orientation; the acceleration jitter amplitude is the change in the UAV's linear acceleration relative to the average acceleration, reflecting the flight stability.

[0065] The formula is designed to give higher scores to images with stable attitude and smooth flight. The image with the highest score is selected as the reference image, and its coordinate system is defined as a unified coordinate system.

[0066] Based on the connectivity relationships in the atlas, adjacent images are resampled to the reference coordinate system level by level through geometric transformation parameters. Overlapping regions are processed using pixel-level fusion techniques, such as the weighted average method: if pixel P has a value of P1 in I1 and a value of P2 in I2, then the fused value is... ,in The weights can be dynamically adjusted based on the distance of each pixel to the overlapping boundary, with pixels closer to I1 being assigned a higher I1 weight. Furthermore, multi-resolution blending, such as pyramid blending, can be used to further eliminate stitching artifacts.

[0067] Virtual images are processed in the same way as real images when used in stitching, but because they are generated by interpolation, attribute labeling can be added to the output. For example, in the final stitched image, virtual areas can be marked with colored borders and given semantic labels such as "no-fly zone" or "privacy occlusion zone" to indicate to the user the virtual nature of the content in that area.

[0068] To improve the stitching effect, some embodiments may prioritize the fusion of images with high confidence levels, processing them sequentially according to their scores from highest to lowest. Furthermore, multiple types of attribute labeling are supported for virtual image regions, including "uncaptureable areas," "no-fly zones," or "privacy-obstructed areas," which can be selected based on the actual scenario. These labels not only enhance the interpretability of the images but also avoid the risk of misuse.

[0069] Assume a drone takes images over a city, with a no-fly zone causing a gap in the central area. Images I1 and I2 cover the left side of the city, while I4 and I5 cover the right side. After constructing the map, a break is found between I2 and I4; a virtual node V3 is inserted, generating a virtual image V3'. Through geometric transformation, I1, I2, V3', I4, and I5 are stitched together to form a complete urban landscape, with the V3' area marked as the "no-fly zone." The final stitched image is both coherent and semantically clear.

[0070] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An airborne visible light image automatic stitching method, characterized by, The method comprises the following steps: acquiring a plurality of images collected by a UAV during flight and corresponding spatial position information and attitude information of the images; determining image pairs that can be registered based on spatial position relationships between the images and image overlap information, and constructing a connection graph, the connection graph taking the images as nodes and the image pairs that can be registered as edges, and being used to describe spatial topological relationships between the images; detecting whether there is a spatial connection break caused by missing images in the connection graph, and if there is a spatial connection break, inserting a virtual node at the break position and constructing a corresponding virtual image for the virtual node, the virtual image comprising edge regions of adjacent images and smooth transition regions between the edge regions; for any two connected image nodes in the connection graph, calculating geometric transformation parameters between the images based on spatial position and attitude information of the image nodes; taking any image outside the virtual image as a reference image, resampling all image contents to a unified coordinate system according to the geometric transformation parameters and connection relationships in the connection graph, and performing pixel-level fusion processing in image overlap regions; the step of constructing the virtual node comprises: calculating an intermediate position and an average attitude between two adjacent image nodes with a spatial connection break in the connection graph as spatial attributes of the virtual node based on spatial position information and attitude information of the two adjacent image nodes; constructing edges connected to the front and rear adjacent image nodes with the virtual node as a center, and adding the edges to the connection graph to restore graph connectivity; the step of constructing the virtual image comprises: extracting edge region image contents of adjacent images on both sides of the break position, and projecting the edge images to the same intermediate coordinate system according to spatial position information and attitude information of the two images; in the intermediate coordinate system, performing an interpolation filling operation on a blank region between the two edge images to generate a virtual image containing the edge images and intermediate smooth transition regions, which is used to replace the missing image and participate in the stitching process.

2. The method according to claim 1, wherein the step of determining the image pairs that can be registered comprises: calculating spatial position relationships corresponding to any two images, and judging whether a spatial distance between the shooting positions is less than a preset distance threshold; in the image pairs that meet the spatial position and attitude constraint conditions, further extracting image contents, calculating an overlap degree index of image edge regions, and determining the image pairs as image pairs that can be registered when the overlap degree exceeds a set threshold.

3. The method of claim 1, wherein the method further comprises: the step of detecting whether there is a spatial connection break caused by missing images in the connection graph comprises: constructing a spatial geometric connectivity graph corresponding to the graph based on spatial position information of the image nodes and relative position relationships between the images corresponding to the connection edges; performing connectivity analysis on the geometric connectivity graph, and if there are two image nodes that should be adjacent but are not connected by a connection edge through an image pair, it is determined that there is a connection break between the two image nodes caused by missing images, and the determination is confirmed in combination with manual judgment.

4. The method of claim 1, wherein the method further comprises: the step of calculating geometric transformation parameters between the images for any two connected image nodes in the connection graph comprises: initially estimating a perspective transformation relationship between adjacent images based on spatial position and attitude information of each image, and generating an initial geometric transformation model; On the basis of the initial geometric transformation model, local image features in the two images are extracted, and a feature matching algorithm is used to obtain image corresponding point pairs; Based on the image corresponding point pairs, the initial geometric transformation model is corrected by an optimization method of minimizing projection error to obtain final geometric transformation parameters between the images.

5. The method of claim 1, wherein the method further comprises: The method further includes calculating a corresponding image confidence score based on a rate of change of the attitude of the unmanned aerial vehicle and an acceleration jitter amplitude during acquisition of each image; The image with the highest confidence score is selected as a reference image, and an image coordinate system thereof is defined as a unified coordinate system.

6. The method according to claim 5, wherein The method further includes preferentially fusing high-confidence images according to a sorting result of the image confidence scores.

7. The method of claim 1, wherein the method further comprises: The method further includes marking a spatial region corresponding to a virtual image with an attribute in the output spliced image.

8. The method according to claim 7, wherein The attribute includes a no-fly zone or a privacy shielding zone.

Images