Airborne visible light image automatic splicing method

Abstract

The invention discloses an airborne visible light image automatic splicing method, and relates to the field of unmanned aerial vehicle image processing, and the method comprises the steps: obtaining a plurality of images collected in the flight process of an unmanned aerial vehicle, and the corresponding spatial position information and attitude information; on the basis of the spatial position relation between the images and the image overlapping information, determining image pairs capable of being registered, and constructing a connection map with the images as nodes and the image pairs capable of being registered as edges; detecting whether spatial connection fracture caused by image missing exists in the atlas or not, if so, inserting a virtual node and constructing a virtual image comprising an edge region and a transition region; and further calculating geometric transformation parameters between the images, resampling all image contents to a unified coordinate system, and executing pixel-level fusion processing in an image overlapping region. According to the invention, the problem of discontinuous image splicing of the unmanned aerial vehicle due to the existence of a no-fly zone, a privacy protection zone and the like of the unmanned aerial vehicle is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of unmanned aerial vehicle image processing, and more particularly, to an airborne visible light image automatic stitching method. BACKGROUND

[0002] With the wide application of unmanned aerial vehicle aerial photography technology, airborne visible light images are widely used in fields such as surveying and mapping, agricultural inspection, city modeling, and emergency rescue. Generally, the unmanned aerial vehicle continuously collects ground images through the camera carried during flight, and then stitches multiple images to generate a high-resolution panoramic image or orthographic image covering the area. In order to achieve accurate stitching, it is often necessary to rely on spatial overlap between images, feature matching, and geometric transformation model.

[0003] However, in actual flight tasks, there are often serious image missing problems that affect the continuity and integrity of image stitching due to the following reasons. For example, some areas (such as airports, military restricted areas, and power facilities) are explicitly designated as no-fly zones for unmanned aerial vehicles, and the unmanned aerial vehicle cannot collect images in these areas. For example, in urban environments, some residential areas, private buildings, or sensitive areas are not allowed to be photographed or need to avoid detailed presentation due to privacy protection requirements.

[0004] The above reasons cause holes, breaks, or misalignments in the stitched images, especially when there is image missing in the key connection area, which can cause the overall stitching structure to be interrupted, affecting the integrity and accuracy of the final image. SUMMARY

[0005] The technical problem to be solved by the present application is to provide an airborne visible light image automatic stitching method to solve the problems mentioned in the background.

[0006] In order to achieve the above purpose, the present application adopts the following technical scheme: An airborne visible light image automatic stitching method, comprising the following steps: Obtaining multiple images collected by the unmanned aerial vehicle during flight and their corresponding spatial position information and attitude information; Based on the spatial position relationship and image overlap information between the images, determining the image pairs that can be registered, and constructing a connection graph, the connection graph taking images as nodes and image pairs that can be registered as edges, for describing the spatial topological relationship between images; Detecting whether there is a spatial connection break caused by image missing in the connection graph, if there is a spatial connection break, inserting a virtual node at the break position, and constructing a corresponding virtual image for the virtual node, the virtual image including the edge regions of adjacent images and the smooth transition regions between the edge regions; For any two connected image nodes in the connection graph, based on their spatial position and attitude information, the geometric transformation parameters between the images are calculated. Taking any image outside the virtual image as a reference image, all image contents are resampled to a unified coordinate system according to the geometric transformation parameters and the connection relationship in the connection graph, and pixel-level fusion processing is performed in the image overlapping area.

[0007] In some embodiments, the step of determining the image pairs that can be registered comprises: The spatial position relationship corresponding to any two images is calculated, and it is judged whether the spatial distance between the shooting positions is less than a preset distance threshold; In the image pairs that meet the spatial position and attitude constraint conditions, the image content is further extracted, the overlap index of the edge region of the image is calculated, and when the overlap exceeds a set threshold, it is determined as an image pair that can be registered.

[0008] In some embodiments, the step of detecting whether there is a spatial connection break caused by missing images in the connection graph comprises: Based on the spatial position information of the image nodes and the relative position relationship between the images corresponding to the connection edges, a spatial geometric connected graph corresponding to the graph is constructed; The connectivity analysis is performed on the geometric connected graph, if there are two image nodes that should be adjacent but have not established a connection edge through an image pair, it is determined that there is a connection break between the two image nodes caused by missing images, and the artificial judgment is confirmed.

[0009] In some embodiments, the step of constructing a virtual node comprises: According to the spatial position information and attitude information of the adjacent image nodes with spatial connection break in the connection graph, the intermediate position and the average attitude between them are calculated as the spatial attributes of the virtual node; Taking the virtual node as the center, edges connected with the front and rear two adjacent image nodes are constructed, and they are added to the connection graph to restore the connectivity of the graph.

[0010] In some embodiments, the construction of the virtual image comprises: The edge region image content of the adjacent images on both sides of the break position is extracted, and the edge images are projected into the same intermediate coordinate system according to the spatial position information and attitude information of the two images; In the intermediate coordinate system, the interpolation filling operation is performed on the blank area between the two edge images to generate a virtual image containing the edge images and the intermediate smooth transition region, which is used to replace the missing image and participate in the stitching processing.

[0011] In some embodiments, for any two connected image nodes in the connection graph, the step of calculating the geometric transformation parameters between the images comprises: Based on the spatial position and attitude information of each image, the perspective transformation relationship between adjacent images is preliminarily estimated, and an initial geometric transformation model is generated; On the basis of the initial geometric transformation model, local image features in the two images are extracted, and image corresponding point pairs are obtained by using a feature matching algorithm; Based on the image corresponding point pairs, the initial geometric transformation model is corrected by an optimization method of minimizing projection error to obtain final geometric transformation parameters between the images.

[0012] In some embodiments, the method further comprises: based on the attitude change rate and acceleration jitter amplitude of the unmanned aerial vehicle during image acquisition of each image, calculating a corresponding image confidence score; Selecting an image with the highest confidence score as a reference image, and defining the image coordinate system thereof as a unified coordinate system.

[0013] In some embodiments, the method further comprises: according to the sorting result of the image confidence score, preferentially fusing high-confidence images.

[0014] In some embodiments, the method further comprises: the method further comprises marking a spatial region corresponding to a virtual image with an attribute in the output spliced image.

[0015] In some embodiments, the method further comprises: the attribute comprises a no-fly zone or a privacy shielding zone.

[0016] The present application has the advantages over the prior art that the present application provides an airborne visible light image automatic splicing method, which proposes a splicing strategy combining graph modeling and virtual compensation for the image missing problem often occurring during the flight of an unmanned aerial vehicle, especially for the uncollectable area caused by a no-fly zone and a privacy protection zone. By constructing a connection graph with images as nodes and image pairs as edges, and automatically inserting virtual nodes and corresponding virtual images at the broken places in the graph, the spatial connectivity of the image sequence is effectively restored. The generation of virtual images not only considers the geometric relationship and edge content between images, but also supports attribute marking of the filled area, such as a no-fly zone, a privacy shielding zone or an uncollectable area, providing semantic guidance for subsequent image fusion and display to prevent misinterpretation and misuse. In addition, the present application also introduces a confidence score mechanism based on attitude stability, which makes the splicing process prefer to use images with high stability as a reference, improving the resampling accuracy and fusion quality. The present application supports adaptive splicing processing in complex scenes, significantly improving the splicing integrity, visual consistency and engineering adaptability of unmanned aerial vehicle images, and is particularly suitable for application scenarios such as urban environments, mountainous areas and infrastructure inspection where there is a large area of visual restriction, and has good practicality and promotional value. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a general flowchart of the method of the present application; Figure 2is a schematic diagram of virtual image generation of the present application. DETAILED DESCRIPTION

[0018] The specific embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0019] The present application discloses an airborne visible light image automatic stitching method, aiming to solve the problem of difficult stitching caused by image loss during the flight of a UAV, especially in the case of no-fly zones, privacy protection zones and other non-acquisition areas. Through techniques such as graph modeling, virtual compensation and geometric transformation, this method realizes efficient stitching of multiple images.

[0020] As shown in Figure 1 The method of the present application specifically includes the following steps: Obtaining multiple images collected by a UAV during flight and their corresponding spatial position information and attitude information; Based on the spatial position relationship and image overlap information between images, determining the image pairs that can be registered, and constructing a connection graph, which takes images as nodes and image pairs that can be registered as edges, for describing the spatial topological relationship between images; Detecting whether there is a spatial connection break caused by image loss in the connection graph, and if there is a spatial connection break, inserting a virtual node at the break position and constructing a corresponding virtual image for the virtual node, the virtual image including the edge regions of adjacent images and the transition regions between the edge regions; For any two connected image nodes in the connection graph, based on their spatial position and attitude information, the geometric transformation parameters between images are calculated; Taking any image outside the virtual image as a reference image, resampling all image contents to a unified coordinate system according to the geometric transformation parameters and the connection relationship in the connection graph, and performing pixel-level fusion processing in the image overlap area.

[0021] More specifically, during the implementation of the present application, the UAV collects multiple images through an airborne visible light camera during flight. Each image not only contains pixel content, but also records its corresponding spatial position information and attitude information. Spatial position information usually includes latitude, longitude and height, which can be obtained through the GPS module or inertial navigation system INS of the UAV, with an accuracy generally within the range of meters. Attitude information includes yaw angle, pitch angle and roll angle, which is provided by the onboard inertial measurement unit, with an angle accuracy of about 0.1 degrees. These information provides a basis for subsequent image registration and stitching. For example, when flying over a piece of farmland, the UAV may take multiple images along a straight path at a fixed height, such as 100 meters, and the shooting position and camera orientation of each image are accurately recorded.

[0022] To clarify the spatial relationship between multiple images and lay the foundation for stitching, a connection graph needs to be constructed. This graph takes images as nodes and pairs of registerable images as edges, reflecting the spatial topology between images.

[0023] When determining which images can be registered, the spatial positional relationship of two images is first analyzed. The Euclidean distance between their shooting positions is calculated, and if the distance is less than a pre-set distance threshold (e.g., 50 meters), it is considered that the two images are close enough in space.

[0024] Image pairs that meet the above conditions also need to be further verified for content overlap. The edge regions of the two images are extracted, and an overlap index is calculated. Overlap can be achieved through feature matching methods, such as using SIFT, scale-invariant feature transform to extract key points and calculate the proportion of matching point pairs, or evaluating the similarity of overlapping regions through pixel-level similarity, such as normalized cross-correlation coefficient. If the overlap exceeds a set threshold, e.g., 10% (which can be adjusted according to image resolution and application requirements), the two images are determined to be a registerable image pair. For example, two images have captured adjacent areas of farmland, and if their overlapping area contains the same landmark, such as a tree, the registration relationship can be confirmed.

[0025] Based on the determined registerable image pairs, a connection graph is constructed. Each node in the graph represents an image, and each edge represents a registration relationship between two images. This structure is similar to an undirected graph and can be stored using an adjacency matrix or adjacency list. For example, assume that a UAV has captured five images I1 to I5, where I1 and I2, I2 and I3, I3 and I4, and I4 and I5 have registration relationships. In the graph, a path from I1 to I5 will be formed. This topological structure provides a basis for subsequent detection of missing areas and stitching.

[0026] Due to no-fly zones or some private areas not allowing photography, there may be missing images in the sequence, causing breaks in the connection graph. Detecting and repairing these breaks is a key step to ensure the integrity of the stitching.

[0027] To identify structural breaks caused by missing images during image stitching, the invention introduces a mechanism for constructing and analyzing the connectivity of spatial geometric graphs. Specifically: All image node pairs are traversed, and node pairs with a spatial distance less than the adjacency threshold are selected, denoted as the adjacency candidate set; For each pair of candidate nodes, perform connectivity search in the graph structure, such as using depth-first search, breadth-first search, or Dijkstra path detection algorithm, to determine whether there is a connection path composed of registration edges between the node pairs; If a candidate node pair should be spatially adjacent but lacks a path connection in the graph structure and there are no registration nodes between them, it is identified as a potential image breakpoint. This is then combined with manual assessment to determine whether the location is indeed an image-missing region, such as a no-fly zone.

[0028] In practical applications, if the area covered by the photograph is not particularly large, the attributes of the missing region in the image can be determined manually. Specifically, users can identify no-fly zones, privacy protection zones, or other restricted areas within the flight area based on aviation management documents, local regulations, or mission planning constraints, and determine the geographical location of these areas and their positional relationship with other photographs already taken, thereby determining the location of the image break.

[0029] Upon detecting a break, a virtual node is inserted at the break location to restore connectivity. Assuming the break occurs between I2 and I4, the spatial midpoints of I2 and I4 are calculated, including average latitude, longitude, and altitude, as well as average attitude information such as yaw, pitch, and roll angles. These are used as the spatial attributes of the virtual node V3. Then, connecting edges are added from V3 to I2 and from V3 to I4, centered on V3, updating the graph structure. The addition of the virtual node reconnects the graph, providing a bridge for subsequent stitching.

[0030] like Figure 2 As shown, virtual nodes require corresponding virtual images to participate in the stitching. The edge regions of I2 and I4 are extracted, such as the right edge of I2 and the left edge of I4. Then, based on their spatial position and pose information, these edges are projected into an intermediate coordinate system. This intermediate coordinate system can be calculated through interpolation, for example, by taking the intermediate state of the camera poses of I2 and I4. Next, the blank areas between the two edges are filled. The filling method can be linear interpolation, i.e., generating a transition region based on the color gradient of the edge pixels; or higher-order spline interpolation can be used to obtain a more natural visual effect. For example, if the edge of I2 is a green farmland and the edge of I4 is a yellow wheat field, the virtual image will generate a smooth transition region from green to yellow as a replacement for the missing image.

[0031] For any two connected nodes in the graph, including real and virtual images, it is necessary to calculate the geometric transformation parameters between them in order to achieve image alignment.

[0032] Based on the spatial location and pose information of each image, the perspective transformation relationship between adjacent images is initially estimated. For example, if the camera yaw angle of I1 is 0 degrees and that of I2 is 5 degrees, an affine transformation involving rotation and translation can be assumed. The initial transformation model can be a two-dimensional affine transformation, including rotation, scaling, and translation, or a perspective transformation, taking into account camera perspective distortion.

[0033] On the basis of the initial model, local feature points of the images are extracted, for example, using the SURF algorithm to detect key points and generate descriptors. Then, the feature points are matched by the RANSAC algorithm, and after removing outliers, reliable corresponding point pairs are obtained. Based on these point pairs, the initial model is optimized by the least squares method to minimize the projection error. For example, if the point (x1, y1) in I1 should be mapped to (x2, y2) in I2, the optimization goal is to minimize the sum of squares of the projection errors of all matched points. The final transformation parameters can accurately describe the geometric relationship between the two images.

[0034] With an image outside the virtual image as the reference image, all images are resampled to a unified coordinate system, and the pixel fusion in the overlapping area is processed.

[0035] To ensure the quality of the stitching, a confidence score is calculated based on the state of the unmanned aerial vehicle during image acquisition. The scoring formula can be as follows: ; Among them, the attitude change rate is the change rate of the unmanned aerial vehicle towards the change rate, reflecting the stability of the camera towards the change rate; the acceleration jitter amplitude is the change amplitude of the linear acceleration of the unmanned aerial vehicle relative to the average acceleration, reflecting the flight stability.

[0036] The purpose of the formula design is to give higher scores to images with stable attitude and smooth flight. The image with the highest score is selected as the reference image, and its coordinate system is defined as the unified coordinate system.

[0037] According to the connection relationship in the atlas, adjacent images are resampled to the reference coordinate system through geometric transformation parameters step by step. The overlapping area is processed using pixel-level fusion technology, for example, the weighted average method: if the value of pixel P in I1 is P1 and the value in I2 is P2, then the fusion value is , wherein The weight can be dynamically adjusted according to the distance of the pixel to the overlapping boundary, and the pixel close to I1 is given a higher I1 weight. In addition, multi-resolution fusion such as pyramid fusion can be used to further eliminate stitching marks.

[0038] The virtual image is processed in the same way as the real image when participating in the stitching, but since it is generated by interpolation, it can be marked with attributes in the output. For example, in the final stitched image, the virtual area can be marked with a color border and accompanied by a semantic label such as "no-fly zone" or "privacy shielding area" to prompt the user that the content of this area is virtual.

[0039] To improve the stitching effect, in some embodiments, images with high confidence can be fused first, and then processed in order from high to low score. In addition, the attribute labels of the virtual image region support multiple types, including "no ability to collect region", "no-fly zone" or "privacy shielding area", which can be selected according to the actual scene. These labels not only enhance the interpretability of the image, but also avoid misuse.

[0040] Suppose the UAV is taking pictures in the sky over the city, and the no-fly zone causes the middle region to be missing. The collected images I1 and I2 cover the left region, and I4 and I5 cover the right region. After constructing the graph, it is found that there is a break between I2 and I4, a virtual node V3 is inserted, and a virtual image V3' is generated. Through geometric transformation, I1, I2, V3', I4, and I5 are stitched into a complete cityscape, and the V3' region is labeled as "no-fly zone". The final output stitching image is not only coherent but also has clear semantics.

[0041] The above description is only a preferred embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can make equivalent replacements or changes within the technical range disclosed by the present application according to the technical solution and the inventive concept of the present application, which should be covered within the protection scope of the present application.

Claims

1. An airborne visible light image automatic stitching method, characterized by, The method comprises the following steps: acquiring a plurality of images and corresponding spatial position information and attitude information collected by a UAV during flight; determining image pairs that can be registered based on spatial position relationships and image overlap information between the images, and constructing a connection graph, the connection graph taking the images as nodes and the image pairs that can be registered as edges, and being used to describe spatial topological relationships between the images; detecting whether there is a spatial connection break caused by missing images in the connection graph, and if there is a spatial connection break, inserting a virtual node at the break position and constructing a corresponding virtual image for the virtual node, the virtual image comprising edge regions of adjacent images and smooth transition regions between the edge regions; for any two connected image nodes in the connection graph, calculating geometric transformation parameters between the images based on spatial position and attitude information of the image nodes; taking any image outside the virtual image as a reference image, resampling all image contents to a unified coordinate system according to the geometric transformation parameters and connection relationships in the connection graph, and performing pixel-level fusion processing in image overlap regions.

2. The method according to claim 1, wherein The step of determining image pairs that can be registered comprises: calculating spatial position relationships corresponding to any two images, and determining whether a spatial distance between the image capturing positions is less than a preset distance threshold; in image pairs that satisfy spatial position and attitude constraint conditions, further extracting image contents, calculating an overlap index of image edge regions, and determining the image pairs as image pairs that can be registered when the overlap index exceeds a set threshold.

3. The method of claim 1, wherein the method further comprises: The step of detecting whether there is a spatial connection break caused by missing images in the connection graph comprises: constructing a spatial geometric connectivity graph corresponding to the connection graph based on spatial position information of the image nodes and relative position relationships between the images corresponding to the connection edges; performing connectivity analysis on the geometric connectivity graph, and if there are two image nodes that should be adjacent but are not connected by an image pair, determining that there is a connection break between the two image nodes caused by missing images, and confirming the determination in combination with manual judgment.

4. The method of claim 1, wherein the method further comprises: The step of constructing a virtual node comprises: calculating a middle position and an average attitude between two adjacent image nodes with a spatial connection break in the connection graph as spatial attributes of the virtual node based on spatial position information and attitude information of the two adjacent image nodes; constructing edges connected to the front and rear adjacent image nodes with the virtual node as the center, and adding the edges to the connection graph to restore connectivity of the connection graph.

5. The method of claim 1, wherein the method further comprises: The step of constructing a virtual image comprises: extracting edge region image contents of adjacent images on both sides of the break position, and projecting the edge images to a same intermediate coordinate system based on spatial position information and attitude information of the two images; performing interpolation filling operation on a blank region between the two edge images in the intermediate coordinate system to generate a virtual image comprising the edge images and intermediate smooth transition regions, and using the virtual image to replace the missing image and participate in stitching processing.

6. The method of claim 1, wherein, The step of calculating geometric transformation parameters between images for any two connected image nodes in the connection graph comprises: initially estimating a perspective transformation relationship between adjacent images based on spatial position and attitude information of each image to generate an initial geometric transformation model; On the basis of the initial geometric transformation model, local image features in the two images are extracted, and a feature matching algorithm is used to obtain image corresponding point pairs; Based on the image corresponding point pairs, the initial geometric transformation model is corrected by an optimization method of minimizing projection error to obtain final geometric transformation parameters between the images.

7. The method of claim 1, wherein the method further comprises: The method further includes calculating a corresponding image confidence score based on a rate of change of the attitude of the unmanned aerial vehicle and an acceleration jitter amplitude during acquisition of each image; The image with the highest confidence score is selected as a reference image, and an image coordinate system thereof is defined as a unified coordinate system.

8. The method according to claim 7, wherein The method further includes preferentially fusing high-confidence images according to a sorting result of the image confidence scores.

9. The method of claim 1, wherein, The method further includes marking a spatial region corresponding to a virtual image with an attribute in the output spliced image.

10. The method of claim 1, wherein, The attribute includes a no-fly zone or a privacy shielding zone.

Images