An image splicing method and system based on double-layer splicing

Abstract

The application discloses a kind of based on double-layer splicing image splicing method and system, belong to image processing technical field.The method includes: step 1, the IMU and GPS data estimation image overlap degree using unmanned aerial vehicle aerial photography, remove redundant frame, and image set is clustered into multiple clusters;Step 2, first layer splicing is carried out, in cluster with the frame that the most stable attitude is reference, according to overlap degree and distance determine splicing order, registration, fusion in turn, obtain the splicing result in cluster;Step 3, second layer splicing is carried out, with each cluster center attitude as reference, determine splicing order and registration, combine similar transformation process non-overlapping area and grid local transformation process overlapping area, complete final fusion.The application is preprocessed by IMU and GPS, reduces redundant information interference;Adopt double-layer splicing strategy, frame-by-frame splicing in cluster relieves error accumulation, combine global and local transformation between clusters, improve splicing stability.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image stitching method and system based on two-layer stitching. Background Technology

[0002] In drone image stitching scenarios, due to mission planning or environmental interference, drones often cannot maintain strictly continuous shooting; this problem also exists when multiple drones shoot simultaneously. These factors lead to a lack of stable frame-to-frame correlations between images, making it impossible to directly rely on inter-frame constraints to establish an accurate alignment model. Simultaneously, discontinuous pose changes significantly increase the difficulty of initial alignment and subsequent registration. Existing methods typically rely on continuous viewpoints or stable paths, but often fail to achieve reliable geometric consistency when faced with multi-source shooting and significant pose shifts. Secondly, during aerial photography, the same area is often repeatedly covered by multiple frames, and multiple images tend to exhibit inconsistent local geometric structures in overlapping areas; the continuous superposition of multiple images leads to blurring. Finally, local errors and viewpoint drift accumulate and propagate globally, causing overall structural stretching and deformation. Furthermore, if images come from different altitudes, shooting times, or even different drone platforms, color differences further degrade the fusion quality, resulting in noticeable stitching artifacts if left untreated. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention proposes an image stitching method and system based on two-layer stitching.

[0004] The technical solution adopted in this invention is:

[0005] A two-layer image stitching method includes the following steps:

[0006] Step 1: Determine the overlap between images based on the IMU data and GPS data provided by the drone during aerial photography, remove redundant frames using the overlap, and cluster the image set after removing redundant frames into multiple clusters;

[0007] Step 2: Perform the first layer stitching within the multiple clusters defined in Step 1. First, calculate the pose stability based on the IMU data of each frame within the cluster, and select the frame with the most stable pose as the cluster center and use it as the reference frame. Determine the stitching order based on the overlap and the distance between the projection centers of the two images, and perform image registration. Frames that fail to register are discarded and re-clustered. Finally, stitch and fuse the images to obtain the first layer stitching result.

[0008] Step 3: Perform the second layer stitching based on the first layer stitching result obtained in Step 2. First, select the most stable pose from the poses of the cluster centers of each cluster, and select the first layer stitching result corresponding to the most stable pose as a reference to determine the stitching order and then perform registration. Then, calculate the global transformation based on similarity transformation to process the non-overlapping areas, and combine the local transformation of grid division to process the overlapping areas. Finally, obtain the second layer stitching result through image fusion to complete the image stitching.

[0009] Furthermore, based on the image stitching method based on two-layer stitching, this invention also proposes a corresponding image stitching system based on two-layer stitching, comprising the following modules:

[0010] Preprocessing module: used to determine the overlap between images based on IMU data and GPS data provided by the drone during aerial photography, remove redundant frames using the overlap, and cluster the image set after removing redundant frames into multiple clusters;

[0011] The first layer stitching module is used to calculate the attitude stability based on the IMU data of each frame of the image in the cluster, select the frame with the most stable attitude as the center of the cluster and use it as the reference frame, determine the stitching order based on the overlap and distance, and perform image registration. Frames that fail to register are removed and re-clustered. Finally, the images are stitched together and fused to obtain the first layer stitching result.

[0012] The second-layer stitching module is used to take the pose of the cluster center of each cluster as the pose stability of each stitching result, select the stitching result with the most stable pose as a reference, determine the stitching order and then perform registration, calculate the global transformation based on similarity transformation to process non-overlapping areas, combine the local transformation of grid division to process overlapping areas, and obtain the final stitching result through image fusion.

[0013] Image fusion module: In the first and second layer stitching, it uses the transformation matrix to calculate the overlapping area at each stitching and calculates the minimum bounding rectangle of the overlapping area. For each pixel in the overlapping area, it calculates the shortest distance to the bounding rectangles of the two images and performs weighted fusion.

[0014] The beneficial technical effects of the present invention are as follows:

[0015] (1) This invention proposes a stitching preprocessing scheme based on IMU and GPS assistance. From the perspective of motion and overlap, IMU and GPS are used to filter and organize the image sequence, reduce the interference of repeated information on stitching, and provide a more reasonable data basis for subsequent stable modeling.

[0016] (2) The present invention introduces a two-layer stitching strategy, which uses the attitude and position information provided by IMU and GPS to establish a unified spatial reference and divide the image set into clusters. The first layer stitching is performed frame by frame within the cluster to alleviate the problem of error accumulation during continuous stitching. The second layer stitching combines global similarity transformation and local transformation between clusters to ensure local geometric accuracy while also ensuring global consistency, thereby improving the overall stability of stitching.

[0017] (3) In view of the problem that the stitching boundary is obvious due to the difference in illumination and uneven overlap during the multi-frame fusion process, the present invention designs a targeted fusion and smoothing mechanism from the perspective of visual consistency, so that the final generated panoramic image maintains good consistency and realism in structure, texture and color. Attached Figure Description

[0018] Figure 1 This is a flowchart of the image stitching method based on double-layer stitching of the present invention;

[0019] Figure 2 This is a framework diagram for implementing the method of Embodiment 1 of the present invention;

[0020] Figure 3 This is a schematic diagram of the dataset to be spliced ​​in Embodiment 1 of the present invention;

[0021] Figure 4 This is a schematic diagram of the clustering results in Embodiment 1 of the present invention;

[0022] Figure 5 This is a schematic diagram of the first layer splicing result in Embodiment 1 of the present invention;

[0023] Figure 6 This is a schematic diagram of global and local transformations in image fusion during the second layer stitching of Embodiment 1 of the present invention;

[0024] Figure 7 This is a schematic diagram of the second layer splicing result in Embodiment 1 of the present invention. Detailed Implementation

[0025] This invention utilizes an Inertial Measurement Unit (IMU) and a Global Positioning System (GPS) to assist in image stitching. By leveraging IMU and GPS, it compensates for attitude and position estimation issues during the stitching process. IMU data is used to estimate the rotation between images, while GPS data is used to estimate their positional relationships. By combining both, the overlap between images can be calculated, thus supporting image stitching. The involvement of IMU and GPS improves the image stitching process and also enhances the accuracy of the stitched results.

[0026] This invention focuses on image stitching for UAV top-down photography and proposes a two-layer stitching method. This method utilizes IMU attitude information to obtain three-axis rotation data of the image, and GPS to obtain positional information between images. It estimates the overlap information between images using Ground Sampling Distance (GSD). Before stitching, the images are clustered, and within each cluster, Speeded-Up Robust Features (SURF) are extracted and matched. Feature point pairs are used to estimate the transformation relationship between adjacent images, and the stitching result for each cluster is obtained by stitching frame by frame. Further weighted stitching is performed between clusters. Specifically, when stitching between clusters, local transformations are estimated in overlapping areas, and global transformations are estimated in non-overlapping areas, ultimately yielding the final stitched result. This invention utilizes IMU and GPS for assistance. On the one hand, from the perspective of motion and overlap, IMU and GPS are used to filter and organize image sequences, reducing the interference of repetitive information on stitching and providing a more reasonable data foundation for subsequent stable modeling. On the other hand, by introducing double-layer stitching, the cumulative error of stitching is reduced, ensuring local geometric accuracy while also ensuring global consistency, thereby improving the overall stability of stitching.

[0027] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0028] Example 1:

[0029] like Figure 1 , Figure 2 As shown, an image stitching method based on two-layer stitching includes the following steps:

[0030] Step 1: Estimate the overlap between images based on the IMU and GPS data provided by the drone during aerial photography, use the overlap to remove redundant frames, and then cluster the image set after removing redundant frames into multiple clusters.

[0031] Step 2: Perform the first layer stitching within the multiple clusters divided in Step 1. First, calculate the pose stability based on the IMU data of each frame in the cluster, and select the frame with the most stable pose as the center of the cluster and use it as the reference frame. Determine the stitching order based on the overlap and the distance between the projection centers of the two images, and perform image registration. Frames that fail to register are removed and re-clustered. Finally, stitch the images sequentially and perform image fusion to obtain the first layer stitching result.

[0032] Step 3: Perform the second layer stitching based on the first layer stitching result obtained in Step 2. First, take the pose of the cluster center of each cluster as the pose stability of each stitching result, and select the stitching result with the most stable pose as a reference to determine the stitching order and then perform registration. Then, calculate the global transformation based on similarity transformation to process the non-overlapping areas, combine the local transformation of grid division to process the overlapping areas, and obtain the final stitching result through image fusion.

[0033] Step 1, specifically:

[0034] Step 11: Calculate the ground sampling interval based on the linear relationship between the geographic center distance of the image pair and the translation distance of the image features to assist in the overlap estimation.

[0035] Specifically, the geographic displacement of the centers of adjacent images is calculated using the latitude and longitude coordinates of the UAV images and the Earth's radius. Simultaneously, the transformation matrix between the two images is calculated using feature point matching, and the pixel spatial displacement of adjacent images is estimated. The relationship between multiple geographic displacements and unique pixel ratios is determined, and the ground sampling interval is calculated.

[0036] Step 12: Calculate the overlap based on the obtained ground sampling interval, IMU attitude information, and GPS information. The overlap is defined as the ratio of the intersection area between the ground projection polygons of the two images to the smaller projection area.

[0037] Step 13, set an overlap threshold If the overlap between two images exceeds the threshold, and all projection areas of the redundant frame are contained in the remaining frames, then the frame is deleted.

[0038] Step 14: Based on the image set after removing redundant frames, the images are divided into multiple clusters using the K-Means clustering algorithm according to the overlap relationship between the images.

[0039] To be more specific, the following is an explanation:

[0040] Ground sampling interval (GSD) is defined as the actual length of a pixel on the ground in an image, usually expressed in meters per pixel or centimeters per pixel. Figure 3 This demonstrates a subset of images from the UAV-VisLoc dataset, showing how the latitude and longitude center coordinates (GPS coordinates) of two adjacent UAV images in this dataset are converted to planar coordinates in a local planar coordinate system. and Calculate the center distance on the ground using the difference in planar coordinates between the two frames:

[0041] ;

[0042] Next, SURF is used to estimate the affine transformation matrix between the two images. This allows us to obtain the translation distance between the two images in pixel space. .

[0043] Specifically, feature points are extracted from two corresponding UAV images and matched. The affine transformation matrix between the two images is estimated using the matched point pairs.

[0044] ;

[0045] Among them, the first two rows and the first two columns , , , They are jointly responsible for scaling, rotation, and other transformations. This represents the translation component. From this, we can obtain the translation distance between the two images in pixel space:

[0046] ;

[0047] This distance, measured in pixels, reflects the average translation between adjacent images.

[0048] By selecting the first 100 pairs of adjacent images, their respective... and A set of results was obtained Data points. Assuming the ground sampling interval is approximately constant within a local region, the two should satisfy a linear relationship. The final estimated ground sampling interval GSD is obtained.

[0049] GSD is used to map image dimensions to a ground coordinate system. Since the drone may exhibit yaw, pitch, or roll attitudes during filming, a rotation matrix needs to be introduced. By rotating and correcting the image orientation, the image can be projected onto the ground coordinate system using this rotation matrix, resulting in a projected polygon. .

[0050] Rotation matrix Calculation method: For the yaw angle calculated by the IMU Pitch angle With roll angle The rotation matrices can be represented as follows:

[0051] ;

[0052] ;

[0053] ;

[0054] Combining the three elements yields the overall attitude matrix:

[0055] ;

[0056] Then, the image is rotated and corrected using this rotation matrix to obtain the projected polygon.

[0057] For any two images and Their geographical overlap Defined as the ratio of the intersection area between the ground projection polygons of two images to the smaller projection area:

[0058] ;

[0059] In the formula: For image The resulting projected polygon, For image The resulting projected polygon, The function refers to the projected area. It refers to the intersection. It refers to and The overlapping area of ​​the projected polygons. It refers to and The area of ​​the overlapping region of the projected polygons.

[0060] Set an overlap threshold ,when If two images have a valid overlapping area, they can be considered as a candidate pair for stitching. An excessive overlap threshold can also be set. It is 0.9. For each frame of image Construct an overlay mask and utilize its ground projection polygon. Generate the corresponding binary overlay mask This allows all images to be represented at a uniform ground scale. Then, the global coverage area is calculated; the joint coverage area of ​​all images can be represented as... For any satisfying Image pairs If you want to delete the image Verification required for removal If the coverage area of ​​all remaining images is still equal to the original coverage area, then remove the image. .

[0061] In layman's terms, during image acquisition, multiple images may be captured of the same area. These images may have a high degree of overlap, or even be identical, which greatly increases the image stitching time; hence, they are called redundant frames. Two conditions are used to determine a redundant frame: first, its overlap with other images exceeds a threshold; second, the coverage area of ​​the stitched image set is the same as the coverage area of ​​the stitched image set after removing the redundant frame. If both conditions are met, the frame is determined to be redundant and deleted.

[0062] Finally, the image overlap relationships are transformed into an adjacency structure between images. This adjacency matrix describes an undirected graph structure, where nodes correspond to image frames, and edges represent spatial overlap relationships between two frames that can be used for registration. In other words, the adjacency structure refers to the overlap relationships between images, and the adjacency matrix is ​​a specific matrix describing these relationships. This further forms spatially consistent image clusters. Let the number of images be... The number of clusters is taken Using the image center coordinates K-Means clustering is performed to obtain several clusters with consistent structures, or image clusters. .

[0063] Figure 4 The diagram shows the clustering results of this step, with different colors representing different clusters.

[0064] Step 2, specifically:

[0065] Step 21: Calculate the image's attitude stability index based on the camera's pitch and roll angles during image capture, denoted as... The most stable image in each cluster is selected as the cluster center.

[0066] Step 22: Using the cluster center as the reference frame, calculate the stitching order of the images in each cluster based on distance and overlap; within each cluster, perform SURF registration between adjacent images according to the stitching order, and remove erroneous points using random sampling consistency, and then calculate the transformation matrix of each image to the reference frame; if a frame fails to register, remove the frame from the cluster and add it to other clusters for registration, until successful or all clusters fail to match.

[0067] Specifically, the transformation matrix between adjacent frames is first calculated based on the stitching order. Taking images A, B, and C as an example, where image A is the reference frame, the transformation matrix from image B to image A can be obtained after registration. And the transformation matrix from image C to image B Then the transformation matrix from image C to image A is: Similarly, the transformation matrix from other frames to the reference frame can be calculated.

[0068] Step 23: Use the transformation matrix to stitch together the images in each cluster, and then perform image fusion, weighted fusion based on the distance of each pixel from the boundary of the overlapping area.

[0069] Specifically, the overlapping area is first calculated using the transformation matrix for each stitch, and the minimum bounding rectangle of the overlapping area is calculated. For each pixel in the overlapping area, the shortest distance from it to the bounding rectangles of the two images is calculated and then weighted and fused.

[0070] To be more specific, the following is an explanation:

[0071] Each cluster contains multiple spatially adjacent UAV images with high geographical overlap. To select images with stable attitudes as cluster centers, the attitude stability index for each image frame is defined as follows:

[0072] ;

[0073] in and Describes the camera's pitch and roll angles respectively, where the subscripts are... , representing different images The pitch and roll angles are used to calculate the stability index of all images using this formula. A smaller value indicates less distortion and higher quality due to attitude disturbances. After determining the stable frame, a reasonable stitching order needs to be constructed for the images within the cluster. Specifically, the reference frame is used as the center, with the reference frame as the first frame. The image with the highest overlap with the reference frame is selected as the second frame, the second highest overlap as the third frame, and so on. When there is no overlap with the reference frame, distance is used for selection, stitching the image closest to the reference frame first, thus determining the final stitching order.

[0074] Next, the SURF method is selected to complete the geometric registration between adjacent images, obtaining the homography transformation matrix for pairwise registration. Considering that all images within a cluster need to be stitched onto the reference frame, the homography transformation of the pairwise registration needs to be converted into the homography transformation of the reference image. The specific steps are as follows:

[0075] With images ,image and images This process is illustrated with an example, where the image... Using the reference frame, the image has been obtained through the above steps. and images and images and images The transformation matrices are denoted as follows: and , to image Transform to image The subsequent image is denoted as ,So Correspondingly, the image Transform to image Subsequent images It can be represented as At this point, the image can be calculated. Transform to image Subsequent images It can then be expressed as:

[0076] ;

[0077] According to this formula, all other images within the cluster can be stitched onto the reference frame.

[0078] Additionally, if an image fails to register during the registration process, the failed frame needs to be removed from the cluster, and an image with the highest overlap with the failed frame that belongs to another cluster should be searched for registration. If successful, the image should be added to the cluster for stitching; if it still fails, the frame should be removed directly.

[0079] To achieve a smooth transition between adjacent images, a local averaging fusion method based on distance transform weights is used to fuse overlapping regions. Let the two images be... and ,image The image obtained after transformation Simultaneously calculate the transformation and The overlapping region can be represented as Weighted fusion of pixels is performed in the overlapping region. For points in the overlapping region... At point Source and The pixel values ​​are denoted as follows: and Similarly, to mitigate the impact of lighting differences, the pixel values ​​being merged need to consider point... The distance to the boundary of the overlapping region is denoted as... and They are respectively arrive and The shortest distance between the boundaries of the overlapping regions is the point. fused pixel values ​​at for:

[0080] ;

[0081] For pixels in non-overlapping areas, their pixel values ​​are directly copied from the corresponding image. Finally, this pixel fusion method utilizes distance transformation to construct smooth transition weights, allowing the fused region to gradually transition from one image to another, thereby avoiding obvious stitching seams in overlapping areas and improving the visual consistency of the panoramic image.

[0082] Figure 5 A schematic diagram showing the first layer of splicing results is displayed.

[0083] Step 3, specifically:

[0084] Step 31: Take the attitude stability index of the cluster center of each cluster in the first layer stitching result as the stability index of the stitching result, and then select the first layer stitching result with the most stable attitude as the center of the second layer stitching; with the selected center as a reference, calculate the spatial center coordinates of each first layer stitching result, and determine the order of the second layer stitching according to the distance of the spatial center coordinates of each stitching result from the reference stitching result; for example, it can be sorted from near to far to determine the order of the second layer stitching.

[0085] Step 32: Perform pairwise registration according to the stitching order, estimating only the similarity transformations between them, i.e., rotation, translation, and proportional scaling changes. Finally, calculate the global similarity transformation matrix for each pairwise registration, and then transform it to the transformation matrix of the reference image. Calculate the overlapping region during stitching based on the calculated global transformation matrix. Divide the overlapping region into a grid and calculate the local transformation matrix for each grid. Apply the local transformation to the overlapping region and the global transformation to the non-overlapping region.

[0086] This step actually involves stitching two images together each time. Taking the stitching of the previous two frames as an example, firstly, the images are registered using Scale-Invariant Feature Transform (SIFT) to determine the overlapping and non-overlapping regions. The overlapping and non-overlapping regions of the two images need to be processed separately. A global similarity transformation is applied to the non-overlapping regions, while the overlapping regions are divided into grids, and the local transformation matrix corresponding to each grid is calculated.

[0087] Specifically, the overlapping region is divided into a grid, and then the objective function is minimized. Solve each grid The corresponding homography matrix:

[0088] ;

[0089] in, This is the coefficient matrix for each matched feature point, representing the weight of each feature point. Here, the weights of each feature point are equal. Add constraints to the number of feature point pairs. To ensure The solution is meaningful, After converting back to matrix form, the local transformation matrix for each grid can be obtained. .

[0090] Step 33: Perform weighted fusion of the images based on the distance of each pixel from the boundary of the overlapping area to obtain the second layer of stitching results, thus completing the image stitching.

[0091] Specifically, the overlapping area is first calculated using the transformation matrix for each stitch, and the minimum bounding rectangle of the overlapping area is calculated. For each pixel in the overlapping area, the shortest distance from it to the bounding rectangles of the two images is calculated and then weighted and fused.

[0092] To be more specific, the following is an explanation:

[0093] First, determine a reasonable assembly order for the second layer, and then set up clusters. The included image set is ,image The plane coordinates are The center position of the cluster is obtained by averaging the center coordinates of all images in the cluster. This coordinate reflects the global position of the cluster within the aerial photography area. Furthermore, the pose of the stable frames in each cluster can be used as the pose of that cluster. The most stable cluster among all clusters is selected as the reference image for inter-cluster stitching. The Euclidean distance between the spatial center coordinates of the clusters and other clusters is used as the basis for the stitching order, ultimately resulting in the cluster sequence to be stitched.

[0094] Next, the transformation relationship between adjacent images is calculated, and it is determined whether they belong to overlapping regions, such as... Figure 6 As shown, local transformations are used only in overlapping regions, while global transformations are used directly in non-overlapping regions.

[0095] Inter-cluster stitching first stitches the first two images in the stitching order, then stitches the result with the third image, and so on, to obtain the final stitched result. Therefore, this step actually stitches only two images at a time. Taking the stitching of the first two frames as an example, the overlapping and non-overlapping areas of the two images need to be processed separately. First, SIFT registration is performed on these two images, that is, SIFT is used to match feature points between the images for intra-cluster registration. Based on the registration result, the global similarity transformation matrix of the images is calculated, thereby determining the overlapping and non-overlapping areas.

[0096] In non-overlapping regions, a similarity transformation model is used for inter-cluster registration. SIFT registration is performed on the images according to the inter-cluster stitching order. For any two adjacent intra-cluster stitching results... and Assuming that we have obtained For reliable matching points, the corresponding clusters are respectively the first... The two-dimensional coordinates of the matching points are calculated. Next, the mean coordinates of the two sets of points are calculated, and the feature points in both sets are centered around their respective centroids. These centered points are used for subsequent rotation and scale estimation. Based on this, a transformation is constructed from the image... To the reference image Cross covariance matrix:

[0097] ;

[0098] In the formula: Let K be the cross-covariance matrix, and K represent the number of pairs of matched feature points. Indicates the intra-cluster splicing result The feature point set that is decentralized around its centroid. Indicates the intra-cluster splicing result The set of feature points that are decentered around their centroid, where the centroid refers to the average coordinates of the set of points.

[0099] right Perform singular value decomposition to obtain from To image Optimal rotation matrix Translation vector And further solve for the scale factor. Then write it as Homogeneous matrix form:

[0100] ;

[0101] This matrix provides information from the cluster Image coordinate system to cluster A direct mapping of the pixel coordinate system.

[0102] In obtaining a cluster-level panoramic image arrive Global similarity / rigid body transformation matrix After that, you can Mapped to this transformation In the coordinate system, to achieve the initial alignment of the two cluster-level stitching results on the global structure, let... The transformed image is To facilitate the description of the subsequent calculation process, the following will be... Recorded as ,So:

[0103] ;

[0104] In overlapping regions, a local geometric refinement strategy based on grid partitioning is employed. First, the image is divided into... Then minimize the objective function using rectangular grid cells. Solve for the homography matrix corresponding to each grid cell:

[0105] ;

[0106] in, This is the coefficient matrix for each matched feature point, representing the weight of each feature point. Here, the weights of each feature point are equal. Add constraints to the number of feature point pairs. To ensure The solution is meaningful, After converting back to matrix form, the local transformation matrix for each grid can be obtained. .

[0107] To achieve a smooth transition between adjacent images, a local averaging fusion method based on distance transform weights is used to fuse overlapping regions. Let the two images be... and ,image The image obtained after transformation Simultaneously calculate the transformation and The overlapping region can be represented as Weighted fusion of pixels is performed in the overlapping region. For points in the overlapping region... At point Source and The pixel values ​​are denoted as follows: and Similarly, to mitigate the impact of lighting differences, the pixel values ​​being merged need to consider point... The distance to the boundary of the overlapping region is denoted as... and They are respectively arrive and The shortest distance between the boundaries of the overlapping regions is the point. fused pixel values ​​at for:

[0108] ;

[0109] For pixels in non-overlapping areas, their pixel values ​​are directly copied from the corresponding image. Finally, this pixel fusion method utilizes distance transformation to construct smooth transition weights, allowing the fused region to gradually transition from one image to another, thereby avoiding obvious stitching seams in overlapping areas and improving the visual consistency of the panoramic image.

[0110] Figure 7 Taking the stitching of 100 frames as an example, the final stitching result is shown.

[0111] In summary, this invention utilizes multimodal fusion to fully leverage the camera pose information provided by the IMU and the position information provided by GPS during UAV image acquisition for initial pose estimation to calculate image overlap and remove redundant frames, thereby minimizing issues such as blurring caused by overlapping frames at the same location. To reduce the cumulative error of frame-by-frame stitching, the overlap is used to establish connectivity between images, and the image set is clustered. A two-layer stitching strategy, involving intra-cluster and inter-cluster stitching, is then employed. The first layer performs high-precision registration to obtain locally optimal alignment results; the second layer performs global optimization to effectively suppress the spread of cumulative errors in the stitching sequence. Finally, an improved image fusion algorithm is designed that significantly reduces visual artifacts at stitching boundaries while maintaining brightness and color consistency, thereby generating high-precision, seamless, and realistic panoramic images.

[0112] Example 2:

[0113] This embodiment 2 describes an image stitching system based on two-layer stitching, which is based on the same inventive concept as the image stitching method based on two-layer stitching described in embodiment 1 above.

[0114] An image stitching system based on two-layer stitching includes the following modules:

[0115] Preprocessing module: used to determine the overlap between images based on IMU data and GPS data provided by the drone during aerial photography, remove redundant frames using the overlap, and cluster the image set after removing redundant frames into multiple clusters;

[0116] The first layer stitching module is used to calculate the attitude stability based on the IMU data of each frame of the image in the cluster, and select the frame with the most stable attitude as the center of the cluster and use it as the reference frame. The stitching order is determined based on the overlap and the distance between the projection centers of the two images, and image registration is performed. Frames that fail to register are removed and re-clustered. Finally, the images are stitched together and image fusion is performed to obtain the first layer stitching result.

[0117] The second-layer stitching module is used to take the pose of the cluster center of each cluster as the pose stability of each stitching result, select the stitching result with the most stable pose as a reference, determine the stitching order and then perform registration, calculate the global transformation based on similarity transformation to process non-overlapping areas, combine the local transformation of grid division to process overlapping areas, and obtain the final stitching result through image fusion.

[0118] Image fusion module: In the first and second layer stitching, it uses the transformation matrix to calculate the overlapping area at each stitching and calculates the minimum bounding rectangle of the overlapping area. For each pixel in the overlapping area, it calculates the shortest distance to the bounding rectangles of the two images and performs weighted fusion.

[0119] It should be noted that in the image stitching system based on double-layer stitching in this embodiment 2, the implementation process of the functions and roles of each functional module is detailed in the implementation process of the corresponding steps in the method of embodiment 1 above, and will not be repeated here.

[0120] It should be noted that the above description is only a preferred embodiment of the present invention. For those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention. All components not explicitly stated in this embodiment can be implemented using existing technology.

Claims

1. An image stitching method based on two-layer stitching, characterized in that... Includes the following steps: Step 1: Determine the overlap between images based on the IMU data and GPS data provided by the drone during aerial photography, remove redundant frames using the overlap, and cluster the image set after removing redundant frames into multiple clusters; Step 2: Perform the first layer stitching within the multiple clusters defined in Step 1. First, calculate the pose stability based on the IMU data of each frame within the cluster, and select the frame with the most stable pose as the cluster center and use it as the reference frame. Determine the stitching order based on the overlap and the distance between the projection centers of the two images, and perform image registration. Frames that fail to register are discarded and re-clustered. Finally, stitch and fuse the images to obtain the first layer stitching result. Step 3: Perform the second layer stitching based on the first layer stitching result obtained in Step 2. First, select the most stable pose from the poses of the cluster centers of each cluster, and select the first layer stitching result corresponding to the most stable pose as a reference to determine the stitching order and then perform registration. Then, calculate the global transformation based on similarity transformation to process the non-overlapping areas, and combine the local transformation of grid division to process the overlapping areas. Finally, obtain the second layer stitching result through image fusion to complete the image stitching. Step 2 specifically includes the following steps: Step 21: Calculate the attitude stability of the image based on the camera's pitch and roll angles when the image is captured, and select the image with the most stable attitude in each cluster as the cluster center, with the cluster center as the reference frame. Step 22: Calculate the stitching order of images in each cluster based on the distance and overlap between the projection centers of the two images; within each cluster, register adjacent images according to the stitching order, and remove erroneous points using random sampling consistency; then calculate the transformation matrix of each image to the reference frame; if a frame fails to register, remove it from the cluster and add it to another cluster for registration until successful or all clusters fail to match; stitch the images in each cluster using the transformation matrix. Step 23: Perform image fusion to obtain the first layer stitching result; Step 3 specifically includes the following steps: Step 31: Take the attitude stability index of the cluster center of each cluster in the first layer stitching result as the stability index of the stitching result, and then select the first layer stitching result with the most stable attitude as the center of the second layer stitching; take the selected center as the reference stitching result, i.e. the reference image, calculate the spatial center coordinates of each first layer stitching result, and determine the order of the second layer stitching according to the distance between the spatial center coordinates of each first layer stitching result and the reference stitching result. Step 32: Perform pairwise registration according to the stitching order of the second layer to determine the overlapping and non-overlapping regions. Apply global similarity transformation to the non-overlapping regions, divide the overlapping regions into grids, and calculate the local transformation matrix corresponding to each grid. Step 33: Perform weighted fusion of the images based on the distance of each pixel from the boundary of the overlapping area to obtain the second layer of stitching results, thus completing the image stitching.

2. The image stitching method based on double-layer stitching according to claim 1, characterized in that, Step 1 specifically includes the following steps: Step 11, calculate the ground sampling interval; Step 12: Calculate the overlap between images based on the ground sampling interval; Step 13: When the overlap between two images in the image set exceeds a set threshold, it is identified as a redundant frame. If all the projection regions of the identified redundant frame are contained in the other frames, it is finally determined to be a redundant frame and deleted. Step 14: Based on the image set after removing redundant frames, the images are divided into multiple clusters using the K-Means clustering algorithm according to the overlap relationship between the images.

3. The image stitching method based on double-layer stitching according to claim 2, characterized in that: In step 11, the latitude and longitude center coordinates of two adjacent images are obtained using GPS data and converted into planar coordinates in a local planar coordinate system. and Calculate the center distance on the ground by using the difference in planar coordinates between the two images: ； Then, feature points are extracted from the two corresponding images and matched. The affine transformation matrix between the two images is determined by the matched point pairs. ； Among them, the first two rows and the first two columns , , , Together they are responsible for scaling and rotation transformations. This represents the translation component; thus, the translation distance between the two images in pixel space is obtained: ； This distance, expressed in pixels, reflects the average translation between adjacent images; it is calculated by selecting multiple pairs of adjacent images. and , and obtained a series Data points; Then according to the formula The ground sampling interval is calculated; where GSD is the ground sampling interval. In step 12, the image size is mapped to the ground coordinate system using the ground sampling interval, and a rotation matrix is ​​introduced. The image orientation is rotated and corrected to obtain the projected polygon; Rotation matrix The calculation method is as follows: The yaw angle is first calculated from IMU data. Pitch angle With roll angle The rotation matrices are represented as follows: ； ； ； Then combine the three to obtain the rotation matrix: ； For any two images and Its degree of overlap The calculation formula is as follows: ； In the formula: For image The resulting projected polygon, For image The resulting projected polygon, The function refers to the projected area. It refers to the intersection. It refers to and The overlapping area of ​​the projected polygons. It refers to and The area of ​​the overlapping region of the projected polygons; In step 13, an overlap threshold is set. and excessive overlap threshold ,when Two images are considered to have a valid overlapping area and are thus considered as candidate pairs for stitching; for each frame of the image... Construct an overlay mask and utilize its ground projection polygon. Generate the corresponding binary overlay mask This allows all images to be represented at a uniform ground scale; then, the global coverage area is calculated, and the joint coverage area of ​​all images is represented as... For any satisfying Image pairs If you want to delete the image Verification required for removal If the coverage area of ​​all remaining images is still equal to the original coverage area, then the image is removed. ; In step 14, the overlapping relationships of the images are transformed into an adjacency structure between images. This adjacency structure describes an undirected graph structure, where nodes correspond to image frames, and edges represent spatial overlap relationships between two frames used for registration. This further forms spatially consistent image clusters. Let the number of images be... The number of clusters is taken Using the image center coordinates K-Means clustering is performed to obtain several image clusters with consistent structures. .

4. The image stitching method based on double-layer stitching according to claim 3, characterized in that: In step 21, each frame image is defined. The attitude stability index is : ； in, and Each frame represents an image. The corresponding camera pitch and roll angles; the image corresponding to the minimum attitude stability index is selected as the most attitude-stable image in each cluster; In step 22, the stitching order of the images in each cluster is determined using the following steps: First, the reference frame is used as the first frame. Then, the image with the highest overlap with the reference frame is selected as the second frame, the image with the second highest overlap is selected as the third frame, and so on. When there is no overlap with the reference frame, the image is selected by distance. The image closest to the reference frame is stitched first, thus determining the stitching order. The SURF method is selected to perform geometric registration between adjacent images, obtaining the transformation matrix for each pair of registrations. Then, the transformation matrix of each pair of registrations is transformed to the homography transformation of the reference frame. The specific steps are as follows: With images Using the reference frame, the image is obtained through registration. and images and images and images The transformation matrices are denoted as follows: and , to image Transform to image The subsequent image is denoted as ,So Correspondingly, the image Transform to image Subsequent images Represented as At this point, the image is calculated. Transform to image Subsequent images This is represented as: ； According to the above formula, stitch all other images within the cluster onto the reference frame; If a failed image is encountered during the registration process, i.e. a failed frame, the failed frame needs to be removed from the cluster, and an image with the highest overlap with the failed frame that belongs to another cluster should be searched for registration. If successful, the image is added to the corresponding cluster for stitching. If it still fails, the failed frame is directly removed.

5. The image stitching method based on double-layer stitching according to claim 4, characterized in that: In step 31, let the cluster be... The included image set is ,image The plane coordinates are The center position of the cluster is obtained by averaging the center coordinates of all images in the cluster. The coordinates reflect the global position of the cluster in the aerial photography area. In addition, the pose of the stable frame in each cluster is taken as the pose of the cluster. The most stable cluster is selected as the reference image for inter-cluster stitching. The Euclidean distance between the spatial center coordinates of the clusters and other clusters is used as the basis for the stitching order to obtain the cluster sequence to be stitched.

6. The image stitching method based on double-layer stitching according to claim 5, characterized in that: In step 32, the inter-cluster stitching of the cluster sequence first stitches the first two images in the stitching order, then stitches the result with the third image, and so on, to obtain the final stitching result; First, the two images are registered. Based on the registration results, the global similarity transformation matrix of the images is calculated, and then the overlapping and non-overlapping regions are determined. In non-overlapping regions, for any two adjacent cluster splicing results and Assuming that we have obtained For reliable matching points, the corresponding clusters are respectively the first... The two-dimensional coordinates of the matching points are calculated. Next, the mean coordinates of the two sets of feature points are calculated, and the two sets of feature points are then decentered around their respective centroids. The decentered points are used for subsequent similarity transformation estimation. Based on this, a model is constructed from... arrive Cross covariance matrix: ； In the formula: Let K be the cross-covariance matrix, and K represent the number of pairs of matched points. Indicates the intra-cluster splicing result The feature point set that is decentralized around its centroid. Indicates the intra-cluster splicing result The set of feature points that are decentered around their centroid, where the centroid refers to the average coordinates of this set of points; For the cross covariance matrix Perform singular value decomposition to obtain from arrive Optimal rotation matrix Translation vector And further solve for the scale factor. Then the global similarity transformation matrix: ； This matrix provides information from Image coordinate system to cluster A direct mapping of the pixel coordinate system; In order to obtain from to cluster Global similarity transformation matrix After that, Mapped to this transformation In the coordinate system, to achieve the initial alignment of the two cluster-level stitching results on the global structure, let... The transformed image is To facilitate the description of the subsequent calculation process, the following will be... Recorded as ,So: ； In the overlapping region, the image is first divided into Then minimize the objective function using rectangular grid cells. Solve for the homography matrix corresponding to each grid cell: ； in, This represents the weight of each feature point. Add constraints to the number of feature point pairs. To ensure The solution is meaningful, After converting back to matrix form, we obtain the local transformation matrix for each grid. .

7. The image stitching method based on double-layer stitching according to claim 6, characterized in that, In steps 23 and 33, the image fusion process for the overlapping regions is as follows: Let the two images be respectively and ,image The image obtained after transformation Simultaneously calculate the transformation and The overlapping region is represented as ; Perform weighted fusion of pixels in the overlapping region, for points in the overlapping region At point Source and The pixel values ​​are denoted as follows: and ;remember and Points arrive and The shortest distance between the boundaries of the overlapping regions is the point. fused pixel values ​​at for: ； For pixels in non-overlapping regions, their pixel values ​​are directly copied from the corresponding image.

8. An image stitching system based on dual-layer stitching, characterized in that, The image stitching method based on two-layer stitching as described in any one of claims 1-7 includes the following modules: Preprocessing module: used to determine the overlap between images based on IMU data and GPS data provided by the drone during aerial photography, remove redundant frames using the overlap, and cluster the image set after removing redundant frames into multiple clusters; The first layer stitching module is used to calculate the attitude stability based on the IMU data of each frame of the image in the cluster, and select the frame with the most stable attitude as the center of the cluster and use it as the reference frame. The stitching order is determined based on the overlap and the distance between the projection centers of the two images, and image registration is performed. Frames that fail to register are removed and re-clustered. Finally, the images are stitched together and image fusion is performed to obtain the first layer stitching result. The second-layer stitching module is used to take the pose of the cluster center of each cluster as the pose stability of each stitching result, select the stitching result with the most stable pose as a reference, determine the stitching order and then perform registration, calculate the global transformation based on similarity transformation to process non-overlapping areas, combine the local transformation of grid division to process overlapping areas, and obtain the final stitching result through image fusion. Image fusion module: In the first and second layer stitching, it uses the transformation matrix to calculate the overlapping area at each stitching and calculates the minimum bounding rectangle of the overlapping area. For each pixel in the overlapping area, it calculates the shortest distance to the bounding rectangles of the two images and performs weighted fusion.

Images