Unmanned aerial vehicle image splicing method based on LightGlue and adaptive multi-band fusion

Abstract

The present application relates to the technical field of image stitching, and provides a UAV image stitching method based on LightGlue and adaptive multi-band fusion. The method comprises obtaining quality-enhanced images and comprehensive picture quality scores; using a heterogeneous feature flow adaptive cascade mechanism based on image quality perception, performing feature extraction and matching on the quality-enhanced images to obtain a group of feature matching point pairs; based on the feature matching point pairs, calculating a homography transformation matrix between the images to obtain a registered image pair; constructing a spatial confidence potential field based on the confidence information of the feature matching point pairs, and using the spatial confidence potential field to perform dynamic seam line optimization to generate a locally feathered fusion weight map; using a resource-aware adaptive multi-band fusion strategy, combining the fusion weight map to fuse the registered image pair to generate a seamless panoramic image, and the method realizes high-robustness, high-quality and high-efficiency UAV image stitching.

Description

Technical Field

[0001] This invention relates to the field of image stitching technology, and in particular to a method for stitching UAV images based on LightGlue and adaptive multi-band fusion. Background Technology

[0002] With the application of drone technology in various fields such as photography, surveying, and agriculture, and the advancement of image stitching technology, the demand for fast and accurate drone image stitching methods is increasing. Image stitching technology is a technique that combines multiple images with overlapping parts into a seamless panoramic or high-resolution image. The core process includes image registration and image fusion.

[0003] Early image stitching techniques, such as SIFT and ORB, traditional hand-crafted feature algorithms, typically require complex geometric information for precise content alignment and shape preservation. Compared to traditional methods, deep learning-based image stitching techniques offer better robustness to complex image distortions and lighting variations. LightGlue, a deep learning-based feature matcher, can improve efficiency while maintaining high-quality matching; however, such models often still impose a significant computational burden on embedded devices like drones. In the image fusion stage, methods such as multi-band fusion are widely used to achieve smooth transitions.

[0004] Although image stitching technology has made significant progress, many problems still exist in complex scenarios such as drone aerial photography. On the one hand, due to the attitude jitter, altitude changes, and lighting variations during drone flight, traditional methods are ineffective in areas with weak texture or large parallax. On the other hand, traditional stitching methods often ignore structural and texture information, resulting in ghosting and seam problems. Among existing technologies, the published invention patent CN119273539A proposes a stitching scheme based on local distortion and seam guidance, but its underlying layer is still based on the traditional SIFT method, and seam optimization lacks deep semantic and confidence constraints. Furthermore, for areas with weak texture or abrupt lighting changes, feature point detection may not be dense enough, resulting in insufficient matching points, which in turn affects image alignment accuracy and the stitching result after seam optimization. Invention patent CN120912428A uses external sensor data such as drone attitude changes and acceleration during flight to evaluate image confidence and guide stitching, but this method does not consider the physical image quality and spatial distribution of feature matching, failing to obtain good local fusion weights. Even with stable flight, large registration errors can occur under extreme exposure, resulting in reduced stitching quality. Summary of the Invention

[0005] In view of this, the present invention provides a UAV image stitching method based on LightGlue and adaptive multi-band fusion to achieve highly robust, high-quality and efficient UAV image stitching.

[0006] In a first aspect, the present invention provides a method for stitching UAV images based on LightGlue and adaptive multi-band fusion, the method comprising:

[0007] Step 1: Perform dual-effect linked quality perception and preprocessing on the input UAV sequence images to obtain the enhanced image and comprehensive image quality score;

[0008] Step 2: Based on the comprehensive image quality score, an adaptive cascade mechanism based on image quality perception heterogeneous feature flow is used to extract and match features of the enhanced image to obtain a set of feature matching point pairs.

[0009] Step 3: Based on the feature matching point pairs, calculate the homography transformation matrix between the images to complete the geometric registration of the images and obtain the registered image pairs;

[0010] Step 4: Construct a spatial confidence potential field based on the confidence information of the feature matching point pairs, and use the spatial confidence potential field to perform dynamic stitching optimization to generate a locally feathered fusion weight map.

[0011] Step 5: Using a resource-aware adaptive multi-band fusion strategy, the registered image pairs are fused together with the fusion weight map to generate a seamless panoramic image.

[0012] Optionally, step 1 includes:

[0013] The image data quality assessment strategy is designed by calculating the image sharpness and evaluating the overall brightness distribution to obtain a brightness score. The scoring calculation logic is as follows:

[0014] a. Based on the response variance of the Laplacian operator, the sharpness score is calculated, and its expression is as follows:

[0015] ;

[0016] in, This represents the sharpness score, with a value ranging from 0 to 1; 1000 represents the variance of the Laplacian convolution response of the image; 1000 represents the normalization coefficient.

[0017] b. The brightness score is obtained by normalizing the mean pixel intensity, and its expression is:

[0018] ;

[0019] in, This represents the brightness score, with a value ranging from 0 to 1. This represents the average intensity value of grayscale pixels in the image, with a value range of 0 to 255.

[0020] c. Obtain a clarity score and brightness rating Then, construct a comprehensive image quality scoring function. This evaluates the feature extraction capability of images under current lighting and texture conditions, and integrates the image quality scoring function. The expression is:

[0021] ;

[0022] in, , This represents a non-linear brightness penalty mechanism.

[0023] Optionally, step 2 includes:

[0024] Based on overall image quality score Dynamically generate LightGlue's cascading backoff threshold And the lower limit threshold for homography calculation ;

[0025] When inputting a low-quality, weakly textured image, actively reduce... To trust the matching capabilities of deep learning models; and improve To enhance the robustness of matrix solution; when high-quality images are input, it actively improves... and reduce The expression for the threshold dynamic mapping function is:

[0026] ;

[0027] ;

[0028] in, These represent the lower and upper limits of the backoff threshold, respectively. These represent the lower and upper limits of the threshold for matrix calculation, respectively. Adaptive adjustment coefficient;

[0029] When LightGlue matches the number of points Less than the cascade backoff threshold If the scale-invariant feature transform SIFT can provide sufficient matching, then it will automatically revert to its previous state. If SIFT is used, then ORB algorithm is used; otherwise, the data is output through dimensionality normalization mapping. The expression for the above progressive backoff strategy is:

[0030] ;

[0031] Phase 1: Feature matching is performed using the SIFT algorithm, denoted as... ;

[0032] ;

[0033] ;

[0034] Phase 2: If SIFT matching fails, the ORB algorithm is used for matching, denoted as... ;

[0035] ;

[0036] ;

[0037] in, This represents the final set of optimal matching results; , , These represent the matching results output by the LightGlue, SIFT, and ORB algorithms, respectively. and These represent the number of valid match points for LightGlue and SIFT, respectively. and These represent the cascade backoff threshold and the lower limit threshold for homography calculation, respectively. This indicates the preset maximum number of feature points.

[0038] Optionally, step 4 includes:

[0039] The spatial confidence potential field is constructed, and its expression is:

[0040] ;

[0041] in, For the spatial confidence potential field; For the first The coordinates and confidence scores of each matching point; The Gaussian kernel scale factor is used to control the potential energy diffusion range; These are the currently calculated image pixel coordinates. For the first The pixel coordinates of the matching feature points; This represents the total number of valid match points extracted by LightGlue.

[0042] The spatial confidence potential field is used to guide the optimal suture line to avoid the parallax-sensitive region and converge to the geometrically aligned reliable region. The total energy function includes a confidence repulsion term derived from the spatial confidence potential field, and its expression is as follows:

[0043] ;

[0044] in, Let be the total energy function. The intensity of the pixel difference between the two images; For geometric center constraint terms; This is the weighting coefficient for the potential energy term, used to adjust the influence of the confidence constraint; The confidence exclusion term is expressed as follows:

[0045] ;

[0046] in, This is an energy penalty factor; the lower the confidence level, the lower the penalty factor. The higher the value, the higher the energy cost of traversing the current area;

[0047] Building upon this, and employing a distance-transform-based local feathering technique, a smooth-transition fusion weight map is generated by calculating the Euclidean distance from each pixel to the seam line. Its expression is as follows:

[0048] ;

[0049] in, A fusion weight map for a smooth transition; The width of the feather; and These are the positive and negative Euclidean distances from the pixel to the seam line, respectively. A cutoff function to ensure that the weight values ​​are strictly limited to the interval [0, 1].

[0050] Optionally, step 5 includes:

[0051] A shallow Laplacian pyramid with only two layers is used for fusion; in the high-frequency layers, weighted fusion is performed using a fusion weight map; in the low-frequency layers, broadband mixing is performed for illumination transition; the pyramid reconstruction expression is:

[0052] ;

[0053] in, For the final seamlessly merged image, The result is the fusion of high-frequency details at layer 0 after weighting by the fusion weight map. This is the weighted fusion result of the first-layer low-frequency approximation image.

[0054] Optionally, the pyramid reconstruction includes:

[0055] A texture analysis mechanism was introduced before constructing the weight pyramid; the average gradient magnitude of the overlapping regions was calculated. Dynamic weight selection generation model: low-texture flat region A linear weighting model is adopted, and the Gaussian blur kernel is increased to reduce color difference tortuosity. Preset texture threshold; high texture complexity area Switch to a nonlinear S-shaped weight model and simultaneously reduce the Gaussian blur kernel.

[0056] In a second aspect, embodiments of the present invention provide a computer-readable storage medium comprising a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to execute the UAV image stitching method based on LightGlue and adaptive multi-band fusion in the first aspect or any possible implementation thereof.

[0057] Thirdly, embodiments of the present invention provide an electronic device, including: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the device, cause the device to perform the UAV image stitching method based on LightGlue and adaptive multi-band fusion in the first aspect or any possible implementation of the first aspect.

[0058] The technical solution provided by this invention includes a method that performs dual-effect linked quality perception and preprocessing on the input UAV sequence images to obtain a quality-enhanced image and a comprehensive image quality score; based on the comprehensive image quality score, a heterogeneous feature flow adaptive cascade mechanism based on image quality perception is used to extract and match features in the quality-enhanced image to obtain a set of feature matching point pairs; based on the feature matching point pairs, the homography transformation matrix between images is calculated to complete the geometric registration of the images, resulting in a registered image pair; a spatial confidence potential energy field is constructed based on the confidence information of the feature matching point pairs, and dynamic stitching optimization is performed using the spatial confidence potential energy field to generate a locally feathered fusion weight map; a resource-aware adaptive multi-band fusion strategy is adopted, and the registered image pair is fused in combination with the fusion weight map to generate a seamless panoramic image. This method achieves highly robust, high-quality, and efficient UAV image stitching. Attached Figure Description

[0059] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 A flowchart illustrating the UAV image stitching method based on LightGlue and adaptive multi-band fusion provided in this embodiment of the invention;

[0061] Figure 2 The following is a schematic diagram of the image preprocessing effect provided in the embodiment of the present invention: (a) is a dark night scene image 1 before preprocessing, (b) is a dark night scene image 1 after preprocessing, (c) is a dark night scene image 2 before preprocessing, and (d) is a dark night scene image 2 after preprocessing.

[0062] Figure 3 This is a schematic diagram of the LightGlue-Lite lightweight network architecture after structured pruning provided in an embodiment of the present invention;

[0063] Figure 4 The overall flowchart of the confidence potential-guided dynamic suture optimization and fusion strategy provided in the embodiments of the present invention is shown below.

[0064] Figure 5 The following are line graphs comparing the performance of the LightGlue model provided in this embodiment of the invention under different pruning layers: (a) is a trend graph of the reprojection RMSE value, (b) is a trend graph of the number of matches, (c) is a trend graph of the inference time, and (d) is a trend graph of the inlier rate.

[0065] Figure 6 The following are comparison diagrams of the matching effects of LightGlue-Lite and benchmark algorithms provided in the embodiments of the present invention under typical low-altitude UAV perspectives: (a) SIFT algorithm effect diagram, (b) ORB algorithm effect diagram, (c) LightGlue algorithm effect diagram, and (d) LightGlue-Lite algorithm effect diagram.

[0066] Figure 7 This is a qualitative comparison of the panoramic image generation effects of different stitching methods provided in the embodiments of the present invention on the UDIS-D dataset;

[0067] Figure 8 A comparison of the ablation experiment results of the adaptive multi-band fusion strategy provided in this embodiment of the invention and the standard multi-band fusion on the OpenDroneMap dataset;

[0068] Figure 9 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0070] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention are also intended to include the plural forms unless the context clearly indicates otherwise.

[0071] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0072] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0073] Figure 1 The flowchart of the UAV image stitching method based on LightGlue and adaptive multi-band fusion provided in the embodiments of the present invention is as follows: Figure 1 As shown, the method includes:

[0074] Step 1: Perform dual-effect linked quality perception and preprocessing on the input UAV sequence images to obtain the enhanced image and comprehensive image quality score.

[0075] This invention designs a dual-effect linked image quality perception and adaptive enhancement module. Traditional preprocessing processes are mostly isolated operations, failing to achieve integrated linkage across the entire system. The physical features extracted in this invention not only guide image quality enhancement in this stage but are also passed as global parameters to subsequent feature matching stages, thereby achieving global linkage.

[0076] In this embodiment of the invention, step 1 includes:

[0077] Design an image data quality evaluation strategy. By calculating the clarity of the image (gradient energy based on the Laplace operator) and evaluating the overall brightness distribution to calculate the brightness score, quickly screen out images with poor quality due to severe jitter, overexposure, underexposure, etc. The following is the scoring calculation logic:

[0078] a. Calculate the clarity score based on the response variance of the Laplace operator. Its expression is:

[0079] ;

[0080] Among them, represents the clarity score, and its value range is 0 to 1; represents the response variance of the Laplace convolution of the image; 1000 represents the normalization coefficient;

[0081] b. The brightness score is obtained by normalizing the mean pixel intensity. Its expression is:

[0082] ;

[0083] Among them, represents the brightness score, and its value range is 0 to 1; represents the average intensity value of the grayscale pixels of the image, and its value range is 0 to 255;

[0084] c. After obtaining the clarity score and the brightness score , construct a comprehensive image quality scoring function to evaluate the feature extraction ability of the image under the current lighting and texture conditions, which is convenient for subsequent deep system-level linkage. The expression of the comprehensive image quality scoring function is:

[0085] ;

[0086] Among them, , represents the non-linear brightness penalty mechanism.

[0087] In the embodiments of the present invention, the comprehensive image quality score will be directly input into the heterogeneous feature stream as a global perception variable, and further dynamically solve the subsequent cascade fallback threshold and the lower limit threshold of the homography calculation .

[0088] In the embodiments of the present invention, the image enhancement branch performs dual-track parallel enhancement according to the score situation and dynamically adjusts the processing intensity parameter α: when > 0.3 and 0.3 < When the value is less than 0.7, α = 0.3 (light processing); otherwise, α = 1.0 (standard processing). This adaptive mechanism can effectively ensure image quality and minimize information loss.

[0089] Then, targeted enhancements are performed on low-quality images. Specifically, when the sharpness is below a threshold (set to 0.5 in this embodiment), contrast-limited adaptive histogram equalization is applied to enhance texture details, and when the brightness exceeds the normal range (set to 0.3~0.7 in this embodiment), a safe white balance algorithm based on channel mean adjustment is used to correct color deviation.

[0090] In embodiments of the present invention, such as Figure 2 As shown in (a) to (d), the preprocessed images show that both texture detail and lighting consistency are effectively improved.

[0091] Step 2: Based on the comprehensive image quality score, an adaptive cascade mechanism based on heterogeneous feature flow based on image quality perception is used to extract and match features of the enhanced image to obtain a set of feature matching point pairs.

[0092] In this embodiment of the invention, the heterogeneous feature flow adaptive cascading mechanism includes LightGlue-Lite, which is a lightweight model containing 4 Transformer layers obtained by structurally pruning the original LightGlue model.

[0093] The LightGlue-Lite priority strategy is adopted, which has better robustness to problems such as illumination differences, viewpoint changes, and slight motion blur. Furthermore, it is improved with a lightweight model mainly based on pruning, which ensures that the computational efficiency is effectively improved and the hardware resource consumption is reduced without sacrificing the image feature extraction quality. Therefore, it is used as the main matcher, and the dynamic evaluation of matching quality is the most important basis for the adaptive fault-tolerant decision system.

[0094] The core idea of ​​the heterogeneous feature flow adaptive cascade mechanism is an adaptive feature progression backoff mechanism based on image quality perception. Traditional methods often use fixed preset thresholds, which can easily lead to incorrect judgments when there are drastic changes in lighting or small textures. Therefore, this invention calculates a comprehensive image quality score in real time based on the sharpness and brightness features of the input image. .

[0095] In this embodiment of the invention, step 2 includes:

[0096] Based on overall image quality score Dynamically generate LightGlue's cascading backoff threshold And the lower limit threshold for homography calculation ;

[0097] When inputting a low-quality, weakly textured image, actively reduce... To trust the matching capabilities of deep learning models; and improve To enhance the robustness of matrix solution; when high-quality images are input, it actively improves... and reduce The expression for the threshold dynamic mapping function is:

[0098] ;

[0099] ;

[0100] in, These represent the lower and upper limits of the backoff threshold, respectively. These represent the lower and upper limits of the threshold for matrix calculation, respectively. Adaptive adjustment coefficient;

[0101] When LightGlue matches the number of points Less than the cascade backoff threshold If the scale-invariant feature transform SIFT can provide sufficient matching, then it will automatically revert to its previous state. If SIFT is used, then ORB algorithm is used; otherwise, the data is output through dimensionality normalization mapping. The expression for the above progressive backoff strategy is:

[0102] ;

[0103] Phase 1: Feature matching is performed using the SIFT algorithm, denoted as... ;

[0104] ;

[0105] ;

[0106] Phase 2: If SIFT matching fails, the ORB algorithm is used for matching, denoted as... ;

[0107] ;

[0108] ;

[0109] in, This represents the final set of optimal matching results; , , These represent the matching results output by the LightGlue, SIFT, and ORB algorithms, respectively. and These represent the number of valid match points for LightGlue and SIFT, respectively. and These represent the cascade backoff threshold and the lower limit threshold for homography calculation, respectively. This indicates the preset maximum number of feature points (set to 5000 in this embodiment of the invention).

[0110] If the number of valid matching points extracted by the ORB algorithm is less than the lower threshold for homography calculation If the image fails to stitch together, the current image pair is determined to have failed, and the process is skipped and proceeds to the next image processing step.

[0111] When the traditional matcher is triggered, to ensure the consistency of the data structure output by different feature matching algorithms, the batch dimension is removed. This ensures that even if the system uses traditional algorithms for feature extraction and matching, the output data format remains consistent, providing a consistent input interface for subsequent homography matrix calculations and image transformations.

[0112] Step 3: Based on the feature matching point pairs, calculate the homography transformation matrix between the images to complete the geometric registration of the images and obtain the registered image pairs.

[0113] In this embodiment of the invention, based on the corresponding point pairs output by feature matching, the Random Sampling Consensus (RANSAC) algorithm is used for iterative optimization to find the optimal homography matrix that maximizes the number of interior points. Its expression is:

[0114] ;

[0115] in, Represents the optimal homography matrix; Represents the candidate homography matrix; This represents the total number of feature matching pairs; Indicates the reference image number homogeneous coordinates of feature points Represents the homogeneous coordinates of the corresponding feature points in the image to be registered; This represents the reprojection error threshold (set to 3.0 pixels in this embodiment of the invention).

[0116] The expression for homography matrix transformation is:

[0117] ;

[0118] in, Represents the optimal homography matrix; This represents the currently calculated image pixel coordinates; Represents the pixel coordinates of the target image after transformation; to Representing the homography matrix The 9 element parameters.

[0119] To ensure transformation accuracy, this invention employs multi-dimensional verification of the matching point quality. This includes evaluating the fitting quality of the homography matrix by calculating the root mean square reprojection error of the interior points in the feature point set, performing interior point ratio statistics and verifying the determinant of the transformation matrix, thus avoiding degenerate transformations. For potential perspective distortion in UAV images, a complete 8-DOF homography model is used, supporting rotation, scaling, translation, and perspective correction.

[0120] The aforementioned transformation matrix is ​​used to accurately map the images to be stitched onto the coordinate system of the reference image. Bilinear interpolation is used to maintain the image quality after geometric transformation, which facilitates subsequent fusion.

[0121] Step 4: Construct a spatial confidence potential field based on the confidence information of the feature matching point pairs, and use the spatial confidence potential field to perform dynamic stitching optimization to generate a locally feathered fusion weight map.

[0122] In embodiments of the present invention, such as Figure 4 As shown, step 4 includes:

[0123] Based on the sparse matching points output by LightGlue Based on the given information and their corresponding confidence levels, a continuous spatial confidence potential field is constructed. A Gaussian kernel function is used to diffuse the discrete feature point confidence levels into the spatial domain, and these confidence levels are normalized so that their range is distributed in the [0,1] interval. The spatial confidence potential field is then constructed, and its expression is:

[0124] ;

[0125] in, For the spatial confidence potential field; For the first The coordinates and confidence scores of each matching point; The Gaussian kernel scale factor is used to control the potential energy diffusion range; These are the currently calculated image pixel coordinates. For the first The pixel coordinates of the matching feature points; This represents the total number of valid match points extracted by LightGlue.

[0126] The spatial confidence potential field is used to guide the optimal suture line to avoid the parallax-sensitive region and converge to the geometrically aligned reliable region. The total energy function includes a confidence repulsion term derived from the spatial confidence potential field, and its expression is as follows:

[0127] ;

[0128] in, Let be the total energy function. The intensity of the pixel difference between the two images; For geometric center constraint terms; This is the weighting coefficient for the potential energy term, used to adjust the influence of the confidence constraint; The confidence exclusion term is expressed as follows:

[0129] ;

[0130] in, This is an energy penalty factor; the lower the confidence level, the lower the penalty factor. The higher the value, the higher the energy cost of traversing the current region; the algorithm uses dynamic programming to solve for the minimum energy path.

[0131] Building upon this, and employing a distance-transform-based local feathering technique, a smooth-transition fusion weight map is generated by calculating the Euclidean distance from each pixel to the seam line. Its expression is as follows:

[0132] ;

[0133] in, A fusion weight map for a smooth transition; Feather width (set to 50 pixels in this embodiment of the invention); and These are the positive and negative Euclidean distances from the pixel to the seam line, respectively. A cutoff function to ensure that the weight values ​​are strictly limited to the interval [0, 1].

[0134] This strategy ensures that mixing occurs only in narrow bands near the suture line, while areas far from the suture line retain their original pixel values. This provides precise spatial weight guidance for subsequent frequency domain fusion.

[0135] Step 5: Using a resource-aware adaptive multi-band fusion strategy, the registered image pairs are fused together with the fusion weight map to generate a seamless panoramic image.

[0136] The Laplacian pyramid is strictly truncated into two layers to avoid the enormous computational overhead and large-scale ghosting issues caused by traditional multi-band fusion methods that construct 5-7 layer pyramids. Layer 0 (High-Frequency Layer): Preserves the details and textures of the original image, using a fusion weight map for precise cropping, physically blocking the diffusion paths of ghosting and artifacts. Layer 1 (Low-Frequency Layer): Contains the image's lighting and color information; broadband mixing is performed at this layer to achieve seamless brightness transitions.

[0137] In this embodiment of the invention, step 5 includes:

[0138] A shallow Laplacian pyramid with only two layers is used for fusion; in the high-frequency layers, weighted fusion is performed using a fusion weight map; in the low-frequency layers, broadband mixing is performed for illumination transition; the pyramid reconstruction expression is:

[0139] ;

[0140] in, For the final seamlessly merged image, The result is the fusion of high-frequency details at layer 0 after weighting by the fusion weight map. This is the weighted fusion result of the first-layer low-frequency approximation image.

[0141] In this embodiment of the invention, the pyramid reconstruction includes:

[0142] A texture analysis mechanism was introduced before constructing the weight pyramid; the average gradient magnitude of the overlapping regions was calculated. Dynamic weight selection generation model: low-texture flat region A linear weighting model is adopted, and the Gaussian blur kernel is increased to reduce color difference tortuosity. The preset texture threshold is set to 0.1 under the texture normalization score system; high texture complexity areas The process switches to a nonlinear S-shaped weight model and reduces the Gaussian blur kernel to prevent details from being overly blurred. This process uses vectorized matrix operations instead of the traditional pixel-by-pixel loop to improve computation speed.

[0143] In this embodiment of the invention, it further includes: a built-in circuit breaker mechanism based on canvas size for situations where homography matrix divergence may occur under large parallax conditions; if the number of projected canvas pixels exceeds a safety threshold... (In this embodiment of the invention, the value is set to 8×10) 7 If the pixel count is less than a certain value, the algorithm determines there is a risk of memory overflow, automatically bypasses the pyramid construction module, and downgrades to linear fusion to ensure system robustness. This invention improves the registration robustness and fusion quality in UAV image stitching, overcoming the high computational cost of traditional methods.

[0144] In this embodiment of the invention, a lightweight improvement scheme for the system is designed based on the LightGlue deep learning matcher, taking into account the computational constraints of embedded platforms. Since the LightGlue model has high computational complexity and large memory consumption in its original 9-layer structure, a lightweight strategy based on layer pruning is proposed. The matching accuracy of LightGlue is positively correlated with its computational complexity, as expressed by:

[0145] ;

[0146] in, This represents the i-th layer Transformer module; These represent the feature descriptors of the input image; L represents the total number of layers; This indicates the composition of functions;

[0147] In embodiments of the present invention, such as Figure 3 As shown, the specific method for lightweight improvement is as follows: First, the integrity of the front-end feature extractor SuperPoint is preserved to ensure the quality of basic features; second, the intermediate layers of LightGlue are pruned layer by layer, and the performance of different pruning schemes on the UAV image dataset is compared through large-scale batch testing; where images A and B are used as input images, layers L1-L3 are adaptive decision regions, layers L4-L6 are point-by-point pruning activation regions, and layers L7-L9 are fine-tuning matching regions.

[0148] In this embodiment of the invention, the real-world scene dataset UDIS-D was used for testing. The top 1000 image pairs selected from this dataset were examined to investigate the impact of LightGlue model depth pruning on its matching performance. The number of Transformer layers in LightGlue was gradually pruned from 9 to 4 layers. SIFT and ORB were used as traditional methods for comparison. Evaluation metrics included matching success rate, inlier rate, number of feature points, inference time, and root mean square error of reprojection, to determine the optimal trade-off.

[0149] In embodiments of the present invention, such as Figure 5 Figures (a) to (d) show the experimental results. In terms of robustness, all pruned versions of LightGlue and the ORB algorithm achieved a 100% matching success rate on this dataset, while the SIFT algorithm achieved 99.9%, indicating that LightGlue exhibits good robustness even with shallower network structures. Regarding the number of matches, LightGlue significantly outperforms SIFT (456.1) and ORB (454.4), with an average of over 1600 matches, approximately 3.5 times that of SIFT (456.1) and ORB (454.4). Notably, as the number of layers is moderately reduced (from 9 to 3), due to appropriate pruning preventing excessive feature suppression caused by overfitting, the number of matching points peaks at 1649.1 with 3 layers.

[0150] Analyzing from the perspectives of registration accuracy and robustness, such as Figure 5As shown in (a), the LightGlue model exhibits excellent robustness to different pruning levels. From the initial 9 layers to 4 layers, its RMSE only increased from 1.427 pixels to 1.452 pixels, maintaining a low sub-pixel error. Compared to traditional matching methods, although SIFT (1.025 pixels) and ORB (1.265 pixels) can achieve high accuracy for individual matching points, LightGlue retains similar geometric accuracy because it has a much larger number of matching points. Similarly, in Figure 5 In (d) of the model, the in-point ratio is consistently around 0.995 in versions with 4 layers and above, indicating that the 4-layer model can maintain its geometric registration accuracy even with a large number of matching points.

[0151] The comparison of runtime efficiency shows that inference time decreases linearly with the number of layers. The original 9-layer network required approximately 0.240 seconds for inference, while the simplified LightGlue-4-layer network only required 0.215 seconds, which is faster than SIFT (0.230 seconds) and only 0.02 seconds slower than the renowned ORB algorithm (0.210 seconds). This demonstrates that the method proposed in this invention utilizes pruning techniques to enable LightGlue-Lite to overcome the inherent slowness of deep learning compared to traditional algorithms, achieving an optimal balance between accuracy, density, and speed.

[0152] In embodiments of the present invention, such as Figure 6 As shown in (a), when the SIFT algorithm encounters complex texture regions such as vegetation, the discriminative power of the feature descriptors decreases, resulting in very sparse effective matching points (only 67 pairs), making it difficult to cover the main body of the image; Figure 6 As shown in (b), the ORB algorithm detects more feature points, but has a high false match rate and contains a large number of scattered intersecting line segments in the matching results, lacking geometric consistency; in contrast, as shown in (b), the ORB algorithm detects more feature points, but has a high false match rate and a large number of scattered intersecting line segments in the matching results, lacking geometric consistency. Figure 6 As shown in (c), the original version of LightGlue has good robustness, and its matched line segments are basically parallel; as Figure 6 As shown in (d) of the present invention, the LightGlue-Lite (4 layers) model of the present invention has the same number of matches and the density of matching points as the original 9-layer model. Pruning does not reduce the number of matches, indicating that the pruning scheme greatly saves the amount of computation while not affecting the network's ability to perceive complex environments and aggregate contextual information.

[0153] Based on the above comparison of accuracy, robustness, and efficiency, a Transformer layer configuration of 4 is considered the optimal solution, namely the lightweight model LightGlue-Lite proposed in this invention. Figure 5As indicated by the asterisk, this version significantly reduces computational overhead while maintaining almost identical reprojection root mean square error and interior point rate to the original deep model, making it the best lightweight solution.

[0154] In this embodiment of the invention, a test set containing 1105 pairs of real-world scene images from the UDIS-D dataset was used, and a comprehensive comparative test was conducted with SIFT, ORB, LightGlue-Lite, and LoFTR models equipped with standard multi-band fusion. Processing speed (FPS), structural similarity (SSIM), peak signal-to-noise ratio (PSNR), gradient magnitude similarity deviation (GMSD), and perceptual image similarity (LPIPS) were used as evaluation metrics, with FPS serving as the efficiency metric. Considering that the black background in non-overlapping regions would dilute the effectiveness of the metrics, this embodiment of the invention only calculates the above metrics within the effective overlapping region of two images. All quality evaluation metrics are uniformly formalized as follows:

[0155] ;

[0156] in, This represents the final value of a certain quality indicator. This represents the stitched image generated by fusion; This represents two registered source images; A binary mask representing the overlapping region; Indicates based on mask Extract the minimum bounding box and perform a cropping operation; This represents the corresponding evaluation function. The final value is the arithmetic mean of the fused image and the two source images calculated separately. For PSNR and SSIM, this embodiment of the invention calculates pixel-by-pixel within the mask; for LPIPS and GMSD, it is calculated based on the minimum bounding box extracted from the mask.

[0157] The results of the embodiments of the present invention (using LightGlue-Lite+ adaptive multi-band fusion) are shown in bold. The experimental results are shown in Table 1.

[0158] Table 1. Experimental data comparing image stitching performance based on the UDIS-D dataset.

[0159] ;

[0160] In terms of performance, the result of 5.11 FPS on RTX 2060 is about 235% higher than LoFTR (1.53 FPS) and similar to SIFT (5.22 FPS) and LightGlue-Lite (5.07 FPS). This shows that the combination of multiple methods such as model pruning, adaptive matching and resource-aware fusion can effectively solve the problems of computational latency and computing power.

[0161] From the perspective of peak signal-to-noise ratio (PSNR) for evaluating image fidelity, the embodiment of this invention achieved a result of 26.33dB, which is 2.37% higher than LightGlue-Lite using standard multi-band fusion, i.e., an increase of 0.61dB. This indicates that the method has excellent performance in pixel-level restoration. In terms of LPIPS and GMSD, which reflect the subjective perception of the human eye, the method of this invention is also superior to the benchmark method, reducing them by 1.65% and 3.97% respectively compared to LightGlue-Lite using standard multi-band fusion. This shows that adaptive fusion can effectively eliminate ghosting and improve visual consistency. In terms of structural similarity (SSIM), the embodiment of this invention is only less than 0.5% different from the existing high-performance method LoFTR, which is on the same level. While significantly improving speed, it can preserve the structural information of the image as much as possible, making the transition smoother and more natural.

[0162] A comparison of the stitching performance of this invention's embodiments with other benchmark algorithms on the UDIS-D dataset, such as... Figure 7 As shown, both the traditional algorithm (SIFT / ORB) and the benchmark model (LightGlue-Lite / LoFTR) employ standard multi-band fusion methods. Under conditions where the registration error is not significant, some algorithms exhibit obvious misalignment and breaks in areas with repetitive textures such as railings, steps, and clouds. Furthermore, all four algorithms show blurred ghosting within overlapping areas with large parallax. The adaptive multi-band fusion strategy of this invention significantly overcomes the aforementioned geometric misalignment problems and effectively reduces ghosting at seams.

[0163] In this embodiment of the invention, two representative sets of high-resolution aerial photographs were selected from the OpenDroneMap dataset for experimentation. Ablation experiments were conducted using LightGlue-Lite combined with a standard multi-band fusion algorithm. The stitched results and local magnifications are shown below. Figure 8 As shown, the baseline method suffers from severe blurring, ghosting, and edge breakage at the stitching point. However, the adaptive multi-band fusion method proposed in this invention, by using confidence guidance and stitching line optimization, accurately finds the optimal stitching position, ensuring that the image stitching edge remains clear and sharp, effectively reducing artifacts and significantly improving the appearance of the stitching point.

[0164] The technical solution provided by this invention includes a method that performs dual-effect linked quality perception and preprocessing on the input UAV sequence images to obtain a quality-enhanced image and a comprehensive image quality score; based on the comprehensive image quality score, a heterogeneous feature flow adaptive cascade mechanism based on image quality perception is used to extract and match features in the quality-enhanced image to obtain a set of feature matching point pairs; based on the feature matching point pairs, the homography transformation matrix between images is calculated to complete the geometric registration of the images, resulting in a registered image pair; a spatial confidence potential energy field is constructed based on the confidence information of the feature matching point pairs, and dynamic stitching optimization is performed using the spatial confidence potential energy field to generate a locally feathered fusion weight map; a resource-aware adaptive multi-band fusion strategy is adopted, and the registered image pair is fused in combination with the fusion weight map to generate a seamless panoramic image. This method achieves highly robust, high-quality, and efficient UAV image stitching.

[0165] The various steps in the embodiments of the present invention can be performed by an electronic device. This electronic device includes, but is not limited to, tablet computers, portable PCs, and desktop computers.

[0166] This invention provides a computer-readable storage medium, which includes a stored program. When the program is running, it controls the electronic device containing the computer-readable storage medium to execute the above-described embodiment of the UAV image stitching method based on LightGlue and adaptive multi-band fusion.

[0167] Figure 9 A schematic diagram of an electronic device provided in an embodiment of the present invention, such as... Figure 9 As shown, the electronic device 21 includes a processor 211, a memory 212, and a computer program 213 stored in the memory 212 and executable on the processor 211. When the computer program 213 is executed by the processor 211, it implements the UAV image stitching method based on LightGlue and adaptive multi-band fusion in the embodiment. To avoid repetition, it will not be described in detail here.

[0168] Electronic device 21 includes, but is not limited to, processor 211 and memory 212. Those skilled in the art will understand that... Figure 9 This is merely an example of electronic device 21 and does not constitute a limitation on electronic device 21. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device may also include input / output devices, network access devices, buses, etc.

[0169] The processor 211 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0170] The memory 212 can be an internal storage unit of the electronic device 21, such as a hard disk or RAM of the electronic device 21. The memory 212 can also be an external storage device of the electronic device 21, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or FlashCard equipped on the electronic device 21. Furthermore, the memory 212 can include both internal and external storage units of the electronic device 21. The memory 212 is used to store computer programs and other programs and data required by network devices. The memory 212 can also be used to temporarily store data that has been output or will be output.

[0171] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0172] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for unmanned aerial vehicle image stitching based on LightGlue and adaptive multi-band fusion, characterized in that, The method includes: Step 1: Perform dual-effect linked quality perception and preprocessing on the input UAV sequence images to obtain the enhanced image and comprehensive image quality score; Step 2: Based on the comprehensive image quality score, an adaptive cascade mechanism based on image quality perception heterogeneous feature flow is used to extract and match features of the enhanced image to obtain a set of feature matching point pairs. Step 3: Based on the feature matching point pairs, calculate the homography transformation matrix between the images to complete the geometric registration of the images and obtain the registered image pairs; Step 4: Construct a spatial confidence potential field based on the confidence information of the feature matching point pairs, and use the spatial confidence potential field to perform dynamic stitching optimization to generate a locally feathered fusion weight map. Step 5: Using a resource-aware adaptive multi-band fusion strategy, the registered image pairs are fused together with the fusion weight map to generate a seamless panoramic image. Step 2 includes: Based on overall image quality score Dynamically generate LightGlue's cascading backoff threshold And the lower limit threshold for homography calculation ; When inputting a low-quality, weakly textured image, actively reduce... To trust the matching capabilities of deep learning models; and improve To enhance the robustness of matrix solution; when high-quality images are input, it actively improves... and reduce The expression for the threshold dynamic mapping function is: ； ； in, These represent the lower and upper limits of the backoff threshold, respectively. These represent the lower and upper limits of the threshold for matrix calculation, respectively. Adaptive adjustment coefficient; When LightGlue matches the number of points Less than the cascade backoff threshold If the scale-invariant feature transform SIFT can provide sufficient matching, then it will automatically revert to its previous state. If SIFT is used, then ORB algorithm is used; otherwise, the data is output through dimensionality normalization mapping. The expression for the above progressive backoff strategy is: ； Phase 1: Feature matching is performed using the SIFT algorithm, denoted as... ; ； ； Phase 2: If SIFT matching fails, the ORB algorithm is used for matching, denoted as... ; ； ； in, This represents the final set of optimal matching results; , , These represent the matching results output by the LightGlue, SIFT, and ORB algorithms, respectively. and These represent the number of valid match points for LightGlue and SIFT, respectively. and These represent the cascade backoff threshold and the lower limit threshold for homography calculation, respectively. This indicates the preset maximum number of feature points.

2. The method according to claim 1, characterized in that, Step 1 includes: The image data quality assessment strategy is designed by calculating the image sharpness and evaluating the overall brightness distribution to obtain a brightness score. The scoring calculation logic is as follows: a. Based on the response variance of the Laplacian operator, the sharpness score is calculated, and its expression is as follows: ； in, This represents the sharpness score, with a value ranging from 0 to 1; 1000 represents the variance of the Laplacian convolution response of the image; 1000 represents the normalization coefficient. b. The brightness score is obtained by normalizing the mean pixel intensity, and its expression is: ； in, This represents the brightness score, with a value ranging from 0 to 1. This represents the average intensity value of grayscale pixels in the image, with a value range of 0 to 255. c. Obtain a clarity score and brightness rating Then, construct a comprehensive image quality scoring function. This evaluates the feature extraction capability of images under current lighting and texture conditions, and integrates the image quality scoring function. The expression is: ； in, , This represents a non-linear brightness penalty mechanism.

3. The method according to claim 1, characterized in that, Step 4 includes: The spatial confidence potential field is constructed, and its expression is: ； in, For the spatial confidence potential field; For the first The coordinates and confidence scores of each matching point; The Gaussian kernel scale factor is used to control the potential energy diffusion range; These are the currently calculated image pixel coordinates. For the first The pixel coordinates of the matching feature points; This represents the total number of valid match points extracted by LightGlue. The spatial confidence potential field is used to guide the optimal suture line to avoid the parallax-sensitive region and converge to the geometrically aligned reliable region. The total energy function includes a confidence repulsion term derived from the spatial confidence potential field, and its expression is as follows: ； in, Let be the total energy function. The intensity of the pixel difference between the two images; For geometric center constraint terms; This is the weighting coefficient for the potential energy term, used to adjust the influence of the confidence constraint; The confidence exclusion term is expressed as follows: ； in, This is an energy penalty factor; the lower the confidence level, the lower the penalty factor. The higher the value, the higher the energy cost of traversing the current area; Building upon this, and employing a distance-transform-based local feathering technique, a smooth-transition fusion weight map is generated by calculating the Euclidean distance from each pixel to the seam line. Its expression is as follows: ； in, A fusion weight map for a smooth transition; The width of the feather; and These are the positive and negative Euclidean distances from the pixel to the seam line, respectively. A cutoff function to ensure that the weight values ​​are strictly limited to the interval [0, 1].

4. The method according to claim 3, characterized in that, Step 5 includes: A shallow Laplacian pyramid with only two layers is used for fusion; in the high-frequency layers, weighted fusion is performed using a fusion weight map; in the low-frequency layers, broadband mixing is performed for illumination transition; the pyramid reconstruction expression is: ； in, For the final seamlessly merged image, The result is the fusion of high-frequency details at layer 0 after weighting by the fusion weight map. This is the weighted fusion result of the first-layer low-frequency approximation image.

5. The method according to claim 4, characterized in that, The pyramid reconstruction includes: A texture analysis mechanism was introduced before constructing the weight pyramid; the average gradient magnitude of the overlapping regions was calculated. Dynamic weight selection generation model: low-texture flat region A linear weighting model is adopted, and the Gaussian blur kernel is increased to reduce color difference tortuosity. Preset texture threshold; high texture complexity area Switch to a nonlinear S-shaped weight model and simultaneously reduce the Gaussian blur kernel.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the UAV image stitching method based on LightGlue and adaptive multi-band fusion as described in any one of claims 1 to 5.

7. An electronic device, characterized in that, include: One or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs including instructions that, when executed by the device, cause the device to perform the UAV image stitching method based on LightGlue and adaptive multi-band fusion as described in any one of claims 1 to 5.

Images