A UAV infrared image stitching method based on SuperPoint and global adjacency optimization

Abstract

The present invention belongs to the technical field of image stitching and relates to a method for stitching infrared images of unmanned aerial vehicles (UAVs) based on SuperPoint and global adjacency optimization. The method comprises the following steps: performing grayscale conversion and histogram equalization preprocessing on infrared images continuously taken by a UAV; detecting feature points using a SuperPoint algorithm, matching feature points using a FLANN algorithm, and screening matching pairs; constructing a transformation matrix and estimating the transformation matrix using a global matching method to traverse adjacent image pairs; constructing a constraint matrix and solving a sparse matrix to obtain optimization parameters; determining an output image size according to the optimization parameters, transforming and stitching the images, and generating a panoramic image. The method improves the robustness of feature extraction and matching, reduces cumulative errors, realizes the stitching of aerial images, and overcomes the shortcomings of traditional image stitching algorithms, such as large computational complexity and time-consuming stitching.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method for stitching unmanned aerial vehicle (UAV) infrared images based on SuperPoint and global adjacency optimization. Background Art

[0002] With the rapid development of low-altitude economics and digital photography, remote sensing imagery acquired from drones has become a crucial data collection tool in numerous fields. High-quality panoramic images, which provide comprehensive visual information for scene understanding and target recognition, are a key challenge in remote sensing image processing. Current infrared images acquired from drones suffer from a limited field of view. This is primarily due to the camera's focal length being constrained by flight altitude and payload limitations, making it difficult for a single image to fully capture the scene. Image stitching technology, an effective solution to this problem, can combine images from multiple angles into a single, wide-field, comprehensive panoramic image.

[0003] Currently, image stitching methods are mainly divided into two categories. One is offline stitching methods that rely on professional PC software such as PTGUI and COLMAP. Although these methods have high accuracy, they are computationally complex, have poor real-time performance, and perform poorly in scenes with large lighting changes or viewing angle differences. The other is real-time stitching methods based on feature matching. However, existing real-time stitching algorithms still face many challenges in feature extraction, global alignment, and edge fusion. For example, traditional feature extraction algorithms often have less than ideal matching accuracy when processing images with unclear texture features, such as infrared images. And when global alignment algorithms process large image sequences, the accumulated errors can lead to significant distortion or misalignment in the stitching results.

[0004] Therefore, a UAV infrared image stitching method is needed that can process infrared images in real time and has high-precision global alignment capabilities. Summary of the Invention

[0005] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a UAV infrared image stitching method based on SuperPoint and global adjacency optimization to improve the accuracy and efficiency of image stitching, which is particularly suitable for the task of stitching time-series infrared images taken by UAVs.

[0006] To achieve the purpose of the present invention, the present invention is implemented by adopting the following technical solutions.

[0007] A method for stitching UAV infrared images based on SuperPoint and global adjacency optimization includes the following technical solutions:

[0008] Obtaining time-series infrared images taken by the UAV to be stitched, and preprocessing the input time-series infrared images to obtain time-series infrared images with enhanced contrast;

[0009] A feature point detection network obtained through self-supervised training is used to extract image feature points from the input pre-processed time-series infrared image and generate descriptor vectors. The descriptor vectors are matched using a KD-tree constructed using a fast nearest neighbor search algorithm, and initial matching pairs are screened out in combination with a bidirectional consistency check. The feature point detection network is constructed based on the SuperPoint feature extraction algorithm.

[0010] Based on the initial matching pairs, a weighted adjacency matrix is ​​constructed to describe the adjacent relationship between time-series infrared images. The affine transformation matrix between adjacent time-series infrared image pairs is estimated using a graph-optimized global matching method. The optimization objective function is established by traversing adjacent time-series infrared image pairs.

[0011] A global constraint matrix for global alignment optimization is constructed based on the weighted adjacency matrix and the affine transformation matrix. The global alignment optimization problem is transformed into a sparse linear system using the global constraint matrix. The sparse linear system is solved using a sparse matrix solution method to obtain the optimized global alignment parameters. The global alignment parameters include the rotation angle and translation between the time-series infrared images.

[0012] According to the optimized global alignment parameters, the size of the output canvas is determined by calculating the boundary range of each time-series infrared image. By calculating the affine transformation matrix of each time-series infrared image, each time-series infrared image is affine transformed according to the affine transformation matrix of each time-series infrared image and projected into a unified panoramic coordinate system to generate a panoramic image.

[0013] As a preferred embodiment of the present invention, the pretreatment process comprises the following steps:

[0014] S21, grayscale processing is performed on the input time series infrared image;

[0015] S22 , performing histogram equalization processing on the time series infrared image processed in step S21 .

[0016] As a preferred embodiment of the present invention, the feature point detection network includes a shared encoder, a dual-branch decoder and a temporal infrared image optimization module, wherein:

[0017] A shared encoder is used to perform multi-level feature extraction on the input time-series infrared image and output a downsampled feature map;

[0018] The dual-branch decoder consists of a feature point detection branch and a descriptor generation branch. The feature point detection branch generates the confidence distribution of pixel-level feature points from the downsampled feature map input from the shared encoder and filters the feature point positions through non-maximum suppression. The descriptor generation branch generates a high-dimensional vector descriptor for each feature point, which is normalized and used for feature point matching.

[0019] The infrared image optimization module is used to normalize the dynamic range of infrared radiation values ​​at the input layer and embed a channel attention mechanism in the encoder to enhance the thermal feature response.

[0020] As a preferred embodiment of the present invention, the self-supervised training process of the feature point detection network includes the following steps:

[0021] S41, pre-training the feature point detection network by generating pseudo-label data through synthetic homography transformation;

[0022] S42. Introduce a real time series infrared image dataset to perform transfer learning fine-tuning on the pre-trained feature point detection network.

[0023] As a preferred solution of the present invention, the matching is performed on the image feature points by a FLANN matcher, and the similarity of the high-dimensional vector descriptors is measured using the Euclidean distance:

[0024]

[0025] Among them, p and q are two eigenvectors, and n is the dimension of the eigenvector.

[0026] As a preferred embodiment of the present invention, the optimization method of the initial matching pair is performed by using a distance ratio test method; wherein: the expression of the distance ratio test is:

[0027]

[0028] Among them, q1 is the closest matching point, and q2 is the second closest matching point.

[0029] As a preferred solution of the present invention, the expression of the affine transformation matrix is:

[0030]

[0031] Among them, a 11 、a 12 、a 21 、a 22 Indicates rotation and scaling, t x t y Indicates translation.

[0032] As a preferred embodiment of the present invention, the optimization objective function is:

[0033] E=Σ ij ρ(||x' ij -H i ·x ij || 2 )

[0034] Where: x' ij and x ij is the coordinate of the matching point pair, H i is the transformation matrix of the i-th image, and ρ is the Huber robust kernel function.

[0035] As a preferred embodiment of the present invention, the global alignment optimization process includes the following steps:

[0036] A) Treat each image to be stitched as a node in the graph. If there are feature point matches between the images, edges are established between the corresponding nodes.

[0037] B) Image alignment requires simultaneous estimation of the transformation parameters of all images. Not all images are adjacent, and all constraint matrices are sparse. Therefore, complex nonlinear problems must be converted into sparse linear equations for solution to improve computational efficiency.

[0038] C) Automatically identify and suppress the impact of mismatches through the IRLS optimization algorithm.

[0039] As a preferred solution of the present invention, the global alignment parameters are used to fine-tune the affine transformation matrix to reduce geometric distortion and gap error.

[0040] As a preferred solution of the present invention, in the process of solving the sparse linear system by the sparse matrix solving method, the cumulative error is reduced by iterative optimization.

[0041] As a preferred solution of the present invention, the overlapping areas in the panoramic image are smoothed by a weighted fusion method to eliminate the stitching gaps:

[0042] I 融合 (x,y)=w1I1(x,y)+w2I2(x,y)

[0043] Where w1 and w2 are weights, satisfying w1+w2=1.

[0044] As a preferred solution of the present invention, the adjacency matrix is ​​expressed as A∈R N×N , used to record the matching relationship between time-series infrared images, where:

[0045]

[0046] The adjacency matrix is ​​a sparse matrix that indicates which images have sufficient feature matches. If the number of feature matches between adjacent images reaches a set threshold, it is considered that there is a sufficient match between the two images and this is recorded in the adjacency matrix.

[0047] As a preferred solution of the present invention, the graph model in the graph optimization is constructed based on the initial matching pairs, and is used to describe the graph model of the topological relationship of the time-series infrared images. The graph model is expressed as G = (V, E), where the node V represents the image frame and the edge E represents the matching relationship between images.

[0048] As a preferred solution of the present invention, the global alignment constraint matrix is ​​expressed as:

[0049] Ax=b

[0050] Where A is the coefficient matrix, x is the transformation parameter vector to be solved, and b is the constant vector.

[0051] The core goal of global alignment optimization is to eliminate the stitching error caused by the accumulation of local transformations by minimizing the alignment error between images.

[0052] As a preferred solution of the present invention, the affine transformation matrix of each image is calculated based on the optimized transformation parameters (including rotation angle and translation amount), and the image is projected into a unified coordinate system. The mathematical expression of image projection is:

[0053] x'=Hx

[0054] Where x is the point in the original image, x' is the projected point, and H is the transformation matrix.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] 1. By fusing the characteristic information of multiple images, this invention can more comprehensively capture the thermal radiation characteristics and structural information in infrared images, thereby more accurately establishing the correspondence between images. This feature significantly improves the stitching quality of low-contrast infrared images, resulting in excellent thermal target integrity and scene consistency.

[0057] 2. This paper uses an image stitching method based on SuperPoint feature point detection, combined with graph optimization global matching technology, to significantly improve the matching accuracy and robustness between images. The SuperPoint algorithm extracts semantic-level features through a deep convolutional network, combined with the global consistency constraints of graph optimization, to ensure the accuracy and stability of feature matching in complex infrared scenes, providing a more reliable data foundation for subsequent stitching.

[0058] 3. By solving global constraints within a graph optimization framework, this paper enables joint pose optimization across image sequences, effectively eliminating the cumulative error problem found in traditional methods. The optimization strategy ensures geometric consistency during image stitching, resulting in higher-quality stitching results.

[0059] 4. The global optimization method of this invention combines rotation and translation information between images. By constructing multi-constraint equations and solving a system of linear equations, it ensures global consistency during image stitching. This method effectively eliminates unnatural stitching or misalignment caused by the accumulation of local errors, significantly improving the robustness and visual quality of the stitching. It is particularly suitable for image stitching tasks in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is the image stitching flow chart designed by the present invention;

[0061] Figure 2 This is the network structure diagram of the SuperPoint lightweight model designed by the present invention. DETAILED DESCRIPTION

[0062] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0063] Example 1, as Figures 1 to 2 As shown in the figure, a UAV infrared image stitching method based on SuperPoint and global adjacency optimization includes the following specific contents:

[0064] The method was run on a computer with the following configuration: an Intel Core i5-12400F processor, an NVIDIA GeForce RTX 3060Ti graphics processor (clocked at 2.50GHz), 32GB of RAM, and Windows 10. The method for stitching drone infrared images based on global adjacency matrix optimization was implemented using the PyTorch 1.7.1 deep learning framework toolkit. This method proposes a drone infrared image stitching method based on SuperPoint and global adjacency optimization, specifically comprising the following steps:

[0065] Step 1: Image Preprocessing

[0066] S11. Image data acquisition and unified processing: Acquire the input image data set and uniformly resize and convert the images to ensure that the sizes of all input images are consistent, providing standardized data for subsequent processing.

[0067] S12. Grayscale image conversion: Convert the input image from RGB format to a grayscale image to reduce computational complexity while retaining the main structural information of the image to facilitate subsequent feature point detection and matching.

[0068] S13. Image contrast enhancement: Perform histogram equalization on grayscale images to enhance the contrast of the image, making the feature points in the image more prominent, and providing a high-quality data foundation for subsequent feature detection and matching.

[0069] Step 2: Feature point detection and matching

[0070] S21. Feature point detection: The SuperPoint algorithm based on deep learning is used to detect key feature points in each image and generate high-dimensional feature descriptors. The SuperPoint algorithm is implemented through a lightweight convolutional neural network. Its network structure includes a feature extraction encoder and a dual-task decoder. It can maintain stable feature detection performance under low-contrast conditions of infrared images, and the detection accuracy is improved by more than 40% compared with traditional methods. Figure 2 shown.

[0071] S22. Feature Point Matching: A fast nearest neighbor search algorithm is used to construct a KD-tree to match feature descriptors, and a bidirectional consistency check is used to select initial matching pairs. An adaptive threshold strategy is used during the matching process, dynamically adjusting the matching distance threshold based on the image signal-to-noise ratio to ensure reliable matching results under different lighting conditions.

[0072] S23. Matching point optimization: Apply global consistency constraints based on graph optimization, build a graph model with feature points as nodes, optimize the matching relationship by minimizing the reprojection error function, use iterative weighted least squares method to eliminate mismatched points, and ultimately retain high-quality matching point pairs that meet geometric consistency, with a matching error rate of less than 5%.

[0073] Step 3: Build a graph model and global optimization

[0074] S31. Graph model construction: Based on the feature matching results, a weighted graph structure is constructed, where nodes represent image poses and edges represent matching relationships between images; the weight of the edge W ij Dynamic calculation based on the number of matching points N and the inlier rate η: W ij =η·N / (1+||t ij ||), ensuring that the weights of large displacement image pairs are adaptively reduced; for weakly connected edges (W ij < threshold θ) to prune the branches and retain strong topological connections.

[0075] S32. Joint transformation matrix estimation: Establish a posture optimization model based on Lie algebra, use sparse Cholesky decomposition to solve the normal equations, and achieve global optimal estimation of posture parameters.

[0076] S33, closed loop consistency verification: for the image sequence on the closed loop path, verify the pose closure error δ = ||ΠT ij -I||F, when δ> tolerance threshold, add additional constraints on the closed loop edges for re-optimization.

[0077] Step 4: Global pose optimization

[0078] S41. Sparse system construction: Convert the global optimization problem into a sparse linear system of the form Ax=b, and use the local connectivity of the image topology to construct the block diagonal coefficient matrix A.

[0079] S42. Accelerated solution: The preconditioned conjugate gradient method (PCG) is used for iterative solution, and matrix operations are accelerated through GPU parallelization. The single iteration time is less than 2ms (1080P image).

[0080] S43, dynamic reweighting: Dynamically adjust the Huber kernel width according to the residual size of each iteration; downgrade the weights of edges with errors greater than 3σ to enhance the robustness of the algorithm.

[0081] Step 5: Image fusion and output

[0082] S51, multi-band fusion: perform Laplace pyramid decomposition on the overlapping area, and perform linear weighted fusion on different frequency bands. The weight is determined by the distance of the stitching seam to effectively smooth the transition area at the seam.

[0083] S52, Adaptive Cropping: Detects the valid content area based on the Alpha channel, uses the convex hull algorithm to calculate the minimum enclosing rectangle, retains >95% of the valid pixels, and ensures high-quality output of the stitching result.

[0084] S53, quality assessment: Calculate the SSIM (structural similarity index) index of the stitched image. When SSIM < 0.85, local re-optimization is automatically triggered to further improve the stitching quality.

[0085] Example 2 is an example of a practical application of a UAV infrared image stitching method based on SuperPoint and global adjacency optimization. The specific implementation process of this example is as follows:

[0086] (1) Data collection and preprocessing

[0087] A DJI M300RTK drone equipped with a FLIR Tau2 infrared camera acquired a 640×512 resolution infrared image sequence at an altitude of 100 meters with an 80% overlap ratio (ambient temperature -10°C to 50°C). The original images were subjected to the following procedures: non-uniformity correction (NUC) to remove fixed-pattern noise from the thermal imager, bicubic interpolation to uniformly scale the image to 1024×820 pixels, grayscale conversion, and adaptive histogram equalization (CLAHE, grid size 64×64) to enhance image quality.

[0088] (2) Feature extraction and matching

[0089] 1) Use the quantized SuperPoint lightweight model for feature detection: add a thermal radiation normalization module to the input layer (linearly map the temperature value to [0, 255]), use depthwise separable convolution in the backbone network, output 256-dimensional descriptors, and control the feature point density to 1500-2000 points / frame

[0090] 2) Feature matching uses a cascade filtering strategy, including FLANN coarse matching (KD-Tree, k=2), bidirectional consistency check (bidirectional matching error <1.5 pixels), and graph optimization fine matching (constructing a feature point relationship graph and optimizing the objective function).

[0091] (3) Pose graph construction and optimization

[0092] A weighted pose graph is constructed and two-level global optimization is performed simultaneously: the front-end is incremental pose estimation (processing time per frame < 15ms), and the back-end is pose graph optimization based on the G2O framework (global optimization once every 100 frames).

[0093] (4) Image fusion and output

[0094] Image transformation and weighted fusion of overlapping areas: According to the transformation matrix H of each image, the images are projected into a unified panoramic coordinate system, and the overlapping areas are weightedly fused.

[0095] Cropping and boundary processing: Valid areas are detected through the alpha channel, irregular boundaries are extracted using the alpha algorithm, and a stitched panorama with a resolution of 3840×2160 is output.

[0096] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A method for stitching UAV infrared images based on SuperPoint and global adjacency optimization, characterized by: The following solutions are included: Obtaining time-series infrared images taken by the UAV to be stitched, and preprocessing the input time-series infrared images to obtain time-series infrared images with enhanced contrast; A feature point detection network obtained through self-supervised training is used to extract image feature points from the input pre-processed time-series infrared image and generate descriptor vectors. The descriptor vectors are matched using a KD-tree constructed using a fast nearest neighbor search algorithm, and initial matching pairs are screened out in combination with a bidirectional consistency check. The feature point detection network is constructed based on the SuperPoint feature extraction algorithm and includes a shared encoder, a dual-branch decoder, and a time-series infrared image optimization module, wherein: A shared encoder is used to perform multi-level feature extraction on the input time-series infrared image and output a downsampled feature map; The dual-branch decoder consists of a feature point detection branch and a descriptor generation branch. The feature point detection branch generates the confidence distribution of pixel-level feature points from the downsampled feature map input from the shared encoder and filters the feature point positions through non-maximum suppression. The descriptor generation branch generates a high-dimensional vector descriptor for each feature point, which is normalized and used for feature point matching. The infrared image optimization module is used to normalize the dynamic range of infrared radiation values ​​at the input layer and embed a channel attention mechanism in the encoder to enhance the thermal feature response; Based on the initial matching pairs, a weighted adjacency matrix is ​​constructed to describe the adjacent relationship between time-series infrared images. The affine transformation matrix between adjacent time-series infrared image pairs is estimated using a graph-optimized global matching method. The optimization objective function is established by traversing adjacent time-series infrared image pairs. A global constraint matrix for global alignment optimization is constructed based on the weighted adjacency matrix and the affine transformation matrix. The global alignment optimization problem is transformed into a sparse linear system using the global constraint matrix. The sparse linear system is solved using a sparse matrix solution method to obtain the optimized global alignment parameters. The global alignment parameters include the rotation angle and translation between the time-series infrared images. According to the optimized global alignment parameters, the size of the output canvas is determined by calculating the boundary range of each time-series infrared image. By calculating the affine transformation matrix of each time-series infrared image, each time-series infrared image is affine transformed according to the affine transformation matrix of each time-series infrared image and projected into a unified panoramic coordinate system to generate a panoramic image.

2. The method for stitching UAV infrared images based on SuperPoint and global adjacency optimization according to claim 1, characterized in that: The pre-processing process comprises the following steps: S21, grayscale processing is performed on the input time series infrared image; S22 , performing histogram equalization processing on the time series infrared image processed in step S21 .

3. The method for stitching UAV infrared images based on SuperPoint and global adjacency optimization according to claim 1, characterized in that: The self-supervised training process of the feature point detection network includes the following steps: S31, pre-training the feature point detection network by generating pseudo-label data through synthetic homography transformation; S32. Introduce a real time series infrared image dataset to perform transfer learning fine-tuning on the pre-trained feature point detection network.

4. The method for stitching UAV infrared images based on SuperPoint and global adjacency optimization according to claim 1, characterized in that: The optimization method of the initial matching pair is performed by using a distance ratio test method; wherein: the expression of the distance ratio test is: Among them, d(p,q1) represents the Euclidean distance between p and its nearest neighbor matching point q1, and d(p,q2) represents the Euclidean distance between p and its second closest neighbor matching point q2. q1 is the nearest matching point of p, and q2 is the second closest matching point of p.

5. The method for stitching UAV infrared images based on SuperPoint and global adjacency optimization according to claim 1, characterized in that: The expression of the affine transformation matrix is: Among them, a 11 、a 12 、a 21 、a 22 Indicates rotation and scaling, t x t y Indicates translation.

6. The method for stitching UAV infrared images based on SuperPoint and global adjacency optimization according to claim 1, characterized in that: The optimization objective function is: E=S ij p(||x' ij -H i ·x ij || 2 ) Where: x' ij and x ij is the coordinate of the matching point pair, H i is the transformation matrix of the i-th image, and ρ is the Huber robust kernel function.

7. The method for stitching UAV infrared images based on SuperPoint and global adjacency optimization according to claim 1, characterized in that: The global alignment parameters are used to fine-tune the affine transformation matrix to reduce geometric distortion and seam errors.

8. The method for stitching UAV infrared images based on SuperPoint and global adjacency optimization according to claim 1, characterized in that: In the process of solving sparse linear systems using sparse matrix solving methods, the accumulated error is reduced through iterative optimization.

9. The method for stitching UAV infrared images based on SuperPoint and global adjacency optimization according to claim 1, characterized in that: The overlapping areas in the panoramic images are smoothed by weighted fusion method to eliminate the stitching gaps: Yo 融合 (x,y)=w1I1(x,y)+w2I2(x,y) Where w1 and w2 are weights, satisfying w1+w2=1.

Images