Unmanned aerial vehicle thermal infrared image splicing method and system based on flight path keeping

Abstract

The invention discloses an unmanned aerial vehicle thermal infrared image stitching method and system based on flight path keeping, and relates to the technical field of image stitching, and the method comprises the steps: constructing an image set; obtaining respective matched feature points of any one thermal infrared image of the to-be-spliced unmanned aerial vehicle, and generating an alignment item through a similar transformation matrix of the to-be-spliced thermal infrared image and a re-projection error of the matched feature points; the actual distance between the thermal infrared image of any unmanned aerial vehicle and the center of the reference image is calculated, and a route track protection item is obtained through the pixel distance between the center point of the similar transformation matrix and the reference image; and constructing a global optimization objective function through weighted summation of the alignment item and the route track protection item, and converting the global optimization objective function into a linear least square model constrained by similar transformation for solving to generate a spliced thermal infrared panoramic image. According to the method, accumulated errors in the splicing process are reduced to the maximum extent, and meanwhile the integrity of a global structure is guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of image stitching, and in particular to a method and system for stitching thermal infrared images of an unmanned aerial vehicle (UAV) based on track keeping. Background Art

[0002] In recent years, with the increasing automation of agriculture, drone technology has been widely adopted in agricultural monitoring, becoming a key tool in modern agriculture. Compared to traditional ground-based surveying and satellite remote sensing, drones can flexibly acquire high-resolution imagery across large areas of farmland. This allows farmers and researchers to monitor crop health and other key farmland information in real time, providing strong technical support for the further development of precision agriculture.

[0003] Compared to traditional visible light imagery, thermal infrared imagery can detect differences in thermal radiation energy emitted by ground objects, a characteristic that makes it crucial in fields such as agricultural monitoring. Using drones equipped with thermal infrared remote sensing technology can effectively capture thermal radiation emitted by crops, thereby reflecting temperature changes within the crop canopy. This makes it widely used in agriculture and forestry for assessing soil moisture balance and water stress, identifying crop lodging, and promoting the development of precision agriculture. Thermal infrared remote sensing technology is also widely used in areas such as detailed studies of urban thermal environments and real-time ground semantic segmentation. By stitching together multiple images, large-scale panoramic images can be generated, providing researchers with a complete view of farmland and facilitating a more comprehensive assessment of crop conditions.

[0004] Compared to visible light imagery, thermal infrared imagery faces more challenges during stitching. For example, the effects of wind and temperature drift, as well as low geometric resolution and poor image quality, often lead to large errors in image feature point extraction and matching, thus compromising the stitching quality of the final panoramic image. In recent years, researchers have proposed various improved methods to address the challenges of large-scale thermal infrared image stitching. While these methods have made some progress in improving stitching accuracy or efficiency, they still have limitations. Existing methods rely on mesh deformation, such as OP-GSP, which utilizes POS information from drones during flight and uses the estimated overlap between adjacent images as local matching pair weights to optimize the local alignment term of GSP to improve stitching. However, this significantly increases computational complexity and processing time, making it difficult to meet real-time requirements. Furthermore, some fast global stitching schemes, such as MegaStitch, model image stitching as a global minimization problem of reprojection errors and constrain the transformation space to a similarity transformation. While these methods offer advantages in stitching efficiency, they often suffer from local image alignment failures, severely impacting the global results and the integrity of the stitched structure. Summary of the Invention

[0005] The purpose of the present invention is to provide a method and system for stitching thermal infrared images of unmanned aerial vehicles based on track keeping in order to solve the problems in the prior art.

[0006] The present invention specifically provides the following technical solutions: A method for stitching thermal infrared images of a UAV based on track keeping, comprising: Collect thermal infrared images from multiple drones and build an image collection; Obtain the matching feature points of any pair of thermal infrared images to be stitched in the image set, and obtain the similarity transformation matrix between each thermal infrared image and the reference coordinate system. Generate an alignment term based on the reprojection error between the similarity transformation matrix of the thermal infrared images to be stitched and the matching feature points. Calculate the actual distance between the center of any thermal infrared image and the reference image, convert the actual distance to obtain the pixel distance between the center of any thermal infrared image and the reference image in the x and y directions, and generate the route trajectory protection item based on the pixel distance; A global optimization objective function is constructed by weighted sum superposition of the alignment term and the route trajectory protection term, and the global optimization objective function is converted into a linear least squares model with similarity transformation constraints for solution to generate a spliced ​​thermal infrared panoramic image.

[0007] Preferably, the alignment term is generated by the reprojection error of the similarity transformation matrix of the thermal infrared image to be stitched and the matched feature points, specifically: Based on the feature points matched to any pair of thermal infrared images to be stitched, the similarity transformation matrix of the thermal infrared images to be stitched is subtracted from the product of their respective feature points, and the absolute value is taken and then squared to generate the alignment item of the thermal infrared images to be stitched.

[0008] Preferably, the generating of the route trajectory protection item by the pixel distance is specifically: Get the similarity transformation matrix of the reference image; The similarity transformation matrix of the thermal infrared image to be stitched is subtracted from the similarity transformation matrix of the reference image, and the difference is multiplied by the coordinates of the center pixel of the thermal infrared image to be stitched on the original image. The product is subtracted from the pixel distance, and the absolute value is taken and then squared to generate the route trajectory protection item.

[0009] Preferably, the global optimization objective function is converted into a linear least squares model with similarity transformation constraints for solving to generate a spliced ​​thermal infrared panoramic image, specifically: The similarity transformation matrix parameters of each thermal infrared image to be stitched are arranged in sequence as a vector, and all matching pairs and route constraints are linearized into a sparse matrix form; where the matching pairs are the matched pairs of thermal infrared images to be stitched, and the route constraints are the trajectory constraints through the route trajectory protection term; The sparse matrix form is solved by a linear least squares solver with similarity transformation structure constraints to obtain the optimal vector, and the similarity transformation matrix is ​​restored through the optimal vector to generate a spliced ​​thermal infrared panoramic image with the restored similarity transformation matrix.

[0010] The present invention provides a UAV thermal infrared image stitching system based on track keeping, comprising: The acquisition module is used to collect thermal infrared images of multiple drones and build an image collection; An alignment term generation module is used to obtain the matching feature points of any pair of thermal infrared images to be stitched in the image set, and obtain the similarity transformation matrix between each thermal infrared image and the reference coordinate system, and generate an alignment term based on the reprojection error between the similarity transformation matrix of the thermal infrared images to be stitched and the matching feature points; A protection item generation module is used to calculate the actual distance between the center of any thermal infrared image and the reference image, convert the actual distance to obtain the pixel distance between the center of any thermal infrared image and the reference image in the x and y directions, and generate the route trajectory protection item based on the pixel distance; The stitching module is used to construct a global optimization objective function by weighted sum superposition of the alignment term and the route trajectory protection term, and convert the global optimization objective function into a linear least squares model constrained by similarity transformation for solution to generate a stitched thermal infrared panoramic image.

[0011] The present invention provides a computer device, including a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of the above-mentioned method for stitching thermal infrared images of unmanned aerial vehicles based on track keeping.

[0012] The present invention provides a storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the above-mentioned method for stitching thermal infrared images of a drone based on track keeping are implemented.

[0013] Compared with the prior art, the present invention has the following significant advantages: The present invention generates an alignment term through the reprojection error of the similarity transformation matrix of the thermal infrared image to be stitched and the matched feature points, and simultaneously calculates the actual distance between the thermal infrared image of any UAV and the center of the reference image, and converts the actual distance into a pixel distance to accurately maintain their relative position in physical space. By obtaining the pixel distance between the center of any thermal infrared image and the reference image in the x and y directions, a route trajectory protection term is generated through the pixel distance. Secondly, the alignment term and the route trajectory protection term are combined to construct a global optimization objective function and solve it. This greatly accelerates the iterative optimization speed while minimizing the cumulative error of the stitching process to the greatest extent, while ensuring the integrity of the global structure, and realizing the generation of high-quality stitched panoramic images. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a schematic diagram of the spatial layout in an embodiment of the present invention; Figure 1 (a) is the spatial layout structure that needs to be constrained during the splicing process. Figure 1 (b) is the process of connecting the sampling points with the central sampling point; Figure 2 is the location of the main research area in the embodiment of the present invention; Figure 2 (a) is the Xinjiang jujube tree dataset, Figure 2 (b) is the Inner Mongolia farmland dataset; Figure 3 is a thermal infrared image showing obvious exposure differences in an embodiment of the present invention; Figure 3 (a) is a thermal infrared image with brighter exposure. Figure 3 (b) is a thermal infrared image with darker exposure; Figure 4 This is a comparison chart of the stitching effects of 558 thermal infrared images of a drone collected at a flight altitude of 120m in 2018 using different stitching methods in an embodiment of the present invention; Figure 4 (a) is the Pix4D mosaic image, Figure 4 (b) is the PhotoScan mosaic image, Figure 4 (c) is the stitching image stitched by the stitching algorithm AutoStitch. Figure 4 (d) is the stitching diagram spliced ​​by MegaStitch, Figure 4 (e) is the spliced ​​graph spliced ​​by OP-GSP, Figure 4 (f) is a spliced ​​image spliced ​​by the method of the present invention; Figure 5 This is a comparison chart of the stitching effects of 563 thermal infrared images of drones collected at a flight altitude of 120m in 2019 using different stitching methods in an embodiment of the present invention; Figure 5 (a) is the spliced ​​image spliced ​​by PTGui, Figure 5(b) is the stitching diagram stitched by AutoStitch. Figure 5 (c) is the stitching diagram stitched by MegaStitch, Figure 5 (d) is the spliced ​​graph spliced ​​by OP-GSP, Figure 5 (e) is a spliced ​​image spliced ​​by the method of the present invention; Figure 6 This is a comparison chart of the stitching effects of 774 thermal infrared images of a drone collected at a flight altitude of 20m in 2024 using different stitching methods in an embodiment of the present invention; Figure 6 (a) is the stitching diagram stitched by MegaStitch, Figure 6 (b) is a spliced ​​image spliced ​​by the method of the present invention; Figure 7 is the RMSE value of the evaluation index in different years in the embodiment of the present invention; Figure 7 (a) is the RMSE value of the evaluation index in 2018, Figure 7 (b) is the RMSE value of the evaluation index in 2019. Figure 7 (c) is the RMSE value of the evaluation index in 2024; Figure 8 This is an IDD comparison performed on data captured at different heights in an embodiment of the present invention; Figure 8 (a) was taken at an altitude of 120m in 2018. Figure 8 (b) was taken at an altitude of 120m in 2019. Figure 8 (c) was taken at a height of 20m in 2024; Figure 9 Comparison of stitching time of MegaStitch, OP-GSP and the method of the present invention on 45 sets of thermal infrared data sets in the embodiment of the present invention; Figure 9 (a) is the splicing of MegaStitch, Figure 9 (b) is the splicing of OP-GSP, Figure 9 (c) is the splicing of the method of the present invention; Figure 10 This is a visualization analysis diagram of 519 drone farmland remote sensing images taken at an altitude of 120m in 2019 under the constraint of route keeping in an embodiment of the present invention; Figure 10 (a) is the recorded GPS route map, Figure 10 (b) is the MegaStitch alignment result. Figure 10 (c) is the alignment result of the method of the present invention; Figure 11 This is a visualization analysis diagram of 774 drone farmland remote sensing images taken at a height of 20m in 2024 under the constraint of route keeping in an embodiment of the present invention; wherein, Figure 11(a) is the recorded GPS route, Figure 11 (b) is the MegaStitch alignment result. Figure 11 (c) is the alignment result of the method of the present invention; Figure 12 The results of the method of the present invention and MegaStitch in the embodiment of the present invention on the FTE index are compared; wherein, Figure 12 (a) is a comparison chart of the results of the method of the present invention and MegaStitch on the FTE index in 2018. Figure 12 (b) is a comparison chart of the results of the method of the present invention and MegaStitch on the FTE indicator in 2019. Figure 12 (c) is a comparison chart of the results of the method of the present invention and MegaStitch on the FTE indicator in 2024; Figure 13 This is an overall flow chart of a UAV thermal infrared image stitching method based on track keeping according to the present invention. DETAILED DESCRIPTION

[0015] The following is a clear and complete description of the technical solutions of the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0016] Traditional image stitching techniques primarily involve a basic process, including feature extraction, matching, transformation estimation, and image fusion. Early methods, such as AutoStitch proposed by Brown and Lowe, assumed pure camera rotation and used hand-crafted local features combined with RANSAC to estimate a single global homography matrix for stitching.

[0017] As research deepens, scholars are increasingly interested in the problem of image stitching with parallax. DHW was proposed, combining two homography transformations to approximate complex actual transformation models, but it remains insufficient in scenes with large parallax. The SVA method was proposed, which first uses a global affine transformation to maintain the overall image structure and then compensates for the deviation of the local transformation of each grid, significantly improving local alignment accuracy. APAP was proposed, which uses "Moving DLT" to calculate the local homography for each grid, achieving efficient spatial deformation. However, it can easily cause more severe distortion in scenes with large parallax.

[0018] To address this issue, the GSP method was proposed. This method estimates the scale and rotation of each image and then optimizes the mesh using APAP vertex alignment results and local and global similarity constraints, achieving a balance between alignment accuracy and natural visual quality. Furthermore, the researchers optimized the stitching process by introducing geometric structure constraints. By matching coplanar local subregions using projection invariant features, they preserved the local and global structure of wide-parallax images while reducing artifacts and distortion. A structure-preserving image stitching algorithm was proposed, which further enhanced the geometric consistency of the stitching by preserving the extracted straight and curved structures.

[0019] In the field of UAV remote sensing image stitching, early research focused on improving visual quality through image post-processing. A bisector stitching algorithm based on local image geometric features was proposed, generating visually seamless remote sensing panoramic images. Using graph cut energy minimization, optimal stitching lines were successfully detected for multi-view images, enabling the multi-image stitching of agricultural remote sensing images. However, these methods only mitigate misalignment at the visual level and do not substantially improve geometric alignment errors.

[0020] As research deepens, researchers are focusing on improving stitching results by leveraging auxiliary information collected by drones. One approach proposes using IMU data collected by drones during flight to estimate and correct various steps in the stitching process, thereby improving stitching quality. Using geographic coordinate information collected by drones, a spatial arrangement structure-preserving algorithm based on triangular similarity transformations was proposed, effectively maintaining the geometric layout of farmland images.

[0021] Considering the real-time nature of large-scale data, researchers have proposed a variety of acceleration strategies. Distributed feature extraction and parallel computing techniques have significantly improved the effectiveness of real-time video stitching. A fast anchor point matching strategy driven by position and attitude information was proposed, which achieves high-speed stitching by selecting a small number of key anchor points. An adaptive stitching algorithm was constructed using geotagging and local deformation techniques, which increases stitching speed while improving visual quality. MegaStitch was proposed, which uses linear least squares with similarity transformation constraints to transform stitching into a global minimization of reprojection error, achieving rapid stitching of large-scale drone image collections.

[0022] However, most of the aforementioned methods target visible light or limited stitching scenarios, and research on thermal infrared image stitching is relatively scarce. This paper uses the latitude and longitude data from the flight to calculate the image overlap ratio, which serves as an alignment weight and introduces spatial position constraints. A novel thermal infrared image stitching method based on three-stage homography matrix correction and global error optimization is proposed, achieving high-precision stitching and positioning at the millimeter level.

[0023] However, thermal infrared remote sensing often suffers from factors such as wind speed, temperature drift, and low geometric resolution, which can lead to increased feature extraction and matching errors, thus affecting the quality of the stitching. Therefore, how to achieve both high precision and real-time performance in large-scale thermal infrared remote sensing image stitching remains a key issue in this field.

[0024] How to significantly improve the stitching speed while maintaining a certain stitching quality has become a key technical bottleneck that needs to be broken through in the current thermal infrared remote sensing image processing of drones.

[0025] To address these challenges, this paper proposes a fast UAV thermal infrared image stitching method based on feature alignment and track preservation under similarity transformation constraints. Building on the similarity transformation assumption of MegaStitch, the UAV GPS route prior is used to convert geographic coordinate information into pixel translation estimates. A route trajectory preservation term (FTP) is constructed, imposing additional constraints on the translation component of each thermal infrared image to accurately preserve its relative position in physical space. Finally, the present invention combines the reprojection error and the FTP into a global linear least-squares problem with similarity transformation constraints. This minimizes the cumulative error of the stitching process while ensuring global structural integrity, achieving the generation of high-quality panoramic images. Experimental results on a real thermal infrared dataset demonstrate that this method outperforms several state-of-the-art stitching methods in both objective and subjective evaluations.

[0026] The contributions of this work include the following: The present invention designs a route trajectory protection item based on geographic coordinates. By utilizing the route trajectory and GPS information of the UAV during flight, the displacement of the image in physical space is effectively constrained, ensuring that the stitching result faithfully reflects the actual flight trajectory as a whole and maintains the spatial position invariance of the sequence to be stitched.

[0027] The present invention proposes a stitching method FTP-Stitch based on the joint optimization of feature alignment and route trajectory protection. This method constrains the transformation space to be a similarity transformation and provides a good initial solution by introducing a route trajectory protection term, thereby greatly reducing the parameter complexity and achieving fast and robust global alignment.

[0028] Extensive experiments on a thermal infrared image dataset covering various crop types, different flight altitudes and environmental conditions show that the proposed method improves the overall stitching efficiency by about 80% compared with representative thermal infrared stitching methods while ensuring high-quality stitching.

[0029] The following describes the MegaStitch stitching method based on similarity transformation alignment and its limitations.

[0030] MegaStitch is a fast stitching method designed for large-scale image stitching. Its core idea is to model image stitching as a global minimization reprojection error problem with similarity transformation constraints. All images are globally aligned through a linear least squares optimization strategy, effectively avoiding the error accumulation problem that is prone to occur in traditional incremental methods.

[0031] Specifically, suppose the image The transformation matrix of the coordinate system in the reference image is , the authors proposed to limit the transformation between images to scale, rotation and translation, that is, to adopt the similarity transformation model. Specifically, the transformation matrix It can be described as:

[0032] (1); Further, assuming Represents the rth row of the matrix, and the homogeneous coordinates of any internal feature point in image Ii are , and for the image and The corresponding interior point pair Ideally, the point pass The result of mapping to the reference coordinate system should be the same as the point pass The results after mapping are close, that is:

[0033] (2); This provides a constraint on each row transformation for each corresponding pair of interior points, forming the following two equations: (3); Under this constraint, the transformation matrix is ​​reduced from eight degrees of freedom to four, making the linear least squares solution process more efficient. By solving the above equations, the absolute transformation matrix of all images can be solved at once. , thereby aligning each image to a unified reference coordinate system. Thanks to its global linear solver and simplified parameterization, MegaStitch offers significant speed advantages when processing large-scale image data, effectively suppressing global drift and local misalignment, and demonstrating excellent stability and robustness.

[0034] However, rapid decision-making is crucial for agricultural applications, especially in scenarios such as precision irrigation and disease control that require immediate feedback. In practical applications, especially when dealing with drone thermal infrared images with low texture, strong noise, or significant thermal drift, MegaStitch's traditional feature point matching methods are prone to matching failures, affecting global optimization accuracy and leading to significant deformation or structural distortion in the overall stitching result. Furthermore, to ensure stitching speed, MegaStitch only aligns the images without fusing them, resulting in noticeable seams in the final panoramic image.

[0035] like Figure 13 As shown, in this embodiment, a method for stitching thermal infrared images of a UAV based on track keeping includes the following steps: Step S1: Collect multiple thermal infrared images of drones and construct an image set.

[0036] Step S2: Obtain the feature points that match any pair of thermal infrared images to be stitched in the image set, and obtain the similarity transformation matrix between each thermal infrared image and the reference coordinate system, and generate alignment items through the reprojection error between the similarity transformation matrix of the thermal infrared images to be stitched and the matching feature points.

[0037] Based on the similarity transformation feature alignment of MegaStitch, the proposed route trajectory protection term is added to construct a fast stitching model based on the joint optimization of feature alignment and route trajectory preservation. ,in is the reference image, each thermal infrared image The similarity transformation between the reference coordinate system is given by the matrix Description, the final optimization object is the transformation matrix of each image Considering that agricultural UAVs usually maintain a fixed altitude and quasi-orthophoto imaging mode, the constraint transformation space is a similarity transformation, that is, only scaling, rotation and translation are considered. The specific transformation matrix can be described as Equation (1).

[0038] Next, for any pair of thermal infrared images to be stitched in the image set and , and They are the feature points they match, and the alignment term (Feature Alignment Term) is defined by the reprojection error. The difference between the similarity transformation matrix of the thermal infrared image to be stitched and the product of each feature point is taken, and the absolute value is squared. That is, the alignment term is generated by the reprojection error between the similarity transformation matrix of the thermal infrared image to be stitched and the matched feature points. The specific expression is: (8); in, For alignment items, and For any pair of thermal infrared images to be stitched and The matched feature points, Thermal infrared images to be stitched The similarity transformation matrix of Thermal infrared images to be stitched The similarity transformation matrix of .

[0039] Step S3: Calculate the actual distance between any thermal infrared image and the center of the reference image, convert the actual distance, obtain the pixel distance between any thermal infrared image and the center of the reference image in the x and y directions, and generate the route trajectory protection item based on the pixel distance.

[0040] The GPS system onboard drones accurately records the flight trajectory and the geographic coordinates of each frame, providing a reliable spatial prior for the stitching process. This effectively suppresses cumulative errors in large-scale image stitching and preserves the actual spatial relationships during drone flight. Based on this, a route trajectory preservation method based on geographic coordinates is proposed. This method utilizes the geographic coordinate information of the drone during flight to maintain the relative positional relationships of the images in real physical space.

[0041] For scenes captured by large-scale drones, the shooting location of each image is regarded as a sampling point, and all sampling points are mapped to a unified reference canvas. Figure 1 (a) shows the spatial layout structure that requires constraints during the stitching process. The black dots represent the actual sampling locations of all images obtained from drone aerial photography, the blue dots represent the reference image locations, and the red arrows show the actual flight path of the drone. Specifically, the geometric center of the captured area is first determined based on the longitude and latitude information, and the corresponding image is used as a reference. Subsequently, the remaining sampling points are sequentially connected to the reference point, and their geographic distance is used to constrain the center position of the deformed image. This allows the drone's GPS route to be used as a global spatial layout constraint, ensuring that the stitching result better reflects the actual flight path.

[0042] In order to further improve the alignment accuracy, a route trajectory protection item based on geographic coordinates is introduced. Figure 1 As shown in (b), any sampling point With the central sampling point Connected, the horizontal and vertical pixel distances in the stitching result It can be expressed in latitude and longitude. The specific steps are as follows:

[0043] First, the actual distance between the center of any image i and the center of the reference image is calculated by using the Haversine formula, and its actual distance in the x and y directions is: , They are given by the following formulas: (4); (5); in , , , are the latitude and longitude of image i and the reference image, respectively, and R is the radius of the Earth. By combining the drone's flight altitude h, focal length f, and pixel size s, the actual distance is further converted to pixel distance, thus obtaining the pixel distance between any image i and the reference image in the x and y directions:

[0044] (6); Finally, let is the coordinate of the center pixel of image i on the original image. For each image in the set of all images to be stitched (excluding the reference image), in order to ensure the spatial layout structure of the image sequence represented by the sampling points as much as possible, it is encouraged that the pixel distance between the center point of each image and the reference image after the homography matrix transformation is close to the result calculated by GPS, and the final flight trajectory preservation term (FPT) is obtained. That is, obtain the similarity transformation matrix of the reference image; subtract the similarity transformation matrix of the thermal infrared image to be stitched from the similarity transformation matrix of the reference image, multiply the difference by the coordinates of the center pixel of the thermal infrared image to be stitched on the original image, and subtract the product from the pixel distance, take the absolute value and then square it to generate the route trajectory protection term. The specific expression is:

[0045] (7); in, is the route trajectory protection item, is the similarity transformation matrix of the reference image, is the coordinate of the center pixel of image i on the original image, The pixel distance between the arbitrary image and the reference image in the x and y directions.

[0046] Step S4: A global optimization objective function is constructed by weighted sum superposition of the alignment term and the route trajectory protection term, and the global optimization objective function is converted into a linear least squares model with similarity transformation constraints for solution to generate a spliced ​​thermal infrared panoramic image.

[0047] The global optimization objective function is constructed by the weighted sum of the alignment term and the route trajectory protection term. The specific expression is: (9); in, is the global optimization objective function, G is the generated matching pair set, and Represent the weight coefficients of the reprojection error term and the route trajectory protection term, respectively. In all experiments , .

[0048] Arrange the parameters of each similarity transformation matrix into vectors in order, and restore the similarity transformation matrix with the optimal vector, specifically: By parameters The quadratic function of Route trajectory protection items That is, the alignment term and the trajectory protection term in the constructed global objective optimization function are expressed as quadratic functions of similarity transformation parameters, and Equation (9) is reduced to a large-scale linear least squares problem with similarity constraints.

[0049] Each thermal infrared image to be stitched Similarity transformation matrix The parameters are arranged in order as a vector , and linearize all matching pairs and route constraints into a sparse matrix form; where the matching pairs are the matched pairs of thermal infrared images to be stitched, and the route constraints are the trajectory constraints through the route trajectory protection term; specifically expressed as: (10); in, is the overall constraint matrix, which is composed of the Jacobian matrices corresponding to the feature point matching items and the track keeping items. is the target value vector, which represents the observation residual consisting of the expected matching error and trajectory error.

[0050] The sparse matrix form is solved by a linear least squares solver with similarity transformation structure constraints to obtain the optimal vector , and through the optimal vector The similarity transformation matrix is ​​restored, and a spliced ​​thermal infrared panoramic image is generated using the restored similarity transformation matrix.

[0051] While maintaining the inherent advantages of MegaStitch's fast global alignment, this joint optimization method further suppresses cumulative errors and improves structural consistency by introducing geo-route trajectory preservation terms and similarity transformation constraints. This ensures that the stitched images are both highly accurate in geometric alignment and consistent in geographic space. It is suitable for stitching large-scale, low-texture thermal infrared UAV images.

[0052] Experimental verification: The thermal infrared remote sensing dataset of UAV covers a variety of crop types, meteorological conditions, collection time periods and flight altitudes, mainly including Figure 2 There are two parts on display: the Inner Mongolia farmland dataset and the Xinjiang jujube tree dataset.

[0053] The Inner Mongolia farmland dataset used was publicly released by Cui et al. in their 2021 study. The dataset was collected in the Jiefangzha irrigation area of ​​the Shahaoqu irrigation district, located in the western Hetao Irrigation District, Inner Mongolia Autonomous Region, China. The geographic coordinates range from 40°52′ to 41°00′N, 107°05′ to 107°10′E, with an average elevation of 1050 m. This region has a temperate continental monsoon climate, with an average annual temperature of 6.7°C and an average annual precipitation of 150–250 mm. The main crops are wheat, corn, and sunflower.

[0054] The data covers multiple growth stages of farmland crops (seedling, bud, and maturity) from 2018 to 2019. Data was collected in the morning, midday, and afternoon, as well as under sunny, cloudy, and light rainy weather conditions, to assess the impact of different light exposure times on the splicing results.

[0055] The study area consists of four farmland plots (Fields 1-4), each ranging in size from 150,000 to 250,000 m². The salt content of the four plots increases gradually, with Field 1 having the lowest salt content and Field 4 having the highest. Due to the characteristics of thermal infrared images, the color and texture of the images within each area are highly similar, which poses a challenge for image stitching. Furthermore, thermal infrared images are prone to temperature drift, resulting in significant exposure differences between adjacent images (for example, the following image shows a different exposure). Figure 3 ), which further significantly increases the difficulty of image stitching.

[0056] The data collection equipment used was a DJI M600 multi-rotor drone equipped with a Zenmuse XT thermal infrared camera system with a resolution of 640×512. The drone was flown at an altitude of 120 meters. Thermal infrared remote sensing images were acquired for each study area along a pre-planned flight path, with an 80% forward overlap and a 70% lateral overlap to ensure comprehensive image coverage and sufficient redundancy for subsequent image stitching.

[0057] Xinjiang jujube tree dataset: The Xinjiang jujube tree dataset was collected in Ruoqiang County, Bayingolin Mongol Autonomous Prefecture, Xinjiang Uyghur Autonomous Region, China. The geographic coordinate range is 39°00′ to 39°15′ North Latitude and 88°05′ to 88°20′ East Longitude, with an average elevation of approximately 900 meters. The region has an arid desert climate, with annual precipitation less than 50 mm, high evaporation, and a significant diurnal temperature swing.

[0058] This dataset was collected using a DJI M300 drone equipped with a Zenmuse H20T thermal infrared camera system. To capture more detailed information about the date orchard structure, the drone's flight altitude was lowered to 20 meters. Compared to the farmland dataset, the date orchard dataset presented additional challenges in image stitching: the lower flight altitude and the height of the fruit trees introduced more severe parallax effects. Furthermore, the irregularities and repetitive textures of the tree canopy structure reduced the stability of feature extraction and matching, significantly increasing the difficulty of image alignment and stitching. The specific data collection details are shown in Table 1.

[0059] Table 1 Environmental conditions and key imaging parameters of UAV farmland remote sensing image data Experimental environment and model selection: In this experiment, all tests were performed on a computing server equipped with an Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz (40 cores), four GeForce RTX 2080 Ti GPUs, 128GB of memory, and 4TB of SSD storage, and running the Ubuntu 16.04.7 LTS operating system.

[0060] The proposed stitching method was implemented in Python 3.8. During the feature extraction phase, the SURF+RANSAC algorithm was used to ensure fast and robust feature detection and matching, even in the low-texture conditions of thermal infrared images. During the algorithm solution phase, the lsq_linear algorithm from the Scipy optimization library was used to solve the linear least-squares problem, efficiently obtaining the global transformation matrix for each image. Finally, during the image fusion phase, stitching boundaries were located using an optimal seam detection algorithm, and Poisson fusion was used to smooth the seam regions, significantly improving the naturalness and visual consistency of the final stitched image. To efficiently process large-scale drone imagery datasets, the entire process was accelerated using a parallel computing framework, ensuring both stitching quality and strong real-time processing capabilities.

[0061] Comparison with representative methods: The proposed stitching method was compared with the general remote sensing stitching software Pix4D, PhotoScan and the representative image stitching algorithms AutoStitch, OP-GSP and MegaStitch on 45 different sets of UAV remote sensing image data, and the stitching quality was evaluated from three aspects: subjective evaluation, objective evaluation and stitching time.

[0062] Subjective evaluation: Subjective evaluation is based on direct human visual perception and comprehensively judges the stitching results. Evaluation indicators mainly include: whether the image alignment is accurate, whether the stitching result is natural, and whether the overall structure is intact.

[0063] Figure 4 The results of stitching thermal infrared images taken by a drone at an altitude of 120 meters in 2018 are shown. The results show that the mainstream commercial drone stitching software Pix4D and PhotoScan ( Figure 4 (a) and Figure 4 (b)), due to the low texture characteristics of thermal infrared images, it is difficult to extract effective features, resulting in severe distortion and failure to generate a complete panoramic image. The classic stitching algorithm AutoStitch ( Figure 4 (c)) also shows obvious deformation. MegaStitch( Figure 4 (d) In large-scale image stitching, the accumulated error leads to trapezoidal deformation and no fusion is performed. The seams are obvious and the structural integrity is damaged. As the most advanced thermal infrared image stitching algorithm, OP-GSP ( Figure 4 (e)) achieves a good stitching result overall, but some alignment failures and distortions occur on the local ridge lines.

[0064] In contrast, the method ( Figure 4 (f)) introduces route trajectory preservation while minimizing the global reprojection error, effectively speeding up the stitching speed while maintaining the overall route of the drone, suppressing the cumulative error, and making the stitching result perform best in terms of geometric alignment, structural consistency, and visual naturalness. In addition, compared with the fusion method used by OP-GSP, the method of

[15] can well preserve the temperature information in the thermal infrared image by using the stitching line. For example, Figure 4 The obvious light and dark changes on the right side of (f) correspond to the low temperature information in the shadow area formed under sunlight, which is also well preserved.

[0065] then Figure 5 The results of stitching thermal infrared images taken by a drone at an altitude of 120m in 2019 were verified. The results showed that the commercial stitching software Pix4D failed to stitch on this dataset; PTGui ( Figure 5 (a)) performs well on this set of thermal infrared images with obvious features, but there are still local distortions and some key images are missing; AutoStitch ( Figure 5 (b)) is not ideal for processing large-scale images, and a large degree of distortion occurs; MegaStitch ( Figure 5(c)) is similar to the results in 2018, with cumulative errors and obvious seams in the splicing results; OP-GSP ( Figure 5 Although the overall splicing effect of (d) is good, deformation still occurs in local areas and the overall structure of the ridge cannot be maintained. In contrast, the method proposed in the present invention ( Figure 5 (e)) effectively overcomes the above problems while ensuring high alignment accuracy, and the stitching results perform best in terms of overall structure, detail preservation, and visual naturalness.

[0066] Finally, to verify the algorithm's ability to cope with flight altitude and parallax challenges, the present invention lowered the flight altitude to 20 meters and collected data on Xinjiang jujube trees. A lower flight altitude means more details, but the complex crown features also increase the difficulty of matching. At the same time, parallax between the ground and the fruit trees is unavoidable. Under this challenge, Figure 6 This paper presents the stitching results of thermal infrared images of Xinjiang's jujube trees, taken at a flight altitude of 20 meters in 2024. Mainstream commercial software Pix4D and PhotoScan faced severe challenges, resulting in stitching failures; even the most advanced thermal infrared stitching algorithm, OP-GSP, failed to successfully complete the stitching. Although MegaStitch achieved stitching results, the lack of flight path constraints prevented it from maintaining the overall structure during the large-scale and complex stitching process, and significant errors in local alignment occurred. In contrast, the algorithm proposed in this paper effectively maintained the overall structure of the acquisition area and achieved excellent alignment of both the jujube trees and the ground plane.

[0067] The results show that the FTP-Stitch method proposed in this paper outperforms the comparative methods in terms of alignment accuracy, local alignment and overall naturalness. Especially when processing images with large temperature drift and exposure differences, it can effectively eliminate ghosting and obvious distortion, thereby generating panoramic images with better visual effects.

[0068] Objective evaluation: Because most commercial software (such as Pix4D) is not open source, objective evaluation of them is impossible. Therefore, this evaluation focuses solely on comparing representative stitching methods. This paper uses the root mean square error (RMSE) and image distortion degree (IDD) as objective evaluation metrics to measure the performance of different stitching methods.

[0069] To quantitatively evaluate the alignment accuracy of stitched panoramic images, this paper uses the widely used metric, Root Mean Square Error (RMSE). RMSE calculates the root mean square of the Euclidean distance between feature points in the transformed image, providing a quantitative measure of the deviation between feature points. This deviation reflects the distance between the feature points in the transformed image and their true location, thus reflecting the quality of the image alignment. The calculation formula is generally expressed as:

[0070] (12); Where G is the set of image matching pairs, , are image matching pairs in G feature points, f represents the image transformation model of the thermal infrared image to be stitched, and N is the number of matched feature points. A lower RMSE value indicates a higher image alignment accuracy. In the experiments of this paper, RMSE was used as an objective evaluation metric to compare the alignment performance of different stitching methods.

[0071] To evaluate the alignment accuracy of the proposed method, the present invention calculated the RMSE on data collected at a flight altitude of 120m in 2018 and 2019, and at a flight altitude of 20m in 2024, respectively. Figure 8 (a) and Figure 8 (b) shows the comparison of the RMSE index between the methods proposed in 2018 and 2019 and the representative splicing algorithms, respectively. Figure 8 (c) shows the evaluation results at a flight altitude of 20m in 2024. It is worth noting that since OPGSP and MegaStitch failed to stitch together some datasets, the effective RMSE cannot be calculated, so the corresponding results are presented in a blank form.

[0072] from Figure 7 (a) and Figure 7It can be seen from the results of (b) that at a flight altitude of 120m, the method of the present invention is significantly better than MegaStitch in terms of RMSE indicators, showing higher alignment accuracy. However, compared with OP-GSP, the RMSE results of the present invention are generally better than OP-GSP, and OP-GSP has a slight advantage in some data sets. However, OP-GSP has splicing failures in multiple data sets, while the algorithm of the present invention can run stably, further demonstrating the robustness of the algorithm of the present invention. In addition, since the algorithm of the present invention focuses on fast splicing, while ensuring high efficiency, it can approach the performance of OP-GSP in terms of RMSE indicators, further verifying the effectiveness of the method of the present invention. At the relatively difficult flight altitude of 20m, OP-GSP all experienced splicing failures. The method of the present invention is significantly better than MegaStitch in terms of RMSE indicators, which also proves the splicing performance of the algorithm of the present invention in complex environments.

[0073] In addition to the RMSE evaluation metric, given that agricultural drones typically perform quasi-orthophoto imaging at a fixed altitude, the stitching process primarily involves similarity transformations (scaling, rotation, and translation) without significant projective distortion. Therefore, this paper uses Image Distortion Degree (IDD) as an objective metric for evaluating the naturalness of the stitching results. IDD is primarily based on the uniformity of the area distribution on the reference plane after image transformation. It measures the naturalness of the stitching result by calculating the root mean square (RMS) of the area difference (ADI) after the image is deformed. Specifically, the IDD evaluation metric is defined as follows:

[0074] (13); in represents the area of ​​the i-th deformed image, and It represents the average area of ​​all transformed images. A more uniform area distribution represents a more natural stitching result. Therefore, the smaller the IDD value, the more natural the image.

[0075] the following Figure 8 IDD comparisons are presented for data captured at 120m altitude in 2018 and 2019, and for data captured at 20m altitude in 2024. Since MegaStitch failed to stitch together some datasets, valid RMSE could not be calculated, and therefore the corresponding results are presented as blank. It can be seen that the IDD of our method is much smaller than that of MegaStitch when stitching remote sensing data captured at both high and low altitudes, demonstrating that our algorithm maintains better overall naturalness and produces better stitching results.

[0076] Comparison of splicing efficiency: In addition to subjective and objective stitching quality evaluation, stitching efficiency is also a key dimension to measure algorithm performance. Figure 9 The article presents a comparison of stitching times for MegaStitch, OP-GSP, and the method of the present invention on 45 thermal infrared datasets. Experimental results show that, under the same hardware environment, the method of the present invention significantly outperforms OP-GSP in stitching speed, reducing the overall stitching time by an average of approximately 80%, significantly shortening the entire stitching process, and achieving real-time processing capabilities. Compared to MegaStitch without image fusion, the method of the present invention takes slightly longer overall. However, it is precisely because of the integration of post-processing such as optimal seam detection and Poisson fusion that better stitching quality can be achieved at a slightly increased time cost.

[0077] Analysis of flight trajectory keeping capability: The effectiveness of this method is evaluated from two aspects: first, a visual comparison of the UAV flight trajectory before and after stitching is performed; second, a new route keeping evaluation metric is proposed to quantify the degree of deviation of the stitching result compared to the GPS route.

[0078] To verify the effectiveness of the route keeping constraint in improving the stitching quality, the present invention superimposes the registration center (red dot) of the stitched image on the alignment result and compares it with the original GPS trajectory of the UAV flight. Figure 10 Take the farmland dataset taken at an altitude of 120 m in Inner Mongolia as an example. Figure 10 (a) is the GPS track recorded by the actual flight of the UAV. Figure 10 (b) is the MegaStitch alignment result without adding the route preservation constraint. Its registration center shows obvious offset and distortion compared with the original route, resulting in the overall tilt and misalignment of the image set. Figure 10 (c) shows the alignment result of the method of the present invention. The registration center of each image is consistent with the original route, the ridge lines are straight and smooth, the overall structure is complete and visually natural.

[0079] Likewise, Figure 11 Comparison results from a dataset of jujube trees at a height of 20 meters in Xinjiang were presented, and consistent conclusions were reached after verification. Due to the large parallax and sparse textures in this scene, the stitching process is more prone to local misalignment. However, after introducing the route trajectory protection factor, the registration center of each image is aligned with the original GPS route, and the image alignment remains stable, demonstrating the method's adaptability and robustness even in complex environments.

[0080] The above two groups of comparisons intuitively prove that the introduction of route trajectory protection constraints can effectively suppress cumulative drift under various environmental conditions and significantly improve the geographic spatial consistency and visual naturalness of the stitching results.

[0081] In order to quantitatively evaluate the route keeping capability, this paper proposes a route keeping evaluation metric, Flight Trajectory Error (FTE), which is used to measure the deviation of the splicing result from the original GPS route. It is calculated as follows:

[0082] (14); in, represents the Euclidean distance of the i-th stitched image compared to the reference image calculated based on GPS, and =(\frac{\text{distance}}) represents the Euclidean distance between the i-th stitched image and the reference image in the stitched result. The root mean square error (RMSE) between them is used to measure the deviation of the stitched result from the original flight path. A lower flight path error indicates a higher degree of alignment between the stitched image and the actual UAV flight path, demonstrating that the stitched result better spatially preserves the original flight path.

[0083] Figure 12 The results of the FTE indicator comparison between the method of the present invention and MegaStitch are presented. Experimental results show that MegaStitch, which does not add the route keeping constraint, has a larger FTE value for the stitched image, indicating that the stitching result deviates from the true flight trajectory of the drone under the condition of large-scale reprojection error accumulation. After the route trajectory protection term is introduced, the error is significantly reduced, and the stitching result is more accurate in the geographic space. Further analysis shows that the route keeping constraint proposed in the present invention provides additional global geometric information, so that the stitching process not only relies on local matching features, but also can maintain the spatial consistency of the image over a large range, thereby reducing the impact of cumulative errors.

[0084] The subjective and objective evaluations conducted on thermal infrared datasets covering different crop types, flight altitudes (120m, 20m) and environmental conditions showed that the method of the present invention not only exhibited high geometric alignment accuracy and naturalness in stitching quality, effectively maintaining but also significantly outperformed existing advanced methods in stitching efficiency, demonstrating its practical application potential in large-scale agricultural monitoring.

[0085] The present invention proposes a UAV thermal infrared image stitching system based on track keeping, which includes: an acquisition module, an alignment item generation module, a protection item generation module and a stitching module.

[0086] Among them, the acquisition module is used to collect multiple thermal infrared images of drones and construct an image set; the alignment item generation module is used to obtain the matching feature points of any pair of thermal infrared images to be stitched in the image set, and obtain the similarity transformation matrix of each thermal infrared image and the reference coordinate system, and generate the alignment item through the reprojection error of the similarity transformation matrix of the thermal infrared image to be stitched and the matching feature points; the protection item generation module is used to calculate the actual distance between any thermal infrared image and the center of the reference image, and convert the actual distance to obtain the pixel distance between any thermal infrared image and the center of the reference image in the x and y directions, and generate the route trajectory protection item through the pixel distance; the splicing module is used to construct a global optimization objective function by weighted sum superposition of the alignment item and the route trajectory protection item, and convert the global optimization objective function into a linear least squares model constrained by similarity transformation for solution to generate a spliced ​​thermal infrared panoramic image.

[0087] The present invention also provides a computer device, including a memory and a processor. The memory stores a program. When the program is executed by the processor, the processor executes the steps of a method for splicing thermal infrared images of a drone based on track keeping.

[0088] According to the disclosed embodiments, a computing device may communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth communications, etc.), or with any device that enables a computing device to communicate with one or more other computing devices (e.g., routers, modems, etc.).

[0089] The above content is a further detailed description of the present invention in combination with specific preferred embodiments. For those skilled in the art to which the present invention belongs, several simple deductions or replacements can be made without departing from the concept of the present invention, which should be regarded as falling within the scope of protection of the present invention.

Claims

1. A method for stitching UAV thermal infrared images based on track keeping, characterized in that: include: Collect thermal infrared images from multiple drones and build an image collection; Obtain the matching feature points of any pair of thermal infrared images to be stitched in the image set, and obtain the similarity transformation matrix between each thermal infrared image and the reference coordinate system. Generate an alignment term based on the reprojection error between the similarity transformation matrix of the thermal infrared images to be stitched and the matching feature points. Calculate the actual distance between the center of any thermal infrared image and the reference image, convert the actual distance to obtain the pixel distance between the center of any thermal infrared image and the reference image in the x and y directions, and generate the route trajectory protection item based on the pixel distance; A global optimization objective function is constructed by weighted sum superposition of the alignment term and the route trajectory protection term, and the global optimization objective function is converted into a linear least squares model with similarity transformation constraints for solution to generate a spliced ​​thermal infrared panoramic image.

2. The method for stitching thermal infrared images of a UAV based on track keeping according to claim 1, characterized in that: The alignment term is generated by the similarity transformation matrix of the thermal infrared image to be stitched and the reprojection error of the matched feature points, specifically: Based on the feature points matched to any pair of thermal infrared images to be stitched, the similarity transformation matrix of the thermal infrared images to be stitched is subtracted from the product of their respective feature points, and the absolute value is taken and then squared to generate the alignment item of the thermal infrared images to be stitched.

3. The method for stitching UAV thermal infrared images based on track keeping according to claim 1, characterized in that: The generating of the route trajectory protection item by the pixel distance is specifically: Get the similarity transformation matrix of the reference image; The similarity transformation matrix of the thermal infrared image to be stitched is subtracted from the similarity transformation matrix of the reference image, and the difference is multiplied by the coordinates of the center pixel of the thermal infrared image to be stitched on the original image. The product is subtracted from the pixel distance, and the absolute value is taken and then squared to generate the route trajectory protection item.

4. The method for stitching UAV thermal infrared images based on track keeping according to claim 1, characterized in that: The global optimization objective function is converted into a linear least squares model with similarity transformation constraints for solving the problem to generate a spliced ​​thermal infrared panoramic image, specifically: The similarity transformation matrix parameters of each thermal infrared image to be stitched are arranged in sequence as a vector, and all matching pairs and route constraints are linearized into a sparse matrix form; where the matching pairs are the matched pairs of thermal infrared images to be stitched, and the route constraints are the trajectory constraints through the route trajectory protection term; The sparse matrix form is solved by a linear least squares solver with similarity transformation structure constraints to obtain the optimal vector, and the similarity transformation matrix is ​​restored through the optimal vector to generate a spliced ​​thermal infrared panoramic image with the restored similarity transformation matrix.

5. A UAV thermal infrared image stitching system based on track keeping, characterized in that: include: The acquisition module is used to collect thermal infrared images of multiple drones and build an image collection; An alignment term generation module is used to obtain the matching feature points of any pair of thermal infrared images to be stitched in the image set, and obtain the similarity transformation matrix between each thermal infrared image and the reference coordinate system, and generate an alignment term based on the reprojection error between the similarity transformation matrix of the thermal infrared images to be stitched and the matching feature points; A protection item generation module is used to calculate the actual distance between the center of any thermal infrared image and the reference image, convert the actual distance to obtain the pixel distance between the center of any thermal infrared image and the reference image in the x and y directions, and generate the route trajectory protection item based on the pixel distance; The stitching module is used to construct a global optimization objective function by weighted sum superposition of the alignment term and the route trajectory protection term, and convert the global optimization objective function into a linear least squares model constrained by similarity transformation for solution to generate a stitched thermal infrared panoramic image.

6. A computer device, characterized in that: The invention comprises a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of a method for stitching thermal infrared images of a drone based on track keeping as described in any one of claims 1 to 4.

7. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a method for stitching thermal infrared images of a drone based on track keeping according to any one of claims 1 to 4 are implemented.

Images