A method for stitching UAV remote sensing images based on spatial layout protection

Abstract

This invention provides a method for stitching UAV remote sensing images based on spatial layout protection, belonging to the field of image processing technology. The method includes the following steps: acquiring several UAV remote sensing images; extracting the two-dimensional spatial layout represented by the latitude and longitude coordinates of each UAV remote sensing image capture point; performing triangle sampling on the two-dimensional spatial layout; protecting the spatial layout structure by constraining the similarity changes of the triangles; adding the protection term of the spatial layout structure as a new energy term to the GSP model; constructing an SAP-GSP stitching model; and inputting the UAV remote sensing images to be stitched into the SAP-GSP stitching model for stitching to obtain a panoramic image with spatial layout protection. This invention uses the SAP-GSP model to ensure that the relative spatial positions of all images remain unchanged after stitching, guaranteeing the overall naturalness of the stitching result and thus obtaining a panoramic image with better overall integrity.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for stitching UAV remote sensing images based on spatial layout protection. Background Technology

[0002] In recent years, drones have been widely used in many fields, such as environmental monitoring, military mapping, aerospace, and smart agriculture, due to their small size, low cost, and ease of use. However, drones can only fly at low altitudes, resulting in images with typically limited fields of view. In the field of smart agriculture, image stitching can be used to synthesize panoramic images with a wider field of view from multiple images. Combined with ground data, this allows for the monitoring of crop planting area, crop growth, and yield estimates, providing a basis for guiding agricultural production.

[0003] Although UAV-based remote sensing image stitching of farmland has wide applications in agricultural production, current research is still relatively limited. Most work mainly involves directly applying existing image stitching algorithms to UAV-based farmland remote sensing images. When the UAV flies at a high altitude, the image mainly shows the outline and structure of the farmland with less detail, and these algorithms can achieve good stitching results. However, when the UAV flies at a low altitude, the details and features of the farmland in the image become more complex and diverse, and the number of images increases exponentially, making these methods prone to stitching failures.

[0004] To overcome the poor stitching results caused by relying solely on local matching, the mesh optimization-based method GSP explores the advantages of local and global similarity transformations to achieve more natural stitching results. On the other hand, based on the GSP model, OP-GSP utilizes POS information from the UAV's flight process, using the estimated overlap of adjacent images as weights for local matching pairs to optimize the local alignment term of GSP and improve stitching results; GES-GSP obtains high-quality stitching results by preserving different types of geometric structures extracted from the images. Due to the high similarity in color and texture features of farmland images, feature extraction and matching are difficult. Therefore, when processing UAV-generated farmland remote sensing images, these methods are prone to local deformation, poor visual effects, distortion, warping, and even stitching failure. Summary of the Invention

[0005] This invention provides a UAV remote sensing image stitching method based on spatial layout protection, aiming to solve the problems of poor visual effects and severe distortion in the image stitching process of existing technologies.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for stitching UAV remote sensing images based on spatial layout protection, comprising the following steps:

[0007] Acquire several remote sensing images from drones;

[0008] Extract the latitude and longitude coordinates of the shooting points in each of the UAV remote sensing images, and obtain the two-dimensional spatial layout of the shooting points through the latitude and longitude coordinates;

[0009] Triangle sampling is performed on the two-dimensional spatial layout, and the similarity changes of the constraint triangles in the triangle sampling are used as protection items for the spatial layout structure.

[0010] The protection term of the spatial layout structure is added as a new energy term to the GSP model to construct the SAP-GSP splicing model;

[0011] The UAV remote sensing images to be stitched are input into the SAP-GSP stitching model for stitching to obtain a panoramic image with spatial layout protection.

[0012] Preferably, the step of performing triangle sampling on the two-dimensional spatial layout and using the similarity changes of the constraint triangles in the triangle sampling as a protection item for the spatial layout structure specifically includes the following steps:

[0013] Each drone remote sensing image is used as a sampling point, and triangles are constructed by taking the two farthest sampling points as endpoints and the other sampling points as vertices.

[0014] Select one group of triangles for image stitching, and replace the deformation result of the triangles with the ideal transformation result to obtain the similarity transformation result;

[0015] Obtain the errors between the vertex deformation results and similarity transformation results of all the triangles, and sum all the errors to obtain an optimal vertex set.

[0016] Preferably, the step of using the capture point of each UAV remote sensing image as a sampling point, and constructing triangles with the two farthest capture points as endpoints and the other sampling points as vertices, specifically includes:

[0017] Through any vertex V i With two reference endpoints V a V b Forming a triangle;

[0018] Among them, vertex V i The coordinates are determined by V a V b Specifically, it means:

[0019]

[0020] Among them, u i h i For V i Va and V b Two values ​​were exported.

[0021] Preferably, the h is obtained i u i The specific expression is:

[0022]

[0023]

[0024] Where H(·) and U(·) are respectively:

[0025]

[0026]

[0027] Where H(·) and U(·) represent solving h respectively. i and u i A function of value.

[0028] Preferably, the deformation result of the triangle is replaced with an ideal transformation result to obtain a similar transformation result, including the following steps:

[0029] The triangles are spliced ​​together, and after splicing, V a V b V i Transformed into

[0030] According to the deformed and Combined with h i u i Estimate V i Expected position Make With ΔV a V b V i resemblance;

[0031] The desired location The calculation formula is:

[0032]

[0033] in, Vertex V i With two reference endpoints V a V b The vertices and two endpoints of the deformed structure after splicing.

[0034] Preferably, the desired position The calculation formula is:

[0035]

[0036] Among them, The above formula also holds true when the lines are collinear.

[0037] Preferably, the step of obtaining the error between the vertex deformation results and the similarity transformation results of all the triangles, and summing all the errors, includes the following steps:

[0038] The ΔV a V b V i Deformation results Its similarity transformation result The error between the vertex deformation result and the similarity transformation result is obtained by moving closer together. The specific expression is as follows:

[0039]

[0040] Summing all the aforementioned errors, the specific expression is as follows:

[0041]

[0042] Where N is the number of triangles constructed.

[0043] Preferably, an optimal vertex set is obtained by solving the summation expression of the error, specifically as follows:

[0044]

[0045] The optimal vertex set is: The latitude and longitude coordinates of the i-th image after deformation, where i = 1, 2, ..., n.

[0046] Preferably, the objective function of the SAP-GSP stitching model is:

[0047]

[0048] in, For alignment items, For locally similar terms, For globally similar items, For spatial layout structural protection items, λ l With λ sap These are the coefficients for the local similarity term and the spatial layout structure protection term, respectively.

[0049] Preferably, when the UAV remote sensing image to be stitched is input into the SAP-GSP stitching model for stitching, a set of deformed grid vertex positions are obtained, so that the total energy function contained in the grid vertex is minimized, and a panoramic image with spatial layout protection is obtained.

[0050] Compared with the prior art, the present invention has the following advantages: The present invention performs triangle sampling on the two-dimensional spatial layout represented by the image capture point, and indirectly maintains the spatial layout structure by constraining the triangles to be as similar as possible; then, the proposed spatial layout protection term is added as a new energy term to the GSP stitching model, thereby obtaining the proposed stitching algorithm SAP-GSP, which keeps the relative spatial position of all images unchanged after stitching, ensuring the overall naturalness of the stitching result, and thus obtaining a panoramic image with better overall integrity. Attached Figure Description

[0051] Figure 1 A comparison image showing the stitched results of 89 25m UAV remote sensing images of farmland provided by this invention;

[0052] Figure 2 A two-dimensional spatial layout diagram representing the photo points on the UAV flight path map provided by the present invention;

[0053] Figure 3 The spatial layout diagram of the shooting points based on triangle similarity transformation provided by this invention;

[0054] Figure 4 Location map of the research area provided for this invention;

[0055] Figure 5 Example diagram of wheat lodging provided by the present invention;

[0056] Figure 6 A qualitative evaluation map of 29 sets of drone-based farmland remote sensing data stitched together from 2020 to 2022 at different growth cycles and different drone flight altitudes, provided by this invention;

[0057] Figure 7 The present invention provides 75 drone-generated remote sensing images of farmland taken at an altitude of 25m during the jointing stage of 2021.

[0058] Figure 8 The present invention provides 230 drone-generated remote sensing images of farmland taken at an altitude of 30m during the 2020 sowing season.

[0059] Figure 9 The present invention provides 316 drone-generated remote sensing images of farmland taken at an altitude of 50m during the 2020 sowing season.

[0060] Figure 10 This is a diagram illustrating the image shape transformation during drone-based farmland image stitching provided by the present invention.

[0061] Figure 11 The present invention provides an area RMSE comparison plot between GSP and our method on 29 datasets;

[0062] Figure 12 This is a comparison chart of the splicing results of the 17th set of data GSP and SAP-GSP provided by the present invention;

[0063] Figure 13 Mesh diagrams before and after deformation provided for this invention;

[0064] Figure 14 The present invention provides a comparison chart of RMSE and SD between GSP and SAP-GSP on 29 datasets;

[0065] Figure 15 A comparison diagram showing the global geometry structure assembled using different methods provided in this invention;

[0066] Figure 16 Error analysis diagram of spatial layout protection capability of different splicing algorithms provided by the present invention;

[0067] Figure 17 A comparison diagram of the splicing results of adding lateral matching pairs at different intervals provided by the present invention. Detailed Implementation

[0068] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings. The following examples are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.

[0069] The invention is illustrated by comparing the stitched results of 89 25m remote sensing images of farmland. For example... Figure 1 (a) shows the drone's flight path, illustrating the spatial layout relationships between all initial images. Autostitch requires a fixed projection center for the camera to acquire data, but this requirement is difficult to meet when taking photos with a drone, such as... Figure 1 (b) shows a small area of ​​distortion in the stitched result. Figure 1 (c) shows the stitching result of GSP. One area is clearly misaligned, and the overall structure of the image is severely distorted, resulting in poor visual quality. GES-GSP attempts to preserve the geometric structure of the image, but due to the difficulty of extracting farmland features, such as... Figure 1 (d) shows the overall structural protection failure. OP-GSP solved the alignment problem, but as... Figure 1 The image shown in (e) is severely distorted overall. Figure 1(f) shows the experimental results of this study. Compared with other algorithms, it better preserves the spatial layout of the image and the stitching result is more natural.

[0070] First, before image stitching, this invention describes two aspects: general image stitching and image stitching for UAV remote sensing.

[0071] 1. Image stitching methods for general scenarios:

[0072] Image stitching is the process of combining multiple images into a panoramic image with a wide field of view, typically using warp models to align the images. Early methods used global parametric transformations such as similarity, affine, and projection transformations, with Autostitch being the most representative. It uses invariant local features to find matches between all images, serving as the seminal work for image stitching. However, Autostitch assumes that images are only captured by rotating the camera, limiting the method's usability. Lin et al. introduced smoothly varying affine fields to align images, allowing for locally adaptive adjustments to achieve accurate local alignment. However, the use of affine regularization is suitable for interpolation, which may not be optimal for smooth extrapolation and is insufficient for perspective transformations. Zaragoza et al. proposed an as-projective-as-possible warp model, which improves local alignment by dividing the image into a grid and estimating a set of smooth transformations for each grid. However, because this method attempts to perform a global projective warp, it can lead to severe stretching and uneven magnification of parts of the stitched image. Lou et al. proposed the piecewise planar region matching method to achieve more robust image registration by segmenting and matching planar regions of images. However, this method is unsuitable when scene regions are more complex because non-overlapping regions in the stitching result will be distorted. Therefore, shape-preserving half-projective was proposed, which reduces distortion in non-overlapping regions by combining homography and similarity transformations to stitch images. However, it cannot handle image parallax and fails to achieve good matching results in overlapping regions. Adaptive as-natural-as-possible also uses a global similarity transformation to reduce distortion in non-overlapping regions. This method linearizes homography in regions that do not overlap with any other images and can adaptively determine angles to effectively correct image shapes. However, for some complex scene structures, this method cannot accurately match overlapping regions, resulting in some ghosting in the result. Solving the ghosting problem can be achieved by post-processing the image using optimal stitching methods, but these are often difficult to apply to stitching multiple images due to error accumulation. Xiang et al. argued that these image structures are often destroyed due to insufficient and unreliable point correspondence when stitching low-texture images. Therefore, line features are introduced to calculate local homography, so as to make the matching results more accurate.To better preserve local and global linear structures, LPC introduces global collinear structures to enhance the desired characters for image warping. While it effectively protects linear structures within the image, it neglects curves in the scene. Therefore, Du et al. proposed a stitching model, GES-GSP, to protect the linear and curved structures extracted from the image to obtain high-quality stitching results. However, this method can only protect the geometric structures of overlapping and non-overlapping regions between pairs of images, and cannot protect the geometric structures spanning more images.

[0073] 2. Stitching methods for UAV remote sensing images:

[0074] In recent years, with the continuous development of UAV technology, obtaining high-resolution, wide-field-of-view remote sensing images using image stitching technology has gradually become a research hotspot. However, due to the large amount of image data and complex scenes captured by UAVs, directly applying existing image stitching techniques to the field of remote sensing may lead to problems such as ghosting, cumulative errors, distortion, or even stitching failure. Luo et al. applied the Parallax-Tolerant image stitching method to UAV remote sensing image stitching, effectively solving the ghosting problem caused by large parallax by finding the optimal stitching line in the overlapping area. However, UAV farmland remote sensing images have almost no height difference, and the parallax problem caused by inconsistent object heights is generally not considered. Therefore, the optimal stitching line method does not have an advantage in solving this problem. More and more researchers are focusing on mesh deformation-based methods. GSP obtains more natural stitching results by constraining three different energy terms: alignment, local similarity, and global similarity. However, because it does not consider the lateral adjacency relationship of the images, problems such as misalignment and ghosting occur when stitching UAV farmland remote sensing images. Cui et al. proposed using POS information from UAV flight to calculate the overlap rate between images as the weight of the alignment term, and to calculate the spatial position constraints of the images through the overlap rate, thereby alleviating the misalignment and ghosting problems in GSP. However, due to the excessive spatial position constraints and the fact that the weight of the alignment term is not solely determined by the overlap rate, the results accumulate significant errors.

[0075] Drone-based remote sensing images of farmland are characterized by their flight altitude typically being much higher than the crop height, resulting in almost no height difference. However, high color and texture similarity leads to low reliability in local matching. Although GPS-based stitching models apply similarity constraints to each image, the lack of overall control over the images makes it difficult to guarantee stitching quality. We note that the latitude and longitude coordinates carried by each drone image effectively reflect their spatial layout within the scene. Using this information reflecting the overall scene to guide stitching would clearly be very useful for improving the overall quality of image stitching.

[0076] Second, it is necessary to protect the spatial layout information of images when stitching them together.

[0077] 1. The necessity of protecting image spatial layout information

[0078] like Figure 2 As shown, the drone's flight points can be viewed as a set of sampling points on a two-dimensional plane, reflecting the spatial arrangement of the acquired image sequence within the entire scene. Maintaining this two-dimensional spatial arrangement during the deformation process is highly beneficial for improving the overall quality of the stitching.

[0079] Due to the high similarity in texture and color among farmland images, stitching models based on local feature point matching struggle to produce high-quality panoramic images. Spatial layout information reflects the relative positional relationships of each photographing point. Therefore, if the stitched image can maintain this spatial layout structure, it improves the overall naturalness of the panoramic image and helps suppress local deformation, distortion, or even stitching failure during the image stitching process.

[0080] 2. Spatial representation of the image before and after transformation

[0081] Unlike previous stitching methods that primarily focused on protecting straight lines or curves within an image, this invention protects the spatial layout represented by the latitude and longitude coordinates of the image's capture points—a clear distinction. Without causing confusion, this invention uses V to represent an image or its associated latitude and longitude coordinates. Before stitching, each point in the image sequence is represented using latitude and longitude coordinates. For an image V with a resolution of m×n, its corresponding capture point coordinates are represented as V(longitude, latitude). After image stitching, all images exist in a single coordinate system; image V is warped to a new position. At this point, its corresponding position is represented by an image. The average coordinates of all corresponding pixels in the panoramic image are represented by the following formula:

[0082]

[0083]

[0084] In the formula, Representing images respectively The average x-coordinate and y-coordinate, Representing images respectively The x-coordinate and y-coordinate of the k-th pixel.

[0085] This invention sets different representations of image spatial coordinates before and after transformation, effectively linking the latitude and longitude coordinates of the image capture point with the spatial coordinates of the stitched image.

[0086] The present invention will now be described in detail. The present invention provides a method for stitching UAV remote sensing images based on spatial layout protection, specifically including the following steps:

[0087] S1: Acquire several UAV remote sensing images.

[0088] S2: Extract the latitude and longitude coordinates of the points captured in each UAV remote sensing image, and obtain the two-dimensional spatial layout of the points captured using the latitude and longitude coordinates.

[0089] S3: Perform triangle sampling on the two-dimensional spatial layout, and use the similarity changes of the constrained triangles in the triangle sampling as a protection item for the spatial layout structure to maintain the spatial layout structure.

[0090] S4: Add the protection term of the spatial layout structure as a new energy term to the GSP model to construct the SAP-GSP splicing model.

[0091] S5: Input the UAV remote sensing images to be stitched into the SAP-GSP stitching model for stitching to obtain a panoramic image with spatial layout protection.

[0092] The following sections will provide a detailed explanation of the image stitching method based on spatial layout protection from two aspects: the spatial layout protection module based on triangle similarity transformation and the GSP image stitching model based on spatial layout prior.

[0093] 1. The following is an explanation of step S3:

[0094] Inspired by triangle sampling in geometric structure protection, this invention performs triangle sampling on the two-dimensional spatial layout represented by the image capture points, indirectly preserving the spatial layout structure by constraining the similarity changes of triangles. Specifically, each image capture point is treated as a sampling point, with the two farthest capture points as endpoints and the other sampling points as vertices to construct triangles. In other words, the spatial layout represented by the latitude and longitude coordinates of the drone flight capture points is constrained as a whole.

[0095] like Figure 3 As shown, where Figure 3 (a) For each sampling point V i and reference point V a and V b They all form a set of triangles; Figure 3 (b) uses a set of triangles from (a) as an example; Figure 3 (c) After splicing, the result of the triangle deformation in (b) is: It usually does not satisfy the similarity transformation, while for The ideal transformation result makes

[0096] like Figure 3 As shown, Figure 3 (a) is a spatial layout diagram representing the locations of points captured by a drone. In this diagram, any vertex V... i With two reference endpoints (V) a and V b Each of them forms a triangle. For example... Figure 3 As shown in (b), in V a V b and V i In the triangle formed, V i The coordinates can be obtained from V a V b express:

[0097]

[0098] In the formula, (u i ,h i ) is from V i V a and V b Two derived quantities. The values ​​of this triangle will not change after undergoing a similarity transformation. h i u i The calculation formula is as follows:

[0099]

[0100]

[0101] Where H(·) and U(·) are respectively:

[0102]

[0103]

[0104] like Figure 3As shown in (c), after image stitching, V a V b V i After deformation, it becomes Since the deformations in image stitching are usually dissimilar transformations, ΔV a V b V i and They are usually not similar. However, based on the deformed and Combined with h calculated before deformation i u i Then estimate V i Expected position It makes Still related to ΔV a V b V i Similar. h calculated according to equation (3) i u i , The calculation formula is:

[0105]

[0106] It should be noted that when When collinear, equation (5) still holds true.

[0107] To preserve the spatial layout structure of the image sequence represented by the shooting point as much as possible, ΔV should be encouraged. a V b V i Deformation results It should be similar to the result of its transformation. As close as possible, it can be expressed by the following formula:

[0108]

[0109] for Figure 3 The sum of the errors between the deformation results and the similarity transformation results for all vertices of the triangles in (a) is:

[0110]

[0111] In the formula, N refers to the number of triangles constructed.

[0112] Combining equations (5) and (6), we solve equation (7) to obtain the following equation:

[0113]

[0114] In the formula, Refers to the latitude and longitude coordinates of the i-th image after deformation.

[0115] Therefore, it can be viewed as a linear optimization problem, and an optimal vertex set can be obtained by solving the sparse linear matrix equations.

[0116] 2. As in step 4, construct a GSP image stitching model based on spatial layout priors, specifically including:

[0117] GSP (Geostationary Image Processing) is an image stitching algorithm capable of stitching large-scale image sequences. It constructs an energy function with multiple constraints, using grid vertices as optimization terms, to obtain high-quality panoramic images. This invention uses GSP as a baseline and adds a proposed spatial arrangement preservation module to the GSP model, thus forming the proposed Spatial ArrangementPreservation Based GSP (SAP-GSP) stitching model. The specific objective function is as follows:

[0118]

[0119] in, Calculated using equation (8), λ is the result of all our experiments. l =0.75, λ sap =1.

[0120] The proposed SAP-GSP stitching method attempts to find a set of deformed mesh vertex positions that minimize the total energy function. In the original GSP algorithm, the alignment term... This ensures that matching pairs in overlapping regions are aligned as closely as possible, thereby guaranteeing alignment quality after image deformation; local similarity terms This ensures that each grid undergoes a similarity transformation as much as possible, thus better preserving the local structure of the image. Global similarity terms. This invention aims to ensure that each image undergoes similar transformations as much as possible, thus maintaining a natural appearance after stitching. The spatial layout structure protection item theoretically ensures that the relative spatial positions of all images corresponding to the shooting points remain unchanged in the stitched panoramic image, thus guaranteeing the overall naturalness of the stitching result.

[0121] The above method will be illustrated through the following experiment.

[0122] First, the Wheat-UAV (Unmanned Aerial Vehicle) farmland remote sensing image dataset of this invention is introduced. Second, based on the constructed dataset, the proposed stitching method is compared with representative image stitching methods such as Autostitch, GSP, OP-GSP, and GES-GSP, evaluating its performance from both subjective and objective perspectives. All experiments were conducted on a computer equipped with a 2.3GHz CPU and 32GB of memory, running Windows 10. This invention uses VLFeat to extract SIFT features, with a grid size of 40×40 pixels. In terms of stitching time, the proposed algorithm takes approximately 16 minutes to stitch 100 images, slightly longer than GSP, but significantly faster than OP-GSP and GES-GSP.

[0123] 1. Introduction to the experimental dataset.

[0124] like Figure 4 As shown, the location for collecting UAV remote sensing data of farmland is the wheat breeding experimental field of Caoxinzhuang Experimental Farm, Yangling Agricultural High-tech Industrial Demonstration Zone, western Guanzhong Plain, Shaanxi Province, China. The geographical coordinates are between 107°59′-108°08′ east longitude and 34°14′-34°20′ north latitude, with an average altitude of 530 meters. It belongs to the East Asian warm temperate semi-humid and semi-arid climate zone, exhibiting distinct continental monsoon climate characteristics: warm and windy springs, hot and rainy summers, cool and rainy autumns, and cold and dry winters. The average annual temperature is 12.9℃, the frost-free period is 211 days, the sunshine duration is 2163.8 hours, the total solar radiation is 114.86 kcal / cm², and the precipitation is 635.1 mm.

[0125] The data collection area is mainly divided into two parts based on crop information, such as... Figure 4 Region 1 is planted with rapeseed, and Region 2 with wheat. It can be seen that the color and texture features within the same area are very similar, which poses a challenge to image stitching. The dataset was captured between 2020 and 2022, using the wheat growth cycle as a reference, obtaining remote sensing images of different growth stages (sowing, seedling, tillering, overwintering, jointing, heading, flowering, grain-filling, and maturity). After the jointing stage, the crop leaves become very dense, making the details and features in the captured images more complex and diverse. In addition, the surrounding environment also plays a role; for example, strong winds may cause lodging. Figure 5 As shown, the features of the fallen wheat in the image are blurred and have double images, which greatly increases the difficulty of stitching.

[0126] The data acquisition device was a DJI Phantom 4 RTK drone equipped with a DJI FC6310R visible light camera, with an imaging resolution of 1600×1300. The drone's forward overlap was 85%, and the lateral overlap was 75%. To analyze the generalization of our method to different flight altitudes, we repeatedly flew at different altitudes of 21 degrees during each data acquisition, and the acquisition results are shown in Table 1.

[0127] Table 1. Parameters for UAV Remote Sensing Image Acquisition of Farmland

[0128]

[0129]

[0130] During the sowing and seedling stages, the experimental field consisted mainly of soil and ridges, with no crops on the ground. During the tillering, overwintering, and jointing stages, wheat seedlings grew to 1-2 cm in height, and the ridges between the crop rows were clearly visible. During the heading, flowering, grain-filling, and maturity stages, the wheat grew taller and its leaves became denser, making the crop rows indistinguishable. To obtain better shooting results, we primarily chose to shoot on sunny or cloudy days. During the shooting, to capture more detailed information about the farmland, we mainly flew at altitudes of 25m, 30m, and 50m. Flights at 25m and 30m were very low, resulting in a relatively small field of view, a large number of images, and richer details and features. Flights at 50m were higher, resulting in a wider field of view, fewer images, and a more prominent outline and structure of the farmland, but with relatively fewer details.

[0131] 2. Comparison with representative methods.

[0132] This section evaluates the proposed algorithm from both qualitative and quantitative perspectives. The subjective evaluation, based on human visual perception, primarily includes whether the stitching result is natural, aligned, free of ghosting, has smooth boundary transitions, exhibits significant distortion, and displays blurred image details. The objective evaluation assesses the degree of distortion in the stitching result by quantitatively calculating image distortion.

[0133] 2.1 Subjective evaluation.

[0134] Based on 29 sets of UAV remote sensing images of farmland, this invention compared the proposed SAP-GSP with four representative stitching methods, including Autostitch, GSP, OP-GSP, and GES-GSP.

[0135] Figure 6 This paper presents a qualitative evaluation of 29 sets of drone-generated farmland remote sensing data mosaics taken at different growth cycles and drone flight altitudes from 2020 to 2022. Figure 6(a), (b), and (c) are comparison images of the stitching results at 25m, 30m, and 50m, respectively. We describe the stitching results from worst to best using seven different methods: fail, warning, distortion, unalignment, ghosting, unnatual, and sussess, corresponding to ratings 1-7 on the radar chart. Therefore, the closer the line is to the outer edge, the better the stitching effect. Fail indicates that the stitched result is severely inconsistent with the actual panoramic image, i.e., there is a stitching error (e.g., ...). Figure 8 (a)); warning indicates that the stitched panoramic image is severely distorted (e.g., ...). Figure 7 (d)); distortion indicates the global structural distortion of the stitched panoramic image (e.g., ...). Figure 7 (c)); unalignment indicates that the images in different columns of the stitched panoramic image are severely misaligned (e.g., ...). Figure 8 (b)); ghosting indicates that the overall alignment of the stitched panoramic image is good, but there are very few cases of ghosting (such as...). Figure 9 (c)); unnatural indicates that the stitched panoramic image does not have problems such as stitching misalignment, distortion, and global structural distortion, but the result is unnatural due to uneven overall image scaling (e.g., Figure 7 (a)); success indicates that a well-preserved image stitching result with less distortion was obtained (e.g., ...). Figure 9 (e)). Figure 6 The Seeding stage 2020y refers to the data captured by the wheat Seeding Stage in 2020.

[0136] Research has found that when drones fly at low altitudes (below 50m), existing stitching algorithms such as Autostitch, GSP, OP-GSP, and GES-GSP are prone to stitching failures. Even when stitching is successful, high-quality results are rarely obtained. Our SAP-GSP method achieved high-quality stitching results in 29 datasets, validating the effectiveness and robustness of the method. Figure 7 This presentation showcases the stitched results of a dataset captured by a drone at an altitude of 25m during the growth stage of 2021. At a drone altitude of 25m, the details and features in the images are more complex and diverse, significantly increasing the difficulty of stitching. In the results obtained from Autostitch and OP-GSP, image distortion is evident due to accumulated errors. In the results obtained from GSP and GES-GSP, misalignment occurs due to a lack of matching between the two columns of images, and the overall structure of the results in GES-GSP is severely distorted. In our SAP-GSP results, image alignment is excellent, the overall structure is successfully preserved, and the overall visual effect is more natural.

[0137] 2.2 Objective evaluation.

[0138] Compared to the flight altitude of the drone, the crop height in farmland images varies less, so the transformations in each image are relatively simple and similar, basically consistent with the lens distortion. Figure 10 To examine the distortion of images in the stitched result, we found that each image in a natural stitched result exhibits pincushion distortion, and the more similar the degree of distortion, the more natural the stitched result. This invention, based on this pincushion distortion, objectively evaluates the naturalness of the stitched result from two aspects: uniformity of area distribution and the degree of pincushion distortion. Specifically, Figure 10 (a) is the image before stitching. Figure 10 (b) is the stitched image.

[0139] (1) Image stitching quality evaluation based on uniform area distribution. Under pincushion distortion, the area ratio of each image before and after deformation is within a certain range. Therefore, this invention reflects the difference in the degree of pincushion distortion by calculating the mean square error of the area of ​​all deformed images. The evaluation index is defined as:

[0140]

[0141] In the formula, S i The area of ​​the i-th deformed image. The average area of ​​all deformed images. The more uniform the area distribution, the more natural the stitching result; therefore, the smaller the RMSE, the more natural the image.

[0142] Figure 11 The results of the area RMSE evaluations for GSP and our method are shown. In over 70% of the 29 data sets, SAP-GSP outperformed GSP, indicating that our method is superior to GSP in terms of naturalness.

[0143] Furthermore, we found that because this method only evaluates mesh deformation and not alignment, it is ineffective for meshes such as... Figure 12 In this case, although our overall stitching effect is better, our evaluation index is higher than GSP. This situation applies to 8 out of 29 data sets.

[0144] (2) Evaluation of splicing quality based on pincushion distortion. For example... Figure 10 As shown, since farmland images are primarily characterized by pincushion distortion, the edges of each image are deformed into quadratic curves or straight lines. This invention fits the grid points on the deformed image boundaries into quadratic curves, as shown below. Figure 13 As shown. Clearly, the distance from each grid point on the boundary to the fitted curve reflects the degree of pincushion distortion. This invention uses the mean square error (RMSE) and standard deviation (SD) of this distance to determine the degree of grid deformation to evaluate the image stitching quality; smaller values ​​are considered more natural. Figure 13 (a) represents the grid representation of the image before deformation. Figure 13 (b) is the mesh representation of the image after deformation, and the lines on the boundary represent quadratic curves fitted based on the mesh points on the image boundary.

[0145] The formula for calculating the RMSE index of grid distortion in the i-th image is:

[0146]

[0147] Among them, l x , l y These are the fitted curves for the grid points in column x and column y, respectively. vxy is the grid point at position (x, y). X and Y are the column number and row number of the grid, respectively, and dst(v, l) is the shortest distance from point v to curve l.

[0148] Based on the RMSE of the i-th image, the SD calculation formula for the i-th image is as follows:

[0149]

[0150] Where avgx and avgy are the average values ​​of the residuals of the grid points in the x-th column and y-th row, respectively.

[0151] The evaluation indicators for the results are:

[0152]

[0153]

[0154] In the formula, N is the total number of images in the dataset.

[0155] Figure 14 The evaluation results of GSP and SAP-GSP on 29 datasets are shown. GSP outperforms SAP-GSP in over 80% of the results, with the remaining 20% ​​being due to [unclear - possibly related to GSP or SAP-GSP]. Figure 13 The results are the same, which indicates that our method is superior to GSP.

[0156] 3. Conduct analysis.

[0157] 3.1 Global Structure Preservation Capability Analysis:

[0158] As mentioned earlier, the proposed method uses the spatial layout reflected by the latitude and longitude coordinates carried by each image to guide the stitching, and has the ability to maintain the global structure. Figure 15 A comparison of the proposed SAP-GSP stitching algorithm with GSP, OP-GSP, and GES-GSP in spatial layout protection is presented, with all algorithms using the same image matching pairs. Figure 15 (a) shows the spatial layout formed by the drone's photographic points. The stitching results from different algorithms... Figure 15 As can be seen from (b)-(d), in terms of spatial layout protection, the proposed SAP-GSP is significantly superior to GSP and GES-GSP.

[0159] like Figure 15 (b) shows the splicing result of GSP, which indicates that... Figure 15 (b) shows a section of the road that has become distorted. GES-GSP proposes a geometry preservation method, and the experimental results are as follows: Figure 15 As shown in (c), unfortunately, the road is severely distorted, failing to preserve the large-scale geometry of the image. This is because the edge features of farmland are difficult to extract, and this method only preserves the geometry within a single image. Therefore, this method is not very effective for UAV-based remote sensing images of farmland. Figure 15 (d) shows the splicing result of SAP-GSP, and... Figure 15 (a) Compared with the UAV flight path, the stitching result basically maintains the relative positional relationship between the images and preserves the large-scale road structure in the image very well, which shows the effectiveness of our method.

[0160] Furthermore, we use Equation (7) to calculate the difference between the actual position and the expected position of each image before and after transformation, in order to evaluate the spatial layout preservation capability of the stitching algorithm. Figure 16 The spatial layout protection capabilities of different stitching algorithms on the Wheat-UAV dataset are listed. It can be seen that the SAP-GSP proposed in this invention has the smallest error, and therefore possesses the optimal spatial layout protection capability.

[0161] 3.2 Analysis of the Improved Local Alignment Capability Based on Adding Side Matching Pairs with Different Spacing:

[0162] Another advantage of spatial layout is that we can calculate the two closest images in adjacent columns to the current image using the drone's flight trajectory as its lateral matching relationship, adding more matching pair constraints to local matching and thus improving the local alignment capability of the images. This invention sets different interval sampling numbers to evaluate the impact on the local alignment of the stitching result.

[0163] Figure 17 This refers to the local alignment capability after adding lateral matching pairs at intervals of 13, 9, 5, and 3 images. Figure 17 In (a), when sampling at intervals of 13 images, the red and blue boxes in the image are not aligned. Figure 17 In (b), when sampling at intervals of 9 images, although the content within the red box is well aligned, the content within the blue box is still not aligned, and the overall image structure is slightly distorted. Interval 5 ( Figure 17 (c) Samples of 3 images and 3 images Figure 17In (d), the contents of the red and blue boxes are well aligned, but the image structure is slightly distorted when there are 3 images in between.

[0164] Taking into account the efficiency of the algorithm, this invention selects to add a lateral matching pair every 5 shooting points in the experiment, so as to improve the local alignment capability of the SAP-GSP algorithm while protecting the spatial layout.

[0165] To address the issue of poor image stitching quality caused by the high similarity in color and texture of farmland when stitching large-scale UAV-generated remote sensing images of farmland, this invention, for the first time, utilizes the spatial layout information represented by the latitude and longitude coordinates of the image capture points during UAV flight to propose an image spatial layout-preserving stitching algorithm, thereby obtaining a more natural overall panoramic image. First, this invention performs triangle sampling at the image capture points representing the spatial layout. By constructing a spatial layout protection module, it theoretically performs triangle similarity transformations as much as possible to protect the invariance of the spatial layout, and derives the corresponding numerical solution equations. Second, the proposed spatial layout protection module is added to the GSP algorithm, which is suitable for large-scale image stitching, resulting in the proposed SAP-GSP image stitching algorithm. Comprehensive experimental results show that, on the constructed farmland dataset, the proposed method outperforms representative stitching methods in both subjective and objective criteria. The proposed method can robustly stitch large-scale UAV-generated remote sensing images of farmland, exhibiting good results in both local alignment accuracy and overall stitching naturalness in panoramic images.

[0166] The above-described embodiments are merely preferred embodiments of the present invention, and the scope of protection of the present invention is not limited thereto. Any simple changes or equivalent substitutions of the technical solutions that can be obviously obtained by those skilled in the art within the scope of the technology disclosed in the present invention shall fall within the scope of protection of the present invention.

Claims

1. A method for unmanned aerial vehicle (UAV) remote sensing image stitching based on spatial layout protection, characterized in that, It comprises the following steps: Obtain a plurality of unmanned aerial vehicle remote sensing images; Extract the latitude and longitude coordinates of the photographing points in each of the unmanned aerial vehicle remote sensing images, and obtain the two-dimensional spatial layout of the photographing points through the latitude and longitude coordinates; Triangular sampling is performed on the two-dimensional spatial layout, and the similarity change of the constrained triangle in the triangular sampling is taken as a protection item of the spatial layout structure; The protection item of the spatial layout structure is added to the GSP model as a new energy item to construct a SAP-GSP stitching model; The unmanned aerial vehicle remote sensing images to be stitched are input into the SAP-GSP stitching model for stitching to obtain a panoramic image with spatial layout protection; Triangular sampling is performed on the two-dimensional spatial layout, and the similarity change of the constrained triangle in the triangular sampling is taken as a protection item of the spatial layout structure, comprising the following steps: Take the photographing points of each unmanned aerial vehicle remote sensing image as a sampling point, take the two photographing points farthest apart as endpoints, and take other sampling points as vertices to construct triangles respectively; Select a group of the triangles for image stitching, replace the deformation result of the triangle with an ideal transformation result to obtain a similarity transformation result; Obtain the error between the vertex deformation result and the similarity transformation result of all the triangles, and sum all the errors to obtain an optimal vertex set; The photographing points of each unmanned aerial vehicle remote sensing image are taken as a sampling point, the two photographing points farthest apart are taken as endpoints, and other sampling points are taken as vertices to construct triangles respectively, specifically comprising: By any one vertex With two reference endpoints , Form a triangle; The coordinates of the vertex are represented by , , and specifically: ； wherein , is , and two quantities derived from acquiring the , The specific expression of the above equation is: ； ； wherein and are respectively: ； ； wherein and respectively denote functions that solve h i and u i values; Replace the deformation result of the triangle with an ideal transformation result to obtain a similarity transformation result, comprising the steps of: The triangles are assembled, after assembly, , , are transformed into , , ; According to the deformed and in combination , estimate the expected position so that is similar to . wherein the desired position The calculation formula is: ； wherein , , are the vertices and the two reference endpoints , the deformed vertices and the two endpoints after stitching. 2.The method of claim 1, wherein, the desired position The calculation formula is: ； wherein, in The same holds for the above equation when the lines are collinear. 3.The method of claim 2, wherein, Obtain the error between the vertex deformation result and the similarity transformation result of all the triangles, and sum all the errors, comprising the steps of: The transformation result of the The transformation result of the similarity transformation The error between the vertex transformation result and the similarity transformation result is obtained by approaching, and the specific expression is:​ ； Sum all the errors, and the specific expression is: ； wherein N is the number of triangles of the construction.

4. The unmanned aerial vehicle remote sensing image stitching method based on spatial layout protection according to claim 3, characterized in that, Solve the sum expression of the error to obtain an optimal vertex set, and the specific expression is: ； Among them, the optimal vertex set is (x, y) ,......, ), denotes the longitude and latitude coordinates of the i-th image after deformation, wherein i = 1, 2,..., n.

5. The method of claim 4, wherein the method further comprises: The objective function of the SAP-GSP stitching model is: ； wherein is an alignment term, is a local similarity term, is a global similarity term, is a spatial layout structure protection term, and are coefficients for the local similarity term and the spatial layout structure protection term, respectively.

6. The unmanned aerial vehicle remote sensing image stitching method based on spatial layout protection according to claim 1, wherein, When the unmanned aerial vehicle remote sensing images to be stitched are input into the SAP-GSP stitching model for stitching, a group of deformed grid vertex positions are obtained, so that the total energy function contained in the grid vertex is minimized, and a panoramic image with spatial layout protection is obtained.

Images