Splicing method for high-similarity agricultural unmanned aerial vehicle remote sensing image sequence

Abstract

The invention discloses a splicing method for a high-similarity agricultural unmanned aerial vehicle remote sensing image sequence, which greatly improves the splicing efficiency of unmanned aerial vehicle images, reduces the occurrence of image deformation and ghosting when farmland unmanned aerial vehicle remote sensing images are spliced, and supports large-range and high-precision space-time continuous monitoring after corn ridge sealing. The method comprises the following steps: acquiring remote sensing images of a large-range farmland in a ridge sealing scene by using an unmanned aerial vehicle, and performing preliminary quality correction on the acquired images; optimizing a feature point extraction interval by adopting a sequence mask image registration method; performing image feature point extraction by using an SURF algorithm; feature point matching is carried out by using a KNN algorithm and a PROSAC algorithm; the matching precision is evaluated through a coordinate root mean square error; and finally, determining an optimal splicing edge between the images by using a graph cut algorithm, and carrying out image splicing to obtain a complete farmland image splicing result.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle remote sensing and agricultural precision monitoring, and in particular to a method for splicing highly similar agricultural unmanned aerial vehicle remote sensing image sequences. Background Art

[0002] Rapidly and accurately obtaining agricultural information is the basis for carrying out precision farming operations and optimizing cultivation management measures. Using drones to take multiple images from low altitudes is an important source of high-resolution data to monitor changes in agricultural conditions. It has the characteristics of strong flexibility, high spatial resolution, simple operation, and low cost, and has become a commonly used precision remote sensing platform. However, when drones acquire images, due to the influence of factors such as viewing angle, flight altitude, and camera focal length, the single image collected cannot cover the entire farmland area. In order to obtain image data information of the target area, it is necessary to use drone image stitching technology to stitch multiple images into a panoramic image.

[0003] The UAV image stitching method is based on image registration technology. At present, the most widely used registration algorithms can be roughly divided into image matching methods based on grayscale information, image registration methods based on transform domain, and image registration methods based on features. The matching method based on grayscale information extracts the target area of ​​the reference image as a template and searches for the most similar area in the image to be registered to determine the overlapping area. The MAD algorithm is widely used for its simplicity and efficiency, but it lacks rotation invariance and is easily affected by changes in viewing angle and lighting conditions. The registration method based on transform domain transfers image information to the transform domain through Fourier transform to achieve image stitching. Although it has good rotation and scale invariance, it has a large amount of calculation and high requirements for image overlap and scale, which is not suitable for UAV image stitching. The feature-based registration method realizes registration by extracting image features and establishing corresponding relationships between features. The Harris algorithm and its derivative algorithms, such as SIFT and SURF, are widely used in image stitching because they are invariant to scale scaling, rotation, etc. The SIFT algorithm is known for its stability, but has high computational complexity; the SURF algorithm improves speed while maintaining feature extraction efficiency.

[0004] The method of UAV image stitching technology mainly uses commercial software such as Pix4Dmapper. However, due to the light weight of the UAV platform itself, its main body is easily affected by the weather during flight, resulting in unstable shaking, which in turn causes the swaying and shaking of the UAV payload, resulting in the captured images often accompanied by ghosting, blurring, defocusing, twisting and increasing scale differences. These problems directly affect the image stitching effect and quality of the existing software. Especially when processing farmland images during the corn ridge closure period, the similarity between images is extremely large, and the image stitching effect is often poor. In addition, when studying large-scale farmland that requires a large number of images to ensure full coverage, the existing stitching methods mostly use the method of gradually stitching new images into the stitched parts. As the stitching process progresses, the amount of computation for feature point matching and calculation increases sharply, which not only consumes a lot of computer memory resources, but also may cause matching errors due to the high similarity between images, causing stitching dislocation and image twisting. How to achieve efficient and accurate image stitching in large-scale farmland UAV image stitching, especially in the ridge closure scene, to obtain the required large-scale farmland images has become a hot spot and difficulty in current research.

[0005] Therefore, in order to successfully stitch the required images and obtain the required large-scale drone images of farmland, it is necessary to study an image stitching method with good adaptability in the ridge closure scene to achieve the stitching of larger-scale and high-similarity drone images. Summary of the invention

[0006] In view of the defects in the prior art, the present invention provides a method for splicing a sequence of highly similar agricultural unmanned aerial vehicle remote sensing images.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] A method for stitching highly similar agricultural UAV remote sensing image sequences comprises the following steps:

[0009] Step 1: using a drone to collect images of a large area of ​​farmland in a ridge closure scenario and perform preliminary quality correction on the images;

[0010] Step 2, using the sequential mask image registration method to optimize the feature point extraction interval;

[0011] Step 3, using the accelerated robust feature SURF algorithm to extract feature points of the image;

[0012] Step 4, perform feature point matching, using the feature point rough matching algorithm of K-nearest neighbor algorithm KNN and the feature point fine matching algorithm of progressive sampling consistency algorithm PROSAC;

[0013] Step 5: Use the coordinate root mean square error (RMSE) to evaluate the matching accuracy and calculate the coordinate deviation of all the same-name point pairs in the two images after matching;

[0014] Step 6: Use the graph cut algorithm to determine the best stitching edge between images for image stitching to obtain a complete farmland image stitching result.

[0015] In a possible implementation, the step 1 includes the following steps:

[0016] (1) Using drones to collect images of large areas of farmland in ridge closure scenarios;

[0017] (2) recording the position and posture information of the image, and performing geometric correction of the image according to the position and posture information, wherein the position and posture information includes the geographic location, attitude angle, and flight altitude;

[0018] (3) adjusting the image brightness and contrast by using a histogram matching technique in response to illumination changes in the image;

[0019] (4) Perform necessary cropping on the image to ensure the clarity and consistency of the image edges.

[0020] In a possible implementation, the step 2 includes the following steps:

[0021] (1) Analyze the flight trajectory of the drone to determine the arrangement direction of the images, and use the slope k and distance l of the feature point connection line to calculate and determine the relative position relationship between the images, where:

[0022]

[0023] In the formula, k and l are the slope and length of the line connecting the feature points of the two images respectively. pt1 ,y pt1 ) is the horizontal and vertical coordinates of a feature point in the image to be matched, (x pt2 ,y pt2 ) are the horizontal and vertical coordinates of the corresponding feature point in the other image to be matched, max(k)=0.002, c=0.002;

[0024] (2) According to the arrangement of images, the sequential masking technique is applied to match the feature points between the mask image and the adjacent area of ​​the spliced ​​image;

[0025] (3) The numbers of the images to be stitched are stored in the matrix according to the arrangement positions, and for images in different rows and columns, the image positions are exchanged and the image arrangement directions are adjusted;

[0026] (4) Setting the optimal feature point extraction interval to optimize the recognition rate and matching efficiency of feature points.

[0027] In a possible implementation, step 3 includes the following steps:

[0028] (1) Calculate the integral image to improve computational efficiency and use a box filter for multi-scale analysis;

[0029] (2) Construct the Hessian matrix to identify potential feature points and extract the corresponding feature descriptors;

[0030] (3) Calculate the determinant value of the Hessian matrix to determine the local extreme points as feature point candidates.

[0031] In a possible implementation, step 4 includes the following steps:

[0032] (1) Using the KNN, the Euclidean distance between feature points is calculated;

[0033] (2) Select K nearest neighbor feature points from the candidate feature point set to form a preliminary matching pair set;

[0034] (3) Using the PROSAC, the preliminary matching pair set is finely matched, iteratively executed, continuously sampled and evaluated, and the inliers and outliers are updated until a better model is obtained;

[0035] (4) Fit all internal points through the model to obtain a more accurate matching result.

[0036] In a possible implementation, step 5 includes: analyzing all pairs of points with the same name, determining the coordinate deviation of each pair of matching points in the two images, obtaining the error between the actual coordinates and the theoretical coordinates of each matching point, calculating the RMSE of the matching point pairs, and quantifying the accuracy of feature point matching.

[0037] In a possible implementation, step 6 includes:

[0038] (1) Generate an undirected graph model, use the pixels in the image as nodes of the graph, and set special nodes, source points S and sink points T, to connect with the pixel nodes in the image;

[0039] (2) Use the graph cut algorithm to calculate the minimum cut path in the graph and determine the optimal stitching edge between the two images to achieve seamless stitching;

[0040] (3) The images are stitched along the determined optimal stitching edge to make the transition of the stitching area smooth and natural, thus obtaining a complete farmland image stitching result.

[0041] The beneficial effects of the present invention are:

[0042] The present invention uses the minimum cut technology to calculate the optimal splicing edge line in the image, so that the transition of the splicing area is smooth and natural. By optimizing the energy function, the optimal splicing path is found to reduce the splicing traces. The images are spliced ​​along the determined optimal splicing edge to ensure the overall consistency and seamlessness of the images. Finally, a complete farmland image splicing result is obtained, and high-quality remote sensing image splicing is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A schematic diagram of the process of stitching farmland drone images provided by the present invention;

[0045] Figure 2 A schematic diagram of the extraction interval optimization solution provided by the present invention;

[0046] Figure 3 A schematic diagram of the optimal seam line search method based on the graph cut method provided by the present invention;

[0047] Figure 4 The present invention provides an optimal seam line search method based on the graph cut method and an image stitching process. DETAILED DESCRIPTION

[0048] The embodiments of the present invention are described below in conjunction with the drawings in the embodiments of the present invention. The terms used in the implementation mode of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention. It is known to those of ordinary skill in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present invention are also applicable to similar technical problems. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present invention.

[0049] Combine the following Figure 1-Figure 4 The present invention describes a method for stitching a sequence of highly similar agricultural UAV remote sensing images provided by an embodiment of the present invention. Figure 1 The present invention provides a flow chart of a method for stitching a sequence of highly similar agricultural drone remote sensing images, including: farmland image preprocessing, image feature point extraction, feature point registration and image stitching. Figure 1 The process and specific steps include but are not limited to the following steps:

[0050] Step 1: Use UAV to collect images of a large area of ​​farmland in a closed ridge scenario and perform preliminary quality correction on the images, and collect and preprocess point data reflecting crop conditions and remote sensing images with two different spatiotemporal resolutions.

[0051] The drone images of farmland with little ground information are collected and preprocessed, including: collecting large-scale drone images of farmland in closed ridge scenes; reading image pose information (POS information); farmland image preprocessing: POS-based pose information correction, histogram matching, image cropping, etc.

[0052] In order to obtain remote sensing images of a large area of ​​farmland in the closed ridge scene, the Inspire 2 drone equipped with the Zenmuse X5S gimbal camera was used to collect remote sensing images of a large area of ​​farmland at a flight altitude of 20 meters, 80% heading overlap and lateral overlap. During the flight of the drone, the image's position information, including geographic location, attitude angle, and flight altitude, was automatically recorded.

[0053] For image data and posture information, geometric correction and histogram matching are performed. Image processing software is used to perform orthorectification on the image to eliminate image distortion caused by changes in flight posture. Histogram matching is used to balance the brightness and contrast of the image to reduce the impact of lighting changes on image consistency. The image is cropped according to the actual boundary of the farmland to remove unnecessary edge parts.

[0054] Step 2: Use the sequential mask image registration method to optimize the feature point extraction interval.

[0055] Determine the flight direction, determine whether to change the direction of adding new images when adding new images, add images in the opposite direction of the previous row (column), store the image numbers in the matrix according to the arrangement position, and match the feature points with the adjacent influences through masks. It includes the following steps:

[0056] According to the following formula, the slope and distance of the feature point matching line are obtained to determine the arrangement of all images in a certain flight mission.

[0057]

[0058] In the formula, k and l are the slope and length of the line connecting the feature points of the two images respectively. pt1 ,y pt1 ) is the horizontal and vertical coordinates of a feature point in the image to be matched, (x pt2 ,y pt2) is the horizontal and vertical coordinates of the corresponding feature point in the other image to be matched, c is the allowable deviation threshold due to the deviation of the image due to the jitter of the drone. After testing, when the two images to be stitched are in a positive left-right direction relationship, max(k) = 0.002, so c = 0.002 can be set.

[0059] The rules for distinguishing the flight path patterns of UAVs are shown in Table 1.

[0060] Table 1

[0061]

[0062] like Figure 2 As shown in Figure 1, the acquired image sequence is input into the algorithm in sequence. According to the rules in Table 1, the first two images to be stitched in the image sequence are exchanged in the left-right direction. By comparing the mode of the slope k of the feature point connection line of the two arrangement methods, it is determined whether the flight route of this flight mission belongs to the left-right direction or the up-down direction (the direction is determined for the plane, such as Figure 2 If it is in the left-right direction, the position of the second image relative to the first image is further determined (e.g. Figure 2 As shown in (b), if the second image is located on the left side of the first image, then when there is no up-down relationship between the third image and the second image, it is added to the left side of the second image. If there is an up-down relationship, it is placed in the corresponding position, and when adding a new image in a new row, the direction is reversed (such as Figure 2 (c) shown in the figure, that is, when adding an image in a new row, add the new image to the right of the previous image; if it is in the up and down direction, the operation is similar, except that when adding a new image, it is necessary to determine whether the newly added image has a left-right relationship with the previous image. If so, the direction is changed. After determining the arrangement of each image, store the image number in the matrix, fill the vacant part with 0, and finally press the number matrix (such as Figure 2 As shown in (d), only the images with adjacent numbers are used for feature point extraction, matching and splicing. The range of the mask can also be specifically defined by combining flight parameters such as heading overlap and lateral overlap (the shape of the mask can be referred to in Figure 2 As shown in (e), use the smallest mask range possible to achieve the best stitching effect. The final stitching effect can be as follows Figure 2 (f) shown.

[0063] Step 3: Use the accelerated robust feature SURF algorithm to extract feature points of the image.

[0064] The image feature points are extracted using the SURF algorithm, including: integral image and box filter processing, construction of the Hessian matrix, calculation of the determinant value of the Hessian matrix, detection of local maxima, and calculation of descriptors. Among them, the SURF (Speeded Up Robust Features) algorithm is a fast and robust feature point extraction and description algorithm. The integral image of the image is calculated to improve computational efficiency, and a box filter is applied for multi-scale analysis to enhance the detailed features of the image. Subsequently, the Hessian matrix is ​​constructed and its determinant value is calculated, and potential feature points in the image are identified by detecting local extreme points. These local extreme points are determined as initial feature point candidates.

[0065] A descriptor is calculated for each feature point. The descriptor is constructed based on the local information of the feature point to ensure the stability and invariance of the feature point under different scales, rotations and lighting conditions. The calculation method of the descriptor includes directional analysis of the area around the feature point to obtain a feature descriptor with rotation invariance.

[0066] Step 4, perform feature point matching, using the feature point rough matching algorithm of the K nearest neighbor algorithm KNN and the feature point fine matching algorithm of the progressive sampling consistency algorithm PROSAC.

[0067] Among them, the image feature points are matched, including: calculating similarity, selecting feature points using the KNN algorithm, constructing a rough matching point set, and fine matching using the PROSAC algorithm, including: randomly selecting a certain number of points, calculating the error between all points in the point set and the model, iterative execution, sampling evaluation, updating internal points and external points, and model fitting. Among them, the KNN (K-Nearest Neighbors) algorithm is a basic and widely used classification and regression method. The PROSAC (Progressive Sample Consensus) algorithm is a robust matching algorithm for matching problems in computer vision.

[0068] After the feature points are extracted, the KNN algorithm is first used to perform rough matching of the feature points. The Euclidean distance between each feature point is calculated, and K nearest neighbor feature points are selected from the candidate feature point set to form a preliminary matching pair set.

[0069] The PROSAC algorithm is used to perform fine matching on the preliminary matching pair set. The PROSAC algorithm iteratively selects inliers and outliers in the preliminary matching pair set and optimizes the matching model by calculating the error of the matching pair. The specific process includes gradually increasing the number of point pairs considered to be correctly matched according to the priority sampling strategy, calculating the fitting error of the model and updating the inliers set until the optimal matching model is found.

[0070] Step 5: Use the coordinate root mean square error (RMSE) to evaluate the matching accuracy and calculate the coordinate deviation of all the same-name point pairs in the two images after matching.

[0071] After completing the feature point matching, the matching algorithm accuracy is evaluated, and the accuracy of the feature point matching is expressed by the coordinate root mean square error (RMSE) after the matching of the same-name point pairs. Specifically, first, all the same-name point pairs are analyzed, and the coordinate deviation of each pair of matching points in the two images is calculated. The formula is as follows:

[0072]

[0073]

[0074] Where n is the number of matching feature points in the two images, (x 1i ,y 1i ) are the actual horizontal and vertical coordinates of the i-th pair of feature points in the first image, (x 2i ,y 2i ) are the actual horizontal and vertical coordinates of the i-th pair of feature points in the second image, and H is the change matrix of the two images. (H(x 1i ), H(y 1i )) are the horizontal and vertical coordinates of the second image corresponding to the first image after the H matrix transformation.

[0075] Use RMSE to evaluate the accuracy of matching and ensure that the coordinate deviations of all matching point pairs are within an acceptable range. By evaluating the matching accuracy, you can determine whether there are significant errors in the image stitching process. If the RMSE value is too large, you need to adjust the matching parameters or re-match the feature points to improve the overall accuracy of image stitching.

[0076] Step 6: Use the graph cut algorithm to determine the best stitching edge between images for image stitching to obtain a complete farmland image stitching result.

[0077] Among them, pixel points are generated, and two special nodes, source point S and sink point T, are additionally generated, and undirected edges are formed between all pixel points; a bipartite graph is constructed, in which the nodes are divided into two groups, and the optimal stitching edges between images are determined; a seamless stitching image is output, and the stitching edges are optimized to make the transition smooth and natural.

[0078] like Figure 3 As shown in the figure, an undirected graph model is generated, the pixels in the image are used as nodes, and special nodes, source points S and sink points T, are set to connect with the pixel nodes in the image. The similarity between each node is calculated, and the weight of the connecting edge is established. The similarity calculation function is as follows:

[0079] E(f)=∑ i，j R(i, j)+B(i, j)

[0080] B(i, j) = s x (i,j)δ(f(i,j)-f(i+1,j)+s y (i, j)δ(f(i, j)-f(i, j+1)

[0081] In the formula, R(i, j) represents the region term, B(i, j) represents the boundary term, which is the inconsistency of the pixel point in the four neighborhoods, and S x (i, j) represents a smooth function in the horizontal direction, S y (i, j) represents the smoothing function in the vertical direction, and f(i, j) represents the pixel point of the image.

[0082] like Figure 4 As shown in the figure, the minimum cut technique is used to calculate the optimal stitching edge line in the image, making the transition of the stitching area smooth and natural. By optimizing the energy function, the optimal stitching path is found to reduce the stitching traces. The images are stitched along the determined optimal stitching edge to ensure the overall consistency and seamlessness of the images. Finally, the complete farmland image stitching result is obtained, achieving high-quality remote sensing image stitching.

Claims

1. A method for stitching highly similar agricultural UAV remote sensing image sequences. It is characterized in that The following steps are involved: Step 1: using a drone to collect images of a large area of ​​farmland in a ridge closure scenario and perform preliminary quality correction on the images; Step 2, using the sequential mask image registration method to optimize the feature point extraction interval; Step 3, using the accelerated robust feature SURF algorithm to extract feature points of the image; Step 4, perform feature point matching, using the feature point rough matching algorithm of K-nearest neighbor algorithm KNN and the feature point fine matching algorithm of progressive sampling consistency algorithm PROSAC; Step 5: Use the coordinate root mean square error (RMSE) to evaluate the matching accuracy and calculate the coordinate deviation of all the same-name point pairs in the two images after matching; Step 6: Use the graph cut algorithm to determine the best stitching edge between images for image stitching to obtain a complete farmland image stitching result.

2. A method for stitching highly similar agricultural UAV remote sensing image sequences according to claim 1, It is characterized in that The step 1 includes the following steps: (1) Using drones to collect images of large areas of farmland in ridge closure scenarios; (2) recording the position and posture information of the image, and performing geometric correction of the image according to the position and posture information, wherein the position and posture information includes the geographic location, attitude angle, and flight altitude; (3) adjusting the image brightness and contrast by using a histogram matching technique in response to illumination changes in the image; (4) Perform necessary cropping on the image to ensure the clarity and consistency of the image edges.

3. According to claim 1, a method for stitching a sequence of highly similar agricultural drone remote sensing images, It is characterized in that The step 2 includes the following steps: (1) Analyze the flight trajectory of the drone to determine the arrangement direction of the images, and use the slope k and distance l of the feature point connection line to calculate and determine the relative position relationship between the images, where: In the formula, k and l are the slope and length of the line connecting the feature points of the two images respectively. pt1 ,y pt1 ) is the horizontal and vertical coordinates of a feature point in the image to be matched, (x pt2 ,y pt2 ) are the horizontal and vertical coordinates of the corresponding feature point in the other image to be matched, max(k)=0.002, c=0.002; (2) According to the arrangement of images, the sequential masking technique is applied to match the feature points between the mask image and the adjacent area of ​​the spliced ​​image; (3) The numbers of the images to be stitched are stored in the matrix according to the arrangement positions, and for images in different rows and columns, the image positions are exchanged and the image arrangement directions are adjusted; (4) Setting the optimal feature point extraction interval to optimize the recognition rate and matching efficiency of feature points.

4. According to claim 1, a method for stitching a sequence of highly similar agricultural UAV remote sensing images, It is characterized in that The step 3 includes the following steps: (1) Calculate the integral image to improve computational efficiency and use a box filter for multi-scale analysis; (2) Construct the Hessian matrix to identify potential feature points and extract the corresponding feature descriptors; (3) Calculate the determinant value of the Hessian matrix to determine the local extreme points as feature point candidates.

5. The method for stitching highly similar agricultural UAV remote sensing image sequences according to claim 1, It is characterized in that The step 4 includes the following steps: (1) Using the KNN, the Euclidean distance between feature points is calculated; (2) Select K nearest neighbor feature points from the candidate feature point set to form a preliminary matching pair set; (3) Using the PROSAC, the preliminary matching pair set is finely matched, iteratively executed, continuously sampled and evaluated, and the inliers and outliers are updated until a better model is obtained; (4) Fit all internal points through the model to obtain a more accurate matching result.

6. A method for stitching highly similar agricultural UAV remote sensing image sequences according to claim 1, It is characterized in that The step 5 includes: analyzing all pairs of points with the same name, determining the coordinate deviation of each pair of matching points in the two images, obtaining the error between the actual coordinates and the theoretical coordinates of each matching point, calculating the RMSE of the matching point pairs, and quantifying the accuracy of feature point matching.

7. The method for stitching highly similar agricultural UAV remote sensing image sequences according to claim 1, It is characterized in that The step 6 includes: (1) Generate an undirected graph model, use the pixels in the image as nodes of the graph, and set special nodes, source points S and sink points T, to connect with the pixel nodes in the image; (2) Use the graph cut algorithm to calculate the minimum cut path in the graph and determine the optimal stitching edge between the two images to achieve seamless stitching; (3) The images are stitched along the determined optimal stitching edge to make the transition of the stitching area smooth and natural, thus obtaining a complete farmland image stitching result.

Images