A high-precision iterative binary segmentation-assisted puncture point method based on SIFT algorithm

Abstract

The present invention discloses a high-precision iterative binary division-assisted puncture point method based on SIFT algorithm, including: using GCP ground object images and potential matching photos to execute SIFT algorithm for feature matching to assist the "puncture point" step in the unmanned aerial photography modeling operation. The idea of ​​using the iterative method and the binary division method to process the ground object image to optimize the SIFT feature matching accuracy. Using one of the potential matching photos as a representative photo, and running the high-precision iterative binary division-assisted puncture point method with the GCP ground object image, avoids repeated calculation of the GCP ground object image size. Afterwards, the SIFT algorithm can be directly executed according to the first determined GCP ground object image size to indicate the puncture point. The advantages of the present invention are: reducing the workload of the unmanned aerial photography modeling operation, improving the accuracy, and improving the work efficiency.

Description

Technical Field

[0001] The invention relates to the technical field of unmanned aerial photography modeling, and in particular to a high-precision iterative binary division auxiliary punctum method based on a SIFT algorithm. Background Art

[0002] The Structure from Motion (SfM) algorithm can reconstruct the three-dimensional model of the study area using the original multi-view UAV photo dataset of the study area and the corresponding RTK and PPK data.

[0003] In the SfM algorithm, the SIFT algorithm (and similar SIFT algorithms) are first used to extract feature points between the original photos. After obtaining the feature points, RANSAC and the eight-point method can be used to estimate the fundamental matrix based on the extracted and filtered feature points, and then the track is extracted, and the scene structure is restored through triangulation, bundle adjustment and other processes.

[0004] During the SfM process, each photo accumulates the residual reprojection error of the bundle adjustment, resulting in a huge cumulative error after all photos are matched. Ground control points (GCPs) can provide additional external information about the 3D geometry, and re-iteration of the bundle adjustment through GCPs can improve the accuracy of the UAV model.

[0005] Therefore, it is particularly important to provide GCPs information for photos. In general, after providing the coordinate information of GCP, a sub-image slightly offset from the actual GCP can be automatically determined on the sparse 3D model formed after aerial triangulation, and then the pixel blocks that exactly correspond to the actual GCP are manually visually specified in the sub-image and its vicinity. This process is called "puncture point".

[0006] After the first control point-free aerial triangulation generates a sparse 3D model, the coordinates of the control points measured by RTK can be imported, and then the "point puncture" work can be carried out. For the point puncture, the traditional method is to manually specify the actual position of each control point for all potential matches ( Figure 1 Here, all potential matches for a control point are determined by the SfM software by first directly mapping the precise coordinates of the control point to a spatial position in the sparse 3D model, and then extracting all the photo sets involved in constructing this position in the sparse 3D model.

[0007] For a large-scale, high-resolution drone aerial photography mission, one control point may correspond to many drone aerial photography photos (potential matches). For such a large number of potential matches, if all of them are "punctured" by manual visual means, it will be extremely time-consuming and laborious in some cases.

[0008] Each potential matching photo may have a certain degree of visual difference due to the difference in elevation, light and shadow, lens angle, etc. when shooting. The manual visual correction method is not sensitive to such visual differences. Especially for those potential matching photos with darker colors and biased angles, it is easy to cause the situation of "puncturing points" by mistake, which will not only fail to improve the accuracy of the drone model, but also affect its accuracy.

[0009] Generally speaking, since the photo collection of drone aerial photography itself carries PPK location information, the sparse three-dimensional model established without control points also has a certain degree of accuracy. Therefore, the "puncture point" work is actually a feature matching task within a small range. It is entirely possible to consider using feature matching algorithms such as SIFT to implement the "puncture point" indication.

[0010] However, there are some problems with the "puncture point" operation when using the SIFT algorithm alone, mainly manifested as follows:

[0011] ① It is difficult to determine the size of the GCP image for executing the SIFT algorithm. Using a GCP image that is too large or too small will result in poor accuracy. Generally speaking, we should seek a minimum GCP image that can meet certain accuracy requirements, so that the accuracy is high and the calculation speed is also fast.

[0012] ② In order to achieve the purpose in ①, the natural way is to repeatedly try an optimal GCP image size through trial and error ( Figure 2 ). However, this is inefficient and uncertain, and it is possible to repeatedly try and error between multiple suboptimal values ​​and thus skip the optimal value.

[0013] Definitions of Abbreviations and Key Terms

[0014] SfM: Structure from Motion;

[0015] RTK: Real-time kinematic carrier phase differential;

[0016] PPK: dynamic post-processing;

[0017] SIFT: Scale Invariant Feature Transform;

[0018] RANSAC: Random Sampling Consensus;

[0019] GCP(s): Ground Control Point(s), hereinafter referred to as control point. Summary of the invention

[0020] In view of the defects of the prior art, the present invention provides a high-precision iterative binary division auxiliary puncture point method based on SIFT algorithm.

[0021] In order to achieve the above invention object, the technical solution adopted by the present invention is as follows:

[0022] A high-precision iterative binary division-assisted puncture point method based on SIFT algorithm, comprising the following steps:

[0023] S1: Prepare GCP ground feature images and a set of potential matching photos automatically determined after aerial triangulation.

[0024] S2: First, the initial iteration window size is determined according to the absolute value of the aerial triangulation error (RMSE) and the size of the GCP object; the initial iteration window size is calculated by the following formula:

[0025] Iteration window size = GCP feature size + δ

[0026] δ = RMSE × n

[0027] n=2 i

[0028] δ is the iterative window correction amplification term, n is the amplification coefficient, and i is the iterative step length. In the initial state, i=0.

[0029] S3: Crop the GCP feature image within this iteration window.

[0030] S4: Execute the SIFT algorithm to perform feature matching on the cropped GCP feature image and its GCP potential matching photo, and calculate the F1 value of the SIFT algorithm after obtaining the result;

[0031] S5: Initially give an F1 value threshold. If the calculated F1 value does not reach the threshold, start iteration. i increases by 1, and repeat S2 to S4 until the F1 value meets the threshold, then stop iteration.

[0032] S6: Start binary division, in [2 n-1 ,2 n ] interval, and recalculate an F1-Score according to S3 each time the binary search is performed on δ, until a minimum δ that meets the F1 value threshold is found, and the result corresponding to this δ is used as the feature point matching result.

[0033] S7: Set the upper limit of the number of iterations to max_iteration. When the number of iterations reaches max_iteration, the iteration is stopped. If the iteration window size exceeds the sub-image size or the GCP feature image size, the iteration is also stopped. At this time, the threshold F1 value should be reduced according to the custom rule, n=0 should be reset, and S2 to S5 should be repeated.

[0034] S8: Finally, the feature point matching result is output to indicate the location of the thorn point. At the same time, the iteration window size is determined.

[0035] Furthermore, the F1-Score in S4 is calculated by precision and recall:

[0036]

[0037]

[0038]

[0039] The calculated F1 value is between 0 and 1.

[0040] Furthermore, the SIFT feature matching operation of the subsequent potential matching photos of the GCP is directly performed according to the iteration window size determined in S7.

[0041] Compared with the prior art, the advantages of the present invention are:

[0042] (1) The SIFT algorithm is used to assist the “puncture point” operation, which greatly improves the speed and accuracy of the “puncture point” and reduces the workload of UAV aerial photography modeling operations.

[0043] (2) Avoiding manual trial and error, the optimal GCP image size is determined through continuous iteration according to the algorithm process, thus improving the accuracy of the SIFT algorithm.

[0044] (3) For each potential matching photo of a GCP, we only need to select one of the potential matching photos to run the algorithm to determine δ, and then reuse this δ in the subsequent SIFT process, thereby improving work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic diagram of the manual visual "puncture point" operation in the prior art;

[0046] Figure 2 It is a schematic diagram of assisting puncture points by SIFT algorithm in the prior art;

[0047] Figure 3 This is a flow chart of a high-precision iterative binary division auxiliary punctum method based on SIFT algorithm according to an embodiment of the present invention;

[0048] Figure 4 It is a diagram showing the final implementation effect of the embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples.

[0050] like Figure 1 As shown, a high-precision iterative binary division-assisted puncture point method based on SIFT algorithm includes the following steps:

[0051] S1: Prepare GCP ground feature images and a set of potential matching photos automatically determined after aerial triangulation.

[0052] S2: First, the size of the initial iteration window is determined according to the absolute value of the aerial triangulation error (RMSE) and the size of the GCP object;

[0053] Iteration window size = GCP feature size + δ

[0054] δ = RMSE × n

[0055] n=2 i

[0056] δ is the iterative window correction amplification term, n is the amplification coefficient, and i is the iterative step length. In the initial state, i=0.

[0057] S3: Crop the GCP feature image within this iteration window.

[0058] S4: Execute the SIFT algorithm to perform feature matching on the cropped GCP feature image and its GCP potential matching photos. After obtaining the results, calculate the F1 value of the SIFT algorithm; the F1 value (F1-Score) is calculated by precision and recall:

[0059]

[0060]

[0061]

[0062] The number of duplicate feature point pairs can be obtained through the homography matrix. The method can be found in the article "A performance evaluation of local descriptors". To determine whether a matching point is a pair of feature points, refer to the criteria proposed in the article "Anaffine invariant interest point detector". The calculated F1 value is between 0 and 1.

[0063] S5: Initially set a larger F1 value threshold (such as 0.90). If the calculated F1 value does not reach the threshold, start iteration. i increases by 1, and repeat S2 to S4 until the F1 value meets the threshold, then stop iteration.

[0064] S6: Start binary division, in [2 n-1 ,2 n ] interval, and each time the binary search is performed, an F1-Score is recalculated according to step S3 until a minimum δ that meets the F1 value threshold is found, and the result corresponding to this δ is used as the feature point matching result.

[0065] S7: It should be noted that the number of iterations should have an upper limit value of max_iteration. When the number of iterations reaches max_iteration, the iteration is also stopped. In addition, if the iteration window size exceeds the sub-image size or the GCP feature image size, the iteration is also stopped. At this time, the threshold F1 value should be reduced according to the custom rule, n=0 should be reset, and S2 to S5 should be repeated.

[0066] S8: Finally, the feature point matching result is output to indicate the location of the thorn point. At the same time, the iteration window size is determined.

[0067] S9: The SIFT feature matching operation of the subsequent potential matching photos of this GCP can be directly performed according to the iteration window size determined in S7.

[0068] The specific implementation of the method of the present invention is described below with reference to a specific example.

[0069] (1) During the drone aerial photography mission of a landslide in Ganzi Prefecture, Sichuan Province, more than 2,000 high-resolution aerial images were collected. More than 20 RTK ground control points were obtained through on-site surveys. These control points will be used for the "puncture point" work of the drone aerial photography modeling mission.

[0070] (2) First, the SfM software is used to perform aerotriangulation on the UAV aerial image (with PPK information) to establish a rough 3D model. After this process is completed, the RMSE is 0.31 pixels.

[0071] (3) The initial RMSE is determined to be 0.31 pixels and the F1 threshold is 0.75. δ is calculated iteratively, and after several rounds of iterations, a δ=9.6 pixels that satisfies the F1 threshold is obtained.

[0072] (4) Perform a binary search on δ = 9.6 pixels and re-run the SIFT algorithm to calculate the F1 value. When δ = 8 pixels, a minimum δ that meets the F1 threshold is obtained.

[0073] (5) Applying this δ to the remaining potential matching photos, the SIFT algorithm can directly output a good matching result ( Figure 4 ), which greatly reduces the time of manual visual search when finding the "puncture point".

[0074] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the implementation methods of the present invention, and should be understood that the protection scope of the present invention is not limited to such special statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A high-precision iterative binary assisted punctum method based on SIFT algorithm, characterized in that: The following steps are involved: S1: Prepare GCP ground feature images and potential matching photo sets automatically determined after aerial triangulation; S2: First, the initial iteration window size is determined according to the absolute value of the aerial triangulation error (RMSE) and the size of the GCP object; the initial iteration window size is calculated by the following formula: Iteration window size = GCP feature size + δ δ = RMSE × n n＝2 i δ is the iterative window correction amplification term, n is the amplification coefficient, i is the iterative step length, and i=0 in the initial state; S3: Crop the GCP feature image within this iteration window; S4: Execute the SIFT algorithm to perform feature matching on the cropped GCP feature image and its GCP potential matching photo, and calculate the F1 value of the SIFT algorithm after obtaining the result; S5: Initially give an F1 value threshold. If the calculated F1 value does not reach the threshold, start iteration; i increases by 1, and repeat S2 to S4 until the F1 value meets the threshold, then stop iteration; S6: Start binary division, in [2 n-1 ,2 n ] interval, and recalculate an F1-Score according to S3 each time the binary search is performed on δ, until a minimum δ that meets the F1 value threshold is found, and the result corresponding to this δ is used as the feature point matching result; S7: Set the upper limit of the number of iterations to max_iteration. When the number of iterations reaches max_iteration, the iteration is stopped. If the iteration window size exceeds the sub-image size or the GCP feature image size, the iteration is also stopped. At this time, the threshold F1 value should be reduced according to the custom rule, n=0 should be reset, and S2 to S5 should be repeated. S8: Finally, the feature point matching result is output to indicate the location of the thorn point; at the same time, the iteration window size is determined.

2. According to the high-precision iterative binary division-assisted punctum method based on SIFT algorithm in claim 1, it is characterized by: The F1-Score in S4 is calculated by precision and recall: The calculated F1 value is between 0 and 1.

3. The high-precision iterative binary division-assisted puncture point method based on the SIFT algorithm according to claim 1, characterized in that: The SIFT feature matching operation of the subsequent potential matching photos of the GCP is directly performed according to the iteration window size determined by S7.

Images