An Improved AKAZE Feature-based UAV Image Matching Method

Abstract

The present invention relates to a method for improving the matching of UAV images with AKAZE features, belonging to the technical field of UAV image matching. This method first uses the AKAZE algorithm to construct a non-linear scale space to detect local stable features, constructs a binary descriptor BEBLID to describe the detected feature points, then uses the brute-force matching method to complete the preliminary matching, and then uses the grid motion statistics method to perform the first rough error rejection. Finally, the epipolar constraint condition is added to purify the inliers to obtain a reliable matching result. Through experimental comparative analysis of the algorithm of the present invention and existing algorithms, the results show that the matching accuracy rate of the method of the present invention is not much different from that of the AKAZE algorithm, the matching speed is increased by about 40% compared with the AKAZE algorithm, and is close to the speed of the ORB algorithm. The number of correct matching point pairs is significantly increased compared with several comparison methods, and the spatial distribution of the matching points is scattered and uniform, having obvious advantages and practicability.

Description

Technical Field

[0001] The present invention belongs to the technical field of UAV image matching, and relates to a UAV image matching method for improving AKAZE features, in particular to a method for detecting features by using the AKAZE algorithm and constructing a BEBLI D descriptor for debris flow image matching. Background Technique

[0002] With the rapid development of UAV technology, constructing a ground three-dimensional scene based on UAV images has become a very convenient way, making it one of the important means for emergency disaster relief such as landslides and debris flows. However, restricted by factors such as the sensor's field of view, shooting angle, and platform movement, usually hundreds or even thousands of images need to be taken to completely cover the disaster area, which requires matching image data taken from different positions and perspectives. Compared with other images, UAV images usually have a relatively high spatial resolution and contain more detailed information about ground objects, which greatly increases the difficulty of image matching. Compared with images with relatively rich artificial ground objects, images mainly distributed with natural landforms such as debris flows have characteristics such as large terrain undulations, complex and changeable landforms, repeated or single textures, and blurred boundaries between different ground objects. The difficulty of feature extraction and matching is greater, and the probability of incorrect matching is also greater. When processing UAV data, including image stitching, dynamic structure recovery, and dense three-dimensional scene reconstruction, matching is the basis and prerequisite. Therefore, improving the quality and speed of matching is of extremely important significance.

[0003] Currently, the commonly used UAV image matching is mainly based on image feature matching. Its essence is to identify feature points, feature lines, or feature surfaces, describe them, and then find homologous features by calculating similarity measures. According to the similarities and differences of the scale space methods constructed by local feature matching algorithms, the currently most widely studied and applied feature point extraction and matching algorithms are mainly divided into two categories. The first category uses a Gaussian kernel function to construct a linear scale space, performs Gaussian smoothing on the image, resulting in the loss of contour detail information, blurred boundaries, and reduced stability of feature points at different scales. Typical representatives include the SIFT (scale invariant feature transform) algorithm, SURF (speeded up robust features) algorithm, ORB (oriented FAST and rotated BRIEF) algorithm, Harris algorithm, etc.; the second category is to use the method of nonlinear diffusion filtering to construct a nonlinear scale space. While adaptively filtering out noise and low-frequency information, it retains the key feature information of the image contour, and the matching effect and feature positioning accuracy are significantly improved, but the computational time complexity is relatively high, and the operating efficiency is significantly lower than other algorithms. Typical representatives include the KAZE algorithm and the Accelerated-KAZE (AKAZE) algorithm.

[0004] Given the advantages of the AKAZE algorithm in preserving image contour information and its faster computational speed compared to the SIFT and SURF algorithms, it has been widely studied, improved, and applied to various image matching and registration tasks by scholars. Bo Shan, Zongchun Li, Xiaonan Wang, etc.'s "Accelerated KAZE-SIFT Feature Extraction Algorithm for Oblique Images" constructed the AKAZE-SIFT algorithm, which uses the AKAZE operator for feature detection and combines the SIFT descriptor for feature description. Applied to the feature matching of UAV oblique images, it overcomes the problem of the relatively weak stability of the original descriptor M-LDB of the AKAZE algorithm. Compared with the AKAZE algorithm, both the matching recall rate and precision rate are improved, but the matching time complexity also increases slightly. Xing Changzheng, Sihui Li's "Research on the Matching Algorithm of BOLD Mask Descriptor Based on AKAZE" proposed a matching algorithm of BOLD mask descriptor based on AKAZE, which has a huge improvement in robustness compared to the AKAZE algorithm, but the matching speed decreases slightly. Dan LI, Qiannan XU, Wennian YU, etc.'s "SRP-AKAZE: an improved accelerated KAZE algorithm based on sparse random projection" proposed the SRP-AKAZE algorithm for image matching. On the basis of the AKAZE algorithm completing feature point detection, it further constructs the SRP-SIFT descriptor based on the sparse random projection (SRP) strategy to describe the feature points. This method not only retains the high-efficiency advantage of the AKAZE algorithm in feature detection but also has the recognition ability and stability of the SIFT descriptor. At the same time, the SRP strategy also reduces the dimension of the feature vector, greatly reducing the computational complexity and making the computational efficiency close to that of AKAZE. Yu Xia's "Research on 3D Reconstruction Based on the AKAZE Algorithm" proposed an optimization method for the AKAZE algorithm using the Charbonnier diffusion equation in the step of constructing the nonlinear scale space. In the case of different degrees of noise, scale change, and rotation transformation, the feature points detected after introducing the Charbonnier filter are more accurate, the number of noise points is less, and the average correct matching rate is increased by about 2%. The time spent on image matching is slightly reduced. Shen Xueli, Chen Xintong's "Feature Matching Algorithm of Triplet Descriptor" used the triplet algorithm to establish the binary descriptor LATCH (learned arrangements of three patch codes) and proposed the AKAZE-LATCH algorithm in combination with the AKAZE algorithm for image matching experiments. It was found that this method has a 10% improvement in the matching accuracy rate and a 7% improvement in the running speed compared to the original AKAZE algorithm, and has better robustness and real-time performance.The "Image Stitching Algorithm Improving AKAZE Features and RANSAC" by Wu Lushen and Chen Xiaodu trained a general CNN descriptor on the GL3D dataset by building an L2-Net network structure to replace the traditional M-LDB descriptor. The registration accuracy of this method is higher than that of the traditional AKAZE algorithm, but the calculation time of this descriptor is significantly increased. The "UAV Image Matching Method with Multi-Matching Strategy Fusion" by Li Ruixiang, Zhao Haitao, Ge Xiaosan, etc. uses the RootSIFT descriptor, combines it with the AKAZE algorithm, and further fuses multiple matching strategies such as the ratio of the nearest neighbor distance, bidirectional matching, and cosine similarity constraint for UAV image feature matching, obtaining a relatively high matching accuracy and precision. However, the overall calculation time is increased compared to the AKAZE algorithm.

[0005] When using AKAZE features for image matching, the above several methods for improving AKAZE have improved accuracy, but the overall computational time complexity is high and the timeliness is low. It is still difficult to meet the speed requirements for data automated processing when applied to scenarios such as disaster emergency. Therefore, how to overcome the deficiencies of the existing technology is an urgent problem to be solved in the field of UAV image matching technology. Summary of the Invention

[0006] The purpose of the present invention is to solve the deficiencies of the existing technology, and provide a method for improving AKAZE features using the BEBLI D descriptor for debris flow image matching. The AKAZE feature detection algorithm is combined with the boosted efficient binary local image descriptor (BEBLID) to propose the AKAZE-BEBLLID algorithm, which is applied to the matching of UAV remote sensing images in the debris flow area. Further, a motion smoothing statistical strategy and epipolar constraint are introduced to eliminate false matching points and refine them, and a more accurate matching result can be obtained more quickly.

[0007] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0008] A method for improving AKAZE features in UAV image matching, including the following steps:

[0009] Step (1), detecting and extracting the AKAZE features with sub-pixel positioning accuracy of the left and right images of the stereo image pair as feature points;

[0010] Step (2), constructing a BEBLID binary descriptor for the detected feature points;

[0011] Step (3), using the Hamming distance brute-force matching method to preliminarily match the feature points with the constructed descriptor;

[0012] Step (4), divide the grid based on the GMS method, count the support in the neighborhood of the matching points, eliminate the gross errors, and obtain the initially screened set of matching points;

[0013] Step (5), introduce the epipolar constraint condition, calculate the fundamental matrix by the robust algorithm RANSAC method, further purify the inliers, and then obtain the correct matching result.

[0014] Furthermore, preferably, in step (1), the feature point detection of AKAZE is achieved by finding the local maximum points of the Hessian after normalizing different scales; when finding the extreme points, each pixel point is compared with its upper and lower two layers and its own layer, a total of 26 adjacent points. When it is greater than all 26 adjacent points, it is an extreme point; after finding the feature points, sub-pixel precise positioning is performed. At the feature points, the second-order Taylor expansion formula is used to interpolate to obtain the sub-pixel precise positioning solution as the feature points.

[0015] Furthermore, preferably, in step (2), when constructing the BEBLID binary descriptor for description, the feature extraction function f(x) is defined as shown in formula (8);

[0016]

[0017] where: I(t) is the gray value of pixel t; R(p 1 ,s) is a square block centered at p 1 , with size s; R(p 2 ,s) is a square block centered at p 2 , with size s;

[0018] Given a threshold T, perform a threshold judgment on f(x) to obtain the feature h(x) ≡ h k (x; f, T) in formula (9);

[0019]

[0020] Finally, -1 is converted to 0, and +1 is converted to 1 to obtain the BEBLID binary descriptor.

[0021] Furthermore, preferably, in step (4), divide the grid based on the GMS method, count the support S i in the neighborhood of the matching points. When S i is greater than α, m i is a correct match and is retained. When S i is less than α, m i is an incorrect match and is eliminated;

[0022]

[0023] β is a hyperparameter, and K n represents the total number of neighborhood matching pairs.

[0024] Furthermore, preferably, β takes the value of 8.

[0025] Furthermore, preferably, in step (5), when calculating the fundamental matrix by the robust algorithm RANSAC method, the unknown parameters in the fundamental matrix are calculated using more than 8 pairs of corresponding points; the epipolar constraint condition is that the maximum distance from a point to the epipolar line is set to 2 pixels.

[0026] The present invention also provides a UAV image matching system for improving AKAZE features, including:

[0027] The first processing module is used to detect and extract AKAZE features with sub-pixel positioning accuracy of the left and right images of the stereo image pair as feature points;

[0028] The second processing module is used to construct BEBLID binary descriptors for the detected feature points;

[0029] The preliminary matching module is used to perform preliminary matching on the feature points with constructed descriptors by the Hamming distance brute-force matching method;

[0030] The third processing module is used to divide the grid based on the GMS method, count the support degree within the neighborhood of the matching points, and eliminate gross errors to obtain the initially screened matching point set;

[0031] The final matching module is used to introduce epipolar constraint conditions for limitation, calculate the fundamental matrix by the robust algorithm RANSAC method, further purify the inliers, and then obtain the correct matching result.

[0032] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the UAV image matching method based on the improved AKAZE features as described above are implemented.

[0033] The present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the UAV image matching method for improving AKAZE features as described above are implemented.

[0034] In the present invention, the left and right images are relative concepts. In UAV image shooting, two photos with a certain overlap degree are called 1 stereo image pair, and the two photos are respectively called the left and right images.

[0035] Compared with the prior art, the beneficial effects of the present invention are:

[0036] The terrain in the debris flow distribution area has large undulations, deep ravines, and there are local shadows in the images. When using the AKAZE algorithm for matching in the UAV images of this area, the effect is good but the computational efficiency is relatively low. Aiming at the problems existing in the prior art, the present invention proposes a UAV image feature matching algorithm that combines the AKAZE feature detection algorithm with the binary descriptor BEBLID and integrates the grid motion statistics (GMS) strategy. First, use the AKAZE algorithm to construct a non-linear scale space to detect local stable features, construct the binary descriptor BEBLID, then complete the initial matching with the brute-force matching method, and then use the grid motion statistics (GMS) method for the first rough error rejection. Finally, add the epipolar constraint to further purify the inliers and obtain a reliable matching result. The algorithm of the present invention is experimentally compared with the classical SIFT algorithm, ORB algorithm, AKAZE algorithm, and AKAZE-LATCH algorithm. The results show that the matching accuracy rate of the method of the present invention is not much different from that of the AKAZE algorithm, the matching speed is increased by about 40% compared with the AKAZE algorithm, and the speed is close to that of the ORB algorithm. The number of correct matching point pairs is significantly improved compared with several comparison methods, and the spatial distribution of the matching points is scattered and uniform, with obvious advantages and practicability. Description of the Drawings

[0037] Figure 1 It is a flowchart of image matching;

[0038] Figure 2 It is a schematic diagram of BEBLID descriptor extraction; among them, each pair of solid-line boxes and dashed-line boxes is a window pair of different sizes generated by the weak learning method, f(x) is the feature extraction function, T is the threshold, and h(x) is the feature; the dotted line represents one correct matching pair, the solid line represents the correct matching pairs within the neighborhood of the matching pair, and the dashed line represents the wrong matching pairs.

[0039] Figure 3 It is a schematic diagram of image gridification and matching point pairs;

[0040] Figure 4 It is an experimental data graph; note: the experimental data mainly selects the image pairs with obvious shadow occlusion areas, large terrain undulations, and deep ravines in the captured low-altitude remote sensing image set;

[0041] Figure 5 It is the feature matching result;

[0042] Figure 6 It is the comparison result of matching efficiency;

[0043] Figure 7 It is a schematic diagram of the structure of the UAV image matching system that improves the AKAZE feature of the present invention;

[0044] Figure 8 It is a schematic diagram of the electronic device structure of the present invention. Detailed implementation manners

[0045] The present invention will be further described in detail below in conjunction with embodiments.

[0046] Those skilled in the art will understand that the following embodiments are only used to illustrate the present invention and should not be construed as limiting the scope of the present invention. For those not specified in the embodiments regarding specific techniques or conditions, the techniques or conditions described in the literature in this field or according to the product specifications shall be followed. For those materials or equipment without indicating the manufacturer, they are all conventional products that can be obtained through purchase.

[0047] Embodiment 1

[0048] An improved AKAZE feature-based UAV image matching method includes the following steps:

[0049] Step (1), detecting and extracting the AKAZE features with sub-pixel level positioning accuracy of the left and right images of the stereo image pair as feature points;

[0050] Step (2), constructing a BEBLID binary descriptor for the detected feature points;

[0051] Step (3), performing preliminary matching on the feature points with constructed descriptors by the Hamming distance brute-force matching method;

[0052] Step (4), dividing grids based on the GMS method, counting the support degrees within the neighborhoods of the matching points, and removing outliers to obtain a set of initially screened matching points;

[0053] Step (5), introducing epipolar constraint conditions, calculating the fundamental matrix by the robust algorithm RANSAC method, further purifying the inliers, and then obtaining the correct matching result.

[0054] Embodiment 2

[0055] An improved AKAZE feature-based UAV image matching method includes the following steps:

[0056] Step (1), detecting and extracting the AKAZE features with sub-pixel level positioning accuracy of the left and right images of the stereo image pair as feature points;

[0057] Step (2), constructing a BEBLID binary descriptor for the detected feature points;

[0058] Step (3), performing preliminary matching on the feature points with constructed descriptors by the Hamming distance brute-force matching method;

[0059] Step (4), dividing grids based on the GMS method, counting the support degrees within the neighborhoods of the matching points, and removing outliers to obtain a set of initially screened matching points;

[0060] Step (5), introduce the epipolar constraint condition limit, calculate the fundamental matrix by the robust algorithm RANSAC method, further purify the inliers, and then obtain the correct matching result.

[0061] In step (1), the feature point detection of AKAZE is achieved by finding the local maximum points of the Hessian after normalization at different scales; when finding the extreme points, each pixel point is compared with 26 adjacent points in its upper and lower two layers and its own layer. When it is greater than all 26 adjacent points, it is an extreme point; after finding the feature points, sub-pixel precise positioning is performed. At the feature point, a second-order Taylor expansion is used, and the sub-pixel precise positioning solution is interpolated as the feature point.

[0062] In step (2), when constructing the BEBLID binary descriptor for description, the feature extraction function f(x) is defined as shown in formula (8);

[0063]

[0064] In the formula: I(t) is the gray value of pixel t; R(p 1 ,s) is a square block centered at p 1 , with size s; R(p 2 ,s) is a square block centered at p 2 , with size s;

[0065] Given the threshold T, perform a threshold judgment on f(x) to obtain the feature h(x) ≡ h k (x; f, T) in formula (9);

[0066]

[0067] Finally, -1 is converted to 0, and +1 is converted to 1 to obtain the BEBLID binary descriptor.

[0068] In step (4), based on the GMS method, divide the grid and count the support degree S i in the neighborhood of the matching points. When S i is greater than α, m i is a correct match and is retained. When S i is less than α, m i is an incorrect match and is excluded;

[0069]

[0070] β is a hyperparameter, and K n represents the total number of matching pairs in the neighborhood.

[0071] Example 3

[0072] An improved method for UAV image matching of AKAZE features, comprising the following steps:

[0073] Step (1), detecting and extracting AKAZE features with sub-pixel level positioning accuracy of the left and right images of the stereo image pair as feature points;

[0074] Step (2), constructing a BEBLID binary descriptor for the detected feature points;

[0075] Step (3), using the Hamming distance brute force matching method to preliminarily match the feature points with the constructed descriptors;

[0076] Step (4), dividing the grid based on the GMS method, counting the support degree within the neighborhood of the matching points, and removing the gross errors to obtain the initially screened set of matching points;

[0077] Step (5), introducing the epipolar constraint condition, calculating the fundamental matrix through the robust algorithm RANSAC method, further purifying the inliers, and then obtaining the correct matching result.

[0078] In step (1), the detection of AKAZE feature points is achieved by finding the local maximum points of the Hessian after normalization at different scales; when finding the extreme points, each pixel point is compared with its upper and lower two layers and its own layer, a total of 26 adjacent points. When it is greater than all 26 adjacent points, it is an extreme point; after finding the feature points, sub-pixel precise positioning is performed. At the feature points, a second-order Taylor expansion formula is used to interpolate to obtain the sub-pixel precise positioning solution as the feature points.

[0079] In step (2), when constructing the BEBLID binary descriptor for description, the feature extraction function f(x) is defined as shown in formula (8);

[0080]

[0081] Where: I(t) is the gray value of pixel t; R(p 1 ,s) is a square block with the center at p 1 , and the size of s; R(p 2 ,s) is a square block with the center at p 2 , and the size of s;

[0082] Given the threshold T, a threshold judgment is made on f(x) to obtain the feature h(x)≡h k (x; f, T) in formula (9);

[0083]

[0084] Finally, -1 is converted to 0, and +1 is converted to 1 to obtain the BEBLID binary descriptor.

[0085] In step (4), grids are divided based on the GMS method, and the support degree S within the neighborhood of the matching points is statistically calculated. i , when S i is greater than α, m i is a correct match and is retained; when S i is less than α, m i is an incorrect match and is excluded.

[0086]

[0087] β is a hyperparameter, and K n represents the total number of matching pairs in the neighborhood.

[0088] The value of β is 8.

[0089] In step (5), when calculating the fundamental matrix by the robust algorithm RANSAC method, the unknown parameters in the fundamental matrix are calculated using more than 8 pairs of corresponding points; the epipolar constraint condition is that the maximum distance from a point to the epipolar line is set to 2 pixels.

[0090] Embodiment 4

[0091] As Figure 7 shown, the UAV image matching system with improved AKAZE features includes:

[0092] The first processing module 101 is used to detect and extract the AKAZE features of the sub-pixel level positioning accuracy of the left and right images of the stereo image pair as feature points.

[0093] The second processing module 102 is used to construct BEBLID binary descriptors for the detected feature points.

[0094] The preliminary matching module 103 is used to perform preliminary matching on the feature points with constructed descriptors by the Hamming distance brute-force matching method.

[0095] The third processing module 104 is used to divide grids based on the GMS method, statistically calculate the support degree within the neighborhood of the matching points, eliminate gross errors, and obtain the initial screening matching point set.

[0096] The final matching module 105 is used to introduce epipolar constraint conditions for limitation, calculate the fundamental matrix by the robust algorithm RANSAC method, further purify the inliers, and then obtain the correct matching result.

[0097] In an embodiment of the present invention, the first processing module 101 detects and extracts AKAZE features with sub-pixel positioning accuracy of the left and right images of the stereo image pair as feature points; the second processing module 102 constructs BEBLID binary descriptors for the detected feature points; the preliminary matching module 103 performs preliminary matching on the feature points with constructed descriptors by using the Hamming distance brute-force matching method; the third processing module 104 divides the grid based on the GMS method, counts the support degree within the neighborhood of the matching points, eliminates outliers, and obtains a set of initially screened matching points; the final matching module 105 introduces the epipolar constraint condition limit, calculates the fundamental matrix through the robust algorithm RANSAC method, further purifies the inliers, and then obtains the correct matching result.

[0098] An improved AKAZE feature-based UAV image matching system provided by an embodiment of the present invention is applied to the matching of UAV remote sensing images in debris flow areas. The matching speed is increased by about 40% compared with the AKAZE algorithm, and is close to the speed of the ORB algorithm. The number of correct matching point pairs is significantly increased compared with several comparison methods, and the spatial distribution of the matching points is scattered and uniform, having obvious advantages and practicability.

[0099] The system provided by the embodiment of the present invention is used to execute the above method embodiments. For the specific process and detailed content, please refer to the above embodiments and will not be elaborated here.

[0100] Embodiment 5

[0101] Figure 8 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. Referring to Figure 8 , the electronic device may include: a processor 201, a communication interface 202, a memory 203, and a communication bus 204. Among them, the processor 201, the communication interface 202, and the memory 203 complete mutual communication through the communication bus 204. The processor 201 can call the logical instructions in the memory 203 to execute the following methods: detecting and extracting AKAZE features with sub-pixel positioning accuracy of the left and right images of the stereo image pair as feature points; constructing BEBLID binary descriptors for the detected feature points; performing preliminary matching on the feature points with constructed descriptors by using the Hamming distance brute-force matching method; dividing the grid based on the GMS method, counting the support degree within the neighborhood of the matching points, eliminating outliers, and obtaining a set of initially screened matching points; introducing the epipolar constraint condition limit, calculating the fundamental matrix through the robust algorithm RANSAC method, further purifying the inliers, and then obtaining the correct matching result.

[0102] In addition, when the logical instructions in the above-mentioned memory 203 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, external hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0103] Embodiment 6

[0104] On the other hand, the embodiments of the present invention also provide a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is used to implement the method for matching UAV images with improved AKAZE features provided in the above-mentioned various embodiments. For example, it includes: detecting and extracting AKAZE features with sub-pixel positioning accuracy of the left and right images of a stereo image pair as feature points; constructing BEBLID binary descriptors for the detected feature points; performing preliminary matching on the feature points with constructed descriptors by the Hamming distance brute-force matching method; dividing grids based on the GMS method, counting the support degrees within the neighborhoods of the matching points, removing gross errors, and obtaining a set of initially screened matching points; introducing epipolar constraint conditions for limitation, calculating the fundamental matrix through the robust algorithm RANSAC method, further purifying inliers, and then obtaining the correct matching result.

[0105] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative efforts.

[0106] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the prior art can be embodied in the form of a software product, and this computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disc, etc., including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0107] Embodiment 7

[0108] 1. Image matching

[0109] The overall flow of this algorithm is as Figure 1 shown.

[0110] The basic idea is as follows:

[0111] 1) Detect and extract AKAZE features with sub-pixel level positioning accuracy for left and right images;

[0112] 2) Construct BEBLID binary descriptors for the detected feature points for description;

[0113] 3) Complete the initial matching by the Hamming distance brute-force matching method;

[0114] 4) Divide the grid based on the GMS method, count the support degree within the neighborhood of the matching points, eliminate outliers, and obtain the initial screening matching point set;

[0115] 5) Introduce the epipolar constraint condition limit, calculate the fundamental matrix through the robust algorithm RANSAC method, and further purify the inliers;

[0116] 6) Obtain the correct matching result.

[0117] 1.1 AKAZE feature detection

[0118] The AKAZE feature detection algorithm uses a non-linear diffusion filtering method to construct a scale space, which can better retain the edge information of the image. Specifically, it can be described by a non-linear partial differential equation, as shown in Equation (1).

[0119]

[0120] In the formula: L is the image brightness; is the gradient; div is the divergence; (x, y) are the image coordinates; c(x, y, t) is the conduction function that determines the diffusion mode of the image. By setting an appropriate conduction function c(x, y, t), the diffusion can be made to adapt to the local structure of the image. The time t serves as a scale parameter, and the larger its value, the simpler the representation form of the image. The construction method of the conduction function is shown in Equation (2).

[0121]

[0122] In the formula: is the image L after Gaussian smoothing σ gradient. The function g() in the AKAZE algorithm is shown in Equation 3.

[0123]

[0124] In the formula: The parameter λ is a contrast factor that controls the diffusion level and can determine how much edge information is retained. It preferentially retains regions with larger widths. The larger its value, the less edge information is retained. In the AKAZE algorithm, the value of the parameter λ is the value at the 70th percentile of the histogram of the gradient image

[0125] The construction of the non-linear scale space of AKAZE features is similar to that of the SIFT algorithm. The scale levels increase logarithmically. There are a total of O groups of pyramid images, and each group has S layers. Different from downsampling layer by layer in each group in SIFT, each level in AKAZE uses the same resolution as the original image. Different groups and layers are marked by the serial numbers o and s respectively, and correspond to the scale parameter σ through Equation (4):

[0126]

[0127] In the formula: σ i is the scale parameter, o is the group, and s is the layer. The non-linear diffusion filtering model is in units of time t. Therefore, the scale parameter σ in pixels needs to be i converted to the time unit. In the Gaussian scale space, convolving the image with a Gaussian kernel with a standard deviation of σ is equivalent to filtering the image for a duration t i = σ i 2 / 2, and t i is called the evolution time. As the scale level increases, except for the image edge pixels corresponding to the target contour, the conduction function values corresponding to most pixels will tend to a constant value.

[0128] ​The feature point detection of AKAZE is achieved by finding the local maxima of the Hessian after normalization at different scales. The calculation of the Hessian matrix is shown in Equation (5).

[0129]

[0130] In the formula: σ i,norm is the normalization scale factor; and are the second-order horizontal and vertical differentials respectively; is the second-order cross differential. When finding the extreme points, each pixel point is compared with its upper and lower two layers and its own layer, a total of 26 adjacent points. When it is greater than all 26 adjacent points, it is the extreme point.

[0131] After finding the feature points, sub-pixel precise positioning is carried out. At the feature points, the second-order Taylor expansion formula (Equation (6)) is used to interpolate to obtain the sub-pixel precise positioning solution (Equation (7)).

[0132]

[0133]

[0134] In the formula: X = (x, y, σ) is the offset of the key point; L is the value of L(x, y, σ) at the key point; is the extreme value of X.

[0135] 1.2 Construction of BEBLID Feature Descriptor

[0136] The Boosted Effective Binary Local Image Descriptor (BEBLID) is a very efficient descriptor. Its execution time is close to ORB, its accuracy is similar to SIFT, and it has certain affine invariance. This algorithm uses the AdaBoost algorithm to help select weak learners (WL), selects K features. The features use the integral image to calculate the difference between the average gray values of a pair of image square regions, and combines them to generate a strong description, outputting local description information with a central region close to the Gaussian distribution. The schematic diagram of BEBLID descriptor extraction is as Figure 2 shown. The solid-line box and the dashed-line box represent square blocks centered at p with size s, denoted by R(p 1 , s) and R(p 2 , s) respectively. The feature extraction function f(x) calculates the difference between the average gray values of the pixels inside the solid-line box and the dashed-line box. Specifically, the feature extraction function f(x) is defined as shown in Equation (8).

[0137]

[0138] Where: I(t) is the grayscale value of pixel t; R(p, s) is a square block centered at p with size s. Given a threshold T, a threshold judgment is made on f(x) to obtain the feature h(x) in formula (9): h(x) ≡ h k (x; f, T)

[0139]

[0140] Finally, in order to make the output {0, 1}, -1 was converted to 0 and +1 was converted to 1 (as Figure 2 ), to obtain the Boosted Effective Binary Local Image Descriptor (BEBLID).

[0141] 1.3 Feature Matching and Gross Error Rejection

[0142] During feature matching, the similarity of feature descriptors is compared by calculating the Hamming distance, and a brute-force match is used to obtain a preliminary matching result. Since the ability to distinguish true and false matches based on neighborhood support increases with the number of feature matching pairs, no other screening is done here, and as many matching point pairs as possible should be retained, and then the gross error rejection is completed using the grid motion statistics method.

[0143] The Grid Motion Statistics (GMS) algorithm believes that the lack of sufficient correct matches is not because the number of matching pairs is too small, but because it is difficult to distinguish correct and incorrect matches. The basic idea of this algorithm is: the smoothness of motion causes the feature points in the neighborhood of correct matches to often maintain geometric consistency, while there is often no such consistency relationship between incorrect matching pairs, that is, there should be multiple matching pairs supporting a correct matching pair in the neighborhood of two images, while there are few or no matching pairs supporting an incorrect matching pair in its neighborhood. Therefore, by evaluating the number of matching pairs contained in the neighborhood of the feature point pairs to be matched, correct and incorrect matches can be distinguished.

[0144] First, divide the images I a , I b into non-overlapping grids, Figure 3 is the feature distribution map of correct and incorrect matches, I a is the matching image, and I b is the image to be matched. Among them, the 3×3 bold grid represents the neighborhood of grids a 5 , b 5 , and the matching pair is represented as m i .

[0145] Then calculate the neighborhood support degree S i of the correct matching pair m i , as shown in formula (10).

[0146]

[0147] Where: grid a 5 and b 5 The number of matching pairs in the 9 grids of the neighborhood is respectively denoted as x i (i = 1, 2, …, 9), subtracting 1 means subtracting the matching pair itself from the total number. Since the matching point pairs are independently distributed, S i Belongs to the binomial distribution, and the distribution of the matching point pairs can be expressed as Equation (11).

[0148]

[0149] Where: S i Represents the neighborhood support degree, B represents the binomial distribution, m i Represents the matching point pairs of the matching area and the area to be matched, K n Represents the total number of matching pairs in the neighborhood, p t Represents the correct matching rate, p f Represents the wrong matching rate.

[0150] According to Equation (11), S i Overall shows a bimodal distribution, and the formulas for the mean E and standard deviation δ of the matching point pair distribution are as follows respectively:

[0151] 1) When m i Is correctly matched, E t (m i ) = K n p t ,

[0152] 2) When m i Is wrongly matched, E f (m i ) = K n p f ,

[0153] Thus, the probability evaluation standard function can be obtained, as shown in Equation (12).

[0154]

[0155] Where: P is the matching probability, it can be seen that Indicates that the more image feature points are extracted, the larger the obtained P value is, and the higher the matching accuracy is. Therefore, the experiment increases the number of detection points to improve the matching probability.

[0156] Furthermore, by setting a suitable threshold, it can effectively judge whether the matching is correct, as shown in Equation (13).

[0157]

[0158] Where: β is a hyperparameter for adjusting the threshold, and the empirical value is selected as 4 - 6. When S i is greater than α, m i is a correct match and is retained. When S i is less than α, m i is an incorrect match and is deleted. In the present invention, since the image itself is large in size, after being divided into a 20×20 grid, the number of matching pairs falling in each grid is considerable, and there are a large number of repeated textures in the natural landform area. To eliminate the incorrect matching pairs brought by the repeated textures, the experimental hyperparameter is set to a more stringent 8. This is because the image itself is large in size. After being divided into a 20×20 grid, the number of matching pairs falling in each grid is considerable, and there are a large number of repeated textures in the natural landform area. The original empirical parameters 4 - 6 are not sufficient to completely eliminate the incorrect matching pairs. To better eliminate the incorrect matching pairs, it is found through experimental parameter adjustment that setting the hyperparameter to 8 can achieve a better effect.

[0159] 1.4 Refinement of Epipolar Constraint Inliers

[0160] The matching point pairs obtained based on the Grid-based Motion Statistics (GMS) strategy may still have incorrect matching pairs, and the matching accuracy needs to be further improved. According to the epipolar geometric relationship in photogrammetry, in stereo matching, for any point on an epipolar line, its corresponding image point on the other image must be located on its corresponding epipolar line. This geometric relationship is expressed by the fundamental matrix (or basic matrix), and the corresponding unknown parameters can be calculated through more than 8 pairs of corresponding points.

[0161] Specifically, on the reference image and the image to be matched, when the distance between the feature vectors of two feature points p and p′ is very small, it is determined that these two feature points correspond to the same scene and are considered as corresponding point pairs. The coordinates of the point pairs are p(x 1 ,y 1 ) and p′(x 2 ,y 2 ). The theoretical position of the feature point (x 1 ,y 1 ) on the reference image mapped to the image to be matched by the fundamental matrix F is which can be expressed as:

[0162] p′ T Fp = 0 (14)

[0163]

[0164] Here, the fundamental matrix is calculated through the RANSAC algorithm, and then all the data are tested using the calculated fundamental matrix to eliminate the point pairs that do not satisfy the epipolar constraint. The feature point (x 1 ,y 1)The maximum distance from the point mapped to the image to be matched by the fundamental matrix F to the corresponding epipolar line is set to 2 pixels. That is, the points with a distance less than or equal to 2 pixels are inliers and are retained, while the points with a distance exceeding 2 pixels are outliers and are deleted, so as to achieve the purpose of purifying the inliers and improving the reliability of the matching.

[0165] The present invention is an unmanned aerial vehicle (UAV) image feature matching algorithm that combines the AKAZE feature detection algorithm with the binary descriptor BEBLID and integrates the grid motion statistics (GMS) strategy. First, the AKAZE algorithm is used to construct a non-linear scale space to detect local stable features. Instead of constructing the original descriptor M-LDB for the detected feature points, the binary descriptor BEBLID is constructed. Then, the brute-force matching method is used to complete the preliminary matching, and the grid motion statistics (GMS) method is used for the first gross error rejection. Finally, the epipolar constraint is added to further purify the inliers and obtain a reliable matching result. The matching accuracy rate of the method of the present invention is close to that of the AKAZE algorithm, the matching speed is increased by about 40% compared with the AKAZE algorithm, the problem of the slow calculation speed of the original descriptor M-LDB of the AKAZE algorithm is overcome, the number of correct matching point pairs is also significantly increased, and the spatial distribution of the matching points is scattered and uniform, having obvious advantages and practicability.

[0166] Application Example

[0167] To verify the feasibility and effectiveness of the method of the present invention, 754 images of debris flow mountainous areas were taken by a DJI Phantom 4 UAV for the experimental data, with a size of 5472 pixels × 3648 pixels, as Figure 4 shown. The present invention uses 5 algorithms to perform matching experiments on 6 groups of typical debris flow natural landform coverage images selected, and the number of inliers, inlier rate, and matching time are selected to quantitatively evaluate the matching effect. All experiments are completed in the hardware environment of an Intel(R) Core(TM) i5-9400H CPU - 8 Core Processor 2.5 GHz, 40 GB RAM, and in the software environment of Windows 10 PyCharm Community Edition 2020 based on the OpenCV 4.5.1 computer vision library of Python 3.6.

[0168] For each group of images, the SIFT, ORB, and AKAZE key-point detection algorithms are used to extract feature points from the left and right images respectively. The specific number of extracted feature points is shown in Table 1. To ensure the fairness of the matching running time of each method in the experiment, the number of feature points extracted from each pair of images is set to be as close as possible. Taking the number of feature points extracted by the AKAZE method as a reference, the number of feature points extracted by the SIFT and ORB algorithms is set to be as close as possible to the number of feature points of the AKAZE algorithm. The initially extracted feature points of the present invention are subjected to brute-force matching without any initial screening. Therefore, the number of homologous points obtained in the initial matching is the number of feature points extracted from the image with fewer feature points among the two images. The points obtained by GMS are the rough matching point numbers of the method of the present invention in Table 2, and the rest are the excluded outlier points. Therefore, the more feature points retained after matching each pair of images in the present invention, the fewer points are excluded. Therefore, the situation of outlier exclusion is not discussed further in the text. Figure 5 are the matching results. It can be seen from the figure that when using the algorithm of the present invention for feature matching, more correct matching point pairs are obtained than those obtained by the SIFT, ORB, AKAZE, and the method in the literature (the feature matching algorithm of the triple descriptor by Shen Xueli and Chen Xintong). The spatial distribution of the matching points is scattered and uniform, which is very beneficial to structure from motion (SFM) and 3D scene reconstruction.

[0169] Table 1 Number of detected feature points in each image (unit: piece)

[0170]

[0171]

[0172] Table 2 Number of rough matching and fine matching point pairs (unit: pair)

[0173]

[0174] In Table 2, the rough matching is the number of matching point pairs after the rough error elimination of six groups of image pairs, and the fine matching is the number of matching points after the inlier purification. Statistical results Figure 6For the comparison of the matching efficiencies of various algorithms, they are the matching accuracy rate and the matching running time respectively. It can be found from the statistical results that: the method proposed in the present invention obtains the largest number of correct matching points in the debris flow area image experiment. As can be seen from Table 1, among the comparison methods, the classical AKAZE algorithm obtains the largest number of correct matching points compared with other methods. However, except for being slightly less than the AKAZE algorithm in Group B, the method of the present invention has a larger number than the AKAZE algorithm in other groups of data, especially with a larger increase in Groups A, D, E, and F. It can be seen that the BEBLID descriptor used in the method of the present invention can better describe the detected feature points than the original M-LDB descriptor of AKAZE and the LATCH descriptor used in the literature method. The AKAZE-LATCH algorithm proposed in the literature method also has good applicability in the natural disaster images of the present invention, with a relatively high matching accuracy rate, but the overall number of correct matching points obtained is not as many as that of the method of the present invention. The running speed of the method of the present invention is increased by nearly 20%-33% compared with the AKAZE-LATCH algorithm.

[0175] The efficiency comparison of five image matching methods for six groups of data is shown in Figure 6 . From Figure 6 it can be found that: the matching accuracy rate of the method of the present invention is basically equivalent to that of several comparison methods and is relatively stable; the time used by the method of the present invention is less than half of that of SIFT, is also significantly faster than the AKAZE algorithm, is slightly faster than the literature method, and is close to the ORB algorithm. The running speed of the method of the present invention is increased by nearly 40% compared with the AKAZE algorithm because the BEBLID descriptor used is a descriptor with a faster calculation speed than the original descriptor M-LDB of the AKAZE algorithm; the running speed of the method of the present invention is increased by about 20%-33% compared with the literature method, indicating that the BEBLID descriptor has a faster operation speed than the LATCH descriptor; the method of the present invention is slightly slower than the ORB algorithm because the AKAZE feature detection algorithm adopted by the algorithm of the present invention takes more time for feature detection than the ORB algorithm.

[0176] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. What is described in the above embodiments and the specification only illustrates the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. An improved AKAZE feature-based UAV image matching method, characterized in that, it includes the following steps: Step (1), detecting and extracting AKAZE features with sub-pixel positioning accuracy of the left and right images of the stereo image pair as feature points; Step (2), constructing a BEBLID binary descriptor for the detected feature points; Step (3), performing preliminary matching on the feature points with the constructed descriptor by the Hamming distance brute-force matching method; Step (4), dividing the grid based on the GMS method, counting the support degree within the neighborhood of the matching points, and removing outliers to obtain the initially screened matching point set; Step (5), introducing the epipolar constraint condition limit, calculating the fundamental matrix by the robust algorithm RANSAC method, further purifying the inliers, and then obtaining the correct matching result; In step (2), when constructing the BEBLID binary descriptor for description, the feature extraction function f(x) used is defined as shown in equation (8); Where: I(t) is the gray value of pixel t; R(p 1 , s) is a square block centered at p 1 , with size s; R(p 2 , s) is a square block centered at p 2 , with size s; Given a threshold T, a threshold judgment is made on f(x) to obtain the feature h(x) ≡ h in formula (9) k (x; f, T); Finally, -1 is converted to 0, and +1 is converted to 1 to obtain the BEBLID binary descriptor; In step (4), the grid is divided based on the GMS method, and the support S within the neighborhood of the matching points is statistically calculated. i , when S i is greater than α, m i is a correct match and is retained. When S i is less than α, m i is an incorrect match and is excluded. β is a hyperparameter, and K n represents the total number of neighborhood matching pairs.

2. The improved AKAZE feature-based UAV image matching method according to claim 1, characterized in that, In step (1), the detection of AKAZE feature points is realized by finding the local maximum points of the Hessian after normalization at different scales; when finding the extreme points, each pixel point is compared with its upper and lower 2 layers and its own layer, a total of 26 adjacent points. When it is greater than all 26 adjacent points, it is the extreme point; after finding the feature points, sub-pixel precise positioning is carried out; at the feature point, a second-order Taylor expansion formula is used to interpolate to obtain the sub-pixel precise positioning solution as the feature point.

3. The improved AKAZE feature-based UAV image matching method according to claim 1, characterized in that, The value of β is 8.

4. The improved AKAZE feature-based UAV image matching method according to claim 1, characterized in that, In step (5), when calculating the fundamental matrix by the robust algorithm RANSAC method, the unknown parameters in the fundamental matrix are calculated by using more than 8 pairs of corresponding points; the epipolar constraint condition is that the maximum distance from the point to the epipolar line is set to 2 pixels.

5. An improved AKAZE feature-based UAV image matching system, characterized in that, it includes: The first processing module is used to detect and extract AKAZE features with sub-pixel positioning accuracy of the left and right images of the stereo image pair as feature points; The second processing module is used to construct a BEBLID binary descriptor for the detected feature points; The preliminary matching module is used to perform preliminary matching on the feature points with the constructed descriptor by the Hamming distance brute-force matching method; The third processing module is used to divide the grid based on the GMS method, count the support degree within the neighborhood of the matching points, and remove outliers to obtain the initially screened matching point set; The final matching module is used to introduce the epipolar constraint condition limit, calculate the fundamental matrix by the robust algorithm RANSAC method, further purify the inliers, and then obtain the correct matching result; When constructing the BEBLID binary descriptor for description, the feature extraction function f(x) used is defined as shown in equation (8); Where: I(t) is the gray value of pixel t; R(p 1 , s) is a square block centered at p 1 , with size s; R(p 2 , s) is a square block centered at p 2 , with size s; Given a threshold T, a threshold judgment is made on f(x) to obtain the feature h(x) ≡ h in formula (9) k (x; f, T); Finally, -1 was converted to 0 and +1 was converted to 1, obtaining the BEBLID binary descriptor; Divide the grid based on the GMS method and count the support S within the neighborhood of the matching points i , when S i is greater than α, m i is a correct match and is retained. When S i is less than α, m i is an incorrect match and is excluded; β is a hyperparameter, and K n represents the total number of neighborhood matching pairs.

6. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, when the processor executes the program, the steps of the UAV image matching method for improving the AKAZE feature according to any one of claims 1 to 5 are implemented.

7. A non-transitory computer-readable storage medium, on which a computer program is stored, characterized in that, when the computer program is executed by a processor, the steps of the UAV image matching method for improving the AKAZE feature according to any one of claims 1 to 5 are implemented.

Images