Image matching enhancement method based on feature sparse region detection

Abstract

The present application relates to a kind of image matching enhancement methods based on feature sparse area detection, belong to multi-source image matching technical field.The method is:1) obtain the pair of to-be-matched heterogeneous images, carry out feature extraction and feature matching, obtain initial matching result;2) in the pair of to-be-matched heterogeneous images, select any image, carry out feature sparse area detection to the selected image, obtain each feature sparse area of the selected image, according to initial matching result, solve the affine transformation model of the pair of to-be-matched heterogeneous images;According to affine transformation model, obtain the corresponding feature sparse area of each group;3) carry out feature extraction to each group of feature sparse area, according to the corresponding feature screening threshold set, carry out feature screening to each group of feature sparse area, obtain local feature;Local feature is aggregated, according to the feature after aggregation, carry out feature matching, obtain the matching enhancement result of feature sparse area.The present application can obtain sufficient, evenly distributed robust matching result.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of multi-source image matching, and particularly relates to an image matching enhancement method based on feature sparse region detection. BACKGROUND

[0002] With the diversification of remote sensing imaging modes, multi-sensor, multi-resolution and multi-temporal remote sensing image data have emerged in an explosive manner, and the demand for joint processing of heterogeneous remote sensing images and mining of their complementary information has also increased. Among them, the joint processing of unmanned aerial vehicle images and satellite images is a major research hotspot in recent years. Satellite reference images with accurate geographic location information can provide reliable reference information for target recognition and positioning of unmanned aerial vehicle images, three-dimensional modeling, disaster assessment and land resource change. To realize the joint processing of unmanned aerial vehicle images and satellite images, high-precision matching control points are required. The matching control points not only require accurate positioning, but also require sufficient quantity and highly uniform distribution. However, due to the differences in imaging mode, imaging time and nonlinear radiation distortion between unmanned aerial vehicle images and satellite images, the matching between the two is difficult to meet the above requirements.

[0003] Remote sensing image matching algorithms have been widely used in the prior art. For example, the SURF (Speed-Up Robust Feature) algorithm, which is based on the classic matching algorithm SIFT (Scale-Invariant Feature Transform), detects feature points by using the determinant value of the Hessian matrix as the response and uses integral images to speed up the operation. The discriminability and computational speed of the SURF algorithm exceed those of the SIFT algorithm. However, traditional handcrafted features represented by the SIFT algorithm and the SURF algorithm cannot adapt to the complex nonlinear radiometric distortion between multi-source remote sensing images. To address this issue, the HAPCG (Histogram of Absolute Phase-Consistent Gradient) algorithm is proposed in the paper “Heterogeneous Image Matching Considering Anisotropic Weighted Power Moments and Absolute Phase Consistent Gradient” (Authors: Yao Yongxiang et al., Journal: Wuhan University Journal, Vol. 46, No. 11, 2021). This algorithm obtains absolute phase direction feature descriptions through the histogram of absolute phase direction gradients and ultimately obtains reliable matching results. The algorithm performs well in multi-source image matching. With the rapid development of deep learning, CNN (Convolutional Neural Network) features have increasingly strong generalization expression capabilities. CNN features are more adaptable to the nonlinear radiometric distortion between multi-source remote sensing images than traditional handcrafted features. Based on CNN features, the DELF (Deep Local Feature) algorithm, the CMM-Net (Cross-Modality Matching Network) algorithm, and the SuperPoint algorithm are proposed in the papers “Large-scale image retrieval with attentive deep local features” (Authors: Noh H et al., Conference: Proceedings of the IEEE International Conference on Computer Vision, pp. 3456-3465, 2017), “Deep learning algorithm for feature matching of heterogeneous remote sensing images” (Authors: Lan Zhaozhen et al., Journal: Acta Geodaetica et Cartographica Sinica, Vol. 50, No. 2, 2021), and “Superpoint: Self-supervised interest point detection and description” (Authors: DeTone D et al., Conference: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 224-236, 2018), respectively. Among them, the SuperPoint algorithm uses a self-supervised training method to supervise the training of the fully convolutional neural network through a key point detector rather than manual annotation. Ultimately, a feature descriptor detection and extraction network with strong multi-view geometric adaptability and good generalization is obtained. Due to its excellent feature extraction response speed, the algorithm is widely used in remote sensing image matching.SuperGlue and LoFTR algorithm are graph neural network (GNN) matching algorithms that have emerged in recent years. Both algorithms focus on learning the affine transformation relationship in stereo image pairs, i.e., the spatial position relationship between feature points, for matching. Among them, the SuperGlue algorithm encodes the position information and feature descriptors of key points through an attention mechanism, fuses the visual context information between feature points through a cross-attention mechanism, so that the same name features have stronger correlation, and finally uses an optimal matching layer to complete feature matching and mismatch elimination at the same time. Compared with the traditional KNN matching algorithm, the SuperGlue algorithm has made great advantages in matching speed and matching quantity, but it relies on a local feature extraction algorithm with excellent performance to extract a sufficient number of local feature points with strong generalization ability.

[0004] The premise of high-quality matching of unmanned aerial vehicle images and satellite images is to obtain a sufficient number of local features with uniform distribution. The SuperGlue matching algorithm uses SuperPoint features to achieve good results in multi-source image matching, but for the problem of ground object differences caused by temporal differences between unmanned aerial vehicle images and satellite images and the problem of feature sparsity caused by the large number of texture sparse areas, the SuperPoint algorithm cannot extract a sufficient number of local features with uniform distribution on the entire image.

[0005] In summary, most of the current local feature extraction and matching algorithms have poor matching ability for unmanned aerial vehicle images and satellite images with large seasonal differences or a large number of texture sparse areas. SUMMARY

[0006] The purpose of the present application is to provide an image matching enhancement method based on feature sparse area detection to solve the problem of poor matching ability of existing local feature extraction and matching algorithms for multi-source remote sensing images with large seasonal differences or a large number of texture sparse areas.

[0007] The technical scheme of the image matching enhancement method based on feature sparse area detection provided by the present application to solve the above technical problems is as follows: the method comprises the following steps:

[0008] 1) Obtain a pair of to-be-matched heterogeneous images, perform feature extraction and feature matching on the pair of to-be-matched heterogeneous images, and obtain an initial matching result of the pair of to-be-matched heterogeneous images;

[0009] 2) selecting an image in the to-be-matched heterogeneous image pair, performing feature sparse region detection on the selected image according to the feature points of the selected image obtained in step 1), obtaining each feature sparse region of the selected image, and calculating an affine transformation model of the to-be-matched heterogeneous image pair according to the initial matching result of the to-be-matched heterogeneous image pair; performing affine transformation on each feature sparse region in the selected image according to the affine transformation model of the to-be-matched heterogeneous image pair, so as to obtain corresponding feature sparse regions of each group;

[0010] 3) performing feature extraction on the corresponding feature sparse regions of each group, setting a corresponding feature screening threshold for each group of corresponding feature sparse regions, screening the features extracted from the corresponding feature sparse regions of each group according to the feature screening threshold, obtaining local features of the corresponding feature sparse regions of each group, aggregating the local features of the corresponding feature sparse regions of each group to obtain global feature sparse region local features of the to-be-matched heterogeneous image pair, and performing feature matching on the global feature sparse region local features of the to-be-matched heterogeneous image pair to obtain a matching enhanced result of the feature sparse regions of the to-be-matched heterogeneous image pair.

[0011] The method of the present application can detect feature sparse regions, guide feature sparse regions to perform feature extraction and adaptive score threshold feature screening one by one, aggregate the reliable local features obtained, and finally obtain a stable matching result with sufficient quantity and uniform distribution through a matching algorithm.

[0012] In order to improve the accuracy of feature sparse region detection, the method for performing feature sparse region detection on the selected image in step 2) is as follows:

[0013] a) dividing the selected image into at least one detection region;

[0014] b) traversing each detection region, storing a detection region that meets a first condition, and performing quadtree splitting on a detection region that meets a second condition to obtain a new detection region; the first condition is that the feature points in the detection region are not greater than a set threshold and the area of the detection region is not less than a minimum detection area; the second condition is that the feature points in the detection region are greater than the set threshold and the area of the detection region is not less than the minimum detection area;

[0015] c) performing step b) each time a new detection region is split out, until none of the new detection regions meets the second condition or the number of splitting times for splitting the new detection region reaches a set splitting number, and storing each detection region as each feature sparse region of the selected image.

[0016] Further, the calculation formula of the set splitting number k is as follows: Wherein, W, H are length and width of the image respectively, S is the minimum detection area, the value of S is obtained according to the test.

[0017] Further, the calculation formula of the feature screening threshold θ of each group of feature sparse regions is as follows: Wherein, C n is the score of the n th local feature in each group of feature sparse regions, N is the number of independently extracted local features in each group of feature sparse regions.

[0018] Further, the process of obtaining the local features of each group of feature sparse regions in step 3) is: first, the local features of each group of feature sparse regions are extracted region by region to obtain the initial local features of each group of feature sparse regions, and then the local feature key points and descriptors with scores greater than the feature screening threshold are screened from the initial local features of each group of feature sparse regions, so as to obtain the local features of each group of feature sparse regions.

[0019] Further, the local features of each group of feature sparse regions are aggregated to obtain the global feature sparse region local features in the following manner:

[0020]

[0021] Wherein, Q represents the number of feature sparse regions, p q , c q and e q represent the position, score and descriptor of the local feature screened in the q th group of feature sparse regions, wherein q = 1, 2, 3, …, Q.

[0022] Considering that there may be certain errors in the calculation of the corresponding feature sparse region by means of the affine transformation model obtained from the initial matching result, the method of detecting the feature sparse region of the selected image in step 2) further includes step d) diffusing the area of each detected region to the surrounding according to the corresponding set distance, and taking each detected region after area diffusion as each feature sparse region of the selected image. Through adaptive area enlargement, the influence of the above-mentioned errors can be avoided, and the accuracy of feature sparse region detection is further improved.

[0023] Further, when the selected image is divided into a detection region, the area calculation formula of each new detection region after each split is as follows: Wherein, W, H are length and width of the selected image, k is the current split number of the selected image.

[0024] Further, in step 1), a SuperPoint feature extraction network model is used to extract features of the to-be-matched heterogeneous image pair, to obtain initial local features of the to-be-matched heterogeneous image pair; a SuperGlue network model is used to perform feature matching on the initial local features of the to-be-matched heterogeneous image pair, to obtain an initial matching result of the to-be-matched heterogeneous image pair.

[0025] Further, in step 3), a SuperPoint feature extraction network model is used to extract local features of each group of feature sparse regions, and a SuperGlue network model is used to perform feature matching on the local features of the global feature sparse region. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is a flowchart of a feature sparse region detection and matching enhancement algorithm according to an embodiment of the present application;

[0027] Figure 2 is a structural schematic diagram of a SuperPoint feature extraction network model according to an embodiment of the present application;

[0028] Figure 3 is a structural schematic diagram of a SuperGlue network model according to an embodiment of the present application;

[0029] Fig. 4(a) is a schematic diagram of the first time splitting of a detection region in a feature sparse region detection algorithm according to an embodiment of the present application; D is a schematic diagram of the first time splitting of a detection region in a feature sparse region detection algorithm according to an embodiment of the present application;

[0030] Fig. 4(b) is a schematic diagram of the first time splitting of a detection region in a feature sparse region detection algorithm according to an embodiment of the present application;

[0031] Fig. 4(c) is a schematic diagram of the kth time splitting of a detection region in a feature sparse region detection algorithm according to an embodiment of the present application;

[0032] Fig. 4(d) is a schematic diagram of the stopping of splitting of a detection region in a feature sparse region detection algorithm according to an embodiment of the present application;

[0033] Figure 5 is a flowchart of an adaptive threshold feature screening algorithm according to an embodiment of the present application;

[0034] Fig. 6(a) is a first group of unmanned aerial vehicle images and satellite remote sensing images according to an embodiment of the present application;

[0035] Fig. 6(b) is a second group of unmanned aerial vehicle images and satellite remote sensing images according to an embodiment of the present application;

[0036] Fig. 6(c) is a third group of unmanned aerial vehicle images and satellite remote sensing images according to an embodiment of the present application;

[0037] Fig. 6(d) is a fourth group of unmanned aerial vehicle images and satellite remote sensing images according to an embodiment of the present application;

[0038] Figure 7 This is a schematic diagram illustrating the different minimum detection area matching enhancement quantities for four pairs of UAV images and satellite remote sensing images according to an embodiment of the present invention;

[0039] Figure 8(a) is a schematic diagram of the initial matching results of the first set of UAV images and satellite remote sensing images in an embodiment of the present invention;

[0040] Figure 8(b) is a schematic diagram of the initial matching results of the second set of UAV images and satellite remote sensing images in an embodiment of the present invention;

[0041] Figure 8(c) is a schematic diagram of the initial matching results of the third set of UAV images and satellite remote sensing images in an embodiment of the present invention;

[0042] Figure 8(d) is a schematic diagram of the initial matching results of the fourth set of UAV images and satellite remote sensing images in an embodiment of the present invention;

[0043] Figure 9(a) is a schematic diagram of the feature sparse region detection results of the first set of UAV images and satellite remote sensing images in an embodiment of the present invention;

[0044] Figure 9(b) is a schematic diagram of the characteristic sparse region detection results of the second set of UAV images and satellite remote sensing images in an embodiment of the present invention;

[0045] Figure 9(c) is a schematic diagram of the feature sparse region detection results of the third set of UAV images and satellite remote sensing images in an embodiment of the present invention;

[0046] Figure 9(d) is a schematic diagram of the feature sparse region detection results of the fourth set of UAV images and satellite remote sensing images in an embodiment of the present invention;

[0047] Figure 10(a) is a schematic diagram of the similarity measurement results of the first pair of sparse feature regions in an embodiment of the present invention;

[0048] Figure 10(b) is a schematic diagram of the similarity measurement results of the second pair of feature sparse regions in an embodiment of the present invention;

[0049] Figure 10(c) is a schematic diagram of the third set of similarity measurement results of sparse regions of features in an embodiment of the present invention;

[0050] Figure 10(d) is a schematic diagram of the fourth set of similarity measurement results of sparse feature regions in the embodiment of the present invention;

[0051] Figure 11(a) is a schematic diagram of the matching results of the first group of UAV images and satellite images using the SD-ME algorithm in an embodiment of the present invention;

[0052] Figure 11(b) is a schematic diagram of the matching results between the second group of UAV images and satellite images using the SD-ME algorithm in an embodiment of the present invention;

[0053] Fig. 11(c) is a schematic diagram of the matching results of the third group of UAV images and satellite images in the SD-ME algorithm of the embodiment of the present application;

[0054] Fig. 11(d) is a schematic diagram of the matching results of the fourth group of UAV images and satellite images in the SD-ME algorithm of the embodiment of the present application;

[0055] Fig. 12(a) is a schematic diagram of the horizontal direction region in the embodiment of the present application;

[0056] Fig. 12(b) is a schematic diagram of the vertical direction region in the embodiment of the present application;

[0057] Fig. 12(c) is a schematic diagram of the 45° direction region in the embodiment of the present application;

[0058] Fig. 12(d) is a schematic diagram of the 135° direction region in the embodiment of the present application;

[0059] Fig. 12(e) is a schematic diagram of the center and peripheral region in the embodiment of the present application. DETAILED DESCRIPTION

[0060] The specific embodiments of the present application will be further described below in conjunction with the accompanying drawings.

[0061] The technical concept of the image matching enhancement method based on feature sparse region detection in the present application is as follows: first, local features depend on the gray information with obvious changes, while the gray changes of texture sparse regions are small, and the number of local features extracted from the texture sparse regions is small under the global consistent parameters, so the enhancement method of the present application involves screening and detecting the local feature sparse regions, and then inputting them to the feature extraction network separately, thereby solving the problem of uneven distribution of local features caused by texture differences under the global consistent parameters. Second, since the local features of the texture sparse regions are not obvious, the feature point scores of this region are often low, so the adaptive score threshold feature screening algorithm is used to retain the local feature points with relatively high scores to the greatest extent. After the joint processing of the feature sparse region detection algorithm and the adaptive score threshold feature screening algorithm, the obtained local features are aggregated and input to the feature matching network, and finally the robust matching results with sufficient number and uniform distribution are obtained.

[0062] Based on the above technical concept, the specific implementation steps of the present application are as follows:

[0063] 1) Obtain a pair of to-be-matched heterogeneous images, perform feature extraction and feature matching on the pair of to-be-matched heterogeneous images, and obtain an initial matching result of the pair of to-be-matched heterogeneous images;

[0064] 2) selecting an image in the to-be-matched heterogeneous image pair, performing feature sparse region detection on the selected image according to the feature points of the selected image obtained in step 1), obtaining each feature sparse region of the selected image, and calculating an affine transformation model of the to-be-matched heterogeneous image pair according to the initial matching result of the to-be-matched heterogeneous image pair; and performing affine transformation on each feature sparse region in the selected image according to the affine transformation model of the to-be-matched heterogeneous image pair, so as to obtain each group of corresponding feature sparse regions.

[0065] 3) performing feature extraction on each group of corresponding feature sparse regions, setting a corresponding feature screening threshold for each group of corresponding feature sparse regions, screening the features extracted from each group of corresponding feature sparse regions according to the feature screening threshold, obtaining local features of each group of corresponding feature sparse regions, aggregating the local features of each group of corresponding feature sparse regions to obtain global feature sparse region local features of the to-be-matched heterogeneous image pair, and performing feature matching on the global feature sparse region local features of the to-be-matched heterogeneous image pair to obtain a matching enhanced result of the feature sparse regions of the to-be-matched heterogeneous image pair.

[0066] Further, in order to improve the accuracy of feature sparse region detection, the application provides a sparse feature region detection algorithm, which processes and analyzes the initial matching result of the to-be-matched image pair through the feature sparse region detection algorithm, and then detects each group of feature sparse regions of the to-be-matched image pair. The detection algorithm comprises the following steps:

[0067] a) dividing the selected image into at least one detection region;

[0068] b) traversing each detection region, storing a detection region that meets a first condition, and performing quadtree splitting on a detection region that meets a second condition to obtain a new detection region; the first condition is that the feature points in the detection region are not greater than a set threshold and the area of the detection region is not less than a minimum detection area; and the second condition is that the feature points in the detection region are greater than the set threshold and the area of the detection region is not less than the minimum detection area;

[0069] c) performing step b) each time a new detection region is split out, until none of the new detection regions meets the second condition or the number of splitting times for splitting the new detection region reaches a set splitting number, and each stored detection region is a feature sparse region of the selected image.

[0070] d) calculating an affine transformation model of the to-be-matched heterogeneous image pair according to the initial matching result of the to-be-matched heterogeneous image pair; and performing affine transformation on each feature sparse region in the selected image according to the affine transformation model of the to-be-matched heterogeneous image pair, so as to obtain each group of corresponding feature sparse regions.

[0071] If the feature points in step b) are smaller than a set threshold, the set threshold is set according to the selected minimum detection area, such as 0, 1, etc.

[0072] Furthermore, considering that there might be some error in calculating the corresponding sparse feature regions using the affine transformation model obtained from the initial matching results, an adaptive area enlargement process was performed on the obtained sparse feature regions. Adaptive area enlargement refers to setting a corresponding diffusion distance for each sparse feature region and expanding the area outwards from the sparse feature region according to the set distance, resulting in a larger region. Adaptive area enlargement can avoid the influence of this error and improve the accuracy of sparse region detection.

[0073] The implementation methods for each of the above steps are explained below.

[0074] 1. Obtaining the initial matching results.

[0075] When performing feature extraction and feature matching on heterogeneous image pairs to be matched, this embodiment preferably uses the SuperPoint feature extraction network model and the SuperGlue network model to extract and match features on heterogeneous image pairs to obtain initial matching results.

[0076] 1) SuperPoint feature extraction network model.

[0077] The SuperPoint algorithm extracts features from the entire image using a fully convolutional neural network architecture. Its local feature extraction and descriptor construction network are as follows: Figure 2 As shown.

[0078] Depend on Figure 2 As can be seen, the SuperPoint algorithm consists of three modules: a feature encoder, a keypoint decoder, and a descriptor decoder. The feature encoder is constructed from a lightweight fully convolutional neural network modified from the VGG style. A fully convolutional neural network replaces the fully connected layers at the end of a traditional convolutional neural network with convolutional layers. The main function of the feature encoder is to extract features after dimensionality reduction of the image, thereby reducing the computational cost of subsequent meshing. After passing through the feature encoder, the input image... Encoded as an intermediate tensor

[0079] The keypoint decoder consists of convolutional layers, a softmax activation function, and a tensor deformation function. The main function of the convolutional layers is to transform the intermediate tensors output by the encoder into feature maps. Then, the probability distribution of interest points is obtained through the Softmax activation function. The feature map; finally, the upsampling model constructed by multiple deconvolutional layers and unpooling layers is replaced by a tensor deformation function, directly... The feature map is flattened to obtain a heat map tensor for key point extraction Each channel vector on the feature map corresponds to a heat value of a 8x8 region of the heat map.

[0080] The key point decoder is composed of a convolution layer, a Softmax activation function and a tensor deformation function, which calculates a probability for each pixel of the image, indicating the possibility of being a feature point. In the key point decoder, the main function of the convolution layer is to convert the intermediate tensor output by the encoder into a feature map Then the Softmax activation function is used to obtain a feature map representing the probability distribution of interest points; the tensor deformation function uses a sub-pixel convolution method to replace the up-sampling model constructed by multiple deconvolution layers and de-pooling layers, which has the advantage of reducing the model calculation amount while restoring the feature map resolution, directly flattening the feature map to obtain a heat map tensor for key point extraction Each channel vector on the feature map corresponds to a heat value of a 8x8 region of the heat map. Each value of the final output heat map tensor represents the probability of the pixel being a feature point.

[0081] The descriptor decoder is used to generate a semi-dense descriptor feature map, which first outputs a semi-dense descriptor (i.e. 1 output per 8 pixels) through a convolution layer, which can effectively reduce the memory and calculation amount and maintain the running efficiency. Then the decoder performs bicubic interpolation on the descriptor to obtain pixel-level accuracy, and finally obtains a dense feature map

[0082] 2) SuperGlue network model.

[0083] After obtaining the initial local features of the to-be-matched heterogeneous image pair, the SuperGlue algorithm is used for matching. SuperGlue is a network for feature matching and outlier rejection, which enhances the features of key points based on GNN network and converts the feature matching problem into a differentiable optimization transfer problem. The structure of the SuperGlue graph neural network model is shown in Figure 3 The algorithm mainly includes two modules, which are the feature enhancement module based on attention graph neural network and the optimal matching module. Among them, the feature enhancement module encodes the position information and descriptor information of the feature points, then performs feature fusion, and then processes L rounds through self-attention layers and cross-attention layers to aggregate the intra-image and inter-image context information respectively, to obtain more specific feature matching vectors f for matching.

[0084] The optimal matching module calculates the inner product between the feature matching vectors f to obtain the matching score matrix S ∈ R M×N , M, N represent the number of feature points in images A and B respectively. Due to the influence of occlusion and other problems, part of the feature points do not exist matching points, so the algorithm proposes a garbage can mechanism, as shown in formula (2), the garbage can is a new row and a column added on the basis of the score matrix, which is used to identify whether the feature has a matching point. SuperGlue regards the final matching result as a distribution problem, and constructs an optimization problem by calculating the distribution matrix P ∈ R M×N and the score matrix C, and solves P by maximizing the total score ∑ i,j C i,j P i,j . The optimal feature distribution matrix P is quickly iteratively solved by Sinkhorm algorithm on GPU.

[0085]

[0086] In the formula, <·, ·> represents the inner product, f i A and are the feature matching vectors of images A and B output by the feature enhancement module.

[0087]

[0088] In the formula, N and M represent the garbage cans in A and B, that is, the number of matches in each garbage can is the same as the number of key points in the other group, and z is the number of matches in each garbage can.

[0089] 2, Obtain the corresponding feature sparse area in the to-be-matched heterogeneous image pair.

[0090] The more and more uniform the levels of feature points distributed in space, the more accurately the spatial geometric relationship between two heterogeneous images can be expressed. However, the texture difference between the UAV image and the satellite image under the global consistent parameter will lead to uneven distribution of the initial matching result, and the feature points in the dense texture area are relatively dense, and the feature points in the sparse texture area are few. In view of this problem, the initial matching result is processed and analyzed by the feature sparse area detection algorithm, and then the feature sparse area is detected, and the adaptive expansion of the feature sparse area and the feature re-extraction are guided. The main idea of the feature sparse area detection algorithm is four-tree region segmentation, and its advantage is that it can more evenly divide the sub-regions, which is conducive to more evenly detecting the feature sparse area and avoiding the redundant feature extraction near the feature dense area.

[0091] The principle and process of the feature sparse area detection algorithm are shown in Algorithm 1 and Figures 4(a) to 4(d) .

[0092]

[0093]

[0094] The calculation formula of the total number of splits K is:

[0095]

[0096] In the formula, W and H are the length and width of the image respectively, and S is the minimum detection area, which is obtained according to test.

[0097] In Algorithm 1, after the feature sparse region is obtained, adaptive area expansion processing is performed on the feature sparse region, that is, area diffusion is performed around the sparse region to obtain a region with larger area.

[0098] The feature sparse region detection algorithm is used to obtain the feature-inconspicuous region of the remote sensing image and detect the relatively reliable feature points in the region. The minimum detection area directly affects the overall number and representativeness of the detected feature sparse region. If the minimum detection area is too large, the overall number of the detected feature sparse region may be too small, which cannot cover most of the feature sparse regions and has low representativeness. If the minimum detection area is too small, the feature detector cannot effectively identify and extract the local features. Therefore, the appropriate minimum detection area is a decisive factor for the performance of the feature sparse region detection algorithm. Since the feature sparse region detection algorithm is based on the principle of four-tree segmentation for detection, the detection area of the small facet is shown in formula (4).

[0099]

[0100] In the formula, W and H are the length and width of the image respectively, and k is the current number of splits of the image.

[0101] 3. Adaptive threshold feature screening.

[0102] After the feature sparse region detection algorithm is processed, one-to-one corresponding feature sparse regions are obtained in the image pair, which mainly include texture sparse regions, feature difference regions, etc. When the local features are initially extracted globally, the feature points with relatively low scores in the feature sparse region are screened out due to the high global consistency screening threshold. If the global consistency screening threshold is lowered during the initial matching, the feature points will still be concentrated in the texture dense region due to the global consistency texture difference, and an uneven matching result will be obtained. The method of the embodiment extracts features from each independent feature sparse region obtained by the feature sparse region detection algorithm. Compared with the feature points in the feature sparse region obtained from the global heat map tensor graph, the feature points obtained by independently extracting the heat map tensor on the small facet of the feature sparse region are relatively more abundant, uniform and have higher confidence.

[0103] The adaptive threshold feature screening algorithm is used to extract local features of SuperPoint for each pair of feature sparse regions, and the algorithm flow is as shown in Figure 5 .

[0104] As shown in Figure 5 , first, the local features of each region are extracted to obtain relatively rich local features on the small facet texture sparse region; the local features in each region are screened based on the score, and the adaptive score threshold calculation is shown in formula (5), and the local feature key points and descriptors with a score greater than the adaptive score threshold of the region are screened out; finally, the features screened in all independent regions are aggregated to form the global feature sparse region local features, as shown in formula (6).

[0105]

[0106] In the formula, C n is the score of the nth local feature in each group of feature sparse regions, and N is the number of local features independently extracted in each group of feature sparse regions.

[0107]

[0108] Where Q represents the number of feature sparse regions, p q , c q and e q represent the position, score and descriptor of the local feature screened in the qth group of feature sparse regions, where q = 1, 2, 3, …, Q.

[0109] After obtaining the global feature sparse region local features by the adaptive threshold feature screening algorithm, they are input into the SuperGlue feature matching network for matching, and finally the robust matching enhancement result of the feature sparse region is obtained.

[0110] The enhancement method of the present embodiment is a feature sparse region detection and matching enhancement algorithm proposed on the basis of the SuperPoint and SuperGlue feature matching algorithm, thereby enhancing the matching effect of SuperGlue. The enhancement method of the present embodiment is referred to as SD-ME algorithm (Feature Sparse Region Detection and Matching Enhancement Algorithm, SD-ME), and the flowchart of the method is as shown in Figure 1The method of the embodiment can be implemented in an Ubuntu 18.04 operating system, a programming language environment is Python 3.6, and a programming platform is Pycharm. A hardware platform uses a notebook computer with an I7 CPU, 31G memory, and a GeForce RTX 2060 graphics card (6 GB of video memory). The SD-ME algorithm uses the official pre-trained model of the SuperPoint algorithm and the SuperGlue algorithm for testing.

[0111] The specific implementation process of the method of the application is further illustrated below by taking four groups of satellite images (left) and unmanned aerial vehicle images (right) as shown in the drawings as examples. Figures 6(a) to 6(d)

[0112] Figures 6(a) to 6(d) In the four groups of satellite images (left) and unmanned aerial vehicle images (right) shown in the drawings, the first group of images has dense buildings and sparse texture areas such as farmland, and the unmanned aerial vehicle images have a certain degree of non-linear distortion due to the distortion of the camera lens; the second group of unmanned aerial vehicle images is a spring 2020 image, and the satellite image is a summer 2018 image, so the difference in time phase and season leads to obvious differences in ground objects, and there are many sparse texture areas such as forests and wastelands; the third group of unmanned aerial vehicle images is taken during the Henan flood disaster in 2021, and the satellite image is a summer 2018 image, so the local gray scale difference between the two images is large, and the ground object difference is obvious; the fourth group of unmanned aerial vehicle images is a winter image, and the satellite image is a summer 2016 image, so the image size difference and resolution difference between the two images are large.

[0113] Figures 6(a) to 6(d) The selected four groups of images have many sparse texture areas, large differences in time phase, obvious differences in ground objects, and certain non-linear spectral radiation distortion, so matching is difficult. The above four groups of images are selected to verify the method of the application, which has strong representativeness.

[0114] First, the minimum detection area S optimal value determination test is performed.

[0115] In the SD-ME algorithm, selecting an appropriate minimum detection area is beneficial to the feature sparse area detection algorithm to obtain effective detection results. Since the length and width of the image size of most images to be matched are inconsistent and are not necessarily an integer multiple of 4, the minimum detection area calculated by formula (6) for different images may have differences within a certain range, so the minimum detection area fluctuation range is expressed in the form of an area interval. According to the principle of four-tree region segmentation in the feature sparse area detection algorithm, the area interval can be derived by formula (7).

[0116] S = 4 l ~ 4 l+1 (7)

[0117] Wherein, l is the number of inverse merging of the cell.​

[0118] To select the minimum detection area most suitable for the feature sparse area detection algorithm, the feature sparse area detection algorithm is used to detect the feature sparse area of the four image pairs in Figures 6(a) to 6(d) . The minimum detection areas are set to 64, 256, 1024, and 4096 respectively, and the results are shown in Figure 7 .

[0119] As shown in Figure 7 , when the minimum detection area S is 256, the number of matching points obtained in the feature sparse area of the four image pairs is the largest. Therefore, the optimal value of the minimum detection area of the feature sparse area detection algorithm in this paper is set to 256 to ensure that the feature sparse area detection algorithm can assist in obtaining the best matching point pairs in terms of quantity and quality.

[0120] Secondly, the feature sparse area detection and similarity measurement are performed.

[0121] The SD-ME algorithm uses the feature sparse area detection algorithm to obtain the feature sparse area between the UAV images and the satellite images, and then performs feature re-extraction and matching through the adaptive threshold feature screening algorithm to finally complete the matching enhancement. To verify the reliability of the feature sparse area detection in the SD-ME algorithm, the feature sparse area detection test is performed on the four image pairs in Figures 6(a) to 6(d) . The initial matching results of the test images by SuperGlue combined with SuperPoint features are shown in Figures 8(a) to 8(d) .

[0122] As shown in Figures 8(a) to 8(d) , SuperGlue combined with SuperPoint features achieves a relatively ideal matching effect on UAV images and satellite remote sensing images, but the number of matching point pairs is small in the texture sparse area (vegetation, water area, farmland, etc.), which cannot play a role in global uniform control in UAV image rectification and registration applications. The SD-ME algorithm extracts the feature sparse area based on the initial matching results, sets the optimal value of the minimum detection area to 256, and performs the feature sparse area extraction test on the four UAV images in Figures 6(a) to 6(d) . The feature sparse area detection algorithm stores each extracted feature sparse area in a linked list structure. To more intuitively display the feature sparse area detection results, the feature sparse areas are combined and displayed, and the results are shown in Figures 9(a) to 9(d) .

[0123] As shown in Figures 9(a) to 9(d) .As shown, the feature sparse region detection algorithm accurately obtains a more reliable feature sparse region, but the key to successfully extracting local features and matching the pair of feature sparse regions is that the feature sparse regions have a strict one-to-one correspondence. The feature similarity of different feature sparse regions is very low. By measuring the feature similarity of the obtained pair of regions, the effective correspondence of the feature sparse region detection algorithm is verified.

[0124] To measure the similarity of the region features, the local features extracted in the region need to be aggregated. In this paper, the vector of aggregate locally descriptors (VLAD) is used to aggregate the local feature descriptors of the feature sparse region. The VLAD algorithm is a classic algorithm in the field of image retrieval. VLAD trains a codebook with several visual words for images through clustering methods. The codebook is used to calculate the nearest codebook clustering center for each image feature, and then the difference between all features and the clustering center is superimposed to obtain a d-dimensional vector corresponding to the number of visual words in the codebook. The concatenation of the d-dimensional vector can obtain the VALD vector of the corresponding image.

[0125] The feature similarity between the pair of images is low due to the differences in ground objects and nonlinear radiation distortion between UAV images and satellite images. However, the feature similarity still has global consistency, that is, the VLAD vector similarity between the pair of feature sparse regions is consistent with the VLAD vector similarity between the whole images, while the VLAD vector similarity between the pair of feature sparse regions is much lower than the VLAD vector similarity between the whole images. The Euclidean distance of the VLAD feature vector is used as a measure for feature similarity measurement. The larger the Euclidean distance, the lower the feature vector similarity, and the smaller the Euclidean distance, the higher the feature vector similarity. The Euclidean distance between feature vectors V1=(x1, x2, …, xd) and V2=(y1, y2, …, yd) is calculated as shown in equation (8). d d

[0126]

[0127] In the formula, Ed is the Euclidean distance, u represents the u-th element in the d-dimensional vector, x u is the u-th element of feature vector V1, y u is the u-th element of feature vector V2, and u=1, 2, …, d.

[0128] ​​The test proves the one-to-one correspondence of the detection results of the feature sparse region detection algorithm and the local feature similarity by testing the similarity of the VLAD vectors of the paired feature sparse regions obtained by the feature sparse region detection algorithm on the image pair based on the Euclidean distance similarity between the VLAD feature vectors of the paired images. Figures 10(a) to 10(d)

[0129] Figures 10(a) to 10(d) In the figure, the horizontal axis represents 36 pairs of randomly selected feature sparse regions, and the vertical axis represents the Euclidean distance between the VLAD vectors of the paired regions. Ed_a represents the VLAD feature vector of the first pair of images, and Ed_a_random represents the VLAD vector of the feature sparse region randomly selected from the first pair of images. Figures 8(a) to 8(d) It can be seen from the figure that the Euclidean distances between the VLAD vectors of the feature sparse regions of the four groups of unmanned aerial vehicle images and the satellite reference images extracted by the feature sparse region detection algorithm are mostly less than the Euclidean distances between the VLAD vectors of the entire images, which proves that the feature sparse regions extracted by the feature sparse region detection algorithm have a strict one-to-one correspondence.

[0130] Finally, the matching performance of the SD-ME algorithm is compared with that of the traditional SIFT algorithm, the ContextDesc algorithm and the SuperGlue algorithm.

[0131] In order to verify the excellent performance of the SD-ME algorithm in feature sparse region detection and matching enhancement, the test performs matching performance test on the four groups of image pairs in Figures 6(a) to 6(d) Among them, SIFT is a local feature descriptor with certain affine invariance and anti-interference; the ContextDesc algorithm is a deep learning algorithm suitable for multi-modal image matching, which enhances the original feature descriptors such as DELF through high-order image visual information and geometric information of key point distribution.

[0132] The test uses the number of correct matching points (P) and the matching time (t) to compare the performance of the above several matching algorithms. The correct matching point refers to the number of points whose actual position on the feature point of the image to be matched and the actual position of the feature point on the reference image are within a threshold, and the correct matching point is verified by formula (9). The number of correct matching points (P) refers to the number of matching points that meet the above conditions, which can reflect the basic performance of the feature matching algorithm.

[0133]

[0134] ​In the formula, H is an affine transformation model fitted by artificial point selection to replace the real affine transformation model of two multi-source remote sensing images, and the distance between the feature point (x u ', y u ') after affine transformation and its homonym (x u , y u ) is less than the threshold value ε (the threshold value is set to 3 in this embodiment), and it is judged to be a correct matching point. Artificial point selection mainly uses various matching methods to judge whether it is a homonym pair one by one by artificial selection; for the area that cannot be identified by various matching algorithms, image magnification visual interpretation is used for selection. With the above two ways, 36 evenly distributed homonym pairs are selected for affine transformation model fitting.

[0135] The comparison results of the above three algorithms and the SD-ME algorithm in the number of correct matching points (P) and the matching time (t) are shown in Table 1.

[0136] Table 1 Comparison of matching test results

[0137]

[0138] As shown in Table 1, the classic SIFT algorithm has poor matching effect, and it cannot be completely applied to unmanned aerial images and satellite images with large differences in non-linear radiation distortion and ground objects. ContextDesc algorithm obtains a certain number of matching point pairs on the four groups of images, but the overall matching accuracy is low, and the time consumption is much higher than that of other algorithms. SuperGlue algorithm has achieved good results on this type of heterogeneous images, while maintaining excellent matching accuracy, it obtains a large number of matching point pairs. SD-ME algorithm enhances the matching of SuperGlue algorithm, and the time consumption of this algorithm is slightly higher than that of SuperGlue algorithm, but it obtains a large number of correct matching point pairs with more uniform distribution, and the matching effect is shown in Figures 11(a) to 11(d) .

[0139] Figures 11(a) to 11(d) It is intuitively shown that the SD-ME algorithm has strong adaptability to unmanned aerial images and satellite reference images. SD-ME algorithm enhances the initial matching result of SuperGlue algorithm, and obtains a large number of correct matching point pairs. SD-ME algorithm compensates for the poor matching performance of SuperGlue algorithm in sparse feature areas, and obtains a sufficient and reliable number of feature matching point pairs in sparse feature areas, and the matching result is more uniformly distributed. In order to further verify the uniformity advantage of the matching enhancement result of SD-ME algorithm, the matching point distribution uniformity of SD-ME algorithm and SuperGlue algorithm is compared.

[0140] According to the literature "Evaluation method of image feature point distribution uniformity" (Author: Zhu Haifeng, Journal: Journal of Daqing Normal University, Vol. 30, No. 3, 2021), the matching point distribution uniformity is calculated according to the distribution uniformity of the matching results in five directions. The image is divided into five directions, a total of ten regions, as shown in Figures 12(a) to 12(e)

[0141] According to the statistical principle, the sample variance is used to represent the difference of the number of matching points in the image blocks in five different directions. If the distribution of matching point pairs in five directions is uniform, the sample variance of the number of matching point pairs in five directions is small, otherwise it is large. The matching point distribution uniformity is shown in formula (10). The greater the matching point distribution uniformity, the more uniform the matching point distribution, otherwise it is not uniform.

[0142]

[0143] In the formula, V g is the gth region statistical distribution vector, which is composed of the number of matching points in ten regions.

[0144] The matching point pair distribution uniformity calculation steps are shown in Algorithm 2.

[0145]

[0146] According to Algorithm 2, the matching point pair distribution uniformity of SD-ME algorithm and SuperGlue algorithm is calculated, and the results are shown in Table 2.

[0147] Table 2 Comparison of matching point distribution uniformity

[0148]

[0149] From Table 2, it can be seen that the matching point pair distribution uniformity of SD-ME algorithm on 4 groups of images is greater than that of SuperGlue algorithm. The matching point uniformity uses logarithmic operation, which reflects the variance of five direction distribution. The matching point uniformity of SD-ME algorithm is obviously better than that of SuperGlue algorithm. This experiment proves the effectiveness of SD-ME algorithm in detecting sparse feature area and matching enhancement, which greatly improves the distribution uniformity of matching results.

[0150] ​In summary, in view of the difficulty in matching the texture sparse area between the UAV image and the satellite image, the embodiment enhances the matching effect of the SuperGlue algorithm through feature sparse area detection and adaptive threshold feature extraction and screening, and proposes a SD-ME algorithm more suitable for matching the UAV image and the satellite image. The SD-ME algorithm detects the feature sparse area, guides the feature extraction and adaptive score threshold feature screening, obtains sufficient and uniformly distributed local features, and finally obtains reliable and robust matching results through the SuperGlue algorithm again. In order to verify the effectiveness of the SD-ME algorithm, four groups of representative high-difficulty image pairs are tested. In the case that the traditional classic SIFT algorithm and the deep learning ContextDesc algorithm have poor matching effect on the four groups of images, the SD-ME algorithm achieves excellent results on the four groups of images. Compared with the SuperGlue algorithm, the number of correct matching points is significantly improved and gathered in the feature sparse area, which makes up for the deficiency of the SuperGlue algorithm in matching the texture sparse area between the UAV image and the satellite image.

[0151] In addition, the additional contribution of the SD-ME algorithm is that it can not only be used to enhance the matching effect of the SuperGlue algorithm, but also be adapted to most CNN feature matching algorithms, and can be applied to other matching scenarios, and has high application value.

Claims

1. An image matching enhancement method based on feature sparse region detection, characterized in that, The method comprises the following steps: 1) obtaining a to-be-matched heterogeneous image pair, performing feature extraction on the to-be-matched heterogeneous image pair, screening the extracted feature points according to a global consistency screening threshold to obtain initial local features, and then performing feature matching to obtain an initial matching result of the to-be-matched heterogeneous image pair; 2) selecting an arbitrary image in the to-be-matched heterogeneous image pair, performing feature sparse region detection on the selected image according to the feature points of the selected image obtained in step 1), obtaining each feature sparse region of the selected image, and calculating an affine transformation model of the to-be-matched heterogeneous image pair according to the initial matching result of the to-be-matched heterogeneous image pair; performing affine transformation on each feature sparse region in the selected image according to the affine transformation model of the to-be-matched heterogeneous image pair, thereby obtaining corresponding feature sparse regions of each group; 3) performing feature extraction on the corresponding feature sparse regions of each group, setting a corresponding feature screening threshold for each group of corresponding feature sparse regions, the feature screening threshold being smaller than the global consistency screening threshold in step 1), screening the features extracted from each group of corresponding feature sparse regions according to the feature screening threshold to obtain local features of each group of corresponding feature sparse regions; and aggregating the local features of each group of corresponding feature sparse regions to obtain global feature sparse region local features of the to-be-matched heterogeneous image pair; performing feature matching on the global feature sparse region local features of the to-be-matched heterogeneous image pair to obtain a matching enhanced result of the feature sparse regions of the to-be-matched heterogeneous image pair; In steps 1) and 3), a SuperPoint feature extraction network model is used for feature extraction. 2.The method of claim 1, wherein, In step 2), the method for performing feature sparse region detection on the selected image is as follows: a) dividing the selected image into at least one detection region; b) traversing each detection region, storing a detection region that meets a first condition, and performing quadtree splitting on a detection region that meets a second condition, thereby obtaining a new detection region; The first condition is that the feature points in the detection region are not greater than a set threshold and the area of the detection region is not less than a minimum detection area; and the second condition is that the feature points in the detection region are greater than the set threshold and the area of the detection region is not less than the minimum detection area; c) each time a new detection region is split out, step b) is executed until none of the new detection regions meets the second condition or the number of splitting times for splitting the new detection region reaches a set splitting number, and each stored detection region is a feature sparse region of the selected image. 3.The method of claim 2, wherein, The set number of splits k The calculation formula is: Wherein, W , H Length and width of the image respectively, S The minimum detection area, S The value is obtained according to the test. 4.The method of claim 1, wherein, The calculation formula of the feature screening threshold θ of each group of feature sparse regions is as follows: wherein, C n The score of the i-th local feature in each group of feature sparse regions is n N The number of local features independently extracted in each group of feature sparse regions.​ 5. The method of claim 1, wherein the feature sparse region detection-based image matching enhancement method is characterized by, In step 3), the process of obtaining the local features of each group of feature sparse regions is as follows: first, performing region-by-region feature extraction on each group of feature sparse regions to obtain initial local features of each group of feature sparse regions, and then screening local feature key points and descriptors with scores greater than the feature screening threshold from the initial local features of each group of feature sparse regions, thereby obtaining the local features of each group of feature sparse regions.

6. The method of claim 1, wherein the feature sparse region detection based image matching enhancement method is characterized by, The way of aggregating the local features of each group of feature sparse regions to obtain global feature sparse region local features is as follows: ， Where Q represents the number of sparse feature regions. p q , c q and e q Representing the first q The local feature locations, scores, and descriptors selected from sparse regions of the group features are: q =1,2,3,…,Q.

7. The method of claim 2, wherein the feature sparse region detection is based on a feature matching method. The method for detecting the feature sparse areas of the selected image in step 2) further comprises step d) diffusing the area of each detected area according to a corresponding set distance to the periphery, and taking each area after the area diffusion as each feature sparse area of the selected image.

8. The image matching enhancement method based on feature sparse region detection according to claim 2, characterized in that, When the selected image is divided into a detection region, the area of each new detection region after each split is calculated by the following formula: wherein, W , H is the length and width of the selected image, k is the current split number of the selected image.

9. The method of claim 1-8, wherein, In step 1), the SuperGlue network model is used for feature matching of the initial local features extracted from the to-be-matched heterogeneous image pair, to obtain an initial matching result of the to-be-matched heterogeneous image pair.

10. The feature sparse region detection based image matching enhancement method of claim 9, wherein, In step 3), the SuperGlue network model is used for feature matching of the local features of the global feature sparse areas.

Images