An Adaptive Method and System for Selecting UAV Image Matching Pairs

Abstract

The present invention provides an adaptive drone image matching pair selection method and system, comprising: S1: acquiring drone images, performing feature extraction on the drone images using the SIFT algorithm, and obtaining a local feature set of the drone images; S2: constructing a vector codebook, performing feature aggregation on the local feature set using the vector codebook, and obtaining a feature matrix set; vectorizing the feature matrix set to obtain a global feature descriptor vector set; S3: performing image retrieval on the global feature descriptor vector set based on a graph index structure to obtain a drone image matching pair set; S4: performing three-dimensional reconstruction on the drone image matching pair set to obtain a three-dimensional reconstruction model. The present invention uses global feature descriptor vectors to replace word frequency calculation based on local features, and provides a graph-based indexing strategy to achieve efficient overlapping image search, making drone image matching pairs more efficient and accurate.

Description

Technical Field

[0001] The present invention relates to the field of image matching, and in particular to an adaptive drone image matching pair selection method and system. Background Art

[0002] Unmanned aerial vehicles (UAVs) are currently one of the primary sources of imagery in remote sensing and mapping. Compared to satellites and aerial RS platforms, UAVs offer greater flexibility, timeliness, and high resolution. These advantages have led to their widespread use in geographic surveying and mapping, agricultural yield estimation, and disaster prevention and mitigation. The high timeliness and high resolution of UAV imagery are highly suitable for 3D reconstruction.

[0003] Compared to traditional aerial triangulation (Aerotriangulation), Incremental Structure from Motion (ISfM) is a technique for recovering camera parameters and 3D point orientation that is independent of image order and prior parameters. It directly recovers camera pose and 3D points from overlapping images. Feature matching is a key step in the SfM workflow, and its efficiency directly impacts the overall efficiency of SfM. The goal of matching pair selection is to search for spatially overlapping image pairs and guide subsequent feature matching, avoiding exhaustive matching and improving the reliability of SfM reconstruction. Therefore, efficiently and accurately finding suitable matching pairs is a core challenge in large-scale SfM. In traditional vocabulary tree-based image retrieval, image similarity is calculated using a pre-built inverted index. For high-resolution and large-scale drone imagery, this approach is extremely inefficient due to the significant time cost of constructing the codebook and calculating word frequencies. Summary of the Invention

[0004] To solve the above technical problems, the present invention provides an adaptive drone image matching pair selection method, comprising:

[0005] S1: Obtain drone images, extract features from them using the SIFT algorithm, and obtain a local feature set of the drone images;

[0006] S2: Construct a vector codebook, perform feature aggregation on the local feature set through the vector codebook to obtain a feature matrix set; vectorize the feature matrix set to obtain a global feature descriptor vector set;

[0007] S3: Perform image retrieval on the global feature descriptor vector set based on the graph index structure to obtain a set of drone image matching pairs;

[0008] S4: Perform 3D reconstruction of the collection through drone image matching to obtain a 3D reconstructed model.

[0009] Preferably, step S1 is specifically as follows:

[0010] S11: Obtain local features of each drone image using SIFT algorithm;

[0011] S12: Preserve drone imagery In the figure, the feature scale is from large to small. Local features make up the drone image The local feature set of , where i is the number of the drone image.

[0012] Preferably, step S2 is specifically as follows:

[0013] S21: construct a pre-trained vector codebook, train the pre-trained vector codebook using the training data set, and obtain a vector codebook;

[0014] S22: Perform feature clustering on each local feature set through the vector codebook, so that each drone image is represented as a The characteristic matrix of

[0015] S23: For each The characteristic matrix is vectorized, and each row in the characteristic matrix is sequentially spliced to obtain the corresponding dimensional global feature descriptor vectors, and compose each global feature descriptor vector into a global feature descriptor vector set.

[0016] Preferably, step S3 is specifically as follows:

[0017] S31: constructing an index structure by using each global feature descriptor vector in the global feature descriptor vector set as a vertex of the index structure, and the index structure has a total of H layers;

[0018] S32: Send the drone image to be queried The vertex corresponding to the global feature descriptor vector Input index structure, where i is the number of the drone image, traverse all vertices in the hth layer, and find the distance vertex Nearest Vertex As the input of h-1 layer; h is the layer number of the index structure, and the initial value is H;

[0019] S33: Repeat step S32 H times to obtain the input vertex at the bottom of the index structure , traverse all the vertices at the bottom layer to find the distance vertex Recent vertices, As drone imagery of matching pairs, where For drone images The number of similar image retrievals;

[0020] S34: Repeat steps S32-S33 until all matching pairs of drone images are obtained, and all matching pairs are combined into a drone image matching pair set.

[0021] Preferably, The specific setting process is as follows:

[0022] S331: Preliminary estimate of drone imagery Similar images with settings ;

[0023] S332: Get the number of local features involved in aggregation in all drone images , where n is the total number of drone images;

[0024] S333: Found Maximum value and minimum value ;

[0025] S334: Drone Imagery The number of similar images retrieved The calculation formula is:

[0026] .

[0027] An adaptive drone image matching pair selection system, comprising:

[0028] The local feature acquisition module is used to obtain UAV images, extract features from them using the SIFT algorithm, and obtain a set of local features of the UAV images.

[0029] The vector acquisition module is used to construct a vector codebook, perform feature aggregation on the local feature set through the vector codebook, and obtain a feature matrix set; vectorize the feature matrix set to obtain a global feature descriptor vector set;

[0030] The retrieval and pairing module is used to perform image retrieval on the global feature descriptor vector set based on the graph index structure to obtain the drone image matching pair set;

[0031] The 3D reconstruction module is used to perform 3D reconstruction of the collection through drone image matching to obtain a 3D reconstructed model.

[0032] The present invention has the following beneficial effects:

[0033] The present invention adopts a global feature descriptor vector to replace the word frequency calculation based on local features, and provides a graph-based indexing strategy to achieve efficient overlapping image search, making the matching of drone images more efficient and accurate, and realizing high-efficiency and high-precision positioning and orientation of drone images in three-dimensional reconstruction work. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flow chart of a method according to an embodiment of the present invention;

[0035] Figure 2 Schematic diagram of feature clustering of local features;

[0036] Figure 3 It is a structural diagram of the index structure;

[0037] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0038] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0039] Reference Figure 1 The present invention provides an adaptive UAV image matching pair selection method, comprising:

[0040] S1: Obtain drone images, extract features from them using the SIFT algorithm, and obtain a local feature set of the drone images;

[0041] S2: Construct a vector codebook, perform feature aggregation on the local feature set through the vector codebook to obtain a feature matrix set; vectorize the feature matrix set to obtain a global feature descriptor vector set;

[0042] S3: Perform image retrieval on the global feature descriptor vector set based on the graph index structure to obtain a set of drone image matching pairs;

[0043] S4: Perform 3D reconstruction of the collection through drone image matching to obtain a 3D reconstructed model.

[0044] In this embodiment, step S1 is specifically as follows:

[0045] S11: Obtain local features of each drone image using SIFT algorithm;

[0046] Specifically, drone images can provide multi-angle observation of ground objects and more realistically reflect the actual situation of objects. However, due to the large viewing angle and geometric deformation, it is difficult to obtain reliable feature homonyms. In view of the characteristics of drone images, the study uses the SIFT (Scale Invariant Feature Transform) feature extraction method to extract local features. This method has the advantages of illumination invariance, rotation invariance, and scale invariance, and can well adapt to multi-angle, high-resolution drone images. In the specific implementation process, the present invention uses SIFTGPU implemented by GPU (Graphic Processing Unit) to extract local features of drone images.

[0047] S12: Preserve drone imagery In the figure, the feature scale is from large to small. Local features make up the drone image The local feature set of , where i is the number of the drone image;

[0048] Specifically, due to the high resolution advantage of drone images, a large number of local features can be extracted from each image. Considering the large number of drone images, the processing efficiency of the massive local features of all images is extremely low. To address this problem, the present invention eliminates the local features extracted from each image; in order to minimize the accuracy of subsequent processing by the elimination step, the idea is to eliminate features with weak expressive ability and leave a small number of features with strong expressive ability.

[0049] For a single image The invention sorts all the extracted local features of the image from large to small according to the feature scale, and then retains The largest scale features are processed later, where It will be adaptively adjusted according to different images. The adjustment strategy is as follows:

[0050] Statistics of this image The number of local features extracted

[0051] Pick ,default is 5;

[0052] Since subsequent feature aggregation does not require the same number of local features in the image, the adaptation here can be completed. On the contrary, for images with a large number of local features, using more large-scale features to participate in subsequent aggregation can further highlight the expressive power of the aggregated features. Therefore, the adaptation here is reasonable.

[0053] In this embodiment, in image retrieval technology, a bag of words (BoW) model is usually used to convert local image features into word frequency expressions. Although this method can effectively obtain image expressions, the computational cost of this method increases significantly as the number of images increases. To address this problem, the present invention uses VLAD (Vector of Locally Aggregated Descriptors) vector aggregation technology to aggregate a large number of low-dimensional local feature vectors into a small number of high-dimensional global feature vectors in a short period of time. The specific steps are shown in S2;

[0054] Step S2 is specifically as follows:

[0055] S21: construct a pre-trained vector codebook, train the pre-trained vector codebook using the training data set, and obtain a vector codebook;

[0056] Specifically, feature aggregation requires a vector codebook. Considering that codebook training is very time-consuming, the present invention adopts the idea of offline codebook training. A universal codebook is obtained by training with a general training dataset. All data processing is implemented using this codebook, which greatly reduces time consumption.

[0057] The codebook training steps are as follows:

[0058] The SIFT GPU algorithm used in the present invention is used to extract local features of the images on the general training data set to obtain all local features of all images;

[0059] Use Kmeans algorithm to cluster local feature vectors to obtain Cluster centers:

[0060] ① Randomly determined The local feature vectors are set as cluster centers ,at the same time

[0061] get clusters ;

[0062] ② Assign each local feature vector to a cluster according to the Euclidean distance principle ;

[0063] ③Calculation clustering The mean of all vectors in is used as the new cluster center ;

[0064] ④If the cluster center changes, repeat ②③ until convergence or a certain number of iterations are met;

[0065] Will The cluster centers are saved as codebook vectors. vectors constitute the pre-trained codebook ;

[0066] After the above steps, the vector codebook can be obtained ,This vector codebook can be used for subsequent UAV image feature aggregation;

[0067] S22: The local features in the local feature set are clustered by the vector codebook, so that each drone image is represented as a The characteristic matrix of

[0068] Specifically, the core idea of VLAD vector aggregation technology is to accumulate the residual vector between the local feature descriptor vector and its corresponding cluster center vector, such as Figure 2 As shown in the figure, for some local features of the image obtained by screening and eliminating, the aggregate feature descriptor is obtained by counting all the features of the same cluster center and calculating the sum of the residuals between these features and the cluster center. The vector codebook obtained in the previous step exists Cluster centers , then we can calculate The characteristic matrix of The calculation formula is:

[0069] (1)

[0070] in the formula Indicates the first Row, No. Elements of the column, Indicates the local features, Indicates the Cluster centers, It means the residual value of the local feature falling on the cluster center; is a symbolic function if Not a cluster center ,but The value of is 0; if Belongs to the cluster center ,but The value of is 1;

[0071] After the above calculations, each image can be represented as a The characteristic matrix of is the number of matrix rows and the number of centers of the pre-trained vector codebook Equal values, and also related to the ability of the feature matrix to express the image, The larger the value, the stronger the matrix's ability to express images, but the calculation efficiency is reduced; The smaller the value, the weaker the matrix's ability to express images, but the higher the calculation efficiency; in this invention The value can be freely set according to the requirements of efficiency and accuracy; is the number of matrix columns, which is equal to the dimension of local features. In this invention, SIFT is used to extract local features of drone images. The default value is 128;

[0072] S23: For each The characteristic matrix is vectorized, and each row in the characteristic matrix is sequentially spliced to obtain the corresponding dimensional global feature descriptor vectors, and compose each global feature descriptor vector into a global feature descriptor vector set.

[0073] In this embodiment, step S3 is specifically as follows:

[0074] S31: constructing an index structure by using each global feature descriptor vector in the global feature descriptor vector set as a vertex of the index structure, and the index structure has a total of H layers;

[0075] Specifically, after obtaining the global feature descriptor vector of the image, the similarity between the vectors can be determined by directly calculating the Euclidean distance between the vectors, thereby reflecting the similarity between the images. However, for drone images with a large number of images, the time consumption of pairwise comparison is very long. To this end, the idea of the present invention is to use an efficient index structure to implement ANN (Approximate Nearest Neighbor) retrieval of vectors. HNSW (Hierarchical Navigable SmallWorld) is a graph-based approximate nearest neighbor retrieval algorithm that uses a hierarchical structure graph to construct a vector index graph, such as Figure 3 As shown; the present invention uses the global feature descriptor vector of the image as the vertex set of the graph to construct an index graph to achieve efficient retrieval;

[0076] Before searching, all calculated global feature descriptor vectors need to be established as an index structure, adding only one vector vertex at a time. The specific steps are as follows:

[0077] (1) Calculate the newly added vertices The highest layer number L projected to is calculated as follows:

[0078] (2)

[0079] in, is a random number between 0 and 1. is a normalization factor to ensure that the calculated number of layers meets actual needs;

[0080] (2) If there are no vertices in the index structure, directly in the layer Add Vertex Otherwise: at the top level Traverse all vertices to find the distance vertex Nearest Vertex As The input of the layer;

[0081] (3) Traverse all vertices in the next layer to find the distance vertex Nearest Vertex As input to the next layer;

[0082] (4) Repeat step (3) until you find Input vertices of the layer , in the layer Traversing vertices Neighbors find distance Recent vertices, the vertices Insertion layer With this Nodes establish connections and become neighbors;

[0083] (5) Take the Vertices are used as input for the next layer and traversed in the next layer Nodes find the distance Recent nodes, vertices Insert this layer with the newly found Nodes establish connections and become neighbors;

[0084] (6) Repeat step (5) until all layers have completed vertices Insertion;

[0085] in, is a parameter set manually. In the present invention, Can be set freely; usually, The larger the value, the more complex the index structure, the longer it takes to create the index, and the higher the accuracy; The smaller the value, the simpler the index structure, the shorter the indexing time, and the lower the accuracy;

[0086] S32: Send the drone image to be queried The vertex corresponding to the global feature descriptor vector Input index structure, where i is the number of the drone image, traverse all vertices in the hth layer, and find the distance vertex Nearest Vertex As the input of h-1 layer; h is the layer number of the index structure, and the initial value is H;

[0087] S33: Repeat step S32 H times to obtain the input vertex at the bottom of the index structure , traverse all the vertices at the bottom layer to find the distance vertex Recent vertices, As drone imagery of matching pairs, where For drone images The number of similar image retrievals;

[0088] S34: Repeat steps S32-S33 until all matching pairs of drone images are obtained, and all matching pairs are combined into a drone image matching pair set.

[0089] In this embodiment, The specific setting process is as follows:

[0090] S331: Preliminary estimate of drone imagery Similar images with settings ;

[0091] S332: Get the number of local features involved in aggregation in all drone images , where n is the total number of drone images;

[0092] S333: Found Maximum value and minimum value ;

[0093] S334: Drone Imagery The number of similar images retrieved The calculation formula is:

[0094] (3)

[0095] In this formula, the final number of images retrieved is According to the number of local features involved in the aggregation The more local features that are aggregated, the more images that are retrieved, because images with a large number of local features can often match more images in subsequent feature matching.

[0096] Step S4 is specifically as follows:

[0097] After obtaining the matching pairs of drone images, feature matching is performed on each pair of images to determine the connection relationship between the images. Based on the number of correctly matched features and the matching area between the images, image pairs with weaker connections can be eliminated, thereby constructing an image topology connection diagram. Based on the generated image topology connection diagram, the normalized cut (NC) algorithm can be used to divide the scene into a specified number of sub-scenes. Then, parallel processing technology is used to simultaneously perform positioning and orientation on multiple sub-scene images, ultimately obtaining positioning and orientation models for all sub-scenes, which are then merged to obtain the final 3D reconstructed model.

[0098] The present invention provides an adaptive drone image matching pair selection system, comprising:

[0099] The local feature acquisition module is used to obtain UAV images, extract features from them using the SIFT algorithm, and obtain a set of local features of the UAV images.

[0100] The vector acquisition module is used to construct a vector codebook, perform feature aggregation on the local feature set through the vector codebook, and obtain a feature matrix set; vectorize the feature matrix set to obtain a global feature descriptor vector set;

[0101] The retrieval and pairing module is used to perform image retrieval on the global feature descriptor vector set based on the graph index structure to obtain the drone image matching pair set;

[0102] The 3D reconstruction module is used to perform 3D reconstruction of the collection through drone image matching to obtain a 3D reconstructed model.

[0103] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0104] The serial numbers of the embodiments of the present invention are for descriptive purposes only and do not represent superiority or inferiority of the embodiments. In a unit claim that lists several means, several of these means may be embodied by the same item of hardware. The use of the terms first, second, and third, etc., does not denote any order and should be construed as identifiers.

[0105] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An adaptive drone image matching pair selection method, characterized in that: include: S1: Obtain drone images, extract features from them using the SIFT algorithm, and obtain a local feature set of the drone images; S2: Construct a vector codebook, perform feature aggregation on the local feature set through the vector codebook to obtain a feature matrix set; vectorize the feature matrix set to obtain a global feature descriptor vector set; S3: Perform image retrieval on the global feature descriptor vector set based on the graph index structure to obtain a set of drone image matching pairs; S4: 3D reconstruction of the collection is performed through drone image matching to obtain a 3D reconstructed model; Wherein, step S3 is specifically as follows: S31: constructing an index structure by using each global feature descriptor vector in the global feature descriptor vector set as a vertex of the index structure, and the index structure has a total of H layers; S32: The drone image to be queried The vertex corresponding to the global feature descriptor vector Input index structure, where i is the number of the drone image, traverse all vertices in the hth layer, and find the distance vertex Nearest Vertex As the input of h-1 layer; h is the layer number of the index structure, and the initial value is H; S33: Repeat step S32 H times to obtain the input vertex at the bottom of the index structure , traverse all the vertices at the bottom layer to find the distance vertex Recent vertices, As drone imagery of matching pairs, where For drone images The number of similar image retrievals; S34: Repeat steps S32-S33 until all matching pairs of drone images are obtained, and all matching pairs are combined into a drone image matching pair set.

2. The adaptive drone image matching pair selection method according to claim 1, characterized in that: Step S1 is specifically as follows: S11: Obtain the local features of each drone image using the SIFT algorithm; S12: Preserve drone imagery In the figure, the feature scale is from large to small. Local features make up the drone image The local feature set of , where i is the number of the drone image.

3. The adaptive drone image matching pair selection method according to claim 1, characterized in that: Step S2 is specifically as follows: S21: construct a pre-trained vector codebook, train the pre-trained vector codebook using the training data set, and obtain a vector codebook; S22: Perform feature clustering on each local feature set through the vector codebook, so that each drone image is represented as a The characteristic matrix of S23: For each The characteristic matrix is vectorized, and each row in the characteristic matrix is sequentially spliced to obtain the corresponding dimensional global feature descriptor vectors, and compose each global feature descriptor vector into a global feature descriptor vector set.

4. The adaptive drone image matching pair selection method according to claim 1, characterized in that: The specific setting process is as follows: S331: Preliminary estimate of drone imagery Similar images with settings ; S332: Get the number of local features involved in aggregation in all drone images , where n is the total number of drone images; S333: Found Maximum value and minimum value ; S334: Drone Imagery The number of similar images retrieved The calculation formula is: 。 5. An adaptive drone image matching pair selection system, characterized in that: Used to implement the adaptive drone image matching pair selection method described in any one of claims 1 to 4.

Images