Unmanned aerial vehicle image matching method and device, equipment, storage medium and program product

Abstract

The invention provides an unmanned aerial vehicle image matching method and device, equipment, a storage medium and a program product, and relates to the technical field of photogrammetry and computer vision, and the method comprises the steps: obtaining a global feature vector of an unmanned aerial vehicle image, and carrying out the image index processing of the global feature vector, and obtaining an image matching pair; performing data scheduling processing of iterative matrix bandwidth reduction on the image matching pair to generate an image scheduling block; and performing feature matching on the image scheduling block, and eliminating matching error data in a feature matching result to obtain an unmanned aerial vehicle image matching result. According to the method, based on a data scheduling strategy of matrix bandwidth reduction, the sparsely connected image matching graph is divided into compact sub-blocks, the sub-blocks can be loaded and processed more efficiently under the condition that the GPU memory is limited, meanwhile, the feature matching speed is increased by combining feature matching and gross error elimination of cascade hash, and the matching efficiency is improved. Therefore, the rapid feature matching requirement of large-scale unmanned aerial vehicle images is met.

Description

Technical Field

[0001] The present invention relates to the field of photogrammetry and computer vision technology, and in particular to a method, device, equipment, storage medium and program product for unmanned aerial vehicle image matching. Background Art

[0002] Unmanned Aerial Vehicle (UAV) is an important data source in the field of photogrammetry and remote sensing. UAVs are highly flexible, timely, and high-resolution, and are widely used in terrain mapping, agricultural yield estimation, disaster prevention and mitigation, etc. High-resolution UAV images with high timeliness are the key to ensuring their widespread application.

[0003] At present, Structure from Motion (SfM) is a key link in UAV image processing. Among them, fast feature matching is a necessary prerequisite for SfM to efficiently process large-scale UAV images. The existing exhaustive matching strategy based on the nearest neighbor search of high-dimensional feature descriptors is very time-consuming and difficult to process large-scale UAV images. Compared with the approximate nearest neighbor search based on KD-Tree, Cascade Hashing, which combines Graphics Processing Unit (GPU) acceleration and data scheduling, has received a lot of attention in feature matching. However, the existing technical solutions cannot meet the needs of fast feature matching of large-scale UAV images. Summary of the invention

[0004] The present invention provides a drone image matching method, device, equipment, storage medium and program product to solve the defect that the prior art cannot meet the demand for rapid feature matching of large-scale drone images, thereby meeting the demand for rapid feature matching of large-scale drone images and improving the feature matching efficiency of drone images.

[0005] The present invention provides a drone image matching method, comprising the following steps: Obtaining a global feature vector of the drone image, and performing graph index processing on the global feature vector to obtain an image matching pair; performing data scheduling processing of iterative matrix bandwidth reduction on the image matching pairs to generate image scheduling blocks; Feature matching is performed on the image scheduling block, and matching error data in the feature matching result is eliminated to obtain the drone image matching result.

[0006] According to a drone image matching method provided by the present invention, the data scheduling process of iterative matrix bandwidth reduction is performed on the image matching pair to generate an image scheduling block, including: Construct an adjacency matrix based on the image matching pairs; Construct a symmetric adjacency matrix based on the adjacency matrix, and generate an image ID list based on the image indices corresponding to the rows or columns of the adjacency matrix; Construct a matrix bandwidth reduction MBR matrix based on the permutation order of the symmetric adjacency matrix, and update the image ID list based on the permutation order; the permutation order is used to reorder the image indices of the rows and columns of the symmetric adjacency matrix; Under the condition of the memory limit of the graphics processing unit GPU, create a new scheduling block based on the MBR matrix; After performing feature matching based on the new scheduling block, update the MBR matrix and the image ID list; Iteratively optimize the bandwidth of the updated MBR matrix to generate the image scheduling block.

[0007] According to a method for unmanned aerial vehicle image matching provided by the present invention, the obtaining of the global feature vector of the unmanned aerial vehicle image includes: Extract the local features of the unmanned aerial vehicle image; Randomly sample the unmanned aerial vehicle image, and sort the local features of the randomly sampled unmanned aerial vehicle image according to a scale factor to obtain a sorting result; the scale factor is the size or resolution of the local feature in different scale spaces; Perform hierarchical clustering on multiple target local features selected based on the sorting result to obtain a feature set; Use the feature set to aggregate the multiple target local features of all the unmanned aerial vehicle images to obtain the global feature vector.

[0008] According to a method for unmanned aerial vehicle image matching provided by the present invention, the performing of graph index processing on the global feature vector to obtain image matching pairs includes: Use the global feature vector as a graph node, and connect the graph nodes according to the similarity between the global feature vectors to establish a vector index graph; Perform approximate nearest neighbor retrieval based on the vector index graph to obtain the image matching pairs.

[0009] According to a method for unmanned aerial vehicle image matching provided by the present invention, the performing of feature matching on the image scheduling block includes: Load the feature descriptors of the image matching pairs in the image scheduling block into the GPU, calculate the hash code and bucket ID of the feature descriptors to construct a hash lookup table; the bucket ID is used to uniquely identify a hash bucket; Perform hash feature matching based on the hash lookup table to obtain candidate feature descriptors; Perform a ratio test screening on the candidate feature descriptors to eliminate the unmatched candidate feature descriptors; Repeat the feature matching process of the image scheduling block until the image scheduling block is processed completely, and obtain the candidate feature descriptors of the image matching pairs.

[0010] According to a UAV image matching method provided by the present invention, eliminating the mismatched data in the feature matching result to obtain the UAV image matching result includes: Perform local geometric constraint and global geometric constraint processing on the candidate feature descriptors in the feature matching result to determine the candidate feature descriptors with matching errors; Eliminate the candidate feature descriptors with matching errors to obtain the UAV image matching result.

[0011] The present invention also provides a UAV image matching device, including the following modules: An acquisition module, configured to acquire the global feature vector of the UAV image, and perform graph indexing processing on the global feature vector to obtain image matching pairs; An image scheduling block generation module, configured to perform data scheduling processing for iterative matrix bandwidth reduction on the image matching pairs to generate image scheduling blocks; A feature matching module, configured to perform feature matching on the image scheduling blocks, and eliminate the mismatched data in the feature matching result to obtain the UAV image matching result.

[0012] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the UAV image matching method described in any one of the above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the UAV image matching method described in any one of the above is implemented.

[0014] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, the UAV image matching method described in any one of the above is implemented.

[0015] The UAV image matching method, device, equipment, storage medium and program product provided by the present invention obtain the global feature vector of the UAV image, perform graph indexing processing on the global feature vector to obtain image matching pairs; perform data scheduling processing on the image matching pairs for iterative matrix bandwidth reduction to generate image scheduling blocks; perform feature matching on the image scheduling blocks, and eliminate the mismatched data in the feature matching results to obtain the UAV image matching result. Based on the data scheduling strategy of matrix bandwidth reduction, the present invention divides the sparsely connected image matching graph into compact sub-blocks, which can be more efficiently loaded and processed under the limited GPU memory. At the same time, by combining feature matching and gross error elimination of cascaded hashing, the matching speed of features is improved, so as to meet the rapid feature matching requirements of large-scale UAV images. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0017] Figure 1 is one of the schematic flowcharts of the UAV image matching method provided by the present invention.

[0018] Figure 2 is the second schematic flowchart of the UAV image matching method provided by the present invention.

[0019] Figure 3 is the schematic flowchart of obtaining the VLAD vector provided by the present invention.

[0020] Figure 4 is the schematic diagram of the original adjacency matrix provided by the present invention.

[0021] Figure 5 is the schematic diagram of the MBR matrix provided by the present invention.

[0022] Figure 6 is the schematic diagram of constructing the nearest neighbor and its corresponding matching graph of the Delaunay triangulation provided by the present invention.

[0023] Figure 7 is the schematic structural diagram of the UAV image matching device provided by the present invention.

[0024] Figure 8 is the schematic structural diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0025] To make the objectives, technical solutions, and advantages of the present invention more clear, the following will, in conjunction with the accompanying drawings of the present invention, clearly and completely describe the technical solutions in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without making creative efforts fall within the scope of protection of the present invention.

[0026] The following will be combined with Figures 1 - 8 describe the UAV image matching method, device, equipment, storage medium, and program product of the present invention.

[0027] Figure 1 is one of the flow schematic diagrams of the UAV image matching method provided by the present invention. As Figure 1 shown, the method includes the following: Step 101, obtain the global feature vector of the UAV image, and perform graph indexing processing on the global feature vector to obtain image matching pairs.

[0028] Specifically, UAV images contain rich visual information, and the global feature vector aims to generally summarize the key features of these images. For example, based on the method of local feature aggregation, the local features in each UAV image can be extracted first, and then the local features are integrated through coding and aggregation methods to form a global feature vector that can represent the entire image. Among them, the global feature vector can reflect the comprehensive features such as the texture, shape, and color distribution of the image to a certain extent, laying a foundation for the subsequent retrieval of image matching pairs.

[0029] Graph indexing is an efficient data organization and retrieval method, which stores and associates data in the form of a graph structure. For the global feature vector of UAV images, by constructing a graph index, nodes and connection edges can be established in the graph according to the similarity relationship between global feature vectors. When performing retrieval, other vectors similar to the target feature vector in the graph structure can be quickly found, and the corresponding images can be determined to form image matching pairs. For example, after obtaining the global feature vector of each UAV image, the global feature vector is inserted into the constructed graph index as a node, and the connection edges between nodes are constructed according to the similarity between global feature vectors. When retrieving a certain image matching pair, using the global feature vector of this image as the query starting point, traverse and search in the graph index according to the hierarchical search strategy, and find the images corresponding to other vectors with a higher similarity to the query vector to form image matching pairs.

[0030] It is understandable that the two images in an image matching pair have similar local or global features. For example, after a certain area has been photographed by a drone multiple times, two images of the same object (such as the same building) taken from different angles can be found, and these two images form an image matching pair.

[0031] Step 102, perform data scheduling processing for iterative matrix bandwidth reduction on the image matching pair to generate an image scheduling block.

[0032] Specifically, adopt a data scheduling strategy of iterative matrix bandwidth reduction (Matrix Band Reduction, MBR) to schedule and process the image matching pair to generate an image scheduling block. Among them, matrix bandwidth is a measure of the distribution of non-zero elements in a matrix. A narrower bandwidth means that the matrix elements are more compact in structure. The MBR technology continuously reduces the bandwidth of the matrix through iteration to optimize the matrix structure. The data scheduling strategy refers to a scheme for determining the allocation, transfer, and invocation of data among different computing units (such as CPUs, GPUs), storage units, and different processing stages when facing a large amount of drone image data, with the aim of improving the overall efficiency of data processing.

[0033] In one embodiment, performing data scheduling processing for iterative matrix bandwidth reduction on the image matching pair to generate an image scheduling block includes: constructing an adjacency matrix based on the image matching pair; constructing a symmetric adjacency matrix based on the adjacency matrix, and generating an image ID list based on the image indexes corresponding to the rows or columns of the adjacency matrix; constructing a matrix bandwidth reduction MBR matrix based on the permutation order of the symmetric adjacency matrix, and updating the image ID list based on the permutation order; the permutation order is used to reorder the image indexes of the rows and columns of the symmetric adjacency matrix; in the case of the memory limit of the graphics processing unit GPU, create a new scheduling block based on the MBR matrix; after performing feature matching based on the new scheduling block, update the MBR matrix and the image ID list; iteratively optimize the bandwidth of the updated MBR matrix to generate an image scheduling block.

[0034] A data scheduling strategy of iterative matrix bandwidth reduction MBR can be adopted to generate an image scheduling block. Matrix bandwidth reduction MBR is a technology for optimizing matrix representation, aiming to reduce the bandwidth of the matrix, thereby reducing the computational complexity, saving memory, and improving the numerical operation efficiency. The goal of matrix bandwidth reduction is to reduce the bandwidth of the matrix to the smallest possible value by rearranging the rows and columns of the matrix (for example, adopting a certain specific permutation rule). Matrix bandwidth refers to the distribution range of non-zero elements in a matrix relative to the main diagonal. Specifically, matrix bandwidth describes the relative positions of the rows and columns of non-zero elements in the matrix, especially the distance from the main diagonal.

[0035] Assume that the total number of drone images is Zhang images, according to image matching pairs Construct an adjacency matrix . The adjacency matrix The construction process is as follows: For each pair of image matching pairs , set the corresponding entry to 1, that is , indicating that there is an image matching pair between image and image ; otherwise, set the corresponding entry to 0, that is , indicating that there is no image matching pair between image and image ; . .

[0036] Data scheduling based on the iterative matrix bandwidth reduction MBR may include the following steps: (1) Initialization: Construct a symmetric adjacency matrix according to the original adjacency matrix , and generate a list of image IDs using the image indices derived from the rows or columns of . For example, the symmetric adjacency matrix can be obtained by adding it to its transpose matrix : : ; where represents the transpose.

[0037] (2) Calculate the permutation order: By using the MBR algorithm (such as the Gibbs-Poole-Stockmeyer (GPS) MBR algorithm), generate a permutation order , where the permutation order is used to reorder the image indices of the rows and columns of the symmetric adjacency matrix. Then, construct the MBR matrix according to the permutation order . For example, reorder the image indices of the rows and columns of the symmetric adjacency matrix according to the permutation order to obtain the MBR matrix . The purpose is to reduce the matrix bandwidth, make the matrix elements more compactly distributed, thus optimizing the data structure and facilitating subsequent data processing and calculation. Especially when dealing with large-scale data (such as UAV image data), it can improve the processing efficiency.

[0038] While constructing the MBR matrix , according to the generated permutation order Update the image ID list . Among them, the image ID list is used to record the identification information of the images. By updating it, it can accurately correspond to the rearranged matrix data, ensuring that in the subsequent data processing process, the image data can be correctly processed and scheduled according to the new order and identification information, guaranteeing the accuracy and coherence of data processing.

[0039] (3)Create a new scheduling block: Under the GPU memory limit, use the MBR matrix to create a new scheduling block. As Figure 5 shown, the GPU memory size is limited by the grid, indicating the maximum number of descriptor data that can be loaded into the GPU memory. By defining the block size , a new scheduling block rendered as a rectangle in Figure 5 is created, where and and represent the row and column indices of each block respectively. The block stores its corresponding image matching pairs: the original adjacency matrix of the coverage area corresponding image matching pairs .

[0040] For example, assume is 100 and is 10. The entire data space can be divided into multiple 10×10 scheduling blocks. In actual operation, according to the information in the MBR matrix , the relevant image matching pairs are assigned to these scheduling blocks. Based on this, large-scale data can be segmented into smaller blocks suitable for GPU processing, improving the efficiency and parallelism of data processing.

[0041] (4)Update the adjacency matrix and the image ID list: After the feature matching guided by the scheduling block , update the MBR matrix by setting the cell values of the corresponding image matching pairs to zero and deleting the rows and columns without image matching pairs. In addition, update the image ID list correspondingly.

[0042] (5)Termination: Repeat steps (2)-(4) until the MBR matrix is empty. Finally, the image scheduling block can be obtained.

[0043] The embodiment of the present invention can reduce the loading and release of data in the GPU as much as possible by using the MBR algorithm to perform adjacency matrix compression and block generation, and at the same time, as many image matching pairs as possible can be packed into a scheduling block, so as to achieve high GPU utilization, and balance the data scheduling burden and GPU computing power utilization. On the other hand, through the matrix bandwidth reduction technology, the non-zero elements of the sparse matrix can be integrated into a more compact central area around the diagonal, so that the originally sparse connections become relatively concentrated to a certain extent. Based on this relatively concentrated structure, the image matching graph can be more conveniently divided into compact sub-blocks. In the case of limited GPU memory, these sub-blocks can be loaded and processed more efficiently, because the data in each sub-block is relatively more closely related, reducing the jumps and discontinuities in the data loading and processing process, thereby improving the efficiency and speed of data processing.

[0044] Step 103, performing feature matching on the image scheduling block, and eliminating matching error data in the feature matching result to obtain the drone image matching result.

[0045] Based on the data scheduling strategy of iterative matrix bandwidth reduction (MBR), the drone image data is divided into multiple image scheduling blocks, and then feature matching is performed on these image scheduling blocks, that is, the matching relationship between images is found and determined based on the image features (such as local feature descriptors) within each scheduling block and between different scheduling blocks. For example, for an image in one scheduling block and an image in another scheduling block, by comparing their features, it is determined whether they are images of the same object or scene at different perspectives, thereby determining the matching relationship between them.

[0046] In the feature matching process, due to the limited distinguishing ability of local feature descriptors, large viewing angle deformation, noise and other factors, some false matches will inevitably occur, thus affecting the accuracy and reliability of the final image matching results. In order to obtain accurate drone image matching results, it is necessary to eliminate false matches based on the feature matching results, identify and remove false matches that do not conform to geometric relationships or other rules from the preliminary feature matching results, and retain truly valid matching pairs.

[0047] After feature matching of the image scheduling block and elimination of false matches, the final result is the drone image matching result. The drone image matching result can reflect the correspondence between different images taken by the drone and can be used for various subsequent applications, such as image stitching (stitching multiple drone images into a large panoramic image), 3D reconstruction (building a 3D model of the shooting scene based on image matching relationships), target detection and tracking (accurately identifying and tracking targets in different images), etc.

[0048] The UAV image matching method provided by the embodiment of the present invention obtains the global feature vector of the UAV image, performs graph indexing processing on the global feature vector to obtain image matching pairs, performs data scheduling processing of iterative matrix bandwidth reduction on the image matching pairs to generate image scheduling blocks, performs feature matching on the image scheduling blocks, and eliminates the mismatched data in the feature matching results to obtain the UAV image matching result. Based on the data scheduling strategy of matrix bandwidth reduction, the present invention divides the sparsely connected image matching graph into compact sub-blocks, which can be more efficiently loaded and processed under the limited GPU memory. At the same time, by combining the feature matching and gross error elimination of cascaded hashing, the matching speed of features is improved, thus meeting the fast feature matching requirements of large-scale UAV images.

[0049] In one embodiment, the obtaining of the global feature vector of the UAV image includes: Step 111, extracting the local features of the UAV image; Step 112, randomly sampling the UAV image, and sorting the local features of the randomly sampled UAV image according to the scale factor to obtain a sorting result; the scale factor is the size or resolution of the local feature in different scale spaces; Step 113, performing hierarchical clustering on multiple target local features selected based on the sorting result to obtain a feature set; Step 114, aggregating the multiple target local features of all the UAV images by using the feature set to obtain the global feature vector.

[0050] The local features of UAV images are extracted using the SIFT (Scale Invariant Feature Transform) feature extraction method. This method has the advantages of illumination invariance, rotation invariance, and scale invariance, and can well adapt to multi-view and high-resolution UAV images. In the specific implementation process, the SIFTGPU library implemented by GPU is used for local feature extraction of UAV images. Specifically, first, the original images taken by the UAV are preprocessed, including operations such as image format conversion, cropping, and scaling, to make them meet the input requirements of the SIFTGPU library. For example, the images can be converted to grayscale images because the SIFT algorithm is based on grayscale images for feature extraction, which can reduce the amount of calculation and data volume. Then, on the preprocessed images, the SIFTGPU library is called through the programming interface. The library function will automatically utilize the parallel computing power of the GPU to perform local feature extraction on the images according to the process of the SIFT algorithm. During the local feature extraction process, developers do not need to care about the underlying details of the GPU. They only need to set some necessary parameters, such as the threshold for feature extraction, the number of pyramid levels, etc. The SIFTGPU library will automatically complete the feature extraction task and return the extracted local features and their feature descriptors.

[0051] UAV images have high redundancy, which is mainly reflected in two aspects: the redundancy of the number of images and the redundancy of the number of features. The redundancy of the number of images means that the UAV takes a large number of similar or duplicate images; the redundancy of the number of features means that due to the high resolution of UAV images, the number of feature points extracted from each image is too large, which contains a lot of similar or duplicate feature information.

[0052] To solve the problem of redundancy in the number of images, a random sampling strategy can be adopted to select a subset of images with a given proportional from the original dataset. Among them, the value of the proportion can be 20%, that is, 20 percent of the number of input images. For example, assuming there are 1000 UAV images, through the random sampling strategy, only 200 of them need to be processed, which reduces the amount of calculation and data storage requirements. At the same time, without losing important information, the number of images to be processed can be reduced, improving the efficiency of subsequent processing.

[0053] To eliminate the feature redundancy caused by the high resolution of UAV images, a scale constraint strategy can be adopted. The specific steps are as follows: First, the local features of the UAV images obtained by random sampling are scaled according to the scale factor Arrange them in descending order, which means that local features with larger scale factors will be ranked ahead in the sorting. Among them, the scale factor is the size or resolution of the local feature in different scale spaces, reflecting the relative change degree of the size or resolution of the features in the image. For example, for images taken by drones, different scale factors can correspond to ground objects or features of different sizes in the image.

[0054] Then, select the first local features from the sorted local features as the feature subset. Among them, can take the value of 1500. If the number of local features of a certain image is less than , then keep all local features. The purpose of doing this is to screen out the most representative and important local features in each image, remove redundant local features that contribute less to subsequent processing, thereby reducing the number of features and improving the quality and processing efficiency of the features. For example, an image extracts 3000 initial local feature points through the SIFT algorithm. According to the scale constraint strategy, the first 1500 local feature points are selected for subsequent operations such as vocabulary tree training.

[0055] After processing the redundancy of the number of images through the above random sampling strategy and the redundancy of the number of features through the scale constraint strategy, all the remaining local features will form the training features of the vocabulary tree, which are used for subsequent tasks such as image matching, classification, and retrieval, providing a more efficient and targeted data basis for the further analysis and application of drone images. Through the random sampling strategy and the scale constraint strategy, on the premise of ensuring information integrity, the data volume and feature redundancy are effectively reduced, providing an optimized data basis and processing process for subsequent image processing and analysis.

[0056] Further perform hierarchical clustering on multiple target local features selected based on the sorting results to obtain a feature set (i.e., the codebook). Specifically, in order to be able to quantify and compare local features, a feature descriptor is generated for each local feature. Among them, the feature descriptor is a set of data or vectors used to describe the visual information of the local area around the local feature, containing various information about the local feature, such as position, scale, direction, and gradients of surrounding pixels, etc. Use hierarchical k - means clustering algorithm for feature descriptor clustering, where k value represents the number of branches of the vocabulary tree. It can include the following steps: 1) Definition of the distance function: For any two feature descriptors and , use the squared Euclidean distance as the distance function of k - means clustering, and its formula is: ; Among them, using the squared Euclidean distance can effectively reflect the difference degree between different feature descriptors, providing a basis for subsequent clustering operations.

[0057] 2) Determine the initial clustering centers: Use the K-means++ algorithm to determine initial clustering centers.

[0058] 3) First-layer clustering division: According to the nearest distance measure, divide the input set of feature descriptors into subsets to form the first layer of the vocabulary tree (the root node is the zeroth layer). Among them, the nearest distance measure is based on the squared Euclidean distance defined above. Specifically, for each feature descriptor, calculate its distance from initial clustering centers, and then assign it to the subset corresponding to the nearest clustering center. Based on this, the training feature set can be initially divided into non-overlapping subsets, and each subset can be regarded as a branch of the vocabulary tree.

[0059] 4) Iterative clustering to construct the vocabulary tree: According to the above steps, perform iterative clustering operations on each subset until the number of layers or the number of leaf nodes of the vocabulary tree reaches a given threshold. Specifically, take each subset as a new training feature set, and use the K-means++ algorithm again to determine the new initial clustering centers, and then divide them according to the nearest distance measure to obtain the subsets of the next layer. Repeat this iterative process to continuously subdivide the feature subsets, forming a hierarchical vocabulary tree structure. When the number of layers of the vocabulary tree reaches a preset value (such as 5 layers) or the number of leaf nodes reaches a certain number (such as 100), stop the iteration.

[0060] By constructing a pre-trained codebook through hierarchical mean clustering algorithm, a large number of training features can be structured and represented, forming a vocabulary tree with a hierarchical relationship. This structure can reflect the similarity and hierarchical relationship between training features to a certain extent, providing an efficient indexing and matching method for subsequent image processing tasks.

[0061] Further utilize the feature set to aggregate multiple target local features of all UAV images to obtain a global feature vector. Specifically, the VLAD (Vector of Locally Aggregated Descriptors) vector aggregation technology can be adopted to aggregate a large number of low-dimensional SIFT local feature vectors into a small number of high-dimensional VLAD global feature vectors. Among them, the core idea of the VLAD vector aggregation technology is to accumulate the residual vectors between the local feature descriptor vectors and their corresponding cluster center vectors. For the feature set after hierarchical clustering, it is divided into the associated features of the cluster center according to the principle of the closest Euclidean distance, and the sum of the residuals between the cluster center and its associated features is calculated, that is, the VLAD feature descriptor is obtained. For cluster centers of the codebook, the feature matrix of can be calculated, where the matrix element has the following calculation formula: ; where represents the element in the th row and th column of the feature matrix; represents the rd local feature; represents the th cluster center; represents the residual value when the local feature falls within this cluster center; represents a sign function; if does not belong to the cluster center , then is 0; if belongs to the cluster center , then is 1.

[0062] After the above calculations, each image can be represented as a feature matrix. Among them, is the number of rows of the matrix, which is equal to the number of centers of the pre-trained codebook; is the number of columns of the matrix, which is equal to the dimension of the local features. Among them, can take the value of 128. Since the matrix does not conform to the input of subsequent retrieval, the calculated feature matrix will be vectorized, and each row in the feature matrix is sequentially concatenated, the feature matrix of becomes a VLAD vector of

[0063] In an embodiment of the present invention, by comprehensively considering the feature information of each local region in the image, local features are aggregated into a global feature vector, which reduces the data volume and improves the efficiency of subsequent processing while retaining the main information of the image.

[0064] In one embodiment, the obtaining of the image matching pairs by performing graph indexing processing on the global feature vector includes: Step 121: Using the global feature vector as a graph node, and connecting the graph nodes according to the similarity between the global feature vectors to establish a vector index graph; Step 122: Performing approximate nearest neighbor search based on the vector index graph to obtain the image matching pairs.

[0065] After obtaining the VLAD global feature vectors of each UAV image, the similarity between vectors can be judged by directly calculating the Euclidean distance between the vectors, so as to reflect the similarity between the images. For example, the HNSW (Hierarchical Navigable Small World) graph indexing structure can be used to implement approximate nearest neighbor search, which constructs a vector index graph using a hierarchical structure graph.

[0066] Specifically, in HNSW, each VLAD global feature vector is used as a graph node, and the graph nodes are connected according to the similarity of the VLAD global feature vectors (for example, using Euclidean distance or cosine similarity metric). Among them, the connected graph nodes can be nodes with higher similarity, thus forming a graph structure with an approximate shortest path, that is, a vector index graph. HNSW organizes nodes through a hierarchical structure. The lower levels contain more nodes and fewer connections; while the higher levels contain fewer nodes, but each node has more connections with other nodes. This structure can accelerate the nearest neighbor search and reduce the amount of calculation.

[0067] For each image to be queried, its approximate nearest neighbors in the HNSW graph index are obtained by querying its VLAD global feature vector, and this process can be accelerated by the hierarchical search strategy in the HNSW graph. The specific steps may include: starting from the top layer (with fewer nodes) of the graph to find the most similar node; then gradually entering the next layer according to the query result of the top layer until the candidate nodes at the bottom layer are found; when searching at the bottom layer, several of the most similar nodes are selected by comparing the distances between the query point and the candidate nodes. After finding the candidate approximate nearest neighbor nodes (i.e., the VLAD global feature vectors of other images), the similarity between these images and the query image can be judged by calculating the distances between them (such as Euclidean distance or cosine similarity). If the similarity is greater than the set threshold, it is determined that the two images match, that is, an image matching pair. According to the nearest neighbor information returned by HNSW and combined with the similarity threshold, a set of image matching pairs can be filtered out.

[0068] In the embodiment of the present invention, by combining the VLAD global feature vector and the HNSW graph index, image matching pairs are determined, realizing efficient processing of large-scale image data, while improving the calculation efficiency and reducing the time consumption.

[0069] In one embodiment, the feature matching of the image scheduling block includes: Step 131: Load the feature descriptors of the image matching pairs in the image scheduling block into the GPU, calculate the hash codes and bucket IDs of the feature descriptors to construct a hash lookup table; the bucket ID is used to uniquely identify the hash bucket; Step 132: Perform hash feature matching based on the hash lookup table to obtain candidate feature descriptors; Step 133: Perform ratio test screening on the candidate feature descriptors to eliminate the unmatched candidate feature descriptors; Step 134: Repeatedly execute the feature matching process of the image scheduling block until the image scheduling block is processed, and obtain the candidate feature descriptors of the image matching pairs.

[0070] The Cascade Hashing feature matching method can be adopted to perform feature matching on the image scheduling block.

[0071] Cascade Hashing feature matching is performed by using the generated image scheduling block Since the sum of the amounts of image data in each block row has considered the GPU memory limit, the feature descriptors of each sub-block in the block row can be loaded into the GPU at one time. Assuming that the generated block is composed of rows and Column composition, current block row Is represented as . The Cascade Hashing feature matching can proceed as follows: (1) Loading feature descriptors: The feature descriptors of the image matching pairs in each block are sequentially loaded into the GPU, and hash codes and bucket IDs are calculated to construct a hash lookup table.

[0072] For example, first, the feature descriptors in the first block are sequentially loaded into the GPU. In the GPU, hash codes and bucket IDs are calculated for each feature descriptor. The hash function can map a 128-dimensional feature descriptor to a shorter hash code (e.g., a 32-bit hash code), and then the hash bucket corresponding to the feature descriptor is determined according to the hash code. The bucket ID is used to uniquely identify the hash bucket, and the bucket ID can be the number or identifier of the hash bucket. Based on the hash code and bucket ID of the feature descriptor, a hash lookup table is constructed.

[0073] (2) Cascade hashing matching: Based on the established hash lookup table, hash feature matching is performed, and the initial candidate matches (i.e., candidate feature descriptors) that pass the ratio test and are guided by the image matching pairs are retained.

[0074] For example, for a query image, its feature descriptor is extracted and hash code and bucket ID are calculated. Then, according to the bucket ID of the query image feature descriptor, the corresponding hash bucket is found in the hash lookup table, and all the feature descriptors in the bucket (i.e., candidate feature descriptors) are obtained. For each candidate feature descriptor, its similarity with the query image feature descriptor is calculated (methods such as Euclidean distance, cosine similarity, etc. can be used), and then a ratio test is performed. For example, for a candidate feature descriptor A , calculate its distance from the query image feature descriptor d A , and at the same time find the feature descriptor that is the second closest to the query image feature descriptor in the same hash bucket as A , calculate the distance B . If d B . If d A / d B is less than a set threshold (e.g., 0.8), then A is considered a good initial candidate match and is retained; otherwise, it is discarded. By screening through the ratio test, some false matches can be removed, and the feature descriptors that are more likely to be true matches are retained.

[0075] (3) Data block release: Steps (1) and (2) are repeatedly executed until the current block row All blocks in are processed. Since the data of the first block will no longer be used when processing the next block row, it is released from the GPU to free up memory space. The data of the first block will no longer be used when processing the next block row, so it is released from the GPU to free up memory space.

[0076] For each block row , execute the above process to obtain the initial candidate matches of all relevant image matching pairs. For example, load the feature descriptors of the second block into the GPU, and then repeat the above hash calculation, cascaded hash matching, and ratio test processes. In this way, all blocks in the block row are processed in sequence to obtain the candidate feature descriptors of all relevant matching pairs. The feature descriptors of the second block are loaded into the GPU, and then the above hash calculation, cascaded hash matching, and ratio test processes are repeated. In this way, all blocks in the block row are processed in sequence to obtain the candidate feature descriptors of all relevant matching pairs.

[0077] Embodiments of the present invention have significant advantages in processing the feature matching task of large-scale UAV image data through reasonable GPU memory utilization, efficient feature matching processes, and optimized data processing methods, and can improve the processing speed, efficiency, and scalability.

[0078] In one embodiment, eliminating the mis-matched data in the feature matching results to obtain the UAV image matching results includes: Step 141, performing local geometric constraint and global geometric constraint processing on the candidate feature descriptors in the feature matching results to determine the candidate feature descriptors with mis-matches; Step 142, eliminating the candidate feature descriptors with mis-matches to obtain the UAV image matching results.

[0079] When performing feature matching, relying solely on local feature descriptors for matching will inevitably result in incorrect matches in the initial matching results due to the limited discrimination ability of local feature descriptors themselves and problems such as large-angle deformation, thus affecting subsequent image processing and analysis tasks, such as image stitching, target recognition, etc. Therefore, it is necessary to eliminate incorrect matches.

[0080] Perform local geometric constraint and global geometric constraint processing on the candidate feature descriptors in the feature matching results to determine the candidate feature descriptors with mis-matches, and then eliminate the candidate feature descriptors with mis-matches to obtain the UAV image matching results.

[0081] Specifically, false match rejection is performed by combining the local geometric constraint of Spatial Angle Order (SAO) and the global geometric constraint of RANSAC (Random Sample Consensus). Among them, the local geometric constraint of SAO is based on the relative spatial angle relationship between feature points and is used to ensure that the relative geometric relationship between the matched feature points is reasonable in the two images. In image matching, RANSAC is used to estimate geometric transformations (such as homography matrix or fundamental matrix), and by rejecting false match points that do not conform to the transformation model, accurate matches are finally obtained.

[0082] Combining the local geometric constraint of SAO and the global geometric constraint of RANSAC for false match rejection can be divided into the following steps: Step 1: Preliminary matching and angle order screening: First, use feature extraction and matching algorithms to match the two images. After obtaining the preliminary matching point pairs, apply the Spatial Angle Order (SAO) to screen out the matching point pairs that do not conform to the geometric constraints. By calculating the relative angles between the matching points, false matches with inconsistent angle orders are rejected. It can include the following steps: 1.1 Calculate the angle order between the matching point pairs in Image 1; 1.2 Correspondingly, calculate the angle order between the matching point pairs in Image 2; 1.3 Reject false matches that do not conform to the geometric constraints through angle order consistency.

[0083] Step 2: Apply RANSAC for global optimization: Apply RANSAC for global optimization on the set of matching points after SAO screening. Estimate the transformation model (such as homography matrix) through RANSAC and reject the matching points that do not conform to the model. It can include the following steps: 2.1 On the matching points after SAO screening, use RANSAC to estimate the geometric transformation model (such as homography matrix); 2.2 RANSAC will identify the inliers that best conform to the transformation model and reject the outliers that do not conform.

[0084] Step 3: Final matching: After optimization by SAO and RANSAC, the remaining matching points are the final accurate matching points.

[0085] Optionally, the local geometric constraint can be realized by using Delaunay triangulation to construct the nearest neighbor and its corresponding graph. The specific implementation steps can include: 3.1 Feature points and nearest neighbors: In image matching, feature points need to be extracted from two images through a feature extraction algorithm first. Then, these feature points can be regarded as a point set. On this point set, Delaunay triangulation is used to construct a triangular network. The connection relationship (i.e., adjacency relationship) between each feature point and other surrounding feature points forms local spatial geometric constraints. For a certain feature point, its nearest neighbor is determined by the adjacent points in the Delaunay triangular network. The Delaunay triangular network can effectively find the local neighborhood of each point and establish reasonable connections for each pair of neighborhood points.

[0086] 3.2 Constructing local constraints between matching points: Using the adjacency relationship of the Delaunay triangular network, a point pair graph can be constructed, where each node represents a feature point and the edge represents the geometric relationship between feature points. The edges in the graph can be based on the following geometric constraints: Relative positions of adjacent points: Through the adjacent points in the Delaunay triangular network, it can be determined whether the relative positions of the matching points meet the expectations. The matching points in Image 1 and Image 2 should maintain similar relative positions, that is, the relationships such as angles and distances between them should be consistent.

[0087] Angle order: For each pair of matching points and their adjacent points, the angle order between them can be calculated. Through the adjacency relationship constructed by the local triangular network, it can be ensured that the angle relationship of the matching points conforms to geometric constraints. For example, the relative angle between points A and B in Image 1 should also be the same as the angle between the corresponding points A' and B' in Image 2.

[0088] 3.3 Rejecting incorrect matches: Using the geometric constraints of the Delaunay triangular network, it can effectively judge whether a match is reasonable and reject incorrect matches that do not conform to geometric constraints. Specifically, it can include: Geometric consistency check: For each pair of matching points, by checking the geometric relationships (such as angle order, adjacency relationship, etc.) between them and their nearest neighbor points, it is judged whether they satisfy local geometric constraints. If there is an inconsistency in the adjacency relationship of a certain pair of points in the two images, it is considered that this match may be incorrect.

[0089] Local geometric model: In the local triangular network, the relative positions and angles of all points follow certain geometric laws. If the relative geometric relationship of the matching points does not conform to this model with the relationship of their adjacent points, it can be judged as an incorrect match and this match is rejected.

[0090] 3.4. Construct the corresponding graph: The local geometric constraint relationships established through the Delaunay triangulation can be further used to construct a corresponding graph. The nodes in this graph represent feature points, and the edges represent the geometric constraint relationships between feature points. The matching candidates for each feature point can be screened through the geometric constraints of this graph, thereby removing unreasonable matches. Through local geometric constraints, the edges in the corresponding graph can be weighted, and the edges that are inconsistent with the geometric relationships of surrounding nodes can be removed. Finally, the remaining edges represent the correct matches that conform to the local geometric constraints.

[0091] The embodiments of the present invention can effectively improve the accuracy of feature matching by combining the SAO local geometric constraint and the RANSAC global geometric constraint. The SAO is used to screen out the incorrect matches that do not conform to the angular order, and then the RANSAC is used to further remove the matching points that do not conform to the global transformation model. Finally, a more robust and accurate matching result is obtained.

[0092] To further analyze and explain the UAV image matching method proposed by the present invention, the following embodiments are referred to.

[0093] The embodiments of the present invention specifically propose an efficient UAV image matching method. As Figure 2 shown, the specific implementation steps include the following parts: Step 1, Retrieval of image matching pairs using VLAD global descriptor and HNSW graph index.

[0094] Step 1.1, SIFT local feature extraction: The SIFT feature extraction method is used to extract the local features of UAV images. The SIFT feature extraction method has the advantages of illumination invariance, rotation invariance, and scale invariance, and can well adapt to multi-view and high-resolution UAV images. In the specific implementation process, the SIFTGPU library implemented by GPU can be used to extract the local features of UAV images.

[0095] Step 1.2, Selection of training features for the codebook: The high redundancy of UAV images is mainly manifested in two aspects, including the redundancy of the number of images and the redundancy of the number of features. For the redundancy of the number of images, a random sampling strategy can be adopted to select a given proportion of the image subset from the original dataset. Among them, the value of the proportion can be 20%, that is, twenty percent of the input number of images.

[0096] To eliminate the feature redundancy caused by the high resolution of UAV images, a scale constraint strategy can be adopted to select a given number of feature subsets from the initial feature points of each image. The specific steps of feature selection can include: 1) Sort the initial feature points according to the scale factor Arrange in descending order; 2) Select the first feature points from the sorted feature points. Among them, the number of features can take the value of 1500. If the initial number is less than 1500, then all feature points are retained. After the above processing, all the retained feature points form the training features of the vocabulary tree.

[0097] Step 1.3, Construction of the pre-trained codebook: For the training features selected in Step 1.2, use hierarchical k-means clustering algorithm for feature descriptor clustering. Among them, value represents the number of branches of the vocabulary tree. For any two feature descriptors and , the distance function of k-means clustering is defined as the squared Euclidean distance of the feature descriptors, and the formula is as follows: ; For the training feature set, 1) First, use the K-means++ algorithm to determine initial cluster centers; 2) Then, according to the nearest distance measure, divide the input feature descriptor set into subsets to form the first layer of the vocabulary tree (the root node is the zero layer); 3) According to the above steps, perform iterative clustering operations on each subset until the number of layers or the number of leaf nodes of the vocabulary tree reaches a given threshold.

[0098] Step 1.4, Local feature aggregation and index construction: The VLAD vector aggregation technology can be used to aggregate a large number of low-dimensional SIFT local feature vectors into a small number of high-dimensional VLAD global feature vectors. The core idea of the VLAD vector aggregation technology is: accumulate the residual vectors between the local feature descriptor vectors and their corresponding cluster center vectors, as Figure 3 shown. For the filtered local feature subset of the image, divide it into the associated features of the cluster center according to the principle of the nearest Euclidean distance, and calculate the sum of the residuals between the cluster center and its associated features, that is, the VLAD feature descriptor is obtained. For cluster centers of the codebook, a feature matrix can be calculated, where the matrix element is calculated by the formula: ; Among them, represents the element in the th row and th column of the feature matrix; represents the th local feature; represents the cluster centers; represents the residual value of the local feature falling within the cluster center; is represented as a sign function; if does not belong to the cluster center , then has a value of 0; if belongs to the cluster center , then is 1.

[0099] After the above calculations, each image can be represented as a feature matrix. Among them, is the number of rows of the matrix, equal to the number of centers of the pre-trained codebook; is the number of columns of the matrix, equal to the dimension of the local feature. Among them, can take a value of 128. Since the matrix does not conform to the input of subsequent retrieval, the calculated feature matrix will be vectorized, and each row in the feature matrix will be sequentially concatenated, the feature matrix of becomes a VLAD vector of

[0100] dimensions, that is, the global feature vector. .

[0101] Step 2, generation of image scheduling blocks based on iterative matrix bandwidth reduction MBR.

[0102] All feature descriptors cannot be loaded into the GPU with limited memory at once, and reasonable data scheduling between the hard disk and the GPU is required. In related technologies, data scheduling strategies can be divided into three categories: The first type of strategy is to perform feature matching on all selected image matching pairs in sequence, which causes the GPU to frequently load and release data. The second type of strategy is to first generate a data loading and release list based on the connection relationship of images, and then alternately execute data reading in the loading list and data clearing in the release list. Since only the newly loaded images are matched with the images in the GPU, this strategy does not fully utilize the computing power of the GPU. The third type of strategy is to divide the images into sub-blocks that can be loaded into the GPU at once, and then perform feature matching between the loaded images. This method has high efficiency in exhaustive feature matching, but its performance significantly degrades for sparsely connected images.

[0103] An embodiment of the present invention proposes a data scheduling strategy based on iterative matrix bandwidth reduction (MBR). Assume that there are images in the UAV images, and use the retrieved image matching pairs to construct an adjacency matrix . The construction process is as follows: For each pair of image matching pairs , set the corresponding item to 1; otherwise, set . Use the adjacency matrix to generate sub-blocks for data scheduling and guide feature matching. The core idea of the MBR data scheduling strategy is: reduce the bandwidth of the original adjacency matrix through the matrix bandwidth reduction algorithm, and generate scheduling blocks from the compressed MBR matrix under the GPU memory limit. Matrix bandwidth reduction is a numerical analysis technique aimed at optimizing the storage and computational efficiency of sparse matrices. The bandwidth of a matrix consists of the diagonal containing the non-zero elements of the matrix and its adjacent rows and columns. Figure 4 and Figure 5 illustrate the effect of MBR. The original adjacency matrix has sparse image connections, as shown in Figure 4 . After applying MBR, the image connections are as close as possible to the diagonal position, as shown in Figure 5 . In other words, MBR integrates the non-zero elements of the sparse matrix into a more compact central area around the diagonal. Based on the basic idea of matrix bandwidth reduction, the data scheduling strategy can be implemented according to the following steps: (1) Initialization: Construct a symmetric adjacency matrix from the original adjacency matrix , and generate an image ID list using the image indices derived from the rows or columns of .

[0104] (2) Calculate the permutation order: Generate a permutation order by using the MBR algorithm that permutes the row and column indices and creates the MBR matrix . For example, the Gibbs-Poole-Stockmeyer (GPS) MBR algorithm can be adopted because of its low time cost and memory requirement. In addition, update the image ID list according to .

[0105] (3) Create a new scheduling block: Under the GPU memory limit, use the MBR matrix to create a new scheduling block. As Figure 5 shown, the GPU memory size is limited by the grid, which represents the maximum number of descriptor data that can be loaded into the GPU memory. By defining the block size , a new scheduling block Figure 5 rendered in a rectangle in is created, where and represent the row and column indices of each block respectively, and the block stores its corresponding image matching pairs.

[0106] (4) Update the adjacency matrix and the image ID list: After the feature matching guided by the scheduling block , update the MBR matrix by setting the cell values of the corresponding image matching pairs to zero and deleting the rows and columns without image matching pairs. In addition, update the image ID list accordingly.

[0107] (5) Termination: Repeat steps (2)-(4) until the MBR matrix is empty. Finally, the scheduling block can be obtained.

[0108] By using the MBR algorithm for adjacency matrix compression and block generation, data loading and release in the GPU can be minimized as much as possible. At the same time, as many image matching pairs as possible can be packed into a scheduling block, high GPU utilization can be achieved, and the data scheduling burden and GPU computing power utilization can be balanced.

[0109] Step 3, Cascade Hashing feature matching with local geometric constraints and gross error rejection.

[0110] The Cascade Hashing feature matching is carried out by using the generated image scheduling block . Since each block row The sum of the amounts of image data therein has considered the GPU memory limit, and the feature descriptors of each sub-block in the block row can be loaded into the GPU at one time. Assume that the generated block is composed of row and column. The current block row is represented as . The Cascade Hashing feature matching can be processed as follows: (1) Loading feature descriptors: The feature descriptors of the image matching pairs in each block are sequentially loaded into the GPU, and the hash codes and bucket IDs are calculated to construct a hash lookup table.

[0111] (2) Cascade hashing matching: Based on the established hash lookup table, hash feature matching is performed, and the initial candidate matches passing the ratio test guided by the image matching pairs are retained.

[0112] (3) Data block release: Steps (1) and (2) are repeatedly executed until all blocks in the current block row are processed. The data of the first block is released because it will not be used in the next block row.

[0113] For each block row , the above process is executed to obtain the initial candidate matches of all relevant image matching pairs. It should be noted that in step (1), redundant data loading operations can be avoided. As Figure 5 shown, when processing the second block row, only the last block is loaded into the GPU because all other required data has been loaded when processing the first block row. In addition, after processing each block row, the data of the first block is released. The above alternating release-loading operations run through the entire matching process.

[0114] Due to the limited discrimination ability of local feature descriptors and large-viewpoint deformation, there are inevitably incorrect matches in the initial matches. Further, the embodiments of the present invention combine the spatial angle order SAO local geometric constraint and the RANSAC global geometric constraint to eliminate incorrect matches. Among them, the local geometric constraint can be implemented by using Delaunay triangulation to construct the nearest neighbor and its corresponding graph, as Figure 6 shown. To further improve the efficiency, the above gross error elimination step is executed on the CPU. Therefore, the result retained after gross error elimination is the final feature match.

[0115] The embodiment of the present invention proposes a data scheduling strategy based on matrix bandwidth reduction, which divides the sparsely connected image matching graph into compact sub-blocks. The strategy can well adapt to the connection structure of the image and the memory capacity of the GPU. At the same time, a feature matching workflow based on cascade hashing is designed, which integrates local constraints based on spatial angle order and global verification based on RANSAC to remove false matches, meets the needs of fast feature matching of large-scale UAV images, and improves the feature matching efficiency of UAV images.

[0116] The drone image matching device provided by the present invention is described below. The drone image matching device described below and the drone image matching method described above can be referenced to each other.

[0117] refer to Figure 7 The drone image matching device provided by the present invention includes an acquisition module 701, an image scheduling block generation module 702 and a feature matching module 703.

[0118] An acquisition module 701 is used to acquire a global feature vector of a drone image, and perform graph index processing on the global feature vector to obtain an image matching pair; An image scheduling block generation module 702 is used to perform data scheduling processing of iterative matrix bandwidth reduction on the image matching pair to generate an image scheduling block; The feature matching module 703 is used to perform feature matching on the image scheduling block and eliminate matching error data in the feature matching result to obtain the drone image matching result.

[0119] The drone image matching device provided by the embodiment of the present invention obtains the global feature vector of the drone image, performs graph index processing on the global feature vector to obtain an image matching pair; performs data scheduling processing of iterative matrix bandwidth reduction on the image matching pair to generate an image scheduling block; performs feature matching on the image scheduling block, and eliminates matching error data in the feature matching result to obtain the drone image matching result. The present invention divides the sparsely connected image matching graph into compact sub-blocks based on the data scheduling strategy of matrix bandwidth reduction. In the case of limited GPU memory, the sub-blocks can be loaded and processed more efficiently. At the same time, by combining feature matching and gross error elimination of cascade hashing, the feature matching speed is improved, thereby meeting the demand for fast feature matching of large-scale drone images.

[0120] In one embodiment, the feature matching module 703 is specifically configured to: construct an adjacency matrix based on the image matching pairs; construct a symmetric adjacency matrix based on the adjacency matrix, and generate an image ID list based on the image indexes corresponding to the rows or columns of the adjacency matrix; construct a matrix bandwidth reduction MBR matrix based on the permutation order of the symmetric adjacency matrix, and update the image ID list based on the permutation order; the permutation order is used to reorder the image indexes of the rows and columns of the symmetric adjacency matrix; in the case of the memory limit of the graphics processing unit (GPU), create a new scheduling block based on the MBR matrix; after performing feature matching based on the new scheduling block, update the MBR matrix and the image ID list; iteratively optimize the bandwidth of the updated MBR matrix to generate the image scheduling block.

[0121] In one embodiment, the acquisition module 701 is specifically configured to: extract local features of the UAV image; perform random sampling on the UAV image, and sort the local features of the randomly sampled UAV image according to a scale factor to obtain a sorting result; the scale factor is the size or resolution of the local feature in different scale spaces; perform hierarchical clustering on multiple target local features selected based on the sorting result to obtain a feature set; use the feature set to aggregate multiple target local features of all the UAV images to obtain the global feature vector.

[0122] In one embodiment, the acquisition module 701 is specifically configured to: use the global feature vector as a graph node, and connect the graph nodes according to the similarity between the global feature vectors to establish a vector index graph; perform approximate nearest neighbor search based on the vector index graph to obtain the image matching pairs.

[0123] In one embodiment, the image scheduling block generation module 702 is specifically configured to: load the feature descriptors of the image matching pairs in the image scheduling block into the GPU, calculate the hash code and bucket ID of the feature descriptors to construct a hash lookup table; the bucket ID is used to uniquely identify a hash bucket; perform hash feature matching based on the hash lookup table to obtain candidate feature descriptors; perform a ratio test screening on the candidate feature descriptors to eliminate the unmatched candidate feature descriptors; repeatedly execute the feature matching process of the image scheduling block until the image scheduling block is processed to obtain the candidate feature descriptors of the image matching pairs.

[0124] In one embodiment, the image scheduling block generation module 702 is specifically configured to: perform local geometric constraint and global geometric constraint processing on the candidate feature descriptors in the feature matching result to determine the candidate feature descriptors with matching errors; eliminate the candidate feature descriptors with matching errors to obtain the UAV image matching result.

[0125] Figure 8 An entity structure schematic diagram of an electronic device is illustrated, as Figure 8 shown. The electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communication bus 840. Among them, the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call the logical instructions in the memory 830 to execute the UAV image matching method. The method includes: obtaining a global feature vector of the UAV image, performing graph indexing processing on the global feature vector to obtain image matching pairs; performing data scheduling processing of iterative matrix bandwidth reduction on the image matching pairs to generate image scheduling blocks; performing feature matching on the image scheduling blocks, and removing the matching error data in the feature matching results to obtain the UAV image matching result.

[0126] In addition, when the logical instructions in the above-mentioned memory 830 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0127] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the UAV image matching method provided by the above-mentioned various methods. The method includes: obtaining a global feature vector of the UAV image, performing graph indexing processing on the global feature vector to obtain image matching pairs; performing data scheduling processing of iterative matrix bandwidth reduction on the image matching pairs to generate image scheduling blocks; performing feature matching on the image scheduling blocks, and removing the matching error data in the feature matching results to obtain the UAV image matching result.

[0128] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the UAV image matching method provided by the above-mentioned various methods. The method includes: obtaining a global feature vector of a UAV image, performing graph indexing processing on the global feature vector to obtain image matching pairs; performing data scheduling processing for iterative matrix bandwidth reduction on the image matching pairs to generate image scheduling blocks; performing feature matching on the image scheduling blocks, and eliminating matching error data in the feature matching results to obtain a UAV image matching result.

[0129] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0130] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disc, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for unmanned aerial vehicle image matching, characterized in that Including: Obtain the global feature vector of the UAV image, and perform graph indexing processing on the global feature vector to obtain image matching pairs; Perform data scheduling processing for iterative matrix bandwidth reduction on the image matching pairs to generate image scheduling blocks; Perform feature matching on the image scheduling blocks, and eliminate the mismatched data in the feature matching results to obtain the UAV image matching results.

2. The UAV image matching method according to claim 1, wherein The performing data scheduling processing for iterative matrix bandwidth reduction on the image matching pairs to generate image scheduling blocks includes: Based on the image matching pairs, construct an adjacency matrix; Construct a symmetric adjacency matrix based on the adjacency matrix, and generate an image ID list based on the image indexes corresponding to the rows or columns of the adjacency matrix; Based on the permutation order of the symmetric adjacency matrix, construct a matrix bandwidth reduction MBR matrix, and update the image ID list based on the permutation order; the permutation order is used to reorder the image indexes of the rows and columns of the symmetric adjacency matrix; Under the condition of the memory limit of the graphics processing unit GPU, create a new scheduling block based on the MBR matrix; After performing feature matching based on the new scheduling block, update the MBR matrix and the image ID list; Iteratively optimize the bandwidth of the updated MBR matrix to generate the image scheduling blocks.

3. The UAV image matching method according to claim 1 or 2, characterized in that The obtaining the global feature vector of the UAV image includes: Extract the local features of the UAV image; Randomly sample the UAV image, and sort the local features of the randomly sampled UAV image according to the scale factor to obtain a sorting result; the scale factor is the size or resolution of the local feature in different scale spaces; Perform hierarchical clustering on multiple target local features selected based on the sorting result to obtain a feature set; Use the feature set to aggregate multiple target local features of all the UAV images to obtain the global feature vector.

4. The method for matching drone images according to claim 1 or 2, characterized in that The performing graph indexing processing on the global feature vector to obtain image matching pairs includes: Use the global feature vector as graph nodes, and connect the graph nodes according to the similarity between the global feature vectors to establish a vector index graph; Perform approximate nearest neighbor retrieval based on the vector index graph to obtain the image matching pairs.

5. The UAV image matching method according to claim 1, wherein, The performing feature matching on the image scheduling blocks includes: Load the feature descriptors of the image matching pairs in the image scheduling blocks into the GPU, calculate the hash codes and bucket IDs of the feature descriptors to construct a hash lookup table; the bucket ID is used to uniquely identify the hash bucket; Perform hash feature matching based on the hash lookup table to obtain candidate feature descriptors; Perform ratio test screening on the candidate feature descriptors to eliminate the mismatched candidate feature descriptors; Repeat the feature matching process of the image scheduling blocks until the image scheduling blocks are processed to obtain the candidate feature descriptors of the image matching pairs.

6. The UAV image matching method according to claim 5, wherein The eliminating the mismatched data in the feature matching results to obtain the UAV image matching results includes: Perform local geometric constraint and global geometric constraint processing on the candidate feature descriptors in the feature matching results to determine the mismatched candidate feature descriptors; Eliminate the candidate feature descriptors with matching errors to obtain the UAV image matching result.

7. An unmanned aerial vehicle image matching device, characterized in that, Including an acquisition module, configured to acquire the global feature vector of the UAV image, and perform graph indexing processing on the global feature vector to obtain image matching pairs; an image scheduling block generation module, configured to perform data scheduling processing for iterative matrix bandwidth reduction on the image matching pairs to generate image scheduling blocks; a feature matching module, configured to perform feature matching on the image scheduling blocks, and eliminate the data with matching errors in the feature matching results to obtain the UAV image matching result.

8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the UAV image matching method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the UAV image matching method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the UAV image matching method according to any one of claims 1 to 6.

Images