A cross-view geolocation method based on global-local feature matching and key point guidance

Abstract

The application provides a cross-view geolocation method based on global-local feature matching and key point guidance, and belongs to the technical field of computer vision and remote sensing image processing technology. First, satellite images and unmanned aerial vehicle images are preprocessed and features are extracted, then a key point guided feature pyramid network is constructed to fuse the extracted multi-scale features. Based on the fused features of the satellite images and the unmanned aerial vehicle images, global similarity and local feature matching scores are calculated. Finally, the global similarity and the local feature matching scores are added to calculate a comprehensive matching score, and a positioning result is output. The application breaks the limitation of traditional methods that only use local features in the training stage, and significantly improves the accuracy of cross-view matching in complex environments.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and remote sensing image processing technology, specifically relating to a cross-view geolocation method based on global-local feature matching and key point guidance. Background Technology

[0002] Cross-view geolocation uses images from drone or ground-based perspectives to match satellite remote sensing images, thereby determining the photographer's geographical location. This technology does not rely on satellite navigation signals, thus playing a crucial role in areas such as autonomous drone navigation in GPS-denied or jammed environments, emergency rescue, augmented reality, and urban monitoring. With the rapid development of computer vision and deep learning technologies, methods based on convolutional neural networks and Transformers have become mainstream in this field.

[0003] Despite significant progress in existing cross-view geolocation technologies, challenges remain in practical applications. Current solutions are mainly categorized into keypoint-based methods and metric learning-based methods. Keypoint-based methods focus on capturing the geometric structure of a scene but often neglect texture and environmental context, and require extremely high accuracy in keypoint detection. While metric learning-based methods can effectively extract global features, they are prone to mismatches when faced with scenes that have similar global appearances but different local structures (such as regular urban blocks). Furthermore, existing keypoint-based methods typically treat keypoints only as auxiliary tasks, failing to deeply integrate them into the backbone network for feature extraction; while existing metric learning methods, although extracting local features, often discard local information directly during the inference stage, using only global features for retrieval, resulting in a waste of fine-grained information and difficulty in achieving high-precision positioning in complex geographical environments. Therefore, researching a robust positioning method that can balance global contextual information with local fine-grained structure is of significant strategic importance for improving the survivability and operational capabilities of UAVs in complex environments. Summary of the Invention

[0004] To address the problem that existing cross-view geolocation methods over-rely on global features and lack guidance from local structures, leading to mismatches in scenarios with similar global appearances but different local structures, this invention proposes a cross-view geolocation method based on global-local feature matching and keypoint guidance. This method enhances local feature representation through a keypoint-guided feature pyramid network and explicitly utilizes local cues during the inference phase through a global-local feature matching strategy, thereby improving the robustness of geolocation.

[0005] The cross-view geolocation method based on global-local feature matching and key point guidance has the following specific steps:

[0006] Step 1: Image preprocessing and feature extraction.

[0007] First, acquire the satellite images to be matched. and drone images And adjust them to a uniform size.

[0008] Then, a pre-trained keypoint detection network is used to process the input satellite and drone images to generate corresponding keypoint augmented images. and These images contain significant structural information about the scene.

[0009] Finally, the original satellite and UAV images are input into a shared backbone network to extract multi-scale feature maps, denoted as... and , The number of layers in the feature map, the th The spatial resolution of layer feature maps varies with The number of layers increases by half, while the number of hidden layers doubles.

[0010] Step 2: Construct a key-point guided feature pyramid network and perform multi-scale feature fusion.

[0011] By using keypoint maps as spatial priors, the fusion of backbone network features is guided through a cross-scale attention mechanism, enabling deep features to perceive the fine-grained structural information of shallow layers.

[0012] Specifically:

[0013] Step 201: Define the cross-scale attention mechanism. Taking satellite imagery as an example, it utilizes shallow features. As a query, key features As the key, a query vector is generated through linear projection and convolutional layers, respectively. and key vector :

[0014]

[0015] in, and This is a learnable weight matrix.

[0016] Step 202, Calculate the attention map An attention map is calculated using the scaled dot product of the query and the key, reflecting the association between keypoint regions and image features.

[0017]

[0018] in, For the hidden layer dimension.

[0019] Step 203, Feature Fusion. For Perform spatial downsampling to adapt to deep features The resolution, and used for weighted calculation from The obtained value matrix , obtain fusion features :

[0020]

[0021] Step 204: Processing via a feedforward neural network (FFN) The characteristics after fusion are obtained. ;

[0022] Step 205, will As new shallow features, hierarchical iteration is performed, repeating steps 201-204, and fused with the features of the next layer, until the M-th layer feature map is obtained, finally yielding the feature representation of the satellite image that integrates local structural cues and high-level semantic information. .

[0023] Similarly, the fused feature representation of the UAV image is obtained. .

[0024] Step 3: Based on the fused features, a global-local feature matching strategy is used to calculate the similarity, and local feature blocks are explicitly compared during the inference stage.

[0025] Specifically:

[0026] First, calculate the global similarity score. After performing global average pooling on the feature map, the scaled inner product is calculated:

[0027]

[0028] in, The channel dimension of the fused deep features;

[0029] Then, calculate the local similarity matrix. Flatten the feature map and calculate the similarity between each local feature vector of the UAV image and all local feature vectors of the satellite image to obtain the similarity matrix. :

[0030]

[0031] Next, the best local match is found. For each local region in the UAV image, the maximum similarity value is found in all local regions of the satellite image, and the best matching set is retained. :

[0032]

[0033] Finally, reliability filtering and scoring. Setting thresholds. Remove For unreliable matches with similarity values ​​below the threshold, the average similarity of the remaining valid best matches is calculated to obtain the final local matching score. :

[0034]

[0035] Step 4: Comprehensive scoring and positioning.

[0036] Step 401, Weighted Fusion. The global similarity score is added to the local similarity score to obtain the final cross-view image pair similarity score. :

[0037]

[0038] Step 402, Location Retrieval. Calculate the similarity between the query drone image and all satellite images in the database, sort them from highest to lowest similarity, and retrieve the most matching satellite image to determine the drone's geographical location.

[0039] The advantages of this invention are:

[0040] (1) A novel cross-perspective geolocation framework is proposed, which combines key point guidance with global-local joint matching. This effectively solves the mismatch problem of traditional methods in scenarios with similar global appearance but different local structures, and improves the robustness of feature alignment.

[0041] (2) A global-local feature matching strategy that explicitly participates in the computation during the inference stage is designed. Local feature blocks are explicitly searched and matched during the inference stage, and noise interference (such as occlusion and view distortion) is eliminated through an adaptive filtering mechanism. This breaks the limitation of traditional methods that only utilize local features during the training stage and significantly improves the accuracy of cross-view matching in complex environments.

[0042] (3) A key point-guided feature pyramid network was designed, using the key point map as prior knowledge to guide multi-scale feature fusion, which enhanced the network’s attention to local salient areas (such as road intersections and specific building structures) and improved the discriminative power of features.

[0043] (4) The method outperforms the state-of-the-art methods on mainstream benchmark datasets (University-1652 and SUES-200), demonstrating its effectiveness and generalization ability. Attached Figure Description

[0044] Figure 1This is a schematic diagram of the overall process of a cross-view geolocation method based on global-local feature matching and key point guidance proposed in this invention. Detailed Implementation

[0045] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0046] This invention proposes a cross-view geolocation method based on global-local feature matching and keypoint guidance, aiming to solve the mismatch problem of existing technologies in scenarios with similar global appearance but different local structures. The method mainly includes four steps: image preprocessing and feature extraction, keypoint-guided feature pyramid network construction, global-local feature matching calculation, and comprehensive scoring and localization.

[0047] like Figure 1 As shown, the specific implementation steps are as follows:

[0048] Step 1: Image Preprocessing and Feature Extraction

[0049] Step 101, Data preparation and preprocessing.

[0050] Obtain the satellite image to be matched. And drone images (Drone Image) This embodiment uses the mainstream University-1652 and SUES-200 datasets. University-1652 contains multi-view images of 1652 buildings; SUES-200 contains drone images at different heights (150m to 300m). The datasets are divided into training and testing sets.

[0051] To adapt to the input requirements of the neural network and ensure the effectiveness of feature extraction, all input satellite and drone images were uniformly adjusted to... Pixel size. During the training phase, data augmentation operations such as random pruning and random flipping are used to improve the model's generalization ability and prevent overfitting. During the inference phase, only standard size adjustments and normalization are performed.

[0052] Step 102, key point generation.

[0053] The XFeat network, a pre-trained keypoint detection network, is used to process the input satellite images. and drone images The process is performed to generate corresponding keypoint-enhanced images, denoted as follows: and .

[0054] Specifically, key point diagram and The size is These keypoint maps contain salient structural information about the scene, such as road intersections and building corners, providing important spatial prior knowledge for subsequent feature fusion and compensating for the shortcomings of purely visual features in representing geometric structures. Compared to traditional handcrafted features, deep learning-based keypoint detectors exhibit better robustness under changes in lighting and viewpoint.

[0055] Step 103, multi-scale feature extraction.

[0056] Original satellite images and drone images Input the shared backbone network ConvNeXt-B.

[0057] In this embodiment, by default, three different scales of stage features are extracted, denoted as... and (here) The deeper scales correspond to feature maps with doubled hidden dimensions but halved spatial resolution. It should be noted that the number of layers in the feature map... It is not absolutely fixed; its specific value depends on the hierarchical architecture of the chosen backbone network and the actual trade-off between accuracy and computational complexity. When using visual backbone networks that typically employ sequential multi-stage feature structures (such as the Block downsampling mechanism typical of ResNet and ConvNeXt), or This is a typical configuration designed to cover a reasonable gradient from fine-grained textures to high-level abstract semantics.

[0058] Specifically, in this embodiment, the spatial dimensions of the feature maps are respectively , and The corresponding number of channels The values ​​are 256, 512, and 1024, respectively. First layer features. :size Number of channels This layer is a shallow feature layer, containing rich textures and geometric details. Second layer features... :size Number of channels Third layer features :size Number of channels This layer represents deep features and contains highly abstract semantic information. This multi-scale design allows the network to simultaneously capture both low-level texture details and high-level semantic information.

[0059] Step 2: Construct a key-point guided feature pyramid network and perform multi-scale feature fusion.

[0060] To enhance the sensitivity of features to local structures, this invention designs a keypoint-guided feature pyramid network. This network uses the keypoint map generated in step 102 as a spatial prior and guides the fusion of backbone network features through a cross-scale attention mechanism, enabling deep features to perceive fine-grained structural information in shallow layers.

[0061] Taking the satellite imagery branch as an example, the fusion process is as follows (the same applies to the UAV branch):

[0062] Step 201: Construct the query and key.

[0063] Utilizing shallow features (dimension is) As a query, key features (dimension is) The query vector is used as the key. A linear projection layer and a convolutional layer are used to generate the query vector. and key vector The calculation formula is as follows:

[0064]

[0065] in, To query the learnable weight matrix of the projection; The learnable weight matrix for key projection is used to align the dimensions of keypoint features with those of image features.

[0066] Step 202: Calculate the attention map.

[0067] By query vector and key vector Scaling dot product to calculate attention map This attention map reflects the degree of association between keypoint regions and image features, i.e., which regions contain significant geometrical information:

[0068]

[0069] in, The dimension of the hidden layer, divided by This is to prevent the gradient from vanishing due to an excessively large dot product result.

[0070] Step 203, feature weighting and downsampling.

[0071] To inject shallow structural information into deep features, attention maps need to be processed. Perform spatial downsampling to adjust its resolution and depth features. (dimension is) Matching. At the same time, for The value matrix is ​​obtained by performing a linear projection. The value matrix is ​​weighted using the downsampled attention map to obtain a feature output that incorporates keypoint information. :

[0072]

[0073] in, The weight matrix for the projection of values; Indicates a downsampling operation; Used to normalize attention weights.

[0074] Step 204, feedforward network processing.

[0075] Output attention The input is processed by a feedforward neural network (FFN) through nonlinear transformation and normalization to obtain the fused features. :

[0076]

[0077] This step further enhances the expressive power of the features.

[0078] Step 205, hierarchical iteration.

[0079] Repeat steps 201 to 204 above, and As new shallow features, compared with deeper features The data is then fused. Through this hierarchical approach, the final feature representation, containing rich local structural cues and high-level semantic information, is obtained, denoted as follows: and These feature maps have a dimension of 1. (In this embodiment, it is) ).

[0080] Step 3: Calculate similarity based on global-local feature matching.

[0081] In the inference phase, in order to overcome the problem of misjudgment in similar scenarios caused by traditional methods relying solely on global features, this invention adopts a global-local feature matching strategy to explicitly compare local feature blocks.

[0082] Step 301: Calculate the global similarity.

[0083] The final feature map obtained in step two and Global average pooling is performed to obtain the global feature vector. The global similarity score is then calculated using the scaled inner product. :

[0084]

[0085] The channel dimension of the fused deep features is compared with the relatively shallow hidden layer dimension mentioned above. satisfy ;

[0086] Global similarity reflects the degree of similarity between images in terms of overall semantics and appearance.

[0087] Step 302: Calculate the local similarity matrix.

[0088] feature map and Flatten the image into a sequence of local feature vectors. Calculate the dot product similarity between each local feature vector in the UAV image and all local feature vectors in the satellite image to obtain a similarity matrix. :

[0089]

[0090] matrix The size is , of which The row representing the drone image The similarity between a local region and all local regions in the satellite image is calculated. This step simulates the process of a drone searching for corresponding areas on the ground during flight, allowing local features to find the best match in the entire image, thus exhibiting a certain degree of translation invariance.

[0091] Step 303, optimal local matching search.

[0092] For each local region (i.e., matrix) of the UAV image For each row, find its best match in the satellite image (i.e., the maximum value in that row), forming the best match set. :

[0093]

[0094] This collection contains the most similar regions found in satellite images for each local region in UAV images, along with their corresponding similarity values.

[0095] Step 304, reliability filtering.

[0096] Due to occlusion, changes in viewpoint, or missing information in image edge regions, some local areas may not find a correct match in satellite imagery. To avoid these unreliable matches interfering with the final score, this invention sets a filtering threshold. Remove medium to low The value is used to retain only the high-confidence matches, resulting in a filtered set. :

[0097]

[0098] Step 305: Calculate the local matching score.

[0099] The final local matching score is obtained by averaging the similarities of the remaining effective best matches after filtering. :

[0100]

[0101] The score It reflects the degree of similarity between image pairs in local details and is key to distinguishing scenes that have similar global appearances but different local structures.

[0102] Step 4: Comprehensive Scoring and Positioning

[0103] Step 401, similarity fusion.

[0104] Global similarity score Local similarity score By adding them together, we obtain the final similarity score between the satellite image and the drone image. :

[0105]

[0106] This formula combines global contextual information with local fine-grained structural information, making the matching results more robust.

[0107] Step 402, location determined.

[0108] Calculate the final similarity between the query drone image and all candidate satellite images in the database. Sort the satellite images from highest to lowest similarity. The top-ranked satellite image is the best match, and its corresponding geographic coordinates (latitude and longitude) are the estimated location of the drone.

[0109] Through the above steps, the present invention can effectively utilize the structural priors and fine-grained information of local features provided by key points, and significantly improve the accuracy and robustness of cross-view geolocation in complex environments.

[0110] Step 5: Model Training and Parameter Optimization

[0111] Step 501, define the loss function. The total loss function of this invention. From global feature mapping terms and local feature mapping terms composition:

[0112]

[0113] in, It includes mean squared error (MSE) loss and InfoNCE loss. MSE is used to narrow the feature distance between positive sample pairs, while InfoNCE is used to maximize the mutual information of positive sample pairs and minimize the mutual information of negative sample pairs. For the local similarity score (calculation method is shown in step 305), maximize... (i.e., minimize) This forces the local features learned by the network to remain highly similar between positive sample pairs, thereby enhancing local discriminative power.

[0114] Step 502, Model Optimization Configuration. The network is trained end-to-end using the training set data. Specific implementation parameters are as follows: The AdamW optimizer, which has better weight decay handling capabilities, is used; the initial learning rate is set to... The learning rate is dynamically adjusted using a cosine annealing strategy to ensure the model's convergence stability in the later stages of training; the batch size is set to... The model is trained for several epochs, and evaluated on a validation set after each epoch. The model weights with the best performance on the validation set are saved as the final model. In terms of hardware, a single NVIDIA GeForce RTX 4090 GPU is used for training. In terms of software, it is implemented based on Python 3.9 and PyTorch 1.11 frameworks.

[0115] Step Six: Performance Verification and Comparison

[0116] To fully verify the effectiveness and advancement of the proposed method, extensive comparative experiments were conducted on two mainstream large-scale public datasets in the field of cross-view geolocation (University-1652 and SUES-200). Detailed dataset information is shown in Table 1. This embodiment employs common evaluation metrics for quantitative analysis, including the core evaluation metrics Recall@K (R@K) and Average Precision (AP). R@K reflects the probability that a true matching object appears in the top K ranked results predicted by the model, directly reflecting the algorithm's recall ability and retrieval precision. AP represents the area under the precision-recall (PR) curve, used to measure the overall retrieval performance of the model at different confidence thresholds.

[0117]

[0118] The experiment compared the present invention with recent state-of-the-art methods (including SDPL, DAC, CAMP, SRLN, Safe-Net, and MEAN), and detailed information on the benchmark models is shown in Table 2. The results show that the present invention demonstrates a leading advantage and achieves the best localization performance on different tasks across both datasets. Specifically, as shown in Table 3, the best results are indicated by bold italics and the second best results by underlined italics. In the University-1652 dataset, which contains dense urban building clusters and numerous similar interference items, UAV-to-satellite map retrieval is a key task for evaluating the autonomous localization capability of UAVs. The present invention achieves an R@1 of 95.99% and an AP of 96.63%. In contrast, the suboptimal baseline method DAC only achieves an R@1 of 94.67% and an AP of 95.50%, respectively, while the present invention achieves significant improvements of 1.32% and 1.13%, respectively. In the reverse engineering task of retrieving data from satellite maps to UAVs, this method also demonstrates outstanding generalization ability, achieving an R@1 of 97.29% and an AP of 94.93%, representing performance improvements of 0.86% and 1.14% respectively compared to the second-best method. This superior performance proves that the present invention possesses extremely high robustness and feature discrimination power when handling urban scenes with a large visual span, dense local occlusion, and highly similar global building appearances.

[0119]

[0120] To further verify the invention's ability to handle scale variations and resolution differences caused by different platform flight altitudes, evaluation and analysis were conducted on the SUES-200 dataset with multiple altitude (150m, 200m, 250m, 300m) testing conditions. Extensive empirical testing clearly shows that as the UAV's flight altitude increases, the scene's field of view gradually widens while local details relatively shrink, demonstrating the invention's outstanding scale invariance. As shown in Table 4, the bold italics represent the best results, and the underlined italics represent the second best results. In the UAV-to-satellite map retrieval task, when the flight altitudes are 150m, 200m, 250m, and 300m, the invention's R@1 reaches 97.63%, 98.85%, 99.60%, and 100.00%, respectively, maintaining extremely high accuracy not only in low-altitude scenarios but also achieving 100% accurate matching in high-altitude scenarios. In the reverse satellite map to UAV retrieval task, the data performance of this invention showed equally high reliability, with R@1 steadily reaching 100.00% from 200m to 300m altitude. For the close-range field of view (150m) where mainstream algorithms are generally prone to errors in this task, this invention still achieved a recall rate of 98.75%.

[0121] In summary, the multi-dimensional data verification from the experiments fully demonstrates that the key-point guided feature pyramid network designed in this invention greatly enhances the deep fusion capability of multi-scale structural features, while the complementary global-local feature joint matching strategy effectively eliminates redundant and ambiguous noise. This deep and organic combination enables the system to achieve high confidence, high robustness, and high accuracy in cross-visual geolocation calculations when dealing with drastic viewpoint shifts, multi-scale height changes, and complex environmental interference.

[0122]

[0123]

Claims

1. A cross-view geolocation method based on global-local feature matching and key point guidance, characterized in that, The specific steps are as follows: Step 1: Acquire satellite and drone images to be matched, perform preprocessing, and generate corresponding keypoint enhanced images through a keypoint detection network, while simultaneously extracting features. Processing input satellite images using a pre-trained keypoint detection network and drone images Generate corresponding key point augmented images and ; Simultaneously, the original satellite images and UAV images are input into the shared backbone network to extract multi-scale feature maps, denoted as... and , The number of layers in the feature map; Step 2: Construct a key-point-guided feature pyramid network and perform multi-scale feature fusion; Specifically: Step 201, define the cross-scale attention mechanism; Taking satellite imagery as an example, shallow features are utilized. As a query, key features As the key, a query vector is generated through linear projection and convolutional layers, respectively. and key vector : in, and The weight matrix is ​​a learnable matrix; Step 202, Calculate the attention map ; An attention map is calculated by scaling the dot product of the query and the key, reflecting the association between keypoint regions and image features: in, For the hidden layer dimension; Step 203, Feature Fusion; for Perform spatial downsampling to adapt to deep features The resolution, and used for weighted calculation from The obtained value matrix , obtain fusion features : Step 204: Processing via a feedforward neural network The characteristics after fusion are obtained. ; Step 205, will As new shallow features, hierarchical iteration is performed, repeating steps 201-204, and fused with the features of the next layer, until the M-th layer feature map is obtained, finally yielding the feature representation of the satellite image that integrates local structural cues and high-level semantic information. ; Similarly, the fused feature representation of the UAV image is obtained. ;; Step 3: Based on the fused features, a global-local feature matching strategy is used to calculate similarity, and local feature blocks are explicitly compared during the inference stage; Step 4: Perform a comprehensive scoring and localization based on the global similarity score and the local similarity score, and output the localization result; Step 401, weighted fusion; combine global similarity scores Local similarity score Add them together to get the final cross-view image similarity. : Step 402, Location retrieval; calculate the similarity between the query drone image and all satellite images in the database, sort them from high to low similarity, retrieve the most matching satellite image, and thus determine the geographical location of the drone.

2. The cross-view geolocation method based on global-local feature matching and key point guidance according to claim 1, characterized in that, Global similarity score The calculation process is as follows: After performing global average pooling on the feature map, the scaled inner product is calculated: in, The channel dimension of the fused deep features.

3. The cross-view geolocation method based on global-local feature matching and key point guidance according to claim 1, characterized in that, The calculation process for local similarity score is as follows: First, calculate the local similarity matrix. Flatten the feature map, calculate the similarity between each local feature vector of the UAV image and all local feature vectors of the satellite image, and obtain the similarity matrix. : Next, the best local match is found; for each local region in the UAV image, the maximum similarity value is found in all local regions of the satellite image, and the best matching set is retained. : Finally, reliability filtering and scoring; setting thresholds. Remove For unreliable matches with similarity values ​​below the threshold, the average similarity of the remaining valid best matches is calculated to obtain the final local matching score. : 。 4. The cross-view geolocation method based on global-local feature matching and key point guidance according to claim 1, characterized in that, The network is trained end-to-end using the training set data, specifically as follows: the AdamW optimizer is used; the initial learning rate is set to... The learning rate is dynamically adjusted using a cosine annealing strategy to ensure the convergence stability of the model in the later stages of training. The model is trained for several epochs and evaluated on the validation set after each epoch. The model weights with the best performance on the validation set are saved as the final model.

5. The cross-view geolocation method based on global-local feature matching and key point guidance as described in claim 4, characterized in that, Total loss function From global feature mapping terms and local feature mapping terms composition: in, It includes MSE loss and InfoNCE loss. MSE is used to shorten the feature distance between positive sample pairs, and InfoNCE is used to maximize the mutual information of positive sample pairs and minimize the mutual information of negative sample pairs; by maximizing This allows the local features learned by the network to maintain high similarity between positive sample pairs, thereby enhancing local discriminative power.

Images