Multiscale cross-view geolocation positioning method, system, device, and medium

Abstract

The application relates to a multi-scale cross-view geographical position positioning method, system, device and medium, the method comprising: acquiring a same-place cross-view remote sensing image and a street view image to be processed; constructing a multi-scale cross-view geographical position positioning model, which is composed of two feature extraction branches including feature extractors with the same structure and an output network; the feature extractor is used for extracting multi-scale features of the input image by adopting a multi-stage model based on a Swin-transformer; the output network is used for determining a geographical position positioning result according to multi-scale features output by the two feature extraction branches; the model is trained in a contrast learning mode according to pre-prepared image samples, and the remote sensing image and the street view image to be processed are input into the trained multi-scale cross-view geographical position positioning model to obtain the geographical position positioning result. The method can improve the geographical position positioning precision and greatly reduce the calculation cost.

Description

Technical Field

[0001] This invention belongs to the field of location positioning technology and relates to a multi-scale, cross-view geolocation method, system, device, and medium. Background Technology

[0002] Cross-view geolocalization is a hot research area in computer vision. It aims to determine the location of a street view image by retrieving the image most similar to the target street view image from a dataset of GPS-tagged remote sensing images; this task can be viewed as an image retrieval task. With the development of deep learning technology, several cross-view geolocalization methods have been proposed, but deep learning-based cross-view geolocalization still faces difficulties and challenges.

[0003] On the one hand, street view images and remote sensing images have significant differences in perspective and field of view. Street view images are panoramic images centered on the shooting location, while remote sensing images are orthophotos with a wider field of view, such as... Figure 1 As shown, (a) is a street view image and (b) is a remote sensing image. This results in the same object having different features and locations in the street view image and the remote sensing image, making the effective alignment of these features a challenging problem. Furthermore, street view images and remote sensing images may have been taken at different times, locations, and seasons. This creates visual differences and environmental variations, increasing the difficulty of cross-view geolocation tasks. The model needs to be able to adapt to these differences to ensure accurate determination of the geographical location of street view images under different conditions, which involves addressing the inconsistencies in visual features caused by changes in time, weather, and season. Additionally, the goal of cross-view geolocation is to locate street view images globally, requiring application and testing on large-scale datasets. Therefore, reducing the computational cost and retrieval time associated with this process is also important. Summary of the Invention

[0004] Based on this, it is necessary to address the above-mentioned technical problems. In this invention, we propose a multi-scale cross-view geolocation method, a multi-scale cross-view geolocation system, a computer device, and a computer-readable storage medium, which can significantly improve the performance of cross-view geolocation.

[0005] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0006] On the one hand, a multi-scale, cross-view geolocation method, which includes:

[0007] Acquire remote sensing images and street view images of the same location from different perspectives.

[0008] A multi-scale, cross-view geolocation model is constructed, which includes feature extraction branches for remote sensing images and street view images, and an output network. The two feature extraction branches include feature extractors with identical structures. The feature extractors are used to extract multi-scale features of the input images using a multi-stage model based on the Swin-transformer. The output network is used to determine the geolocation result based on the multi-scale features output by the two feature extraction branches.

[0009] The multi-scale cross-view geolocation model is trained using a contrastive learning approach based on pre-prepared image samples, resulting in a well-trained multi-scale cross-view geolocation model.

[0010] The remote sensing image to be processed and the street view image are input into the trained multi-scale cross-view geolocation model to obtain the geolocation result.

[0011] In one embodiment, the image samples include multiple sets of cross-view remote sensing images and street view images; during training, a feature extractor sharing weights with the feature extractor is added to the feature extraction branch of the remote sensing images, a strong enhancement module and a weak enhancement module are added to the two feature extractors respectively, and a mapping head is added after each of the two feature extractors.

[0012] The multi-scale cross-view geolocation model used in the testing process was obtained by removing the added strong enhancement, weak enhancement modules, feature extractor, and two mapping heads from the model used in the training process.

[0013] A multi-scale, cross-view geolocation model is trained using contrastive learning based on pre-prepared image samples, resulting in a trained multi-scale, cross-view geolocation model, including:

[0014] The pre-prepared image samples are divided into training samples and test samples.

[0015] The remote sensing images in the training samples are processed using strong enhancement modules and weak enhancement modules respectively to obtain strong enhancement images and weak enhancement images.

[0016] The strongly enhanced image and the weakly enhanced image are respectively input into the two feature extractors of the feature extraction branch of the remote sensing image to obtain the multi-scale features of the strongly enhanced remote sensing image and the weakly enhanced remote sensing image.

[0017] The multi-scale features of the strongly enhanced remote sensing image and the weakly enhanced remote sensing image are respectively input into two mapping heads with shared parameters to obtain the strongly enhanced mapping features and the weakly enhanced mapping features.

[0018] The street view images from the training samples are input into the feature extractor of the feature extraction branch of the street view image to obtain multi-scale features of the street view image.

[0019] Construct the total loss function; the total loss function consists of two retrieval losses and one contrast loss.

[0020] The total loss is determined based on strong enhancement mapping features, weak enhancement mapping features, multi-scale features of street view images, and the total loss function;

[0021] The multi-scale cross-view geolocation model is trained with the total loss to obtain the trained multi-scale cross-view geolocation model.

[0022] The multi-scale cross-view geolocation model was tested using test samples to obtain a trained multi-scale cross-view geolocation model.

[0023] In one embodiment, the feature extractor includes a 4-stage module, two max-pooling layers, and two multilayer perceptrons, wherein each stage module includes several Swing Transformer modules.

[0024] The strongly enhanced image and the weakly enhanced image are respectively input into the two feature extractors of the feature extraction branch of the remote sensing image to obtain multi-scale features of the strongly enhanced remote sensing image and the weakly enhanced remote sensing image, including:

[0025] The enhanced image is input into the first feature extractor in the feature extraction branch of the remote sensing image, which is a 4-stage module, to obtain 4 features with different dimensions.

[0026] The third-dimensional features are processed sequentially through the first max pooling layer and the first multilayer perceptron, and then a matrix inner product operation is performed with the third-dimensional features to obtain the first inner product features.

[0027] The features of the fourth dimension are processed successively through the second max pooling layer and the second multilayer perceptron, and then a matrix inner product operation is performed with the features of the fourth dimension to obtain the second inner product features.

[0028] By concatenating the first inner product feature and the second inner product feature, multi-scale features of the enhanced remote sensing image are obtained.

[0029] The weakly enhanced image is input into the second feature extractor of the feature extraction branch of the remote sensing image to obtain the multi-scale features of the weakly enhanced remote sensing image.

[0030] In one embodiment, the multilayer perceptron includes two fully connected layers.

[0031] In one embodiment, the mapping head includes: a first linearization layer, a ReLU activation function, a batch normalization layer, and a second linearization layer.

[0032] The multi-scale features of strongly enhanced and weakly enhanced remote sensing images are input into two shared parameter mapping heads to obtain strongly enhanced mapping features and weakly enhanced mapping features, including:

[0033] The multi-scale features of the strongly enhanced remote sensing image are input into the first mapping head, and then processed sequentially through the first linearization layer, the ReLU activation function, the batch normalization layer, and the second linearization layer to obtain the strongly enhanced mapping features.

[0034] The multi-scale features of the strongly enhanced remote sensing image are input into the second mapping head to obtain the weakly enhanced mapping features.

[0035] In one embodiment, the total loss function is constructed, including:

[0036] Based on the strong enhancement mapping features and weak enhancement mapping features obtained from the same remote sensing image with strong and weak enhancement, a contrastive loss function is constructed as follows:

[0037]

[0038] in, To compare the losses, d pos-con With d neg-con represents the Euclidean distance between the first matching pair and the first unmatched pair, respectively. The first matching pair consists of strongly enhanced and weakly enhanced images from the same remote sensing image, while the first unmatched pair consists of strongly enhanced and weakly enhanced images from different remote sensing images.

[0039] Based on strong enhancement mapping features, weak enhancement mapping features, and multi-scale features of street view images, two cable loss functions are constructed as follows:

[0040]

[0041]

[0042] in, The first search loss, For the second retrieval loss, d pos-ret1 With d neg-ret1 d represents the Euclidean distance between the second matching pair and the second unmatched pair, where the second matching pair consists of a street view image from the same location and a remote sensing image after strong enhancement, and the unmatched pair consists of a street view image from a different location and a remote sensing image after strong enhancement; pos-ret2 With d neg-ret2represents the Euclidean distance between the third matching pair and the third unmatched pair, respectively. The third matching pair consists of a street view image from the same location and a remote sensing image after weak enhancement, while the unmatched pair consists of a street view image from a different location and a remote sensing image after weak enhancement.

[0043] Based on the contrastive loss function and the two retrieval functions, the total loss function is constructed as follows:

[0044]

[0045] in, This represents the total loss.

[0046] In one embodiment, the remote sensing image to be processed and the street view image are input into a trained multi-scale cross-view geolocation model to obtain geolocation results, including:

[0047] The remote sensing image to be processed and the street view image are respectively input into the feature extractor of the feature extraction branch of the trained multi-scale cross-view geolocation model to obtain the multi-scale features of the remote sensing image and the multi-scale features of the street view image.

[0048] Calculate the cosine similarity between the multi-scale features of the street view image and the multi-scale features of the remote sensing image, sort the remote sensing images according to the magnitude of the cosine similarity, and obtain the geolocation result.

[0049] On the other hand, a multi-scale, cross-view geolocation device is also provided, the device comprising:

[0050] The multi-view image acquisition module is used to acquire remote sensing images and street view images of the same location from different perspectives.

[0051] The multi-scale cross-view geolocation model construction module is used to construct a multi-scale cross-view geolocation model. The multi-scale cross-view geolocation model includes feature extraction branches for remote sensing images and street view images, and an output network. The two feature extraction branches include feature extractors with identical structures. The feature extractors are used to extract multi-scale features of the input images using a multi-stage model based on Swin-transformer. The output network is used to determine the geolocation result based on the multi-scale features output by the two feature extraction branches.

[0052] The multi-scale cross-view geolocation model training module is used to train the multi-scale cross-view geolocation model using a contrastive learning approach based on pre-prepared image samples, resulting in a trained multi-scale cross-view geolocation model.

[0053] The multi-scale cross-view geolocation module is used to input the remote sensing image to be processed and the street view image into the trained multi-scale cross-view geolocation model to obtain the geolocation results of the remote sensing image to be processed and the street view image.

[0054] On the other hand, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described multi-scale cross-view geolocation method.

[0055] Furthermore, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the aforementioned multi-scale cross-view geolocation method.

[0056] The aforementioned multi-scale cross-view geolocation method, system, device, and medium include the following steps: acquiring remote sensing images and street view images of the same location from different perspectives; constructing a multi-scale cross-view geolocation model, which includes feature extraction branches for the remote sensing and street view images, and an output network; the two feature extraction branches each include a feature extractor with identical structure; the feature extractors are used to extract multi-scale features from the input images using a multi-stage model based on the Swin-transformer; the output network is used to determine the geolocation result based on the multi-scale features output by the two feature extraction branches; training the multi-scale cross-view geolocation model using contrastive learning based on pre-prepared image samples to obtain a trained multi-scale cross-view geolocation model; and inputting the remote sensing and street view images to be processed into the trained multi-scale cross-view geolocation model to obtain the geolocation result. This method can improve geolocation accuracy and significantly reduce computational costs. Attached Figure Description

[0057] Figure 1 The differences in viewpoint and field of view between street view images and remote sensing images are shown, where (a) is a street view image and (b) is a remote sensing image.

[0058] Figure 2 This is a flowchart illustrating a multi-scale, cross-view geolocation method in one embodiment;

[0059] Figure 3 This is a structural diagram of a multi-scale, cross-view geolocation model in one embodiment;

[0060] Figure 4 This is a framework for a multi-scale, cross-view geolocation model during training, as described in another embodiment.

[0061] Figure 5 This is a structural block diagram of a multi-scale, cross-view geolocation device in one embodiment;

[0062] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0064] In one embodiment, such as Figure 2 As shown, a multi-scale, cross-view geolocation method is provided, which includes the following steps:

[0065] Step 100: Acquire remote sensing images and street view images of the same location from different perspectives.

[0066] Step 102: Construct a multi-scale cross-view geolocation model. The multi-scale cross-view geolocation model includes feature extraction branches for remote sensing images and street view images, and an output network. The two feature extraction branches include feature extractors with the same structure. The feature extractors are used to extract multi-scale features of the input images using a multi-stage model based on Swin-transformer. The output network is used to determine the geolocation result based on the multi-scale features output by the two feature extraction branches.

[0067] Specifically, to address the feature alignment problem arising from viewpoint and field of view issues, achieving feature alignment between street view images and remote sensing images has become a hot topic in the field of cross-view geolocation. Current mainstream methods include CNN (Convolutional Neural Network) based methods, GAN (Generative Adversarial Network) based methods, and Transformer based methods. CNN-based methods rely on polar coordinate transformation... The polar transformation (or polar transformation) applies polar transformation to remote sensing images, transforming them into images with a similar geometric distribution to street view images. Polar transformation is a mathematical transformation, not a learned parameter, and it requires alignment of the orientation and center point of the street view image and the remote sensing image, lacking generalization ability. This method may fail when the alignment is not satisfied. GANs can generate realistic images, and GAN-based methods leverage this advantage to generate realistic street view images from polar-transformed remote sensing images, using the generated street view images for retrieval. However, GAN-based methods also rely on the alignment of the orientation and center point of the street view image and the remote sensing image, and GAN-based models require significant computational resources and face the problem of model collapse. CNN-based and GAN-based models rely on the alignment of the street view image and the remote sensing image, making it difficult to learn the geometric positional relationship between them. To address this problem, Transformer-based methods were proposed. Transformer-based methods use position encoding techniques to learn the positional relationship between the street view image and the remote sensing image, and employ multi-head attention... Self-attention technology extracts important features from images, achieving feature alignment between street view images and remote sensing images, and this method does not rely on the alignment relationship between the two types of images. Although existing Transformer-based methods can learn the geometric positional relationship between remote sensing images and street view images to some extent, they still have some problems. On the one hand, current Transformer-based methods use a single-stage downsampling strategy to generate a single-resolution feature map, which may lead to information loss. On the other hand, current Transformer-based methods have not yet considered the influence of changing factors such as time, weather, and season. Finally, existing Transformer-based methods perform multi-head attention mechanism calculations globally, resulting in significant computational overhead and long retrieval time.

[0068] A multi-stage model based on the Swin-transformer is used as the backbone network to extract multi-scale features. On one hand, this network can generate feature maps of different resolutions at multiple stages, based on which the network can extract multi-scale features. These features can contain more detailed features. On the other hand, the network employs window-based attention and shifted window-based attention mechanisms to perform multi-head attention calculations locally, reducing computational overhead and retrieval time. The structure of the multi-scale cross-view geolocation model is as follows: Figure 3 As shown.

[0069] This method extracts multi-scale features by generating feature maps of different resolutions using a multi-stage feature extractor based on the Swin-transformer. Multi-scale features contain more detailed information, enhancing the information extraction capability of this method. Furthermore, this method performs multi-head attention computation only locally, significantly reducing computational costs and improving its usability.

[0070] Step 104: Train the multi-scale cross-view geolocation model using contrastive learning based on the pre-prepared image samples to obtain the trained multi-scale cross-view geolocation model.

[0071] Specifically, the contrastive learning approach involves performing strong and weak enhancements on the input remote sensing image in the feature extraction branch of the remote sensing image, and then inputting them into two feature extractors with shared weights. The two mapping heads with shared parameters then obtain two multi-scale mapping features of the remote sensing images. In the feature extraction branch of the street view image, the input street view image is input into the feature extractor to obtain multi-scale features of the street view image. The total loss is determined based on the two multi-scale mapping features of the remote sensing images and the multi-scale features of the street view image. The model is then back-trained to obtain a trained multi-scale cross-view geolocation model.

[0072] The concept of contrastive learning is introduced into the training process of the network. By introducing contrastive learning, the model will be able to cope with the problem of inconsistencies in visual features caused by factors such as time, weather, and season.

[0073] Contrastive learning is a self-supervised learning method that aims to generate meaningful representations by learning the similarities and differences between samples. This method typically involves dividing samples into positive pairs (similar samples) and negative pairs (dissimilar samples), and then optimizing the model so that similar samples are closer together in the embedding space, while dissimilar samples are further apart. By introducing contrastive learning, this method can generate similar feature representations from strongly and weakly enhanced images, increasing the robustness of the features and enhancing the method's ability to handle images affected by time, weather, and seasonal changes.

[0074] Step 106: Input the remote sensing image to be processed and the street view image into the trained multi-scale cross-view geolocation model to obtain the geolocation result.

[0075] The aforementioned multi-scale cross-view geolocation method includes: acquiring remote sensing images and street view images of the same location from different perspectives; constructing a multi-scale cross-view geolocation model, which includes feature extraction branches for the remote sensing images and street view images, and an output network; the two feature extraction branches include feature extractors with identical structures; the feature extractors are used to extract multi-scale features of the input images using a multi-stage model based on the Swin-transformer; the output network is used to determine the geolocation result based on the multi-scale features output by the two feature extraction branches; training the multi-scale cross-view geolocation model using contrastive learning based on pre-prepared image samples to obtain a trained multi-scale cross-view geolocation model; and inputting the remote sensing images and street view images to be processed into the trained multi-scale cross-view geolocation model to obtain the geolocation result. This method can improve geolocation accuracy and significantly reduce computational costs.

[0076] In one embodiment, the image samples include multiple sets of cross-view remote sensing images and street view images; during training, a feature extractor sharing weights with the feature extractor is added to the feature extraction branch of the remote sensing images, a strong enhancement module and a weak enhancement module are added to the two feature extractors respectively, and a mapping head is added after each of the two feature extractors; the multi-scale cross-view geolocation model in the testing process is obtained by removing the added strong enhancement, weak enhancement modules, feature extractor and two mapping heads from the model in the training process; step 104 includes the following steps:

[0077] Step 200: Divide the pre-prepared image samples into training samples and test samples;

[0078] Step 202: Process the remote sensing images in the training samples using the strong enhancement module and the weak enhancement module respectively to obtain the strong enhancement image and the weak enhancement image;

[0079] Step 204: Input the strongly enhanced image and the weakly enhanced image into the two feature extractors of the feature extraction branch of the remote sensing image, respectively, to obtain the multi-scale features of the strongly enhanced remote sensing image and the weakly enhanced remote sensing image;

[0080] Step 206: Input the multi-scale features of the strongly enhanced remote sensing image and the weakly enhanced remote sensing image into two mapping heads with shared parameters, respectively, to obtain the strongly enhanced mapping features and the weakly enhanced mapping features;

[0081] Step 208: Input the street view images from the training samples into the feature extractor of the feature extraction branch of the street view image to obtain the multi-scale features of the street view image;

[0082] Step 210: Construct the total loss function; the total loss function includes two retrieval losses and one contrastive loss;

[0083] Step 212: Determine the total loss based on the strong enhancement mapping features, weak enhancement mapping features, multi-scale features of the street view image, and the total loss function;

[0084] Step 214: Train the multi-scale cross-view geolocation model according to the total loss to obtain the trained multi-scale cross-view geolocation model;

[0085] Step 216: Test the multi-scale cross-view geolocation model input into the testing process using test samples to obtain the trained multi-scale cross-view geolocation model.

[0086] Specifically, Figure 4 The image shows the multi-scale cross-view geolocation model (GeoViewMatch framework) during training, including a feature extraction branch for remote sensing images (containing two feature extractors), two mapping heads with shared parameters, and a feature extraction branch for street view images.

[0087] During the training phase, the network inputs include remote sensing images and street view images.

[0088] For remote sensing images, the network performs both strong and weak augmentation, resulting in two enhanced images. These two enhanced images are then input into two feature extractors sharing weights. For each enhanced image, the feature extractors generate four features f1, f2, f3, and f4 of different dimensions. The model selects features f3 and f4 for max pooling along the channel dimension, and then passes them through a multilayer perceptron. The resulting features are then multiplied by f3 and f4 to obtain feature f. s3 with f s4 , feature f s3with f s4 The features are concatenated to obtain the final multi-scale features f. multi Strongly enhanced and weakly enhanced remote sensing images will yield multi-scale features f respectively. multi-s with f multi-w f multi-w with f multi-s The feature f will be obtained through two mapping heads with shared weights. head-w with f head-s .

[0089] For street view images, the model does not perform strong or weak enhancement; instead, it inputs the original image into feature extraction, obtaining four features f1, f2, f3, and f4. Then, using a multilayer perceptron, the obtained features are multiplied by f3 and f4 to obtain feature f. s3 with f s4 , feature f s3 with f s4 The features are concatenated to obtain the final multi-scale features f. multi-p This multi-scale feature does not pass through the mapping head.

[0090] During training, the model applies strong periodicity and weak augmentation to remote sensing images, introducing a contrastive learning framework. This allows the model to cope with the effects of changes in time, weather, and season on remote sensing images. Simultaneously, the model extracts different features at different stages to obtain multi-scale features, which helps the model capture more information.

[0091] During the testing process, neither the street view images nor the remote sensing images underwent strong or weak enhancement. Multi-scale features were extracted from both the street view and remote sensing images using a feature extractor, and these multi-scale features were not processed by a mapping head. The cosine similarity between the multi-scale features of the street view images and the multi-scale features of the remote sensing images was calculated. The remote sensing images were then sorted according to the magnitude of the cosine similarity to obtain the final retrieval results.

[0092] In one embodiment, the feature extractor includes a 4-stage module, two max-pooling layers, and two multilayer perceptrons, wherein each stage module includes several Swing Transformer modules; step 204 includes:

[0093] The enhanced image is input into the first feature extractor of the remote sensing image feature extraction branch, which is a 4-stage module, to obtain 4 features with different dimensions.

[0094] The third-dimensional features are processed sequentially through the first max pooling layer and the first multilayer perceptron, and then a matrix inner product operation is performed with the third-dimensional features to obtain the first inner product features.

[0095] The fourth dimension features are processed by the second max pooling layer and the second multilayer perceptron, and then a matrix inner product operation is performed with the fourth dimension features to obtain the second inner product features.

[0096] By concatenating the first inner product feature and the second inner product feature, multi-scale features of the enhanced remote sensing image are obtained.

[0097] The weakly enhanced image is input into the second feature extractor of the feature extraction branch of the remote sensing image to obtain the multi-scale features of the weakly enhanced remote sensing image.

[0098] In one embodiment, the multilayer perceptron includes two fully connected layers.

[0099] In one embodiment, the mapping head includes: a first linearization layer, a ReLU activation function, a batch normalization layer, and a second linearization layer; step 206 includes: inputting the multi-scale features of the strongly enhanced remote sensing image into the first mapping head, and processing them sequentially through the first linearization layer, the ReLU activation function, the batch normalization layer, and the second linearization layer to obtain strongly enhanced mapping features; inputting the multi-scale features of the strongly enhanced remote sensing image into the second mapping head to obtain weakly enhanced mapping features.

[0100] In one embodiment, step 210 includes: constructing a contrastive loss function based on the strong enhancement mapping features and weak enhancement mapping features obtained from the strong enhancement and weak enhancement images of the same remote sensing image:

[0101]

[0102] in, To compare the losses, d pos-con With d neg-con represents the Euclidean distance between the first matching pair and the first unmatching pair, respectively, where the first matching pair consists of strongly enhanced and weakly enhanced images from the same remote sensing image, and the first unmatching pair consists of strongly enhanced and weakly enhanced images from different remote sensing images.

[0103] Based on strong enhancement mapping features, weak enhancement mapping features, and multi-scale features of street view images, two cable loss functions are constructed as follows:

[0104]

[0105]

[0106] in, The first search loss, For the second retrieval loss, d pos-ret1 With d neg-ret1d represents the Euclidean distance between the second matching pair and the second unmatched pair, where the second matching pair consists of a street view image from the same location and a remote sensing image after strong enhancement, and the unmatched pair consists of a street view image from a different location and a remote sensing image after strong enhancement; pos-ret2 With d neg-ret2 represents the Euclidean distance between the third matching pair and the third unmatched pair, where the third matching pair is a street view image from the same location and a remote sensing image after weak enhancement, and the unmatched pair is a street view image from different locations and a remote sensing image after weak enhancement.

[0107] Based on the contrastive loss function and the two retrieval functions, the total loss function is constructed as follows:

[0108]

[0109] in, This represents the total loss.

[0110] Specifically, the method proposed in this application mainly includes three losses: retrieval loss. Compared with loss Total loss Defined as:

[0111]

[0112] All three losses employ soft-margin triplet loss:

[0113]

[0114] Where α is the weight, which is set to 10 in this paper. pos With d neg This represents the Euclidean distance between matching and unmatching pairs.

[0115] (1) In retrieval loss In the model, matching pairs consist of street view images from the same location paired with strongly enhanced remote sensing images, while mismatch pairs consist of street view images from different locations paired with strongly enhanced remote sensing images. In the retrieval loss... In this model, matched pairs consist of street view images from the same location paired with weakly enhanced remote sensing images, while mismatched pairs consist of street view images from different locations paired with weakly enhanced remote sensing images. The retrieval loss drives matched pairs to generate similar features, while mismatched pairs generate dissimilar features.

[0116] (2) In comparison of losses In this model, matched pairs are strongly enhanced and weakly enhanced images from the same remote sensing image, while mismatched pairs are strongly enhanced and weakly enhanced images from different remote sensing images. The contrastive loss requires strongly enhanced and weakly enhanced images from the same image to generate similar features, which helps the model cope with the effects of changes in time, weather, and season.

[0117] In one embodiment, step 106 includes: inputting the remote sensing image to be processed and the street view image into the feature extractor of the feature extraction branch of the trained multi-scale cross-view geolocation model to obtain multi-scale features of the remote sensing image and multi-scale features of the street view image; calculating the cosine similarity between the multi-scale features of the street view image and the multi-scale features of the remote sensing image; sorting the remote sensing images according to the magnitude of the cosine similarity to obtain the geolocation result.

[0118] It should be understood that, although Figure 2 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0119] In a validation implementation, experiments were conducted on two datasets: CVUSA and CVACT. The CVUSA dataset: Images were collected in the United States, covering commercial, residential, suburban, and rural areas. It includes 35,532 pairs of training images and 8,884 pairs of test images. The original size of the street view images is 224×1232, and the original size of the remote sensing imagery is 800×800. For the street view images, we scaled them to 128×640. For the remote sensing imagery, we scaled them to 256×256. The CVACT dataset: Images were collected in Australia, covering urban and suburban areas. Similarly, it includes 35,532 pairs of training images and 8,884 pairs of test images, with location information provided. The original size of the street view images is 832×1664, and the original size of the remote sensing imagery is 1200×1200. For the street view images, we first cropped them to 416×1664 and then scaled them to 128×640. For remote sensing images, we scaled them down to 256×256.

[0120] (1) Experimental parameters

[0121] The experiments were conducted using the PyTorch framework on two NVIDIA RTX 4090 GPUs. We used AdamW as the parameter for rate optimization. The weight decay of the AdamW optimizer was set to 1×10⁻⁶. -4 Furthermore, we use Sharpness-Aware Minimization (ASAM) for model optimization to avoid the model falling into local minima. The total number of training epochs is 200. The batch size is set to 64. For the feature extractor in the model, we initialize the Patch Embedding module and the ST Block module with pre-trained weights on ImageNet-1K, and the learning rate of these modules is set to 1×10⁻⁶. -4 The learning rate of the multilayer perceptron and the mapping head in the feature extractor was set to 2×10. -4 We used random color dithering, random grayscale, and Gaussian blur as data augmentation.

[0122] (2) Evaluation indicators and comparison models

[0123] In the experiment, I used Top-K recall rate and two meter-level evaluation metrics as evaluation indicators.

[0124] Top-K recall: The Top-K recall rate is an evaluation metric for retrieval problems. Its calculation process is as follows: Sort the retrieval results according to the cosine similarity, take out the top K remote sensing images with the highest similarity, if these K images contain the correct retrieval results, it is considered a correct retrieval. Divide the number of correct retrievals by the total number of retrievals to get the retrieval accuracy, denoted as R@K.

[0125] Meter-level evaluation 1: A threshold T is selected, and the remote sensing image with the highest cosine similarity is used as the search result. If the distance between the search result and the street view image to be searched is less than or equal to the threshold, it is considered a correct search. The number of correct searches is divided by the total number of searches to obtain the search accuracy rate, denoted as M@T.

[0126] Meter-level evaluation 2: The search results are sorted by cosine similarity. The top K most similar remote sensing images are taken out, and the average distance between the street view image to be searched and these K remote sensing images is calculated. The smaller the average distance, the better the performance.

[0127] To demonstrate the effectiveness of this method, several advanced cross-view geolocation models were used for quantitative comparison: LOCG, CVTF, SAFA, CDTE, and L2LTR.

[0128] (3) Experimental Results

[0129] To verify the effectiveness of our proposed method in generating maps from remote sensing images, we compared it with the methods described above. Table 1 lists the R@K results of each method on the dataset.

[0130] Table 1. R@K results of each method on CVUSA and CVACT

[0131]

[0132] Experimental results show that our proposed model (GeoViewMatch) achieves state-of-the-art performance on two datasets and four R@K results. On the CVUSAshujuji dataset, GeoViewMatch achieves improvements of 1.42%, 0.86%, 0.63%, and 0.06%. On the CVACT dataset, GeoViewMatch achieves improvements of 4.76%, 1.79%, 1.14%, and 0.11%. This demonstrates the effectiveness of our model.

[0133] Tables 2 and 3 show the results of the two meter-level evaluations. Since the CVUSA dataset does not provide image location information, we only performed the evaluation on the CVACT dataset.

[0134] Table 2 Meter-level Assessment 1

[0135]

[0136]

[0137] Table 3 Meter-level Assessment II

[0138]

[0139] Experimental results show that our model exhibits state-of-the-art performance in both meter-level assessments. This indicates that GeoViewMatch can limit the retrieval results to a smaller range, reducing geolocation errors.

[0140] (4) Calculate loss

[0141] To compare the computational costs of each model, Table 4 shows the floating-point operations (GFLOPS) and GPU memory usage during training, as well as the inference time required for testing.

[0142] Table 4 Calculation Cost

[0143]

[0144] Experimental results show that GeoViewMatch significantly reduces the number of floating-point operations and inference time while maintaining a relatively low GPU memory usage. Among the comparison models, the SAFA model has the lowest number of floating-point operations at 42.24 × 10⁻⁶. 9 GeoViewMatch's floating-point operations are only 23% of SAFA's. Among the compared models, L2LTR has the shortest inference time at 82ms, while GeoViewMatch's inference time is only 51% of L2LTR's. Compared to other models, GeoViewMatch's computational cost is significantly reduced, which means the model is more suitable for large-scale geolocation tasks in real-world scenarios.

[0145] In one embodiment, such as Figure 5 As shown, a multi-scale cross-view geolocation device is provided, comprising: a multi-view image acquisition module, a multi-scale cross-view geolocation model construction module, a multi-scale cross-view geolocation model training module, and a multi-scale cross-view geolocation module, wherein:

[0146] The multi-view image acquisition module is used to acquire remote sensing images and street view images of the same location from different perspectives.

[0147] The multi-scale cross-view geolocation model construction module is used to construct a multi-scale cross-view geolocation model. The multi-scale cross-view geolocation model includes feature extraction branches for remote sensing images and street view images, and an output network. The two feature extraction branches include feature extractors with identical structures. The feature extractors are used to extract multi-scale features of the input image using a multi-stage model based on Swin-transformer. The output network is used to determine the geolocation result based on the multi-scale features output by the two feature extraction branches.

[0148] The multi-scale cross-view geolocation model training module is used to train the multi-scale cross-view geolocation model using a contrastive learning approach based on pre-prepared image samples, resulting in a trained multi-scale cross-view geolocation model.

[0149] The multi-scale cross-view geolocation module is used to input the remote sensing image to be processed and the street view image into the trained multi-scale cross-view geolocation model to obtain the geolocation results of the remote sensing image to be processed and the street view image.

[0150] In one embodiment, the image samples include multiple sets of cross-view remote sensing images and street view images. During training, a feature extractor sharing weights with the feature extractor is added to the feature extraction branch of the remote sensing images. A strong enhancement module and a weak enhancement module are added to the two feature extractors respectively, and a mapping head is added after each of the two feature extractors. The multi-scale cross-view geolocation model during testing is obtained by removing the added strong enhancement, weak enhancement modules, feature extractor, and two mapping heads from the model during training. The multi-scale cross-view geolocation model training module is also used to divide the pre-prepared image samples into training samples and test samples. The remote sensing images in the training samples are processed by the strong enhancement module and the weak enhancement module respectively to obtain strongly enhanced images and weakly enhanced images. The strongly enhanced images and weakly enhanced images are respectively input into the two feature extractors of the feature extraction branch of the remote sensing images. In the extraction process, multi-scale features of strongly enhanced and weakly enhanced remote sensing images are obtained. These multi-scale features are then input into two shared parameter mapping heads to obtain strongly enhanced mapping features and weakly enhanced mapping features, respectively. Street view images from the training samples are input into the feature extractor of the street view image feature extraction branch to obtain multi-scale street view image features. A total loss function is constructed, comprising two retrieval losses and one contrast loss. The total loss is determined based on the strongly enhanced mapping features, weakly enhanced mapping features, multi-scale street view image features, and the total loss function. The multi-scale cross-view geolocation model is then trained using the total loss to obtain the trained multi-scale cross-view geolocation model. Finally, test samples are used to test the multi-scale cross-view geolocation model input to the testing process to obtain the trained multi-scale cross-view geolocation model.

[0151] In one embodiment, the feature extractor includes a 4-stage module, two max-pooling layers, and two multilayer perceptrons, wherein each stage module includes several Swing Transformer modules; the multi-scale cross-view geolocation model training module is also used to input the strongly enhanced image into the first feature extractor of the feature extraction branch of the remote sensing image into the 4-stage module to obtain four features of different dimensions; the third-dimensional feature is processed successively through the first max-pooling layer and the first multilayer perceptron and then subjected to a matrix inner product operation with the third-dimensional feature to obtain the first inner product feature; the fourth-dimensional feature is processed successively through the second max-pooling layer and the second multilayer perceptron and then subjected to a matrix inner product operation with the fourth-dimensional feature to obtain the second inner product feature; the first inner product feature and the second inner product feature are concatenated to obtain the multi-scale features of the strongly enhanced remote sensing image; the weakly enhanced image is input into the second feature extractor of the feature extraction branch of the remote sensing image to obtain the multi-scale features of the weakly enhanced remote sensing image.

[0152] In one embodiment, the multilayer perceptron in the multiscale cross-view geolocation model training module includes two fully connected layers.

[0153] In one embodiment, the mapping head includes: a first linearization layer, a ReLU activation function, a batch normalization layer, and a second linearization layer; the multi-scale cross-view geolocation model training module is further used to input the multi-scale features of the strongly enhanced remote sensing image into the first mapping head, and after processing through the first linearization layer, the ReLU activation function, the batch normalization layer, and the second linearization layer in sequence, obtain the strongly enhanced mapping features; and input the multi-scale features of the strongly enhanced remote sensing image into the second mapping head to obtain the weakly enhanced mapping features.

[0154] In one embodiment, the multi-scale cross-view geolocation model training module is further configured to construct a contrastive loss function based on the strong enhancement mapping features and weak enhancement mapping features obtained from the strong enhancement and weak enhancement images of the same remote sensing image:

[0155]

[0156] in, To compare the losses, d pos-con With d neg-con represents the Euclidean distance between the first matching pair and the first unmatching pair, respectively, where the first matching pair consists of strongly enhanced and weakly enhanced images from the same remote sensing image, and the first unmatching pair consists of strongly enhanced and weakly enhanced images from different remote sensing images.

[0157] Based on strong enhancement mapping features, weak enhancement mapping features, and multi-scale features of street view images, two cable loss functions are constructed as follows:

[0158]

[0159]

[0160] in, The first search loss, For the second retrieval loss, d pos-ret1 With d neg-ret1 d represents the Euclidean distance between the second matching pair and the second unmatched pair, where the second matching pair consists of a street view image from the same location and a remote sensing image after strong enhancement, and the unmatched pair consists of a street view image from a different location and a remote sensing image after strong enhancement; pos-ret2 With d neg-ret2 represents the Euclidean distance between the third matching pair and the third unmatched pair, where the third matching pair is a street view image from the same location and a remote sensing image after weak enhancement, and the unmatched pair is a street view image from different locations and a remote sensing image after weak enhancement.

[0161] Based on the contrastive loss function and the two retrieval functions, the total loss function is constructed as follows:

[0162]

[0163] in, This represents the total loss.

[0164] In one embodiment, the multi-scale cross-view geolocation module is further configured to input the remote sensing image to be processed and the street view image into the feature extractor of the feature extraction branch of the trained multi-scale cross-view geolocation model, respectively, to obtain multi-scale features of the remote sensing image and multi-scale features of the street view image; calculate the cosine similarity between the multi-scale features of the street view image and the multi-scale features of the remote sensing image; sort the remote sensing images according to the magnitude of the cosine similarity to obtain the geolocation result.

[0165] Specific limitations regarding the multi-scale, cross-view geolocation device can be found in the limitations of the multi-scale, cross-view geolocation method described above, and will not be repeated here. Each module in the aforementioned multi-scale, cross-view geolocation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute these modules.

[0166] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a multi-scale, cross-view geolocation method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0167] Those skilled in the art will understand that Figure 6The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0168] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiment.

[0169] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program being executed by a processor to perform the steps in the method embodiments described above.

[0170] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0171] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0172] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A multi-scale, cross-view geolocation method, characterized in that, The method includes: Acquire remote sensing images and street view images of the same location from different perspectives; A multi-scale cross-view geolocation model is constructed, which includes feature extraction branches for remote sensing images and street view images, and an output network. The two feature extraction branches include feature extractors with identical structures. The feature extractors are used to extract multi-scale features of the input images using a multi-stage model based on Swin-transformer. The output network is used to determine the geolocation result based on the multi-scale features output by the two feature extraction branches. The multi-scale cross-view geolocation model is trained using contrastive learning based on pre-prepared image samples to obtain a trained multi-scale cross-view geolocation model. During training, a feature extractor sharing weights with the original feature extractor is added to the feature extraction branch of the remote sensing image. Strong enhancement modules and weak enhancement modules are added to the two feature extractors respectively, and a mapping head is added after each feature extractor. The multi-scale cross-view geolocation model in the testing process is obtained by removing the added strong enhancement module, weak enhancement module, feature extractor, and two mapping heads from the model trained in the training process. The strong enhancement module and weak enhancement module are used to process the remote sensing image to generate strongly enhanced and weakly enhanced images, respectively. The remote sensing image to be processed and the street view image are input into the trained multi-scale cross-view geolocation model to obtain the geolocation result; The feature extractor includes a 4-stage module, two max pooling layers, and two multilayer perceptrons, wherein each stage module includes several Swing Transformer modules. The strongly enhanced image and the weakly enhanced image are respectively input into two feature extractors in the feature extraction branch of the remote sensing image to obtain multi-scale features of the strongly enhanced image and the weakly enhanced image, including: The enhanced image is input into the four-stage module of the first feature extractor in the feature extraction branch of the remote sensing image to obtain four features with different dimensions; The third-dimensional features are processed sequentially through the first max pooling layer and the first multilayer perceptron, and then a matrix inner product operation is performed with the third-dimensional features to obtain the first inner product features. The fourth dimension features are processed by the second max pooling layer and the second multilayer perceptron, and then a matrix inner product operation is performed with the fourth dimension features to obtain the second inner product features. The first inner product feature and the second inner product feature are concatenated to obtain the multi-scale features of the strongly enhanced image; The weakly enhanced image is input into the second feature extractor of the feature extraction branch of the remote sensing image to obtain the multi-scale features of the weakly enhanced image; A contrastive loss function is constructed based on the strong enhancement mapping features and weak enhancement mapping features obtained from the strong enhancement and weak enhancement images of the same remote sensing image. Based on the strong enhancement mapping features and the weak enhancement mapping features, as well as the multi-scale features of the street view image, two search loss functions are constructed. Based on the contrast loss function and the two retrieval loss functions, a total loss function is constructed.

2. The method according to claim 1, characterized in that, The image samples include multiple sets of remote sensing images and street view images from different perspectives; The multi-scale cross-view geolocation model is trained using contrastive learning based on pre-prepared image samples to obtain a trained multi-scale cross-view geolocation model, including: The pre-prepared image samples are divided into training samples and test samples; The remote sensing images in the training samples are processed using strong enhancement modules and weak enhancement modules respectively to obtain strong enhancement images and weak enhancement images; The strongly enhanced image and the weakly enhanced image are respectively input into two feature extractors in the feature extraction branch of the remote sensing image to obtain the multi-scale features of the strongly enhanced image and the weakly enhanced image; The multi-scale features of the strongly enhanced image and the weakly enhanced image are respectively input into two mapping heads with shared parameters to obtain the strongly enhanced mapping features and the weakly enhanced mapping features; The street view images from the training samples are input into the feature extractor of the feature extraction branch of the street view image to obtain multi-scale features of the street view image; Construct a total loss function; the total loss function includes two retrieval loss functions and one contrastive loss function; The total loss is determined based on the strong enhancement mapping features, the weak enhancement mapping features, the multi-scale features of the street view image, and the total loss function; The multi-scale cross-view geolocation model is trained according to the total loss to obtain the trained multi-scale cross-view geolocation model. The multi-scale cross-view geolocation model was tested using test samples to obtain a trained multi-scale cross-view geolocation model.

3. The method according to claim 1, characterized in that, The multilayer sensor includes two fully connected layers.

4. The method according to claim 1, characterized in that, The mapping head includes: a first linearization layer, a ReLU activation function, a batch normalization layer, and a second linearization layer; The multi-scale features of the strongly enhanced image and the weakly enhanced image are respectively input into two mapping heads with shared parameters to obtain the strongly enhanced mapping features and the weakly enhanced mapping features, including: The multi-scale features of the strongly enhanced image are input into the first mapping head, and then processed through the first linearization layer, the ReLU activation function, the batch normalization layer, and the second linearization layer to obtain the strongly enhanced mapping features. The multi-scale features of the strongly enhanced image are input into the second mapping head to obtain the weakly enhanced mapping features.

5. The method according to claim 2, characterized in that, Construct the total loss function, including: Based on the strong enhancement mapping features and weak enhancement mapping features obtained from the same remote sensing image with strong and weak enhancement, a contrastive loss function is constructed as follows: in, To compare the losses, and represents the Euclidean distance between the first matching pair and the first unmatching pair, respectively, where the first matching pair consists of strongly enhanced and weakly enhanced images from the same remote sensing image, and the first unmatching pair consists of strongly enhanced and weakly enhanced images from different remote sensing images. Based on the strong enhancement mapping features, the weak enhancement mapping features, and the multi-scale features of the street view image, two retrieval loss functions are constructed as follows: in, The first search loss, For the second retrieval loss, and represents the Euclidean distance between the second matching pair and the second unmatched pair, respectively. The second matching pair consists of a street view image from the same location and a remote sensing image after strong enhancement, while the unmatched pair consists of a street view image from a different location and a remote sensing image after strong enhancement. and represents the Euclidean distance between the third matching pair and the third unmatched pair, where the third matching pair is a street view image from the same location and a remote sensing image after weak enhancement, and the unmatched pair is a street view image from different locations and a remote sensing image after weak enhancement. Based on the contrast loss function and the two retrieval loss functions, the total loss function is constructed as follows: in, This represents the total loss.

6. The method according to claim 1, characterized in that, The remote sensing image to be processed and the street view image are input into a trained multi-scale cross-view geolocation model to obtain geolocation results, including: The remote sensing image to be processed and the street view image are respectively input into the feature extractor of the feature extraction branch of the multi-scale cross-view geolocation model to obtain the multi-scale features of the remote sensing image and the multi-scale features of the street view image. Calculate the cosine similarity between the multi-scale features of the street view image and the multi-scale features of the remote sensing image, sort the remote sensing images according to the magnitude of the cosine similarity, and obtain the geographic location result.

7. A multi-scale, cross-view geolocation device, characterized in that, The device employs the multi-scale cross-view geolocation method described in claim 1 for multi-scale cross-view geolocation positioning; including: The multi-view image acquisition module is used to acquire remote sensing images and street view images of the same location from different perspectives. A multi-scale cross-view geolocation model construction module is used to construct a multi-scale cross-view geolocation model. The multi-scale cross-view geolocation model includes feature extraction branches for remote sensing images and street view images, and an output network. The two feature extraction branches include feature extractors with identical structures. The feature extractors are used to extract multi-scale features of the input images using a multi-stage model based on Swin-transformer. The output network is used to determine the geolocation result based on the multi-scale features output by the two feature extraction branches. The multi-scale cross-view geolocation model training module is used to train the multi-scale cross-view geolocation model using a contrastive learning method based on pre-prepared image samples, so as to obtain a trained multi-scale cross-view geolocation model. The multi-scale cross-view geolocation module is used to input the remote sensing image to be processed and the street view image into the trained multi-scale cross-view geolocation model to obtain the geolocation results of the remote sensing image to be processed and the street view image.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images