Streetscape image geographic positioning method and system based on cross-modal comparative learning and description enhancement

Abstract

The invention relates to the technical field of geographic positioning, in particular to a streetscape image geographic positioning method and system based on cross-modal comparative learning and description enhancement, and the method comprises the steps: obtaining the visual features of a to-be-queried streetscape image and a feature set of a reference image library through a pre-trained image encoder; generating text description features of the to-be-queried streetscape image by using a large language model, and fusing the visual features and the text description features to generate a query feature set of the to-be-queried streetscape image; aligning the to-be-queried streetscape image with the reference image based on the query feature set and the image library feature set to obtain a geographic position of the to-be-queried streetscape image; wherein the image encoder is trained based on the streetscape image sample set and the reference image sample set in the training process by using a preset cross-modal contrast learning loss function. The method can solve the problem of non-panoramic image positioning, improves the accuracy and flexibility of the positioning process, and has a good application prospect in the field of geographic positioning.

Description

Technical Field

[0001] The present invention relates to the field of geographic positioning technology, and in particular to a street view image geographic positioning method and system based on cross-modal contrast learning and description enhancement. Background Art

[0002] Street view geolocalization plays a key role in urban analysis, navigation, and social research, and its main task is to identify the geographic location of a given street view image. Broadly speaking, street view geolocalization mainly uses two frameworks: retrieval-based methods and classification-based methods, both of which rely heavily on visual features extracted from a single region. Among them, retrieval-based methods use the features of the query image to search for similar images in street view or geotagged galleries. Classification-based methods use trained classifiers to directly predict the geographic region of an image. However, both methods are highly dependent on the quality and diversity of the dataset, and their performance is limited by the data coverage. Given the complexity of street view images in terms of viewpoints and contexts, it is a huge challenge to build a comprehensive dataset covering all possible viewpoints and regions around the world. Therefore, both retrieval-based and classification-based methods have limited generalization capabilities across different geographic regions around the world. In addition, existing methods align panoramic street view images with remote sensing images to determine geographic locations. Although these methods are effective for locating panoramic street view images, they are not suitable for locating geographic locations in general street view images, such as images taken with general cameras or smartphones. Therefore, methods that can directly align non-panoramic street view images with remote sensing images are urgently needed, which is crucial for practical applications. Summary of the invention

[0003] To this end, the present invention provides a street view image geolocation method and system based on cross-modal contrastive learning and description enhancement, which solves the problem that existing geolocation is not suitable for non-panoramic street view image alignment.

[0004] According to the design scheme provided by the present invention, on the one hand, a street view image geolocation method based on cross-modal contrast learning and description enhancement is provided, comprising:

[0005] Using a pre-trained image encoder to obtain visual features of a street view image to be queried and a feature set of a reference image library, using a language model to generate text description features of the street view image to be queried, and fusing the visual features and the text description features to generate a query feature set of the street view image to be queried, wherein the reference image is a remote sensing image with a geographical tag;

[0006] Aligning the street view image to be queried with the reference image based on the query feature set and the gallery feature set to obtain the geographic location of the street view image to be queried;

[0007] Among them, during the training process, the image encoder is trained based on the street view image sample set and the reference image sample set and using a preset cross-modal contrastive learning loss function. The cross-modal contrastive learning loss function is composed of a cross-view contrastive learning loss between street view image samples and reference image samples and an inter-view contrastive learning loss between street view image sample pairs and between reference image sample pairs.

[0008] As a street view image geolocation method based on cross-modal contrastive learning and description enhancement of the present invention, further, the image encoder is constructed using a CNN or ViT architecture.

[0009] As a street view image geolocation method based on cross-modal contrastive learning and description enhancement of the present invention, further, the training process of the image encoder includes the following contents:

[0010] Collecting street view image sample data and reference image sample data, and obtaining a street view image sample data set and a reference image sample data set through image enhancement operations, wherein the image enhancement operations include image compression, color jittering, image blurring, and image occlusion;

[0011] Building an image encoder based on a CNN or ViT architecture and setting a cross-modal contrastive learning loss function, wherein the image encoder includes a reference image encoder and a query image encoder;

[0012] The image encoder is trained based on a cross-modal contrastive learning loss function and using a street view image sample dataset and a reference image sample dataset.

[0013] As a street view image geolocation method based on cross-modal contrastive learning and description enhancement of the present invention, further, in the image encoder training process, a language model is used to obtain text description features of street view image samples, and a multi-layer perceptron is used to project the text description features into the visual feature space of the street view image samples to fuse the text description features with the visual features.

[0014] As a street view image geolocation method based on cross-modal contrastive learning and description enhancement of the present invention, further, a cross-modal contrastive learning loss function is set, including:

[0015] An inter-modality contrast loss is constructed based on the positive and negative sample features in the street view image sample dataset and the reference image sample dataset, so as to use the inter-modality contrast loss as the cross-view contrast learning loss;

[0016] Based on the positive and negative sample pairs in the street view image sample dataset and the positive and negative sample pairs in the reference image sample dataset, we construct the street view image intra-modality contrast loss and the reference image intra-modality contrast loss respectively, and combine the two intra-modality contrast losses into the inter-view contrast learning loss.

[0017] The cross-modal contrastive learning loss function is constructed based on the cross-view contrastive learning loss and the inter-view contrastive learning loss and through the balancing factor.

[0018] As a street view image geolocation method based on cross-modal contrastive learning and description enhancement of the present invention, further, the cross-modal contrastive learning loss function is expressed as: in, They are the cross-view contrastive learning loss and the inter-view contrastive learning loss, respectively, and w is the balancing factor.

[0019] As a street view image geolocation method based on cross-modal contrastive learning and description enhancement of the present invention, further, the street view image to be queried is aligned with the reference image based on the query feature set and the gallery feature set, comprising:

[0020] Based on the query feature set and using feature similarity, candidate libraries are retrieved from the library feature set, and the retrieval results are converted into a visual heat map;

[0021] The feature similarity score is used to amplify the contrast between the candidate image library and the query feature set through an exponential function, and the query street view image is aligned with the reference image based on a visualized heat map.

[0022] In another aspect, the present invention further provides a street view image geolocation system based on cross-modal contrast learning and description enhancement, comprising: a feature acquisition module and an image alignment module, wherein:

[0023] A feature acquisition module is used to use a pre-trained image encoder to acquire visual features of the street view image to be queried and a feature set of a reference image library, use a language model to generate text description features of the street view image to be queried, and fuse the visual features and the text description features to generate a query feature set of the street view image to be queried, wherein the reference image is a remote sensing image with a geographical tag;

[0024] An image alignment module, used for aligning the street view image to be queried with the reference image based on the query feature set and the gallery feature set, so as to obtain the geographic location of the street view image to be queried;

[0025] Among them, during the training process, the image encoder is trained based on the street view image sample set and the reference image sample set and using a preset cross-modal contrastive learning loss function. The cross-modal contrastive learning loss function is composed of a cross-view contrastive learning loss between street view image samples and reference image samples and an inter-view contrastive learning loss between street view image sample pairs and between reference image sample pairs.

[0026] Beneficial effects of the present invention:

[0027] The present invention trains an image encoder for street view geolocation based on a cross-modal contrastive learning loss function, wherein the loss function is composed of cross-view (inter-modality) contrastive learning between street view images and remote sensing images, and inter-view (intra-modality) contrastive learning between street view image pairs and remote sensing image pairs. In order to merge the description of the street view image into the query feature, the image semantic description is encoded into a vector using the LLM2Vec language model, which is then fused with the street view image features to retrieve remote sensing images with geo-tags and achieve accurate geolocation. After the model training is completed, the additional description can be merged into the geolocation process through manual input, thereby improving the accuracy and flexibility of the positioning process. The present invention has good application prospects in the field of geolocation. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 The flowchart of geo-positioning of street view images based on cross-modal contrastive learning and description enhancement in an embodiment is shown;

[0029] Figure 2 This is a schematic diagram of the cross-modal contrastive learning principle in the embodiment;

[0030] Figure 3 This is a schematic diagram of the street view geo-positioning visualization result in the embodiment;

[0031] Figure 4 This is an illustration of composite modal retrieval in the embodiment. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the present invention clearer and more understandable, the present invention is further described in detail below in conjunction with the accompanying drawings and technical solutions.

[0033] Street view geolocation aims to predict the geographic location of street view images, which is crucial for applications such as urban planning and navigation. Most existing methods focus on panoramic street views or remote sensing images. Although these methods perform well on experimental datasets, they have two major limitations: they cannot locate the geographic location in general images (such as images taken by cameras or smartphones), and they do not consider adding human input to the localization process. In order to address the potential value of human cues for enhancing geolocation, an embodiment of the present invention provides a street view image geolocation method based on cross-modal contrastive learning and description enhancement, such as Figure 1 As shown, including:

[0034] S101, using a pre-trained image encoder to obtain visual features of a street view image to be queried and a feature set of a reference image library, using a language model to generate text description features of the street view image to be queried, and fusing the visual features and the text description features to generate a query feature set of the street view image to be queried, wherein the reference image is a remote sensing image with a geographic tag;

[0035] S102, aligning the street view image to be queried with the reference image based on the query feature set and the gallery feature set to obtain the geographic location of the street view image to be queried;

[0036] Among them, during the training process, the image encoder is trained based on the street view image sample set and the reference image sample set and using a preset cross-modal contrastive learning loss function. The cross-modal contrastive learning loss function is composed of a cross-view contrastive learning loss between street view image samples and reference image samples and an inter-view contrastive learning loss between street view image sample pairs and between reference image sample pairs.

[0037] like Figure 2 As shown in the figure, the cross-view image geolocation architecture of this scheme is built on a Siamese network, where the image encoder can adopt a CNN or ViT architecture to map street view and satellite imagery into feature embeddings for retrieval. Two ViT-based CLIP-ViT / 14 models can be used as image encoders. The encoder for satellite images is initialized with weights from RemoteCLIP, a variant of CLIP-ViT / 14 that is fine-tuned on a large satellite imagery dataset. For cross-modal fusion, the text encoder LLM2Vec, which can encode long texts, can be used to encode text descriptions. The resulting text features are projected into the image feature space through a multi-layer perceptron (MLP), and then the combiner module integrates the text and image features.

[0038] Among them, the training process of the image encoder can be designed to include the following contents:

[0039] Collecting street view image sample data and reference image sample data, and obtaining a street view image sample data set and a reference image sample data set through image enhancement operations, wherein the image enhancement operations include image compression, color jittering, image blurring, and image occlusion;

[0040] Building an image encoder based on a CNN or ViT architecture and setting a cross-modal contrastive learning loss function, wherein the image encoder includes a reference image encoder and a query image encoder;

[0041] The image encoder is trained based on a cross-modal contrastive learning loss function and using a street view image sample dataset and a reference image sample dataset.

[0042] like Figure 2In the framework shown, the training process is divided into two stages. In the first stage, the two image encoders are trained using cross-modal contrastive loss to enable them to distinguish images from different locations. In the second stage, the image features are combined with the language description using a composite modality representation method. In this stage, the image encoder is frozen, while the MLP network and combiner are trained separately to project text features and perform feature fusion. The LLM2Vec text encoder remains frozen throughout the stage.

[0043] Among them, when setting the cross-modal contrastive learning loss function, it can be designed to include:

[0044] An inter-modality contrast loss is constructed based on the positive and negative sample features in the street view image sample dataset and the reference image sample dataset, so as to use the inter-modality contrast loss as the cross-view contrast learning loss;

[0045] Based on the positive and negative sample pairs in the street view image sample dataset and the positive and negative sample pairs in the reference image sample dataset, we construct the street view image intra-modality contrast loss and the reference image intra-modality contrast loss respectively, and combine the two intra-modality contrast losses into the inter-view contrast learning loss.

[0046] The cross-modal contrastive learning loss function is constructed based on the cross-view contrastive learning loss and the inter-view contrastive learning loss and through the balancing factor.

[0047] make is a set of query images (i.e. street view images), is a set of reference images (i.e., satellite images). The contrastive learning objective is defined based on the InfoNCE loss as follows.

[0048] For the query image I q , there can be:

[0049]

[0050] Where q represents the features of the query image and R represents the feature set of the reference image. + corresponds to q, and the rest of the reference images are used as negative samples. Similarly, for the reference image I r , there can be:

[0051]

[0052] Where r represents the feature of the reference image, Q is the feature set of the query image, and there is a positive sample q + Matches r.

[0053] The symmetric contrast loss is defined as:

[0054]

[0055] Inter-modality contrast loss is crucial for street view geolocation because it enables effective alignment between different modalities. Specifically, for a query image Iq, a query encoder can be used to extract query features Q: = q, where q = Encoder q (I q ). In addition, let is a set of street view images from different viewpoints. The query features from these alternative views are represented as Where q+ = Encoder q (I q+ ). For the reference image, apply a random rotation (90°, 180° or 270°) and extract the corresponding rotation reference feature R: = {r}, where r = Encoder r (I r ). In addition, the original unrotated reference image can be retained and its corresponding feature set R can be obtained + :={r +}, where r + =Encoder r (I r + ). The final inter-modal contrast loss function is defined as:

[0056]

[0057] To enforce feature consistency between different views, an intra-modality contrastive learning objective is utilized to generate similar representations between the query image and the corresponding image from another view. In addition, to improve the robustness of the method to misaligned directions, the same contrastive learning objective is designed for the reference image and its enhanced variants. Overall, the intra-modality contrastive loss is defined as:

[0058]

[0059] By combining the above inter-modality and intra-modality contrastive learning objectives, the training loss of image representation with cross-modality contrastive loss is:

[0060]

[0061] To further improve the performance of street view geolocalization, a multi-modal representation is introduced to combine text descriptions with query images. By combining the visual features of the query image and the semantic information of the text description, the model is able to exploit the complementarity of these modalities. The text description provides additional context that helps disambiguate the location and enrich the feature representation, especially in cases where visual cues alone may not be sufficient.

[0062] In order to obtain a composite representation, a fusion strategy is adopted, that is, the text features are encoded using a frozen language model LLM2Vec. These text features are then integrated with the query image features using a combiner module. The text features are first projected into the same feature space as the image features through an MLP. The projected text and image features are then concatenated, and an additional MLP with sigmoid activation is applied in its last layer to calculate the fusion weights. Finally, these fusion weights are used to perform a weighted average of the concatenated features to obtain the final combined features for the street view geolocation task. Specifically, the fusion weight λ is calculated as follows:

[0063] λ=σ(MLP([t∥q])) (7)

[0064] Where q represents the query image feature, [·][·] represents the feature concatenation operation, and σ represents the S-type activation function.

[0065] t=MLP(Encoder t (d q )) (8)

[0066] The Encoder t is a pre-trained text encoder, d q is the text description corresponding to the query image. Then the composite modal representation is calculated as follows:

[0067] c=λ·q+(1-λ)·t (9)

[0068] Finally, the training objective of the composite modality representation for contrastive learning is:

[0069]

[0070] Where C:={c} is a set of composite features.

[0071] After the model training is completed, the trained model is used to extract features from remote sensing images and street view images to form a gallery and query set, respectively. Then, street view geolocation is performed by retrieving a candidate set from the gallery set using the query features. Specifically, the retrieval results can be converted into a visual heat map; the contrast between the candidate gallery and the query feature set is amplified by an exponential function using the feature similarity score, and the street view image to be queried is aligned with the reference image based on the visual heat map.

[0072] Furthermore, based on the above method, an embodiment of the present invention also provides a street view image geolocation system based on cross-modal contrast learning and description enhancement, comprising: a feature acquisition module and an image alignment module, wherein:

[0073] A feature acquisition module is used to use a pre-trained image encoder to acquire visual features of the street view image to be queried and a feature set of a reference image library, use a language model to generate text description features of the street view image to be queried, and fuse the visual features and the text description features to generate a query feature set of the street view image to be queried, wherein the reference image is a remote sensing image with a geographical tag;

[0074] An image alignment module, used for aligning the street view image to be queried with the reference image based on the query feature set and the gallery feature set, so as to obtain the geographic location of the street view image to be queried;

[0075] Among them, during the training process, the image encoder is trained based on the street view image sample set and the reference image sample set and using a preset cross-modal contrastive learning loss function. The cross-modal contrastive learning loss function is composed of a cross-view contrastive learning loss between street view image samples and reference image samples and an inter-view contrastive learning loss between street view image sample pairs and between reference image sample pairs.

[0076] In order to verify the effectiveness of this solution, the following is a further explanation based on experimental data:

[0077] In the experiment, CLIP-ViT-L / 14 was used as the image encoder, and the satellite image encoder was initialized with weights from Remote-CLIP. Both encoders processed images with an input resolution of 224×224. The loss weight factor w was set to 0.5. In the first stage, the model was trained for 40 epochs with a batch size of 128, and then trained for 15 epochs in the second stage. The AdamW optimizer was used with an initial learning rate of 0.0001 and a cosine learning rate scheduler was applied. All experiments were conducted on three NVIDIA GeForce RTX A800 GPUs. To augment the data, all images input to the image encoder undergo various transformations, including image compression, color jittering, blurring, and occlusion. For LLM2Vec, a supervised version based on Mistral-7B was used.

[0078] By dividing the satellite image into smaller blocks, the similarity between the query image and each satellite image block in the region is calculated and the result is visualized as a heat map. The satellite image is divided into blocks with a resolution of 700×700, which is consistent with the training dataset. To enhance the visualization, an exponential function is applied to the calculated similarity scores to amplify the contrast. Figure 3As shown in the figure, in the local heat map, the white box represents the reference image area; in the global heat map, the white box represents the area corresponding to the local heat map. Two examples are shown from the test set: one represents urban buildings and the other represents suburban roads. The heat map visualization shows that the solution successfully identifies the areas with high similarity and accurately locates the true location of the query image; to illustrate the impact of the composite modality retrieval example, Figure 4 As shown, incorporating image descriptions (either from LVLM or human input) can improve retrieval performance.

[0079] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0080] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0081] The units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementation is not considered to be beyond the scope of the present invention.

[0082] Those skilled in the art will appreciate that all or part of the steps in the above method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk. Optionally, all or part of the steps in the above embodiment can also be implemented using one or more integrated circuits, and accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or in the form of software function modules. The present invention is not limited to any specific form of combination of hardware and software.

[0083] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A street view image geolocation method based on cross-modal contrastive learning and description enhancement, characterized in that: Include: Using a pre-trained image encoder to obtain visual features of a street view image to be queried and a feature set of a reference image library, using a language model to generate text description features of the street view image to be queried, and fusing the visual features and the text description features to generate a query feature set of the street view image to be queried, wherein the reference image is a remote sensing image with a geographical tag; Aligning the street view image to be queried with the reference image based on the query feature set and the gallery feature set to obtain the geographic location of the street view image to be queried; Among them, during the training process, the image encoder is trained based on the street view image sample set and the reference image sample set and using a preset cross-modal contrastive learning loss function. The cross-modal contrastive learning loss function is composed of a cross-view contrastive learning loss between street view image samples and reference image samples and an inter-view contrastive learning loss between street view image sample pairs and between reference image sample pairs.

2. The street view image geolocation method based on cross-modal contrastive learning and description enhancement according to claim 1, characterized in that: The image encoder is constructed using a CNN or ViT architecture.

3. The street view image geolocation method based on cross-modal contrastive learning and description enhancement according to claim 1, characterized in that: The training process of the image encoder includes the following: Collecting street view image sample data and reference image sample data, and obtaining a street view image sample data set and a reference image sample data set through image enhancement operations, wherein the image enhancement operations include image compression, color jittering, image blurring, and image occlusion; Building an image encoder based on a CNN or ViT architecture and setting a cross-modal contrastive learning loss function, wherein the image encoder includes a reference image encoder and a query image encoder; The image encoder is trained based on a cross-modal contrastive learning loss function and using a street view image sample dataset and a reference image sample dataset.

4. The street view image geolocation method based on cross-modal contrastive learning and description enhancement according to claim 1 or 3, characterized in that: During the image encoder training process, the language model is used to obtain the text description features of the street view image samples, and the multi-layer perceptron is used to project the text description features into the visual feature space of the street view image samples to merge the text description features with the visual features.

5. The street view image geolocation method based on cross-modal contrastive learning and description enhancement according to claim 1 or 3, characterized in that: Set the cross-modal contrastive learning loss function, including: An inter-modality contrast loss is constructed based on the positive and negative sample features in the street view image sample dataset and the reference image sample dataset, so as to use the inter-modality contrast loss as the cross-view contrast learning loss; Based on the positive and negative sample pairs in the street view image sample dataset and the positive and negative sample pairs in the reference image sample dataset, we construct the street view image intra-modality contrast loss and the reference image intra-modality contrast loss respectively, and combine the two intra-modality contrast losses into the inter-view contrast learning loss. The cross-modal contrastive learning loss function is constructed based on the cross-view contrastive learning loss and the inter-view contrastive learning loss and through the balancing factor.

6. The street view image geolocation method based on cross-modal contrastive learning and description enhancement according to claim 1, characterized in that: The cross-modal contrastive learning loss function is expressed as: in, They are the cross-view contrastive learning loss and the inter-view contrastive learning loss, respectively, and w is the balancing factor.

7. The method for geo-locating street view images based on cross-modal contrastive learning and description enhancement according to claim 1, characterized in that: Align the street view image to be queried with the reference image based on the query feature set and the gallery feature set, including: Based on the query feature set and using feature similarity, candidate libraries are retrieved from the library feature set, and the retrieval results are converted into a visual heat map; The feature similarity score is used to amplify the contrast between the candidate image library and the query feature set through an exponential function, and the query street view image is aligned with the reference image based on a visualized heat map.

8. A street view image geolocation system based on cross-modal contrastive learning and description enhancement, characterized in that: Contains: feature acquisition module and image alignment module, among which, A feature acquisition module is used to use a pre-trained image encoder to acquire visual features of the street view image to be queried and a feature set of a reference image library, use a language model to generate text description features of the street view image to be queried, and fuse the visual features and the text description features to generate a query feature set of the street view image to be queried, wherein the reference image is a remote sensing image with a geographical tag; An image alignment module, used for aligning the street view image to be queried with the reference image based on the query feature set and the gallery feature set, so as to obtain the geographic location of the street view image to be queried; Among them, during the training process, the image encoder is trained based on the street view image sample set and the reference image sample set and using a preset cross-modal contrastive learning loss function. The cross-modal contrastive learning loss function is composed of a cross-view contrastive learning loss between street view image samples and reference image samples and an inter-view contrastive learning loss between street view image sample pairs and between reference image sample pairs.

9. An electronic device, characterized in that: include: at least one processor, and a memory coupled to the at least one processor; The memory stores a computer program, and the computer program can be executed by the at least one processor to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 7 can be implemented.

Images