A Street View Image Geolocation Inference Method Based on a Large Visual Language Model
By using a visual-text pairing network to filter highly localizable street view images and constructing a large visual language model, the problem of filtering in existing geolocation methods is solved, achieving highly accurate geolocation and reasoning capabilities.
Patent Information
- Application Number
- CN202410501942.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-24
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2044-04-24
AI Technical Summary
Existing street view image geolocation methods rely on the diversity and comprehensiveness of geotagged image libraries, which are difficult to filter. Furthermore, classification-based methods ignore semantic information and lack the reasoning ability to interpret user interpretations.
A visual-text pairing network is used to filter highly localizable street view images, and a large visual language model is constructed. Through inference fine-tuning and location fine-tuning stages, geographic location inference is performed by combining visual and textual information.
It improves the accuracy of street view image geolocation, enhances the model's reasoning ability, and can provide descriptions of the location results and reasoning process.
Smart Images

Figure CN118551066B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of geolocation technology, and in particular to a street view image geolocation reasoning method based on a large visual language model. Background Technology
[0002] Street view image geolocation aims to predict the geographic location of a given street view image. The importance of street view geolocation is evident in a variety of applications, encompassing social research, urban planning, and navigation. Existing frameworks for street view geolocation can be broadly categorized into two types: retrieval-based methods and classification-based methods. Retrieval-based methods involve identifying the most similar images in a geotagged image library and then returning the corresponding geographic location. However, these methods rely on the diversity and comprehensiveness of the geotagged image library, which can lead to difficulties in filtering. On the other hand, classification-based methods divide the Earth's surface into different regions and assign the input image to a specific region. While these methods utilize shared visual features within a single region, they neglect semantic information (such as signage text) that is crucial for geolocation. More importantly, these classification methods typically operate as black-box models, lacking the ability to provide user-interpretable reasoning. Summary of the Invention
[0003] In order to achieve geolocation based on street view images through reasoning capabilities, this invention provides a street view image geolocation reasoning method based on a large visual language model.
[0004] This invention provides a street view image geolocation inference method based on a large visual language model, comprising:
[0005] Step 1: Use a visual-text pairing network to filter out highly localizable street view images from the original street view image set to form a highly localizable street view image dataset;
[0006] Step 2: Fine-tune the visual language model using the highly localizable street view image dataset to obtain the geolocation inference model;
[0007] Step 3: Construct a geolocation reasoning problem according to the given format, input the street view image to be tested and the geolocation reasoning problem into the geolocation reasoning model, and obtain the location result. The location result includes the geographical location of the street view image to be tested and a description of the reasoning process.
[0008] Furthermore, the visual-text pairing network employs a text-image multimodal network model;
[0009] Correspondingly, step 1 specifically includes:
[0010] The text encoder in the text-image multimodal network model is used to generate the text embedding for each descriptive text;
[0011] The image encoder in the text-image multimodal network model is used to generate a segmentation mask for each street view image, and the pixel ratio of each segmentation category is counted to form the image feature vector of the street view image.
[0012] The dot product of the text embedding and the image feature vector yields a result of size [value missing]. The matrix is denoted as the locality matrix; where m represents the size of a text embedding and n represents the number of segmentation categories.
[0013] Regarding the Dimensionality reduction of the matrix yields a The vector, denoted as the weight vector, contains elements that represent the localization weight parameters corresponding to each segmentation category.
[0014] The localizability value of the street view image is measured based on the segmentation mask and weight vector of each street view image;
[0015] Street view images with a locatability value greater than a set threshold and their corresponding descriptive text are combined into an image-text pair and added to the high locatability street view image dataset.
[0016] Furthermore, the localizability value of the street view image is measured based on the segmentation mask and weight vector of each street view image. Specifically, it includes:
[0017]
[0018] in, Segmentation mask representing a street view image The Middle The pixel ratio of each segmentation category Indicates the first The positioning weight parameters for each segmentation category.
[0019] Furthermore, MaskFormer is used as the image encoder in the text-image multimodal network model.
[0020] Furthermore, Sentence-BERT is used as the text encoder in the text-image multimodal network model.
[0021] Furthermore, the training process of the geolocation inference model includes an inference fine-tuning stage and a location fine-tuning stage;
[0022] During the inference fine-tuning phase, the input is a street view image and a first descriptive question, whereby the first descriptive question is a textual description of the geographical location of the street view image at a first positioning accuracy level; the output is the geographical location of the street view image at the first positioning accuracy level and a description of the inference process.
[0023] During the location fine-tuning stage, the input is a street view image and a second descriptive question, which is a textual description of the geographical location of the street view image at a first positioning accuracy level and a second positioning accuracy level, respectively; the output is the geographical location of the street view image at the first positioning accuracy level and the geographical location at the second positioning accuracy level, wherein the second positioning accuracy level is higher than the first positioning accuracy level.
[0024] Furthermore, both the inference fine-tuning stage and the position fine-tuning stage utilize the LoRA method to fine-tune the pre-trained large language model.
[0025] Furthermore, the large visual language model adopts the Qwen-VL model.
[0026] The beneficial effects of this invention are:
[0027] (1) This invention introduces a localization index to filter out street scene images with high localization from a large-scale set of original street scene images for training of a large visual language model, thus solving the challenge of lacking a high-quality street scene image training dataset.
[0028] (2) This invention utilizes the excellent ability of the visual language big model in processing multimodal visual and text input, and incorporates various external knowledge for the reasoning process. While improving the accuracy of positioning, it also promotes the reasoning ability in the geolocation process of street view images. Attached Figure Description
[0029] Figure 1 A flowchart illustrating a street view image geolocation reasoning method based on a large visual language model provided in an embodiment of the present invention;
[0030] Figure 2 A schematic diagram illustrating the generation of localizability metrics based on a visual-text pairing network model, as provided in an embodiment of the present invention.
[0031] Figure 3 The network framework for the geolocation inference model provided in this embodiment of the invention;
[0032] Figure 4 The input and output formats for the inference fine-tuning stage provided in the embodiments of the present invention. Detailed Implementation
[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0034] like Figure 1 As shown, this embodiment of the invention provides a street view image geolocation inference method based on a large visual language model, including:
[0035] S101: Use a visual-text pairing network to filter highly localizable street view images from the original street view image set to form a highly localizable street view image dataset;
[0036] Specifically, the localizability varies among different street view images. For example, images with text signs or prominent landmarks (such as the Eiffel Tower) are easily localized, while images taken in tunnels or obscured by walls have poor localizability. Simply merging all street view images into a large visual language model for training is not an optimal approach, as adding poor-quality data negatively impacts the training efficiency of updating the large visual language model. Therefore, this embodiment of the invention introduces a visual-text pairing network to generate the required localizability metrics for the input street view images, thereby selecting highly localizable street view images from a large-scale street view image set for training the large visual language model.
[0037] As one possible implementation, the visual-text pairing network in this embodiment of the invention employs a text-image multimodal network model. Correspondingly, the process of using the text-image multimodal network model to select highly localizable street view images is as follows:
[0038] S1011: Generate text embeddings for each descriptive text using a text encoder in a text-image multimodal network model;
[0039] S1012: Use the image encoder in the text-image multimodal network model to generate a segmentation mask for each street view image, and count the pixel ratio of each segmentation category to form the image feature vector of the street view image.
[0040] S1013: Perform a dot product between the text embedding and the image feature vector to obtain a vector of size... The matrix, denoted as the locationability matrix, is as follows: Figure 2 As shown; where m represents the size of a text embedding and n represents the number of segmentation categories;
[0041] S1014: Regarding the above Dimensionality reduction of a matrix (including steps such as normalization, thresholding, and mean calculation) yields a... The vector, denoted as the weight vector, contains elements that represent the localization weight parameters corresponding to each segmentation category.
[0042] Specifically, the more helpful the segmentation category (such as buildings, sky, vehicles, pedestrians, etc. in a street view image) is for localization, the higher the corresponding localization weight parameter will be.
[0043] S1015: Measure the localizability value of the street view image based on the segmentation mask and weight vector of each street view image. The specific calculation formula is as follows:
[0044]
[0045] in, Segmentation mask representing a street view image The Middle The pixel ratio of each segmentation category Indicates the first The positioning weight parameters for each segmentation category.
[0046] Furthermore, in the above embodiments, MaskFormer is used as the image encoder in the text-image multimodal network model, and Sentence-BERT is used as the text encoder in the text-image multimodal network model.
[0047] S1016: Add street view images with a locatability value greater than a set threshold and their corresponding descriptive text to the high locatability street view image dataset as an image-text pair.
[0048] Specifically, a higher locability value indicates more visual cues for geolocation in the street view image, and vice versa. Based on experience, this embodiment selects 0.4 as the threshold for filtering locable street view images. Ultimately, over 70,000 highly locable street view images with geotags were passed to the next stage of training the large visual language model.
[0049] S102: Fine-tune the large visual language model using the highly localizable street view image dataset to obtain the geolocation inference model GeoReasoner;
[0050] Specifically, this embodiment selects the Qwen-VL model, a large-scale visual language model, as the model to be fine-tuned. The structure of the large-scale visual language model is as follows: Figure 3As shown, the system mainly includes a visual encoder, a projection layer, and a pre-trained LLM (Large Language Model). The visual encoder adjusts the input street view image to a specific resolution and then divides it into a set of image patches. In this embodiment, the visual encoder module adopts the VisionTransformer (ViT) architecture. The projection layer adjusts the image patches into an LLM-compatible sequential representation. In this embodiment, a cross-attention module is used to compress the visual feature sequence to a fixed length of 256, the same as the baseline, addressing the efficiency challenge posed by a large number of visual feature sequences. Subsequently, the compressed visual feature sequence and the text sequence are passed to the pre-trained LLM module, which acts as a decoder to generate the answer.
[0051] In this embodiment of the invention, the training process of the geolocation inference model includes an inference fine-tuning stage and a location fine-tuning stage.
[0052] During the inference fine-tuning phase, the input is a street view image and a first descriptive question, whereby the first descriptive question is a textual description of the geographical location of the street view image at a first positioning accuracy level; the output is the geographical location of the street view image at the first positioning accuracy level and a description of the inference process.
[0053] Specifically, the goal of this stage is to enhance the model's reasoning ability by utilizing textual cues paired with street view images collected from geolocation games. Due to the granularity displayed in the image-text data pairs, this embodiment of the invention currently uses the country level as the first positioning accuracy level and the city level as the second positioning accuracy level.
[0054] As an example, the input street view image and the first descriptive question, along with the model's output answer, can be formatted as follows: Figure 4 The prompting method is shown below.
[0055] During the location fine-tuning stage, the inputs are a street view image and a second descriptive question, which is a textual description of the geographical location of the street view image at a first positioning accuracy level and a second positioning accuracy level, respectively; the output is the geographical location of the street view image at the first positioning accuracy level and the geographical location at the second positioning accuracy level.
[0056] Specifically, this phase aims to combine prior knowledge of geographic locations (such as country information) at the first positioning accuracy level with highly relocatable street view images and geotags to infer finer-grained location information at the second positioning accuracy level (such as city level). This phase uses a similar prompt format to the first phase, except that it does not describe the reasoning process.
[0057] In this embodiment of the invention, the LoRA method is used to fine-tune the pre-trained LLM of Qwen-VL in both stages. This helps to improve the overall performance of Qwen-VL in the inference and location tuning stages, enabling the model to better capture the complex relationships in image-text data pairs.
[0058] S103: Construct a geolocation reasoning problem according to a given format, input the street view image to be tested and the geolocation reasoning problem into the geolocation reasoning model, and obtain the location result. The location result includes the geographical location of the street view image to be tested and a description of the reasoning process.
[0059] To verify the effectiveness of the method of the present invention, the present invention also provides the following experimental data.
[0060] During the experimental phase, the inventors provided these models with the same input street view images, inference processes, and result formats. Specifically, consistent prompts were used, as follows:
[0061] According to the content of the image, please think step by step anddeduce in which country and city the image is most-likely located and givethe most important reason. Output in JSON format, eg, {'country': '', 'city': '', 'reason':''}.
[0062] Table 1 shows a comparison of the prediction results of GeoReasoner with other methods. Overall, GeoReasoner outperforms other models, especially those based on LVLM. Taking the best-performing Qwen VL as an example, in terms of F1 score, GeoReasoner outperforms it by 25.02% in national-level geolocation and by 38.61% in city-level geolocation. GPT-4V has significantly lower recall performance in geolocation tasks.
[0063] Table 1 Comparison of prediction results between GeoReasoner and other methods
[0064]
[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A street view image geolocation reasoning method based on a large visual language model, characterized in that, include: Step 1: Use a visual-text pairing network to filter highly localizable street view images from the original street view image set to form a highly localizable street view image dataset; the visual-text pairing network adopts a text-image multimodal network model; correspondingly, Step 1 specifically includes: The text encoder in the text-image multimodal network model is used to generate the text embedding for each descriptive text; The image encoder in the text-image multimodal network model is used to generate a segmentation mask for each street view image, and the pixel ratio of each segmentation category is counted to form the image feature vector of the street view image. The dot product of the text embedding and the image feature vector yields a result of size [value missing]. The matrix is denoted as the locality matrix; where m represents the size of a text embedding and n represents the number of segmentation categories. Regarding the Dimensionality reduction of the matrix yields a The vector, denoted as the weight vector, contains elements that represent the localization weight parameters corresponding to each segmentation category. The localizability value of the street view image is measured based on the segmentation mask and weight vector of each street view image; Street view images with a locatability value greater than a set threshold and their corresponding descriptive text are combined into an image-text pair and added to the high locatability street view image dataset; Step 2: Fine-tune the visual language model using the highly localizable street view image dataset to obtain the geolocation inference model; Step 3: Construct a geolocation reasoning problem according to the given format, input the street view image to be tested and the geolocation reasoning problem into the geolocation reasoning model, and obtain the location result. The location result includes the geographical location of the street view image to be tested and a description of the reasoning process.
2. The street view image geolocation reasoning method based on a large visual language model according to claim 1, characterized in that, The localizability value of each street view image is measured based on its segmentation mask and weight vector. Specifically, it includes: in, Segmentation mask representing a street view image The Middle The pixel ratio of each segmentation category Indicates the first The positioning weight parameters for each segmentation category.
3. The street view image geolocation reasoning method based on a large visual language model according to claim 1, characterized in that, MaskFormer is used as the image encoder in the text-image multimodal network model.
4. A street view image geolocation reasoning method based on a large visual language model according to claim 1 or 3, characterized in that, Sentence-BERT is used as the text encoder in the text-image multimodal network model.
5. The street view image geolocation reasoning method based on a large visual language model according to claim 1, characterized in that, The training process of the geolocation inference model includes an inference fine-tuning stage and a location fine-tuning stage. During the inference fine-tuning phase, the input is a street view image and a first descriptive question, whereby the first descriptive question is a textual description of the geographical location of the street view image at a first positioning accuracy level; the output is the geographical location of the street view image at the first positioning accuracy level and a description of the inference process. During the location fine-tuning stage, the input is a street view image and a second descriptive question, which is a textual description of the geographical location of the street view image at a first positioning accuracy level and a second positioning accuracy level, respectively; the output is the geographical location of the street view image at the first positioning accuracy level and the geographical location at the second positioning accuracy level, wherein the second positioning accuracy level is higher than the first positioning accuracy level.
6. The street view image geolocation reasoning method based on a large visual language model according to claim 5, characterized in that, Both the inference fine-tuning stage and the position fine-tuning stage utilize the LoRA method to fine-tune the pre-trained large language model.
7. A street view image geolocation reasoning method based on a large visual language model according to claim 1, 5, or 6, characterized in that, The large-scale visual language model adopts the Qwen-VL model.
Citation Information
Patent Citations
Geographic positioning method and system for street view pictures
CN110472092A
Night streetscape semantic segmentation method based on difficult category perception mechanism
CN116883667A
