Image geographic positioning method and system based on geographic feature reinforcement learning
By designing a reasonable reward function and fine-tuning the visual language model with a fine-grained geolocation dataset, the problem of inconsistent image geolocation in existing technologies is solved, achieving higher accuracy and consistency in geolocation results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANKAI UNIV
- Filing Date
- 2026-01-19
- Publication Date
- 2026-04-24
AI Technical Summary
Existing image geolocation methods rely on thought chain data generated by generative large language models, which leads to inconsistencies with human reasoning. Furthermore, the reward function is poorly designed and ignores the model's efforts to approach the correct answer, resulting in insufficient positioning accuracy and consistency.
Design a reasonable reward function, including spatial similarity reward, semantic similarity reward and consistency reward, and improve the localization ability of the visual language model by constructing a fine-grained geolocation dataset and supervising its fine-tuning.
It improves the image geolocation capability of the visual language model at a finer granular level, ensuring the consistency and accuracy of the inference process generated by the model, and can accurately infer geographical locations from images.
Smart Images
Figure CN121920532A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision technology, specifically relating to an image geolocation method and system based on geographic feature reinforcement learning. Background Technology
[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.
[0003] Image geolocation is a crucial task in computer vision, aiming to infer geographical location from the visual content of an image. Early geolocation methods typically divided the Earth into grid cells, achieving location through classification or retrieval. In recent years, with the development of large-scale language models and visual language models, some methods have begun to employ visual language models to handle geolocation tasks, aiming to achieve better performance in open environments. In particular, with the emergence of reinforcement learning-based methods, some approaches have achieved breakthroughs in performance and interpretability by introducing chained reasoning data and processes to construct reasonable and reliable thought chains for geolocation.
[0004] However, existing methods primarily rely on thought chain data generated by generative large language models to train the reasoning process of visual language models. This may be inconsistent with genuine human reasoning and could amplify the inherent biases of visual language models. Traditional geolocation datasets often only provide GPS coordinates or lack sufficiently fine-grained labels, which differ significantly from the natural language descriptions humans typically use to express geographic locations. Furthermore, inferring precise locations from limited visual cues requires human-like reasoning, and thought chain data generated purely by generative large language models may deviate from actual thought processes. Previous research has also indicated that reinforcement learning-based visual language models often learn superficial patterns rather than genuine reasoning abilities.
[0005] Meanwhile, most existing reward functions designed for reinforcement learning methods directly judge whether the model output is equal to the true label. However, since different descriptions in geographic tasks may point to the same location attribute, this may generate unreasonable reward signals. In addition, such reward function design will ignore the model's efforts to move closer to the correct answer without outputting a completely correct answer. Summary of the Invention
[0006] To address the aforementioned problems, this invention proposes an image geolocation method and system based on geographic feature reinforcement learning. By designing a reasonable reward function, this invention can achieve fine-grained output of geographic location.
[0007] According to some embodiments, the present invention adopts the following technical solution: An image geolocation method based on geographic feature reinforcement learning includes the following steps: A geolocation dataset is constructed, which includes images, location coordinates, three-level location labels, and inference process data. The geolocation dataset is then used to perform supervised fine-tuning of the visual language model. A geographic similarity reward function, which includes spatial similarity rewards and semantic similarity rewards, is constructed. The fine-tuned model is then subjected to reinforcement learning so that it can accurately infer geographic location information from images. Construct a consistency reward function to evaluate the integrity and consistency of the fine-tuned model's reasoning process, ensuring that the reasoning process generated by the model supports its conclusions; The final model is used to process the target image to obtain the image geolocation result.
[0008] As an alternative implementation method, the process of constructing a geolocation dataset includes acquiring manually labeled data in structured language that describes the reasoning process at three levels: country-region-specific location, for supervised fine-tuning. Data is sampled for reinforcement learning; The remaining data is extracted for model evaluation.
[0009] As a further defined implementation method, the process of sampling data for reinforcement learning includes: for each country Comprehensive population Land area Highway mileage ,according to: Calculate the initial sampling weights, where Within each country, establish grids of equal size according to a set specification. For grid population Logarithmic correction is used: ; in, For the country's average population density, finally all grids are... After normalization, samples are drawn without replacement according to probability proportions until the total sample size reaches the set number, forming a dataset for reinforcement learning.
[0010] As an alternative implementation, the geographical similarity reward function is: Among them, spatial similarity It is a function related to the spatial distance difference between the predicted and true label locations, encouraging the model to align with the correct geographical location; semantic similarity It is a function related to the predicted and true structured address semantic relationships, encouraging the model to predict structured information while mitigating the problem of correct answers being misjudged due to aliases and translation differences. , These are the weighting coefficients.
[0011] As a further defined implementation method, spatial similarity The calculation process includes: The natural language address output by the model is inverse encoded into predicted coordinates. Then with the truth value Calculate distance: Where r is the Earth's radius, then spatial similarity is calculated: ； in The temperature parameter is such that the smaller the distance, the closer the reward is to 1, and the gradient slows down as the distance decreases.
[0012] As a further defined implementation method, semantic similarity The calculation process includes: encoding the country / region / precise address into three levels respectively. and Calculate the cosine similarity: ; Then, threshold truncation and weighting are performed: ; These are the weighting coefficients. This is a threshold truncation function, meaning that when a high-level error is achieved, low-level errors are not scored. This reward function allows the model to still obtain a positive signal when the text descriptions are not completely consistent.
[0013] As an alternative implementation method, the process of constructing the consistency reward function includes: the consistency reward function is a hierarchical weighted hit function, which calculates whether a hit occurs at three levels: country, region, and precise address, and then weights them according to their importance. ; in This is an indicator function; it returns 1 if the condition is met, and 0 otherwise. For hierarchical weights; This is a length penalty factor to prevent the model from deceiving the consistency evaluation model with extremely short descriptions.
[0014] As a further defined implementation method, the length penalty factor It is an S-shaped function whose growth rate is slow at first, then fast, and then slows down again as the length of the reasoning process increases. First, let's consider the first... Level of reasoning length Normalization to ,Right now ; Further changes will be made: ; in Control the slope, As the central location, Too short time , lower After the length is appropriate The reward will no longer increase with the number of words. This is the minimum inference length. This represents the maximum inference length.
[0015] An image geolocation system based on geographic feature reinforcement learning includes: The dataset building module is configured to build a geolocation dataset, which includes images, location coordinates, three-level location labels, and inference process data. The geolocation dataset is used to perform supervised fine-tuning of the visual language model. The geographic similarity reward function construction module is configured to construct a geographic similarity reward function that includes spatial similarity rewards and semantic similarity rewards, and to perform reinforcement learning on the fine-tuned model so that it can accurately infer geographic location information from images; The consistency reward function construction module is configured to construct a consistency reward function to evaluate the integrity and consistency of the fine-tuned model inference process, ensuring that the inference process generated by the model supports its conclusions. The image geolocation module is configured to process the target image using the final model to obtain the image geolocation result.
[0016] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the steps in the above method.
[0017] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the steps in the method described above.
[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention improves the image geolocation capability of visual language models at a finer-grained level and designs a reasonable reward function based on geographic attributes. Furthermore, the solution proposed in this invention is a fine-tuning of existing general-purpose visual language models. The image geolocation training strategy proposed in this invention can be embedded into many existing visual language models to enhance their image geolocation capabilities.
[0019] The image geolocation method provided by this invention can be used in the training of various visual language models for this task to improve the image geolocation capabilities of visual language models. Visual language models embedded with this geolocation method can serve as support models for visual tasks including, but not limited to, image geolocation, scene understanding, and spatial reasoning.
[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description
[0021] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.
[0022] Figure 1 This is a schematic diagram illustrating the construction of a fine-grained geolocation dataset incorporating human thought processes, as per the present invention.
[0023] Figure 2 This is a schematic diagram illustrating the working principle of the reward function in this invention, where (a) is the geographical similarity reward function and (b) is the consistency reward function.
[0024] Figure 3 This is a schematic diagram illustrating the effect of image geolocation in the application of this invention. Detailed Implementation
[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0026] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0027] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.
[0028] Where there is no conflict, the embodiments and features described in this application may be combined with each other.
[0029] Example 1 Image geolocation methods based on geographic feature reinforcement learning include: a. Construct a geolocation dataset that includes expert annotations and human reasoning processes. The dataset contains images, GPS coordinates, three-level location labels, and reasoning processes. The dataset was constructed by collaborating with geography experts and geolocation game players to collect their reasoning processes, and then standardizing and formatting their language and structure to create a unified thought chain template. This dataset was then used to perform supervised fine-tuning of a general visual language model.
[0030] b. Design a geographic similarity reward function and train the model through reinforcement learning so that it can accurately infer geographic location information from images; The reward function includes spatial similarity reward and semantic similarity reward. Spatial similarity reward is used to measure the physical distance between the predicted location and the actual location, while semantic similarity reward is used to measure the similarity between the predicted location and the actual location in text description.
[0031] c. Design a consistency reward function to evaluate the completeness and consistency of the model's reasoning process through a consistency evaluation model, ensuring that the reasoning process generated by the model supports its conclusions.
[0032] The consistency reward evaluates the completeness and consistency of the model's reasoning process through a consistency evaluation model, ensuring that the reasoning process generated by the model supports its conclusions. The consistency evaluation model only obtains the model's reasoning process and not its conclusions.
[0033] The following is a detailed introduction. First, step a, the process of constructing a geolocation dataset that includes expert annotations and human reasoning, includes: A team of geography experts and top players in image geolocation games was publicly invited through a volunteer platform to form a labeling team. Volunteers described their reasoning process using structured language, categorized into three levels: country, region, and specific location. This data was then used for supervised fine-tuning.
[0034] Design a sampling algorithm to sample the data, reduce geographical bias, and improve sample diversity. First, for each country... Comprehensive population Land area Highway mileage ,according to
[0035] Calculate the initial sampling weights, where This formula ensures that densely populated and well-connected areas receive higher prior probabilities, avoiding undersampling in sparse areas caused by simple equal-area sampling. Then, within the country, according to... Create a uniform mesh .
[0036] To curb excessive concentration of population in urban centers, grid-based population management is implemented. Logarithmic correction is used:
[0037] in, This represents the average population density of the country. This step ensures that sampling within the same country is biased towards more populated areas without excessively creating class imbalance, further reducing bias. Finally, all grids are... After normalization, samples are drawn without replacement according to probability proportions until the total sample size reaches 20,000 (of course, the specific value can be modified according to needs in other embodiments) to form a dataset for reinforcement learning.
[0038] The above process extracts 3000 images from the remaining OSV-5M data (of course, the specific number can be modified according to requirements in other embodiments), and uses GPT-4o to label the locatability to form a validation set to evaluate the model's capabilities.
[0039] like Figure 2 As shown, this embodiment designs a geographical similarity reward function and a consistency reward function based on the characteristics of geographical tasks for training in the reinforcement learning phase.
[0040] like Figure 2 As shown in (a), the design process of the geographic similarity reward function is as follows: The overall reward structure, the geographic similarity reward function, consists of a linear weighted average of spatial similarity and semantic similarity: ; , These are the weighting coefficients.
[0041] Spatial similarity It is a function related to the spatial distance difference between the predicted and ground truth labels, encouraging the model to gravitate towards the correct geographic location. First, OpenCage is used to inverse-encode the natural language address output by the model into predicted coordinates. Then with the truth value Calculate the great circle distance:
[0042] Next, spatial similarity is calculated:
[0043] in The temperature parameter is used. The smaller the distance, the closer the reward is to 1, and the gradient slows down as the distance decreases, which is consistent with the human habit of coarse-to-fine localization.
[0044] semantic similarity This is a function related to the predicted and true structured address semantic relationships, encouraging the model to predict structured information while mitigating the problem of "correct answers being judged incorrectly" caused by aliases and translation differences. In this embodiment, multilingual Sentence-BERT can be used to encode the three levels of "country / region / precise address" as follows: and Calculate the cosine similarity:
[0045] Then, threshold truncation and weighting are performed:
[0046] When a high-level error occurs, a low-level error is not scored. This reward function allows the model to still receive a positive signal when the text descriptions are not entirely consistent.
[0047] like Figure 2 As shown in (b), the design process of the consistency reward function is as follows: The basic idea of the consistency reward function is to only input the thought chain generated by the model into the consistency evaluation model, without letting it see the final answer. The model is required to reproduce the correct conclusion based solely on the reasoning process. If the reproduction is successful, it means that the reasoning itself is self-sufficient and has no logical breaks.
[0048] The consistency reward function is a hierarchical weighted hit function that calculates hit rate separately for three levels: country, region, and precise address, and then weights them according to their importance.
[0049] in This is an indicator function; it returns 1 if the condition is met, and 0 otherwise. For hierarchical weights; This is a length penalty factor to prevent the model from deceiving the consistency evaluation model with extremely short descriptions.
[0050] In this embodiment, the length penalty factor It is an S-shaped function whose growth rate is slow at first, then fast, and then slows down again as the length of the reasoning process increases. First, let's consider the first... Level of reasoning length Normalization to :
[0051] Further changes will be made:
[0052] in Control the slope, It is located at the center. Too short time , lower After the length is appropriate The reward no longer increases with the number of words, and this formula forces the model to provide a sufficiently detailed and self-consistent thought process.
[0053] To demonstrate the effectiveness of the method provided in this embodiment (hereinafter referred to as "this method"), it is compared with four benchmark methods—Qwen2.5-VL-7B, Qwen2.5-VL-32B, Gemma3-27B, and InternVL3-14B—on the evaluation dataset proposed in this paper. The results are shown in Table 1, demonstrating that this method significantly outperforms the baseline models in the image geolocation task.
[0054] Table 1
[0055] like Figure 3 As shown, in image geolocation tasks, this method not only accurately predicts the geographical location but also outputs a detailed and consistent thought process while capturing as many geographical features as possible. Other models not only have low accuracy but also their thought processes are prone to errors or inconsistencies. Example 2 An image geolocation system based on geographic feature reinforcement learning includes: The dataset building module is configured to build a geolocation dataset, which includes images, location coordinates, three-level location labels, and inference process data. The geolocation dataset is used to perform supervised fine-tuning of the visual language model. The geographic similarity reward function construction module is configured to construct a geographic similarity reward function that includes spatial similarity rewards and semantic similarity rewards, and to perform reinforcement learning on the fine-tuned model so that it can accurately infer geographic location information from images; The consistency reward function construction module is configured to construct a consistency reward function to evaluate the integrity and consistency of the fine-tuned model inference process, ensuring that the inference process generated by the model supports its conclusions. The image geolocation module is configured to process the target image using the final model to obtain the image geolocation result.
[0056] The steps for each module can be referred to the method provided in Implementation Example 1.
[0057] Example 3 A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the steps in the method provided in Embodiment 1.
[0058] Example 4 An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the steps in the method provided in Embodiment 1.
[0059] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of one or more computer-usable storage media (including, but not limited to, disk storage, etc.) containing computer-usable program code. CD - ROM It takes the form of a computer program product implemented on (such as optical memory, etc.).
[0060] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0061] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0062] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0063] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made by those skilled in the art without creative effort within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. An image geolocation method based on reinforcement learning of geographic features, characterized in that, Includes the following steps: A geolocation dataset is constructed, which includes images, location coordinates, three-level location labels, and inference process data. The visual language model is then supervised and fine-tuned using the geolocation dataset. A geographic similarity reward function, which includes spatial similarity rewards and semantic similarity rewards, is constructed. The fine-tuned model is then subjected to reinforcement learning so that it can accurately infer geographic location information from images. Construct a consistency reward function to evaluate the integrity and consistency of the fine-tuned model's reasoning process, ensuring that the reasoning process generated by the model supports its conclusions; The final model is used to process the target image to obtain the image geolocation result.
2. The image geolocation method based on geographic feature reinforcement learning as described in claim 1, characterized in that, The process of constructing a geolocation dataset includes acquiring manually labeled data that describes the reasoning process in structured language at three levels: country, region, and specific location, for supervised fine-tuning. Data is sampled for reinforcement learning; The remaining data is extracted for model evaluation.
3. The image geolocation method based on geographic feature reinforcement learning as described in claim 2, characterized in that, The process of sampling data for reinforcement learning includes: for each country Comprehensive population Land area Highway mileage ,according to: Calculate the initial sampling weights, where Within each country, establish grids of equal size according to a set specification. For grid population Logarithmic correction is used: ; in, For the country's average population density, finally all grids are... After normalization, samples are drawn without replacement according to probability proportions until the total sample size reaches the set number, forming a dataset for reinforcement learning.
4. The image geolocation method based on geographic feature reinforcement learning as described in claim 1, characterized in that, The geographic similarity reward function is: Among them, spatial similarity It is a function related to the spatial distance difference between the predicted and true label locations, encouraging the model to align with the correct geographical location; semantic similarity It is a function related to the predicted and true structured address semantic relationships, encouraging the model to predict structured information while mitigating the problem of correct answers being misjudged due to aliases and translation differences. , These are the weighting coefficients.
5. The image geolocation method based on geographic feature reinforcement learning as described in claim 4, characterized in that, spatial... Similarity The calculation process includes: The natural language address output by the model is inverse encoded into predicted coordinates. Then with the truth value Calculate distance: Where r is the Earth's radius, then spatial similarity is calculated: in The temperature parameter is such that the smaller the distance, the closer the reward is to 1, and the gradient slows down as the distance decreases.
6. The image geolocation method based on geographic feature reinforcement learning as described in claim 4, characterized in that, semantic similarity The calculation process includes: encoding the country / region / precise address into three levels respectively. and Calculate the cosine similarity: ; Then, threshold truncation and weighting are performed: ； These are the weighting coefficients. This is a threshold truncation function, meaning that when a high-level error is achieved, low-level errors are not scored. This reward function allows the model to still obtain a positive signal when the text descriptions are not completely consistent.
7. The image geolocation method based on geographic feature reinforcement learning as described in claim 1, characterized in that, The process of constructing the consistency reward function includes: the consistency reward function is a hierarchical weighted hit function, which calculates the hit status separately for three levels: country, region, and precise address, and then weights them according to their importance. ； in This is an indicator function; it returns 1 if the condition is met, and 0 otherwise. For hierarchical weights; The length penalty factor is used to prevent the model from deceiving the consistency evaluation model with extremely short descriptions. It is an S-shaped function whose growth rate is slow at first, then fast, and then slows down again as the length of the reasoning process increases. First, let's consider the first... Level of reasoning length Normalization to ,Right now ; Further changes will be made: ; in Control the slope, As the central location, Too short time , lower After the length is appropriate The reward will no longer increase with the number of words. To be the minimum inference length, This represents the maximum inference length.
8. An image geolocation system based on reinforcement learning of geographic features, characterized in that, include: The dataset building module is configured to build a geolocation dataset, which includes images, location coordinates, three-level location labels, and inference process data. The geolocation dataset is used to perform supervised fine-tuning of the visual language model. The geographic similarity reward function construction module is configured to construct a geographic similarity reward function that includes spatial similarity rewards and semantic similarity rewards, and to perform reinforcement learning on the fine-tuned model so that it can accurately infer geographic location information from images; The consistency reward function construction module is configured to construct a consistency reward function to evaluate the integrity and consistency of the fine-tuned model inference process, ensuring that the inference process generated by the model supports its conclusions. The image geolocation module is configured to process the target image using the final model to obtain the image geolocation result.
9. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, complete the steps of the method according to any one of claims 1-7.
10. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the steps of the method according to any one of claims 1-7.