Remote sensing multi-modal reasoning method and system based on geographic space thinking chain

Abstract

The invention relates to the technical field of remote sensing vision, and particularly discloses a remote sensing multi-modal reasoning method and system based on a geospatial thinking chain, and the method comprises the steps: constructing a geospatial thinking chain data set which comprises structured reasoning data organized according to a planning-positioning-synthesizing three-section cognitive architecture, the association module is used for establishing verifiable association between visual evidence in a remote sensing image and a text conclusion; constructing a remote sensing visual language basic model, wherein the model comprises a visual encoder, a language decoder and a visual projection layer connecting the visual encoder and the language decoder; according to the method, remote sensing analysis is modeled into a verifiable multi-step reasoning process, so that the model provides a verifiable analysis track while outputting a final answer, the problem that output cannot be verified due to traditional end-to-end training is solved, the performance of a complex analysis task is remarkably improved, and meanwhile, the accuracy of the result is improved. And the conversion from opaque perception to structured and verifiable reasoning is realized.

Description

Technical Field

[0001] This invention relates to the field of remote sensing vision technology, specifically a remote sensing multimodal reasoning method and system based on geospatial thinking chains. Background Technology

[0002] Remote sensing image interpretation, a crucial task in the field of Earth observation, plays an irreplaceable role in high-risk, high-value scenarios such as disaster response, environmental monitoring, and urban planning. In recent years, with the rapid development of deep learning technology, Visual Language Models (VLMs) have made significant progress in remote sensing, demonstrating enormous potential for handling complex analytical tasks such as visual question answering, image description, and fine-grained target counting. However, existing remote sensing visual language models still face insurmountable structural bottlenecks in practical applications, severely limiting their ability to be implemented in critical business scenarios.

[0003] Currently, most mainstream remote sensing visual language models employ an end-to-end implicit mapping training paradigm, directly learning the mapping relationship from input pixels to text output. While this mechanism simplifies model design to some extent, it compresses the complex reasoning process into a single, invisible transformation step, resulting in a significant "black box" characteristic and a lack of process transparency. More seriously, this generation method, lacking intermediate logical constraints, is highly susceptible to generating "illusions"—seemingly reasonable descriptions that are completely erroneous in factual terms. For example, in critical applications such as earthquake relief or flood monitoring, the model might fabricate non-existent buildings or disaster areas. Such unverifiable output poses a significant safety hazard, making users unable to trust the model's decision-making recommendations.

[0004] Secondly, while multimodal thought chain (CoT) techniques have proven effective in enhancing reasoning capabilities in general computer vision, their success often relies on the presence of salient, holistic targets within the image. In contrast, remote sensing images are characterized by vast scenes, strong non-uniformity, and significant differences in target scale and dense distribution. Remote sensing analysis tasks often require systematic search strategies and sophisticated texture analysis, rather than simple holistic recognition. Existing general thought chain methods struggle to address the shift from passive recognition to target-oriented active perception required in remote sensing images, and cannot perform effective reasoning planning in complex geospatial environments, resulting in poor performance in tasks requiring multi-step spatial reasoning, such as "tracking the number of bridges along a river." Recent research has attempted to introduce reasoning mechanisms into remote sensing, but these methods suffer from a fundamental flaw: a lack of perceptual alignment. Existing technologies typically treat reasoning as a semantic interpretation process based on model-parameterized world knowledge, rather than an analytical interpretation process based on visual evidence. Even when some models can output inference text, they are limited to abstract descriptions, lacking direct and verifiable connections to specific pixel regions in the image (such as bounding boxes and coordinates). Due to the lack of explicit spatial reference, the reasoning steps generated by the model are often disconnected from visual perception, making it impossible to prove that its conclusions are based on real objects in the image, thus failing to eliminate the illusion phenomenon. Therefore, existing technologies urgently need an innovative method that can break the end-to-end black box mechanism, adapt to complex remote sensing scenarios, and achieve strict alignment between reasoning steps and visual evidence. Summary of the Invention

[0005] The purpose of this invention is to provide a remote sensing multimodal reasoning method and system based on geospatial thinking chain to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A remote sensing multimodal reasoning method based on geospatial thinking chain, the method comprising: A geospatial thinking chain dataset is constructed, which contains structured reasoning data organized according to a three-stage cognitive framework of "planning-location-integration" to establish verifiable associations between visual evidence in remote sensing images and textual conclusions. A basic model for remote sensing visual language is constructed, which includes a visual encoder, a language decoder, and a visual projection layer connecting the two. The model is trained using a two-stage alignment strategy, including: In the first stage, the model is fine-tuned under supervision using the geospatial thinking chain dataset to inject geospatial cognitive structure and reasoning process. In the second stage, a group relative strategy optimization algorithm is adopted, and the inference results of the model are optimized by reinforcement learning based on a task-oriented reward mechanism.

[0007] As a further embodiment of the present invention, the visual encoder is based on the Vision Transformer architecture and integrates a dynamic position encoding mechanism, wherein the dynamic position encoding is implemented in the following way: The input image is divided into multiple image blocks, and the coordinates of each image block are normalized to obtain continuous grid coordinates. The positional encoding that fits the current input size is generated by sampling from the pre-trained position embedding table using an interpolation method.

[0008] As a further embodiment of the present invention, the calculation formula for the normalization process is as follows: ; in, g norm These are normalized continuous grid coordinates; w , h () represents the original coordinates of the image patch; W p , H p This defines the width and height of the patch grid.

[0009] As a further embodiment of the present invention, the interpolation method is bicubic interpolation, and the adapted position code calculation formula is as follows: ; in, It is a bicubic interpolation function. P orig For the pre-trained position embedding table, P adapted ( g ) is the adapted position code.

[0010] As a further aspect of the present invention, the construction of the geospatial thinking chain dataset includes: Input the remote sensing image and its corresponding ground truth annotation; Generate reasoning text that conforms to the cognitive structure of "planning-positioning-integration" through a multimodal large model; The planning phase involves problem decomposition and search strategy generation, the positioning phase outputs spatial coordinate evidence of the target, and the synthesis phase integrates visual evidence logically and outputs the final conclusion.

[0011] As a further aspect of the present invention, the loss function used in the first stage of supervised fine-tuning is autoregressive log-likelihood loss: ; in, For sequence length, oi The target output sequence contains the steps of a complete thought chain. o i,t Let t be the t-th token in the sequence, and θ be the model parameters.

[0012] As a further embodiment of the present invention, the second-stage group relative strategy optimization includes the following steps: Multiple candidate output sequences are generated by sampling the same input; The reward value for each sequence is calculated based on a task-specific reward function; The relative advantage of each sequence is calculated based on the reward value, which is used for policy updates.

[0013] As a further embodiment of the present invention, the task-specific reward function includes at least one of the following: For visual localization and detection tasks, the reward is calculated based on the intersection-union ratio (IUU) of the predicted bounding box and the ground truth bounding box. For the target counting task, the reward is calculated based on the error between the predicted number and the actual number; For visual question answering and classification tasks, rewards are calculated based on the degree of answer matching.

[0014] This invention also provides a remote sensing multimodal reasoning system based on geospatial thinking chains, the system comprising: The dataset construction module is used to construct a geospatial thinking chain dataset, which contains structured reasoning data organized according to a three-stage cognitive framework of "planning-location-integration" to establish verifiable associations between visual evidence in remote sensing images and textual conclusions. The model building module is used to build a basic model of remote sensing visual language, which includes a visual encoder, a language decoder, and a visual projection layer connecting the two. The training module, employing a two-stage alignment strategy, trains the model, including: In the first stage, the model is fine-tuned under supervision using the geospatial thinking chain dataset to inject geospatial cognitive structure and reasoning process. In the second stage, a group relative strategy optimization algorithm is adopted, and the inference results of the model are optimized by reinforcement learning based on a task-oriented reward mechanism.

[0015] Compared with the prior art, the beneficial effects of the present invention are: This invention employs a three-stage explicit reasoning framework of planning, localization, and synthesis as the core of the model's thinking. The planning stage decomposes complex queries using natural language to formulate a systematic search strategy; the localization stage forces the model to identify specific targets in images and output spatial coordinate evidence, establishing a direct relationship between text and pixels; the synthesis stage logically derives the final conclusion based on visual evidence. This design breaks away from the black-box mechanism of traditional end-to-end models, enhancing the ability to handle complex problems through explicit task decomposition and providing high transparency and verifiability to the reasoning process through mandatory visual localization, thus solving the problem of the disconnect between the reasoning process and visual perception.

[0016] This invention employs a two-stage alignment strategy combining cognitive alignment and reasoning optimization as its core training paradigm. The first stage utilizes structured thought chain data for supervised fine-tuning, establishing the model's basic reasoning format and cognitive framework. The second stage introduces a group relative policy optimization algorithm, directly optimizing the factual correctness of the reasoning results (such as IoU accuracy and counting error) through reinforcement learning. This strategy combines the learning process of reasoning form and factual content, not only allowing the model to acquire analytical methodologies similar to human experts, but also significantly suppressing the generation of multimodal illusions through a result-oriented reward mechanism, thereby improving the model's applicability and credibility in open-world scenarios. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention.

[0018] Figure 1 This is an architecture diagram of a remote sensing multimodal reasoning method based on geospatial thinking chains.

[0019] Figure 2 This is a schematic diagram of the two-stage training of a remote sensing multimodal reasoning method based on geospatial thinking chains.

[0020] Figure 3 This is a schematic diagram of multi-target detection reasoning based on geospatial thinking chain in the embodiment.

[0021] Figure 4 This is a schematic diagram of spatial orientation reference reasoning based on geospatial thinking chain in the embodiment.

[0022] Figure 5 This is a schematic diagram of specific shape target referential reasoning based on geospatial thinking chain in the embodiment.

[0023] Figure 6 This is a schematic diagram of implicit semantic understanding reasoning based on geospatial thought chain in the embodiment. Detailed Implementation

[0024] To make the technical problems, solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0025] In this embodiment of the invention, a remote sensing multimodal reasoning method based on geospatial thinking chain is provided, the method comprising: A geospatial thinking chain dataset is constructed, which contains structured reasoning data organized according to a three-stage cognitive framework of "planning-location-integration" to establish verifiable associations between visual evidence in remote sensing images and textual conclusions. A basic model for remote sensing visual language is constructed, which includes a visual encoder, a language decoder, and a visual projection layer connecting the two. The model is trained using a two-stage alignment strategy, including: In the first stage, the model is fine-tuned under supervision using the geospatial thinking chain dataset to inject geospatial cognitive structure and reasoning process. In the second stage, a group relative strategy optimization algorithm is adopted, and the inference results of the model are optimized by reinforcement learning based on a task-oriented reward mechanism.

[0026] like Figure 1 and Figure 2 As shown, in this embodiment, the model construction involves constructing a remote sensing visual language reasoning model based on geospatial thought chains. This model includes a visual encoder and a dynamic position encoding module, a visual projection layer, and a large language model. The visual encoder extracts visual features from multi-scale remote sensing images. The dynamic position encoding module adjusts the position embedding using an interpolation algorithm to adapt to inputs at different resolutions. The visual projection layer aligns the dimensions of the visual tokens with the hidden dimensions of the LLM. The large language model generates a reasoning thought chain by processing the concatenated visual token sequence and text token sequence. Dataset Construction: Construct a geospatial thought chain dataset, which contains a structured reasoning sequence, and the reasoning thought chain explicitly includes three stages: planning, location, and synthesis; Model training: A two-stage alignment strategy is used to train the model. The first stage is the cognitive architecture learning and remote sensing domain alignment stage, which uses supervised fine-tuning to optimize the autoregressive generation probability of the model. The second stage is the inference optimization stage, which uses a group relative strategy optimization algorithm and combines a task-specific reward function to optimize the model's inference strategy.

[0027] In a preferred embodiment of the present invention, the visual encoder is based on the Vision Transformer architecture and integrates a dynamic position encoding mechanism, wherein the dynamic position encoding is implemented in the following manner: The input image is divided into multiple image blocks, and the coordinates of each image block are normalized to obtain continuous grid coordinates. The positional encoding that fits the current input size is generated by sampling from the pre-trained position embedding table using an interpolation method.

[0028] In this embodiment, a visual language foundation model based on geospatial reasoning (hereinafter referred to as RSThinker) is constructed. This model architecture includes a visual encoder and dynamic location encoding module, a visual projection layer, and a language decoder. To comprehensively evaluate the model's capabilities in the perception-reasoning chain, a system of tasks covering fine-grained perception to complex task reasoning is designed, including visual localization (VG), object detection (Det), object counting (OC), scene classification (SC), image description (IC), and visual question answering (VQA). Through these instruction tasks, RSThinker can identify specific coordinates in an image (e.g., "detect all ships on the dock in the image"), plan search paths (e.g., "first find the dock location, then detect the ships"), and generate verifiable conclusions based on visual evidence, thereby systematically evaluating the model's geospatial reasoning capabilities.

[0029] A visual encoder, used to process input images and generate visual token sequences, addresses the challenge of wide resolution ranges and variable aspect ratios in remote sensing images. This visual encoder is built upon the Aimv2-Huge backbone network and integrates a dynamic positional encoding mechanism. Given the drastic resolution variations and variable aspect ratios in remote sensing images, traditional visual transformers typically require fixed-resolution inputs. This invention employs a dynamic positional encoding mechanism. Unlike traditional interpolation, this mechanism first normalizes the coordinates of each image patch to a continuous grid, and then samples from a pre-trained position table using bicubic interpolation.

[0030] In a preferred embodiment of the present invention, the dynamic positional encoder adapts to input images of arbitrary sizes by interpolating a pre-trained positional embedding table. This mechanism first segments the input image into... W p *H p For any patch in the image, the discrete coordinates of the patch are given. g =( w , h ), and normalize its mapping to In a continuous grid coordinate system, normalized coordinates are obtained, and the calculation formula for the normalization process is as follows: ; in, g norm These are normalized continuous grid coordinates; w , h () represents the original coordinates of the image patch; W p , H p This defines the width and height of the patch grid.

[0031] Subsequently, the bicubic interpolation function was used based on the original position embedding table of the pre-trained model. P orig Sampling is performed to calculate a new positional code adapted to the current resolution. Suppose the input image is segmented into... W p *H p For any patch coordinates g =( w , h ), adapted position encoding The calculation is as follows: ; in, It is a bicubic interpolation function. P orig For the pre-trained position embedding table, P adapted ( g ) is the adapted position code.

[0032] Through dynamic adaptation without destructive scaling, the geometry of the original image is preserved, significantly enhancing the ability to capture fine-grained textures and dense small targets in remote sensing images.

[0033] The visual projection layer acts as a bridge connecting vision and language, mapping the dimensions of visual tokens to the hidden dimensions of the language decoder, ensuring that visual features and textual features are compatible within the same high-dimensional semantic space. Let the visual projection layer be a function... f It is composed of a multilayer perceptron (MLP). For the input visual feature sequence... V The projection layer maps it to V proj This makes its dimension similar to the word embedding dimension of a large language model. D LLM Consistency is achieved, thereby enabling feature alignment between modalities.

[0034] The language decoder is built upon the GLM-4 series of large language models and integrates 3D-RoPE (3D Rotational Position Encoding) technology to enhance its ability to encode the spatial location of two-dimensional images. 3D-RoPE not only encodes sequence length information but also independently encodes the row and column coordinates of visual tokens in the original image, enabling the model to explicitly perceive the two-dimensional spatial distance between visual features during self-attention computation. Visual tokens and text tokens are concatenated into a unified sequence input decoder, which utilizes multi-head self-attention (MSA) and feedforward neural networks (FFN) to handle long dependencies, outputting a text sequence containing structured thought chains.

[0035] In a preferred embodiment of the present invention, the construction of the geospatial thinking chain dataset includes: Input the remote sensing image and its corresponding ground truth annotation; Generate reasoning text that conforms to the cognitive structure of "planning-positioning-integration" through a multimodal large model; The planning phase involves problem decomposition and search strategy generation, the positioning phase outputs spatial coordinate evidence of the target, and the synthesis phase integrates visual evidence logically and outputs the final conclusion.

[0036] In this embodiment, a remote sensing thought chain dataset (Geo-CoT380k) is constructed for model training. The overall process aims to make the implicit reasoning process explicit, constructing a reasoning thought chain in the remote sensing domain, such as... Figure 1 As shown, the process is divided into two stages: first, a basic data pool is formed by aggregating existing remote sensing datasets across various tasks (such as DOTAv2, VRSBench, RSVQA, etc.); then, an automated generation pipeline based on reverse engineering is used to construct structured thinking chain data containing a three-stage logic of planning-localization-integration, for use in subsequent cognitive alignment training of the model.

[0037] Unlike traditional datasets that only contain image-answer pairs, the geospatial thought chain data structure explicitly includes three logically progressive cognitive stages: planning, localization, and synthesis. The generation process of this dataset introduces a General Large Model (GPT-4V) as an annotator and uses the actual image annotations as prior input, forcing the annotator to logically derive the intermediate reasoning path of those actual annotations, thereby ensuring the authenticity and reliability of the thought chain data.

[0038] An automated pipeline design is employed. This dataset construction does not rely on expensive and inefficient manual annotation, but instead utilizes a Generalized Multimodal Large Model (GPT-4V) as the annotation engine, employing a result-oriented reverse inference strategy. The specific steps are as follows: Input a remote sensing image and its corresponding ground truth label, such as "The image contains 5 aircraft, with coordinates [...]". A prompt containing system instructions is designed, forcing the GPT-4V to act as a remote sensing expert and generate an inference path that logically derives the ground truth label from visual features. The generated inference data must strictly adhere to the following Geo-CoT cognitive architecture: Planning Phase: Decompose complex user queries and formulate systematic search strategies. For example, for a counting task, the model needs to generate planning text such as "To count all vehicles, I will first scan the main road area, then check the parking area"; Grounding Phase: When executing the search strategy, the model must identify specific targets in the image and explicitly output the corresponding spatial coordinate evidence. For example, "A vehicle was found above the main road, with coordinates [x1, y1, x2, y2]". This step establishes a direct link between the text description and the pixel region; the synthesis stage: based on the collected visual evidence, logical summarization or calculation is performed to generate the final answer. For example, "In conclusion, there are 3 vehicles on the main road and 2 vehicles in the parking lot, for a total of 5 vehicles."

[0039] A specific implementation example is taken as an object detection task, such as... Figure 3 As shown, the input image is an airport scene, and the instruction is "detect the bounding boxes of all aircraft in the image." The ground truth labels include the bounding boxes of all aircraft. The generated Geo-CoT data is as follows: Planning phase: "To detect all aircraft, I first observe the overall layout of the image and identify the apron and runway areas. Next, I will carefully scan these areas to find targets with fuselage and wing features"; Localization phase: "In the lower right quadrant of the image, I found the first aircraft, with coordinates [661, 522, 854, 645]; in the slightly tilted position in the middle of the image, I found the second aircraft, with coordinates [415, 549, 594, 673]..."; Synthesis phase: "After a full scan, it is confirmed that there are a total of 3 aircraft in the image, and all of them have been labeled. Now output the final list of detection boxes." Taking the visual localization task as an example, as follows: Figure 4As shown, the input image is a parking lot background, and the input command is "Where is the vehicle to the right of the vehicle in the lower left corner?" The ground truth labels include the bounding boxes of all vehicles and major buildings. The generated Geo-CoT data is as follows: Planning phase: "To answer this question, I need to first locate the reference point, namely 'the vehicle in the lower left corner,' and then find the target vehicle based on spatial relationships." Location phase: "First, I located the vehicle in the lower left corner, with coordinates [280, 763, 272, 791]. Next, I searched the area to the right of this vehicle and found another vehicle with coordinates [323, 768, 366, 795]"; Synthesis phase: "It is confirmed that the vehicle is indeed located to the right of the reference vehicle. Therefore, the coordinates of the target vehicle are [323, 768, 366, 795]." The above process was used to construct the Geo-CoT380k dataset, which contains approximately 380,000 high-quality instruction-thought chain data. This dataset overcomes the limitations of traditional remote sensing datasets, which only contain image-label pairs and lack intermediate inference processes, providing rich high-level supervision signals for the model.

[0040] In a preferred embodiment of the present invention, the first stage is Cognitive Architecture Injection (SFT), which initializes the network using pre-trained weights and performs full fine-tuning on the model using the Geo-CoT380k dataset. The goal is to enable the model to learn the structured output format and basic inference logic of Geo-CoT. Specifically, images are used as input, requiring the model to predict elements containing... <think> ...< / think> The complete inference text of the tag and <answer> ...< / answer> The final answer to the label. At this stage, the model shifts from directly guessing the answer to a pattern of first thinking and planning, then finding evidence, and finally drawing conclusions, establishing a basic cognitive framework. The loss function used in the first stage of supervised fine-tuning is autoregressive log-likelihood loss. The autoregressive loss (SFT stage) is used to optimize the model's generative ability by constraining the model to generate high-quality structured thought chains in sequence.

[0041] Given an input image I and the problem Q Maximize the target sequence o i Log-likelihood probability: ; in, For sequence length, o i The target output sequence contains the steps of a complete thought chain. o i,t Let t be the t-th token in the sequence, and θ be the model parameters.

[0042] At this stage, the model not only learns to output the final answer, but also learns the structured format of geospatial thought chains. The training objective is to maximize the target sequence containing the complete reasoning path. o i The joint probability is defined by its loss function as the negative of the autoregressive log-likelihood loss.

[0043] As a preferred embodiment of the present invention, the Group Relative Policy Optimization (GRPO) algorithm is introduced to address the illusion problem that may arise in supervised fine-tuning models. This algorithm does not rely on an additional value network but directly optimizes the policy network through outcome rewards. The Group Relative Policy Optimization loss (GRPO stage) is used to improve the credibility and factual correctness of inference. Although the SFT stage endows the model with structured reasoning capabilities, because it is trained solely on maximum likelihood estimation, the model may still generate illusions with correct format but fabricated content (e.g., generating correct coordinate format, but the coordinate positions do not cover the real target). Therefore, the Group Relative Policy Optimization (GRPO) algorithm is introduced in this stage. This algorithm is an online reinforcement learning method whose core idea is to estimate the dominance function using relative advantages and disadvantages within a group, thereby directly optimizing the policy for hard evaluation metrics (such as IoU, counting error) of the remote sensing task without training an additional value network. The second stage of Group Relative Policy Optimization includes the following steps: Multiple candidate output sequences are generated by sampling the same input; The reward value for each sequence is calculated based on a task-specific reward function; The relative advantage of each sequence is calculated based on the reward value, which is used for policy updates.

[0044] In this embodiment, the second stage is Reliability Optimization (GRPO), which directly optimizes the factual correctness by introducing reinforcement learning (RL). Unlike the PPO algorithm, which relies on an additional value network, GRPO optimizes the policy through group sampling and relative advantage estimation, resulting in higher computational efficiency and greater stability. The specific process is as follows: For each input ( I , Q) Based on the current strategy model Sample to generate a set of k distinct answer sequences (In this embodiment, k=8). Then, the reward value for each answer is calculated using a task-specific reward function, and the model strategy is updated to increase the probability of generating high-scoring answers.

[0045] First, group sampling is performed on a given input remote sensing image. I and query Q Based on the current strategy (Initialized to the model trained in the first stage), Monte Carlo sampling is performed to generate a set of k different candidate inference sequences. In this implementation example, the group size is set to k=8. These sequences share the same input, but due to the randomness of sampling, they will contain different inference paths and prediction results.

[0046] Then, multidimensional rewards are calculated for each generated sequence. o i Calculate its reward value R i To adapt to the specific characteristics of multimodal remote sensing tasks, a result-oriented hard-constraint reward function was designed. For visual localization and detection tasks, the predicted bounding boxes in the inference sequence were parsed. B pred Calculate its relationship with the actual bounding box. B gt The intersection-over-union ratio (IoU) is used as the model's reward. For the objective counting task, the predicted number is parsed out. C pred Calculate its difference from the actual quantity. C gt The relative error is checked to ensure the model performs accurate, one-by-one counting. Regarding format compliance, the output is checked to ensure it strictly adheres to... <think> ...< / think> <answer> ...< / answer> The Geo-CoT tag structure. If the format does not conform, a formatting reward or penalty is applied to maintain the stability of the cognitive architecture.

[0047] Then, group relative advantage estimation is performed. Traditional PPO algorithms require training a Critic network to estimate state values. V ( s This incurs significant memory overhead during large model training. GRPO leverages the characteristics of group sampling, using other outputs within the same group as a baseline. For the i-th output, its dominance value... The calculation is as follows: ; In the formula, and These represent the mean and standard deviation of the reward within the group, respectively. This formula indicates that if the reward of an inference path is higher than the group average, its advantage is positive, and the model will increase the probability of generating that path; conversely, it will decrease.

[0048] Finally, the strategy is updated, and the model is constrained by KL divergence. The advantage value calculated using the above formula is used to update the model parameters θ, maximizing the following objective function: ; In the formula, The probability ratio between the old and new policies; `clip` is the cutoff function used to limit the policy update magnitude. Within the interval. The clip operation limits the step size of policy updates, ensuring the smoothness of training. For KL divergence penalty term: ; This item is used to constrain the current policy. Without deviating from the reference model obtained in the first stage (i.e., the SFT model). β is the penalty coefficient. This constraint is crucial; it prevents the model from focusing solely on the final result and generating incoherent language or disrupting the reasoning logic of Geo-CoT, ensuring that the model maintains the fluency and interpretability of language generation while improving accuracy.

[0049] By employing a two-stage alignment strategy, combining supervised fine-tuning (SFT) and group relative policy optimization (GRPO), and designing a total optimization objective that includes autoregressive likelihood loss and reinforcement learning agent loss, the model's logicality, verifiability, and factual correctness in remote sensing tasks can be improved.

[0050] In a preferred embodiment of the present invention, the task-specific reward function includes at least one of the following: For visual localization and detection tasks, the reward is calculated based on the intersection-union ratio (IUU) of the predicted bounding box and the ground truth bounding box. For the target counting task, the reward is calculated based on the error between the predicted number and the actual number; For visual question answering and classification tasks, rewards are calculated based on the degree of answer matching.

[0051] In this embodiment, to accurately guide model optimization, this invention designs a rule-based reward function for remote sensing. Through this result-based reward feedback mechanism, the model can learn to distinguish between seemingly reasonable incorrect reasoning and fact-based correct reasoning, thereby significantly improving the credibility of the final output. For visual localization and detection tasks, the reward value is based on the intersection-over-union (IoU) ratio between the predicted bounding box and the ground truth bounding box. The predicted bounding boxes in the inference results are then parsed. B pred Calculate its relationship with the actual bounding box. B gt Crossover ratio: ; For the objective counting task, the reward is based on the mean absolute error (MAE) between the predicted and actual counts. The predicted count is then parsed. C pred Calculate its difference from the actual quantity. C gt Relative error: ; For visual question answering and classification tasks, a reward based on exact matching is adopted: ; The two-stage training strategy, loss function, and reward function design of this invention are specifically targeted: Stage I utilizes the Geo-CoT380k dataset for supervised fine-tuning (SFT), which, while preserving basic visual language understanding capabilities, rapidly adapts to remote sensing geospatial scenarios and injects a planning-localization-integration cognitive reasoning architecture; such as Figure 3 As shown, the reference answer text is: "To detect all aircraft in the image, I first analyzed the overall layout of the airport, identifying the apron and runway areas. Then, I identified three densely parked aircraft on one side of the terminal, with coordinates [661, 522, 854, 645], etc.; then I identified two aircraft on the other side of the terminal... Finally, combining all visual evidence, I confirmed that there are a total of 6 aircraft in the image..." The training uses autoregressive log-likelihood loss. The second stage introduces a group relative policy optimization (GRPO) architecture. Through group sampling and dominance estimation, it improves the correctness of the model's reasoning in complex visual tasks while avoiding dependence on additional value networks and reducing computational costs. The pruning proxy loss is combined with the KL divergence penalty term to optimize the model's generation of high-reward (e.g., high IoU, low counting error) policies, while also constraining the model to avoid deviating from the reference policy to prevent format collapse, ultimately improving the model's training effect and inference performance. The output consists of multiple candidate thought chains generated for the same input. The relative advantage is calculated by comparing them with the real labels. High-scoring outputs, such as "accurately frame all oil tanks and count them correctly", will be reinforced. The corresponding reference answer is a reward signal based on task metrics (such as detection box IoU or count MAE). The training uses the GRPO objective function to optimize the generation ability and calibrate the accuracy of visual perception.

[0052] This invention also provides a remote sensing multimodal reasoning system based on geospatial thinking chains, the system comprising: The dataset construction module is used to construct a geospatial thinking chain dataset, which contains structured reasoning data organized according to a three-stage cognitive framework of "planning-location-integration" to establish verifiable associations between visual evidence in remote sensing images and textual conclusions. The model building module is used to build a basic model of remote sensing visual language, which includes a visual encoder, a language decoder, and a visual projection layer connecting the two. The training module, employing a two-stage alignment strategy, trains the model, including: In the first stage, the model is fine-tuned under supervision using the geospatial thinking chain dataset to inject geospatial cognitive structure and reasoning process. In the second stage, a group relative strategy optimization algorithm is adopted, and the inference results of the model are optimized by reinforcement learning based on a task-oriented reward mechanism.

[0053] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0054] The performance of the RSThinker model proposed in this invention is qualitatively evaluated. The results are as follows: Figure 5 , Figure 6 As shown, the model demonstrates its excellent performance in spatial reasoning, fine-grained perception, and logical planning through remote sensing image understanding tasks of varying difficulty.

[0055] In fine-grained shape recognition tasks (see...) Figure 5 Faced with the specific instruction to "detect buildings shaped like the number 8," RSThinker demonstrated a deep understanding of geometric features. The model explicitly described in its thought process: "I noticed a prominent building in the upper region with two large ring structures, resembling a vertical combination of the number 8," and ultimately output an accurate bounding box. This shows that RSThinker not only relies on semantic labels but also combines visual morphological features for reasoning, significantly outperforming models like VHM that rely solely on text matching in its ability to uncover scene details.

[0056] In geographic semantic reasoning question answering tasks (see...) Figure 6 Faced with the abstract query "finding a traditional wedding venue with strong religious customs," RSThinker demonstrates powerful semantic-visual alignment capabilities. It doesn't limit itself to simple label matching but initiates deep chain-of-thought reasoning: first, in the planning phase, it establishes the analytical objective of finding "features of a place of worship or community gathering"; then, in the localization phase, it not only accurately identifies church buildings through visual features such as "dove" and "red roof," but also astutely verifies its legitimacy as a public gathering place by combining environmental contexts such as "courtyard space" and "parking lot," ultimately locking onto the coordinates [475, 332, 857, 622]. This process demonstrates that RSThinker can effectively bridge the semantic gap from high-level user intent (wedding customs) to low-level visual features (architectural form), a gap that typical multimodal models often struggle to understand in terms of implicit causal logic and contextual information.

[0057] RSThinker demonstrates significant advantages in remote sensing image understanding tasks: it effectively combines visual information with explicit reasoning logic, accurately understands the combination of spatial relationships and target features, and extracts core scene features directly and efficiently. By outputting verifiable thought chains (Geo-CoT), the model effectively suppresses hallucinations, and its localization accuracy and reasoning logic are superior to existing state-of-the-art models, maintaining good consistency and generalization ability across different scenarios.

[0058] To verify the model's performance, this invention conducts a systematic comparison with 10 representative large-scale models across multiple publicly available remote sensing benchmarks covering fine-grained perception to overall understanding (see [link to relevant documentation]). Figure 1 The dataset includes closed-source commercial models (Claude-sonnet-4, Gemini-2.0-flash, ChatGPT-5), open-source general-purpose multimodal large models (MiniGPT-v2, Qwen2.5-VL), and open-source remote sensing-specific large models (GeoChat, VHM, SkySenseGPT, EarthDial). The tasks cover six major categories: visual localization (VG), object detection (Det), object counting (OC), scene classification (SC), image description (IC), and visual question answering (VQA).

[0059] Figure 1The performance evaluation of the model on six major task categories is presented, and the specific datasets involved are introduced as follows: In the scene classification task, the RESISC45 dataset is a public benchmark dataset with 45 categories built by Northwestern Polytechnical University, used to test the model's ability to classify rich scenes; the AID dataset is a large-scale aerial image dataset designed to advance the classification research of high-resolution aerial scenes; the WHU-RS19 dataset contains structured high-resolution scenes extracted from satellite images, focusing on scene recognition at different scales; the WHU-SIRI dataset mainly focuses on urban areas, used to distinguish scenes of different urban functional areas; and the UCMerced dataset is a classic land use scene classification dataset, widely used to evaluate the basic classification performance of algorithms. In image captioning tasks, the RSICD dataset is a large-scale dataset for remote sensing image captioning, providing rich image-natural language description pairs; the RSITMD dataset is a fine-grained remote sensing image-text multimodal dataset, particularly suitable for image-text retrieval and generation tasks; the UCM-Caption, Sydney-Caption, and NWPU-Caption datasets are image caption versions extended from the UCMerced, Sydney, and NWPU-RESISC45 classification datasets, respectively, used to test the model's ability to generate corresponding scene descriptions. In visual localization tasks, the DIOR-RSVG dataset is a large-scale optical remote sensing image referencing and understanding benchmark built based on the DIOR detection dataset; the RRSIS-D dataset is specifically designed for remote sensing image referencing and segmentation tasks, testing the model's pixel-level localization capabilities; the RSVG dataset is used to evaluate the model's comprehensive ability to perform language-visual alignment and localization in complex remote sensing scenes. In target detection and counting tasks, the DOTA dataset is a large-scale dataset for directional target detection in aerial imagery, containing dense targets at various scales and orientations; the HRRSD dataset is a high-resolution remote sensing image target detection dataset, covering common ground features at various resolutions; the VisDrone dataset, taken by drones, focuses on the detection and tracking of small targets from complex perspectives; the NWPU-VHR dataset contains 10 classes of geospatial targets and is a classic benchmark for ultra-high resolution detection; the RSOD dataset focuses on the detection of specific targets such as aircraft and oil tanks. In visual question answering tasks, the VRSBench dataset is a comprehensive remote sensing visual language benchmark, covering multiple sub-tasks such as question answering, description, and localization; the RSVQA dataset is a visual question answering dataset specifically designed for remote sensing images, containing template-based generated and manually annotated questions to test the logical reasoning ability of models.

[0060] Visual localization and detection tasks (VG & Det): In the visual localization task (DIOR-RSVG dataset), which demands high spatial awareness, RSThinker achieved an mIoU of 90.2%, while similar remote sensing models like EarthDial only achieved 30.2%, and the general strong model Qwen2.5-VL achieved 36.3%. In the object detection task (DOTAv2 dataset), RSThinker achieved an mAP@0.5 of 77.06%, significantly outperforming GLM-4V-Thinking (40.45%). This indicates that the model, through dynamic position encoding and GRPO training, has achieved a qualitative leap in pixel-level localization capabilities, accurately capturing small targets in remote sensing images.

[0061] Target counting task (OC): In the most challenging dense target counting task (DOTAv2), RSThinker's mean absolute error (MAE) is reduced to 2.728 (lower is better), while Qwen2.5-VL's MAE is as high as 36.059, and EarthDial's is 8.422. This is thanks to the Geo-CoT architecture's requirement for the model to perform "region-by-region counting" inference logic, effectively avoiding count omissions or duplications in large-scale scenarios.

[0062] Visual Question Answering and Classification Tasks (VQA & SC): In the VRSBench-VQA task, RSThinker achieved an accuracy of 82.84%, outperforming the commercial model Gemini-2.0-flash (72.00%) and the remote sensing-specific model SkySenseGPT (57.46%). In the scene classification task (RESISC45), RSThinker achieved an accuracy of 96.89%. This demonstrates that the model not only excels in low-level perception but also possesses state-of-the-art performance in high-level semantic understanding and complex problem reasoning.

[0063] Ablation Study Analysis: To verify the effectiveness of the proposed two-stage training strategy, ablation experiments were conducted. Using only SFT (without CoT), the model achieved an mIoU of 81.80% on the visual localization task. After introducing Geo-CoT structured data (SFT w / CoT), the mIoU improved to 87.70%, demonstrating the necessity of cognitive architecture injection. Further introducing GRPO (SFT w / CoT + GRPO) further increased the mIoU to 89.02%, and the MAE of target counting was further optimized from 2.932 to 2.728. This result fully demonstrates the complementarity and necessity of the two-stage strategy of "cognitive architecture injection + credibility optimization".

[0064] Overall, the RSThinker model not only achieves comprehensive performance improvements across multiple tasks in the remote sensing field, but also outperforms current mainstream general-purpose models and remote sensing-specific models. This result fully demonstrates that, in remote sensing scenarios, constructing a unified large-scale model system based on perception-enhanced thought chains is an effective way to solve the problems of black-box reasoning and perceptual illusions, showcasing the enormous potential and value of RSThinker as a next-generation verifiable intelligent foundational model for remote sensing.

[0065] This invention employs a visual encoder based on the Aimv2-Huge backbone network and a dynamic positional encoding mechanism as the core perception module. The visual encoder, combined with a bicubic interpolation algorithm, maps discrete patch coordinates to a normalized continuous grid, achieving dynamic adaptation to input images of arbitrary resolution and aspect ratio without destructive scaling or cropping. The language decoder uses a large language model with a spatially aware 3D-RoPE positional encoding mechanism. The visual projection layer maps visual tokens to a feature space aligned with the hidden size of the large language model, ensuring precise alignment of visual features and textual instructions in the high-dimensional semantic space. This design effectively solves the perception challenges of remote sensing images with large scale spans and dense target distributions, significantly enhancing the model's ability to capture fine-grained texture features and small targets, while maintaining its ability to understand large-scale geographic topology.

[0066] This invention employs a three-stage explicit reasoning framework of planning, localization, and synthesis as the core of the model's thinking. The planning stage decomposes complex queries using natural language to formulate a systematic search strategy; the localization stage forces the model to identify specific targets in images and output spatial coordinate evidence, establishing a direct relationship between text and pixels; the synthesis stage logically derives the final conclusion based on visual evidence. This design breaks away from the black-box mechanism of traditional end-to-end models, enhancing the ability to handle complex problems through explicit task decomposition and providing high transparency and verifiability to the reasoning process through mandatory visual localization, thus solving the problem of the disconnect between the reasoning process and visual perception.

[0067] This invention employs a two-stage alignment strategy combining cognitive alignment and reasoning optimization as its core training paradigm. The first stage utilizes structured thought chain data for supervised fine-tuning, establishing the model's basic reasoning format and cognitive framework. The second stage introduces a group relative policy optimization algorithm, directly optimizing the factual correctness of the reasoning results (such as IoU accuracy and counting error) through reinforcement learning. This strategy combines the learning process of reasoning form and factual content, not only allowing the model to acquire analytical methodologies similar to human experts, but also significantly suppressing the generation of multimodal illusions through a result-oriented reward mechanism, thereby improving the model's applicability and credibility in open-world scenarios.

[0068] This invention employs an autoregressive cross-entropy loss and a group-relative policy optimization objective function to form the overall optimization objective. The autoregressive loss optimizes the fluency and logic of the structured thought chain generated by the model by maximizing the log-likelihood probability of the target sequence. The group-relative policy objective function combines a reward function based on task metrics (IoU, MAE) with a KL divergence penalty term, directly optimizing the model's inference accuracy while constraining the policy update magnitude to prevent model collapse. The combination of these two approaches significantly improves the model's performance on tasks such as visual localization, object detection, and complex question answering, ensuring that the generated content is both logically consistent and highly factually accurate.

[0069] This invention employs a structured geospatial thought chain dataset (Geo-CoT380k) constructed via an automated data pipeline based on reverse generation as data support. This pipeline utilizes a general large model, under strong constraints from real labels, to reverse-generate intermediate reasoning paths connecting visual evidence and the final answer, constructing structured data that includes task decomposition, evidence localization, and logical synthesis. This method overcomes the limitations of traditional remote sensing datasets, which only contain image-label pairs and lack intermediate reasoning processes. It provides high-level cognitive supervision signals for the model, thereby significantly improving the model's generalization and autonomous planning capabilities when facing complex, unseen geospatial analysis tasks.

[0070] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A remote sensing multimodal reasoning method based on geospatial thinking chain, characterized in that, The method includes: A geospatial thinking chain dataset is constructed, which contains structured reasoning data organized according to a three-stage cognitive framework of "planning-location-integration" to establish verifiable associations between visual evidence in remote sensing images and textual conclusions. A basic model for remote sensing visual language is constructed, which includes a visual encoder, a language decoder, and a visual projection layer connecting the two. The model is trained using a two-stage alignment strategy, including: In the first stage, the model is fine-tuned under supervision using the geospatial thinking chain dataset to inject geospatial cognitive structure and reasoning process. In the second stage, a group relative strategy optimization algorithm is adopted, and the inference results of the model are optimized by reinforcement learning based on a task-oriented reward mechanism.

2. The remote sensing multimodal reasoning method based on geospatial thinking chain according to claim 1, characterized in that, The visual encoder is based on the Vision Transformer architecture and integrates a dynamic position encoding mechanism, wherein the dynamic position encoding is implemented in the following way: The input image is divided into multiple image blocks, and the coordinates of each image block are normalized to obtain continuous grid coordinates. The positional encoding that fits the current input size is generated by sampling from the pre-trained position embedding table using an interpolation method.

3. The remote sensing multimodal reasoning method based on geospatial thinking chain according to claim 2, characterized in that, The calculation formula for the normalization process is as follows: ； in, g norm These are normalized continuous grid coordinates; w , h () represents the original coordinates of the image patch; W p , H p This defines the width and height of the patch grid.

4. The remote sensing multimodal reasoning method based on geospatial thinking chain according to claim 3, characterized in that, The interpolation method is bicubic interpolation, and the formula for calculating the adapted position code is: ； in, It is a bicubic interpolation function. P orig For the pre-trained position embedding table, P adapted ( g ) is the adapted position code.

5. The remote sensing multimodal reasoning method based on geospatial thinking chain according to claim 1, characterized in that, The construction of the geospatial thinking chain dataset includes: Input the remote sensing image and its corresponding ground truth annotation; Generate reasoning text that conforms to the cognitive structure of "planning-positioning-integration" through a multimodal large model; The planning phase involves problem decomposition and search strategy generation, the positioning phase outputs spatial coordinate evidence of the target, and the synthesis phase integrates visual evidence logically and outputs the final conclusion.

6. The remote sensing multimodal reasoning method based on geospatial thinking chain according to claim 1, characterized in that, The loss function used in the first stage of supervised fine-tuning is the autoregressive log-likelihood loss: ； in, For sequence length, o i The target output sequence contains the steps of a complete thought chain. o i,t Let t be the t-th token in the sequence, and θ be the model parameters.

7. The remote sensing multimodal reasoning method based on geospatial thinking chain according to claim 1, characterized in that, The second phase of group relative strategy optimization includes the following steps: Multiple candidate output sequences are generated by sampling the same input; The reward value for each sequence is calculated based on a task-specific reward function; The relative advantage of each sequence is calculated based on the reward value, which is used for policy updates.

8. The remote sensing multimodal reasoning method based on geospatial thinking chain according to claim 7, characterized in that, The task-specific reward function includes at least one of the following: For visual localization and detection tasks, the reward is calculated based on the intersection-union ratio (IUU) of the predicted bounding box and the ground truth bounding box. For the target counting task, the reward is calculated based on the error between the predicted number and the actual number; For visual question answering and classification tasks, rewards are calculated based on the degree of answer matching.

9. A remote sensing multimodal reasoning system based on geospatial thinking chain, used to implement the remote sensing multimodal reasoning method based on geospatial thinking chain as described in any one of claims 1-8, characterized in that, The system includes: The dataset construction module is used to construct a geospatial thinking chain dataset, which contains structured reasoning data organized according to a three-stage cognitive framework of "planning-location-integration" to establish verifiable associations between visual evidence in remote sensing images and textual conclusions. The model building module is used to build a basic model of remote sensing visual language, which includes a visual encoder, a language decoder, and a visual projection layer connecting the two. The training module, employing a two-stage alignment strategy, trains the model, including: In the first stage, the model is fine-tuned under supervision using the geospatial thinking chain dataset to inject geospatial cognitive structure and reasoning process. In the second stage, a group relative strategy optimization algorithm is adopted, and the inference results of the model are optimized by reinforcement learning based on a task-oriented reward mechanism.

Images