A remote sensing image interpretation method and system based on a large language model

Abstract

The application provides a remote sensing image interpretation method and system based on a large language model, which comprises: obtaining a sample set, the data in the sample set comprising user query data and remote sensing images; preprocessing the samples; inputting the user query data into a task planning module for intent analysis and forming a task plan; obtaining entities, relationships, attributes and high-relevance text segments related to the task plan from a structured local knowledge base using a knowledge enhancement module based on the task plan, and generating tool calling instructions based on the entities, relationships, attributes and high-relevance text segments; updating and optimizing the tool calling instructions using a reflection module; controlling the remote sensing tool module according to the updated tool calling instructions, and interpreting the preprocessed remote sensing images to obtain an interpretation result; and using a two-stage training paradigm based on supervised fine-tuning and reinforcement learning to improve the structured and generalizable instruction generation capability of the task planning module in the remote sensing multi-tool chain. The application can maintain higher consistency and repeatability in complex task chains and multi-tool chains, significantly reducing the occurrence rates of false calling, redundant calling, parameter errors, format errors and hallucinations.

Description

Technical Field

[0001] This invention relates to the fields of intelligent agents and remote sensing technology, and in particular to a remote sensing image interpretation method and system based on a large language model. Background Technology

[0002] In recent years, remote sensing imagery has played an increasingly important role in fields such as geographic information extraction, disaster monitoring, military reconnaissance, environmental governance, and ecological change analysis. High-resolution, multi-source, heterogeneous remote sensing data provides humanity with abundant information for Earth observation, but it also brings a series of challenges, including massive data volume, complex features, and significant spatiotemporal differences. Traditional remote sensing interpretation methods often rely on specific algorithm models, such as convolutional neural networks or Vision Transformers, typically designed for single tasks (such as target detection, change detection, or land cover classification). Their generalization ability is limited, making it difficult to achieve unified processing across diverse and complex tasks. Furthermore, these methods often require professional algorithm engineers or remote sensing experts to participate in model selection, parameter configuration, and result analysis, posing a high operational barrier for ordinary users and limiting the popularization and application efficiency of remote sensing technology.

[0003] Breakthroughs in artificial intelligence technology, especially the rise of Large Language Models (LLMs) and Vision-Language Models (VLMs), have provided new directions for intelligent remote sensing interpretation. LLM-based agent frameworks are gradually becoming a new paradigm for intelligent systems, achieving autonomous execution "from instruction to action" through natural language understanding, task planning, and toolchain invocation. Applying LLM to remote sensing interpretation tasks provides new ideas for achieving automated and intelligent remote sensing data processing. However, existing LLM-based remote sensing agents still face several technical bottlenecks in practical applications.

[0004] First, current intelligent agents heavily rely on predefined templates or manual rules for tool invocation and task planning, lacking dynamic adaptability. When faced with complex tasks, the agent's strategies in tool selection, parameter setting, and operation sequence tend to become rigid, leading to instability in the tool invocation process and ultimately causing fluctuations in task performance. Second, methods based on static knowledge bases or traditional retrieval-augmented generation (RAG) are insufficient to meet the requirements of remote sensing tasks for real-time performance, structured knowledge representation, and cross-source information integration. Limited by factors such as slow knowledge base updates, static content, and coarse retrieval granularity, the agent struggles to acquire effective knowledge in a timely manner and cannot support more refined and interconnected reasoning processes.

[0005] In summary, traditional remote sensing interpretation methods struggle to handle the diverse tasks arising from complex, multi-source data, while existing LLM-based agents exhibit significant shortcomings in tool invocation and knowledge acquisition. To address the complexity of tasks caused by heterogeneous, multi-source remote sensing data and the deficiencies of existing LLM agents in tool planning and knowledge acquisition, it is necessary to construct a novel system with greater autonomy, adaptability, and knowledge representation capabilities. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention proposes a remote sensing image interpretation method based on a large language model. The method includes: acquiring user query data and remote sensing data; processing the user query data and remote sensing data using a trained remote sensing image interpretation model to obtain remote sensing image interpretation results; the remote sensing image interpretation model includes a task planning module, a knowledge enhancement module, a reflection module, and a remote sensing tool module.

[0007] Training the remote sensing image interpretation model includes: acquiring a sample set, which includes user query data and remote sensing images; preprocessing the samples; inputting the user query data into the task planning module for intent parsing and generating a task plan; generating initial tool call instructions based on the task plan; detecting the initial tool call instructions, and when the instructions involve domain knowledge or external information requirements, invoking the knowledge enhancement module to retrieve task-related entities, relationships, attributes, and highly relevant text fragments from a structured local knowledge base or online sources, and injecting the retrieval results as knowledge supplements into the model context; updating and optimizing the tool call instructions using a reflection module; controlling the remote sensing tool module according to the updated tool call instructions and interpreting the preprocessed remote sensing images to obtain interpretation results; constructing the model's loss function based on the interpretation results, adjusting the model's parameters, and completing model training when the loss function converges.

[0008] A remote sensing image interpretation system based on a large language model, comprising: a user input module, a task planning module, a knowledge enhancement module, a reflection module, a remote sensing tool library, and a system output module;

[0009] The user input module is used to receive users' natural language queries and multimodal remote sensing data, complete format verification, anomaly detection and necessary preprocessing, and standardize multi-source inputs into a unified request object;

[0010] The task planning module is used to understand and decompose user intent to form a task plan;

[0011] The knowledge enhancement module retrieves entities, relationships, attributes, and highly relevant text fragments related to the task plan from the structured local knowledge base according to the task plan, and generates tool call instructions based on the entities, relationships, attributes, and highly relevant text fragments.

[0012] The reflection module performs consistency, rationality, and boundary condition checks on the tool call instructions generated at each step or stage.

[0013] The remote sensing tool library is used to interpret the preprocessed remote sensing images and obtain the interpretation results;

[0014] The system output module is responsible for merging and presenting the results generated in multiple steps, and the output includes the model's final natural language response and tool call results.

[0015] The beneficial effects of this invention are:

[0016] This invention employs supervised fine-tuning and reinforcement learning to train the agent, enabling it to develop stable and generalizable tool selection and parameter configuration strategies in multi-task, multi-tool scenarios. Compared to schemes relying on templates and heuristic rules, this invention maintains higher consistency and repeatability in complex task chains and multi-tool sets, significantly reducing the incidence of miscalls, redundant calls, parameter errors, format errors, and illusions. This invention utilizes a dual-channel knowledge retrieval mechanism based on knowledge graphs, working in conjunction with online search, to simultaneously provide the system with structured knowledge reasoning, semantic-level retrieval, and dynamic knowledge updating capabilities. The former provides interpretable knowledge support at the entity-relationship-attribute level, while the latter supplements the system with the latest network information through cross-document, cross-modal context completion and online information retrieval. The synergy of these two approaches deepens the agent's understanding of professional terminology, domain norms, and task boundary conditions, ensuring both the professional depth of knowledge and its timeliness and real-time responsiveness, significantly improving the system's adaptability and interpretability in complex task scenarios. Attached Figure Description

[0017] Figure 1 This is a system structure diagram of the present invention;

[0018] Figure 2 This is a flowchart of the strategy optimization training process of the present invention;

[0019] Figure 3 This is a flowchart of the knowledge enhancement method of the present invention;

[0020] Figure 4 This is a system flowchart of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] A remote sensing image interpretation method based on a large language model is disclosed. The method includes: acquiring user query data and remote sensing data; processing the user query data and remote sensing data using a trained remote sensing image interpretation model to obtain remote sensing image interpretation results; the remote sensing image interpretation model includes a task planning module, a knowledge enhancement module, a reflection module, and a remote sensing tool module; training the remote sensing image interpretation model includes: acquiring a sample set, the data in the sample set including user query data and remote sensing images; preprocessing the samples; inputting the user query data into the task planning module for intent parsing and forming a task plan; and generating initial tools based on the task plan. The process involves invoking commands; initially, the tool invocation commands are detected; when commands involve domain knowledge or external information requirements, the knowledge enhancement module is invoked to retrieve task-related entities, relationships, attributes, and highly relevant text fragments from a structured local knowledge base or online sources, and the retrieval results are injected into the model context as knowledge supplementation; the reflection module updates and optimizes the tool invocation commands; the remote sensing tool module is controlled according to the updated tool invocation commands, and the preprocessed remote sensing images are interpreted to obtain interpretation results; the loss function of the model is constructed based on the interpretation results, the model parameters are adjusted, and the model training is completed when the loss function converges.

[0023] This technology aims to address the challenges of intelligently processing massive amounts of remote sensing data, proposing an image interpretation intelligent agent system specifically tailored for the remote sensing field. The system uses a large language model as its core controller, constructing a closed-loop process of "task parsing—knowledge enhancement—instruction generation—reflective correction—tool execution—result aggregation," achieving end-to-end intelligent processing of remote sensing tasks. During the training phase, supervised fine-tuning and reinforcement learning-based optimization strategies are employed, enabling the model to generate formatted and semantically correct tool invocation instructions. During the inference phase, a reflective mechanism facilitates online self-correction. Furthermore, by introducing knowledge enhancement methods based on knowledge graphs and online search, the system can dynamically utilize domain knowledge and the latest data to provide stable and professional instruction-level output for complex remote sensing tasks, meeting the practical needs of efficient remote sensing data processing.

[0024] The overall structure of this system is shown in the attached figure. Figure 1As shown, the system includes a user input interface, a task planning module (composed of a large language model), a knowledge enhancement module (based on knowledge graphs and online search methods), a reflection module, a remote sensing tool library, and a system output interface. After receiving user queries and remote sensing imagery, the task planning module parses the intent and generates a task plan. The knowledge enhancement module retrieves relevant entities, relationships, attributes, and highly relevant text fragments from a structured local knowledge base as needed, or obtains real-time information, geographical knowledge, and reference materials through online searches, integrating them into a directly referable comprehensive knowledge context. Subsequently, it generates tool invocation instructions conforming to the agreed-upon syntax (including tool names, parameter key-value pairs, and placeholders), which can be directly executed by subsequent system processes. To improve the stability of the inference phase, the reflection mechanism performs consistency, rationality, and boundary condition checks after each step or stage of output. If non-compliant or contradictory content is found, rewriting and supplementary retrieval are automatically triggered, thus completing online correction at minimal cost. Finally, the output module summarizes the natural language responses and remote sensing tool outputs.

[0025] In this embodiment, the functional modules include: user input module, task planning module, knowledge enhancement module, reflection module, remote sensing tool library, and system output module.

[0026] User input module: Responsible for receiving users' natural language queries and multimodal remote sensing data, performing format verification, anomaly detection and necessary preprocessing, and standardizing multi-source inputs into a unified request object.

[0027] Task planning module: As the semantic hub of the system, it understands and decomposes user intent based on standardized requests, forming a task plan that includes a sequence of subtasks, dependencies, a set of candidate tools, and a draft parameter list. This module generates structured tool invocation instructions according to a unified syntax standard, clearly defining tool names, parameter keys, and placeholders.

[0028] The knowledge enhancement module consists of two collaborative parts: a knowledge graph-based Relationship Aggregator (RAG) and online search. The RAG part employs a "dual-channel retrieval" mechanism (overall retrieval + independent retrieval). When knowledge enhancement is triggered, the system first extracts key entities and relationships from the task plan and executes them in parallel on the local knowledge graph and semantic index: First, it performs an overall retrieval around the overall semantics of the query to obtain highly relevant core segments; second, it dynamically allocates retrieval quotas based on the keyword importance given by the large model, performing independent retrievals on high-importance keywords, thus simultaneously covering both "overall semantics" and "key sub-semantics." When local knowledge coverage is insufficient or there are timeliness requirements, online search can be triggered to obtain the latest information from the internet. After deduplication, conflict resolution, and credibility assessment, the above results are integrated into knowledge supplements that can be directly injected into the context of the large model to support task execution.

[0029] The Reflection Module, running online during the reasoning phase, checks the consistency, rationality, and boundary conditions of tool call instructions generated at each step or stage. Checks include whether the tool name is valid and matches the task intent, whether parameters are complete and within allowed value ranges, whether instructions are consistent with the graph constraints and retrieved data, and whether there are any contradictions between results across steps. If any non-compliant or contradictory content is found, the Reflection Module automatically generates rewrite prompts, guiding the task planning module to fine-tune parameters, replace alternative tools, or supplement knowledge retrieval, thereby completing online correction with minimal cost and improving the stability and usability of the instruction sequence.

[0030] The remote sensing tool library primarily covers target detection, change detection and disaster assessment, scene classification, semantic / instance segmentation, SAR-specific processing, image denoising, and super-resolution. These capabilities are presented externally with unified tool descriptions, including tool names, inputs, outputs, and parameter constraints, allowing the task planning module to generate structured calling instructions. Through the provision of multi-task, multi-modal, and multi-temporal tools, it supports a complete task chain from target level to scene level, and from detection to assessment, significantly improving the adaptability and professional depth of instruction generation.

[0031] The system output module is responsible for merging and presenting the results generated in multiple steps. The output includes the model's final natural language response and tool call results.

[0032] In this embodiment, inputting user query data into the task planning module for intent parsing includes: jointly understanding the type information of user requests and input remote sensing data, extracting task objectives, objects of interest, spatiotemporal range, accuracy / resolution requirements and output format constraints, mapping the intent to standard remote sensing task types and decomposing the tasks to form a task plan containing the order and dependencies of subtasks, and standardizing the task plan.

[0033] The knowledge enhancement module retrieves entities, relationships, attributes, and highly relevant text fragments related to the task plan from a structured local knowledge base. This includes: extracting search keywords from user requests and task plans as search conditions; performing a search in the structured local knowledge base to return entities, relationships, attributes, and highly relevant text fragments that match the search conditions; and sorting and deduplicating the search results based on relevance to form knowledge enhancement content that can be directly injected into the model context.

[0034] The tool call instruction is updated and optimized using the reflection module, which includes: based on the predefined tool metadata description, parameter constraint rules and task semantic constraints, the tool call instruction is automatically verified and diagnosed in multiple dimensions. The automatic verification includes: (1) tool name legality verification, by matching the tool name generated in the instruction with the tool identifier set in the remote sensing tool library to determine whether the selected tool exists and is in a callable state; (2) task matching verification, based on the current subtask type, task planning stage information and domain knowledge provided by the knowledge enhancement module, to determine whether the selected tool is suitable for the current task semantics; (3) parameter integrity and value range verification, based on the predefined parameter set, parameter type, required / optional markers and numerical or enumerated value range in the tool description, to detect the missing, redundant or out-of-bounds situation of parameters in the instruction; (4) cross-step consistency and boundary condition verification, by comparing the type, spatial range, resolution and semantic constraints of the current instruction and the existing intermediate results in the historical execution trajectory, to detect whether there are inconsistencies between the preceding and following steps, missing dependent results or boundary condition conflicts. After completing the above verifications, the reflection module further enhances the context with knowledge, diagnoses and analyzes detected contradictions, omissions, or unexecutable parameters, and automatically generates targeted correction prompts. When verification fails, it triggers the rewriting and correction of tool selection or parameter configuration, and re-executes the verification process on the updated instructions until tool call instructions that meet consistency and executability constraints are generated.

[0035] The interpretation of preprocessed remote sensing images includes: matching the corresponding remote sensing interpretation tool from the remote sensing tool library according to the final tool call instruction and loading parameters; using the preprocessed remote sensing image as input; and executing the remote sensing interpretation algorithm corresponding to the current subtask type. The remote sensing interpretation algorithm is a pixel-level, target-level, or scene-level analysis algorithm for remote sensing images, outputting structured subtask results. These subtask results include, but are not limited to, the spatial location of the target or region, category labels, confidence scores, change area masks, statistical indicators, or attribute parameters. These results are encapsulated according to a unified data structure and backfilled into the execution trajectory as intermediate states for subsequent subtasks to reference in tool selection, parameter setting, or result constraints. The system iteratively executes the above interpretation process according to the task planning sequence until all subtasks are completed, and then fuses, merges, or statistically analyzes the multi-step interpretation results to obtain the final remote sensing interpretation result.

[0036] The model's loss function is:

[0037]

[0038] The loss consists of supervised learning loss and reinforcement learning loss. The supervised learning loss minimizes the negative log-likelihood of the reference tool invocation instruction sequence given the context, learning structured instruction generation and tool-parameter alignment. The reinforcement learning loss function is...

[0039]

[0040] The model is updated by using a strategy optimization based on group relative advantage and combined with the pruning strategy objective. The reward signal includes the validity of structured syntax and the semantic matching degree of tools / parameters, thereby improving the correctness and stability of system calling tools while ensuring executability.

[0041] In this embodiment, the policy optimization training method based on supervised fine-tuning and reinforcement learning includes: as shown in the appendix. Figure 2 As shown, this invention employs a two-stage training paradigm based on supervised fine-tuning and reinforcement learning to enable the task planning module to acquire structured and generalizable instruction generation capabilities in remote sensing multi-tool links.

[0042] The first stage is Supervised Fine-Tuning (SFT): First, for the remote sensing task set and toolset, a high-quality "correct instruction trajectory" is constructed. This is a complete sequence of multi-step tool call instructions (including tool name and parameter key-value pairs) generated by a strong foundation / expert workflow, after correctness screening. Notated as follows, a trajectory is denoted as... , where sk is the conditional context of step k (task semantics, historical execution trajectory, etc.), and ak is the tool invocation instruction that should be generated in step k. The goal of SFT is to maximize the likelihood of the reference action sequence:

[0043]

[0044] This allows the model to first learn the correct structured output syntax and basic tool-parameter alignment, providing a stable initialization for subsequent reinforcement learning.

[0045] The second stage involves reinforcement learning training, treating "generating the next tool call instruction" as a sequential decision. Therefore, a dense reward system based entirely on textually deterministic criteria is designed to provide immediate evaluation of each output step.

[0046]

[0047] in, To verify whether the structured syntax meets the conventions (such as the requirement to include valid tool names, parameter lists, key-value pair integrity, order, and placeholder specifications); key items Used to measure semantic matching with reference instructions. Let the reference set be... The model generation set is After performing optimal one-to-one matching on the two, scores are assigned and aggregated at three levels: tool name, parameter name, and parameter value.

[0048]

[0049] in The set of tool names in the set. For parameter key set, Representation and Reference Optimal matching prediction. Weight the three items. Linear fusion and scale normalization:

[0050]

[0051] This provides fine-grained, differentiable approximation-based intensive supervision over whether the tool selection is correct, whether the parameters are complete, and whether the values ​​are accurate, and through... Standardize reward criteria to stabilize training.

[0052] The strategy optimization employs Grouped Reinforcement Policy Optimization (GRPO). Several steps (or candidate trajectory segments) within a group are sampled for the same query Q. Estimating by within-group standardized construction advantage:

[0053]

[0054] And update the policy using the cut PPO target:

[0055]

[0056] Ultimately, the model achieved stable policy generation capabilities through the two-stage training process described above: the supervised fine-tuning stage ensured the model mastered the correct structured syntax and tool call templates, while the reinforcement learning stage further optimized the model's decision consistency and parameter generation accuracy in multi-step task chains through rule-based rewards and within-group normalized advantage estimation. After multiple rounds of iterative training, the task planning module can automatically generate standardized, logically sound, and professionally consistent tool call instruction sequences when facing multi-source, heterogeneous, and semantically complex remote sensing tasks, thereby significantly improving the system's intelligence level and the stability of remote sensing task planning.

[0057] As attached Figure 3 As shown, this invention proposes a knowledge enhancement method for remote sensing tasks. It uses a knowledge graph to construct a local vector knowledge base and introduces a dual-channel knowledge retrieval mechanism to work in conjunction with network search. This improves the relevance, coverage, and timeliness of knowledge without increasing the complexity of use.

[0058] To support subsequent enhanced retrieval generation mechanisms, this invention first constructs a vectorized knowledge base based on a knowledge graph, geared towards remote sensing tasks. In the offline phase, this invention segments high-quality remote sensing-related text into fragments according to semantic structure and extracts entities, relationships, and key attributes using a large model to form a traceable initial knowledge graph. Subsequently, standardized key-value representations are generated for each entity and relationship: the key supports efficient semantic retrieval, and the value is a standardized descriptive text rewritten from the fused original text fragments, ensuring that the knowledge can be directly used at the generation stage. In the graph merging stage, this invention performs entity fusion and relationship deduplication based on semantic consistency and contextual constraints, unifying the representation of nodes pointing to the same concept from different sources, making the knowledge structure more compact and globally consistent. After completing structured extraction, this invention further performs unified vectorization processing on entity descriptions, relation semantics, and their associated text, and constructs an efficient vector index while preserving the entity-relationship topology to support cross-entity reasoning and multi-hop association retrieval. The resulting vector knowledge base achieves a unified representation of "structural knowledge + semantic vectors", which can provide high-precision semantic recall, as well as interpretability and scalability, providing a high-quality knowledge foundation for subsequent RAG.

[0059] This module comprises a knowledge graph-based RAG (Remote Sensing Graph) and a web-based search, and introduces a dual-channel retrieval enhancement mechanism during the retrieval phase to simultaneously cover the "overall semantics" and "key sub-semantics" of the query. Offline, a remote sensing domain knowledge graph and semantic index are constructed, accumulating entities, relationships, attributes, and highly relevant text fragments to form an interpretable and traceable domain knowledge foundation. Online, when task planning determines a knowledge gap or requires specialized evidence, local RAG retrieval is initiated and web-based searches are triggered in parallel as needed, uniformly outputting comprehensive knowledge that can be directly injected into the context of large models.

[0060] The dual-channel retrieval enhancement mechanism comprises two paths: "overall retrieval" and "independent retrieval." Overall retrieval focuses on the global semantics of the query, returning core evidence highly relevant to the overall task intent, ensuring semantic coherence and cross-concept dependencies. Independent retrieval targets high-importance keywords within the task, allocating separate retrieval quotas and retrieving corresponding evidence to cover fine-grained key points and long-tail information, avoiding recall bias and missing crucial details caused by single-chain queries. The results from both paths undergo local deduplication, conflict resolution, and credibility assessment. They are then weighted and fused based on source type, timeliness markers, and graph consistency to form knowledge output with a higher degree of relevance to the task.

[0061] When local knowledge coverage is insufficient or there is a clear need for timeliness, the linked network search module retrieves the latest information from whitelisted sources and annotates external results with timestamps, geographical applicability, and source credibility. Network results and local RAG candidates are processed in the same fusion process, undergoing redundancy removal and conflict resolution before being incorporated into the comprehensive knowledge context. This context, along with the evidence chain (including source, time, and fusion decision summary), is injected into the task planning to guide task execution and provide high-value evidence for the online correction by the reflection module.

[0062] Through the collaborative design of "knowledge graph + dual-channel knowledge retrieval + network search", this module significantly improves the relevance, coverage and timeliness of knowledge retrieval without increasing the complexity of use, providing reliable knowledge support for the stable planning and high-quality instruction generation of complex remote sensing tasks.

[0063] In one specific embodiment, this system is used for intelligent assessment of building damage after a fire, such as... Figure 4 The system flowchart shows that the user submits a request in natural language to "automatically detect the fire range and assess the degree of building damage based on pre- and post-disaster remote sensing images" and uploads multi-source images. After the system input module completes verification and encapsulation, it enters the k-th step of task planning. The large language model sequentially plans sub-tasks such as image registration check, change detection, building detection, and disaster damage assessment. In each step, the system first determines whether there is a knowledge gap. If domain knowledge is needed, it triggers RAG and online search to obtain supplementary information, which is then fused with the results of the local knowledge graph and injected into the context. Subsequently, the task planning module generates the formatted tool call instructions for this step and enters a reflection check to verify the legality of the instructions, the completeness of the parameters, and the consistency with the knowledge. If errors are found, it returns to rewrite or supplement the search. After passing the verification, the corresponding tool is executed. The system then determines whether to continue calling subsequent tools. If the task is not completed, it enters the next step, repeating the cycle of planning—knowledge enhancement (on demand)—instruction generation—reflection verification—tool execution until all sub-tasks are completed. Finally, the system output module summarizes the results of each step and generates a structured assessment report that includes the fire range and the level of building damage, realizing multi-step intelligent remote sensing interpretation of fire scenarios.

[0064] A remote sensing image interpretation system based on a large language model, comprising: a user input module, a task planning module, a knowledge enhancement module, a reflection module, a remote sensing tool library, and a system output module;

[0065] The user input module is used to receive users' natural language queries and multimodal remote sensing data, complete format verification, anomaly detection and necessary preprocessing, and standardize multi-source inputs into a unified request object;

[0066] The task planning module is used to understand and decompose user intent to form a task plan;

[0067] The knowledge enhancement module retrieves entities, relationships, attributes, and highly relevant text fragments related to the task plan from the structured local knowledge base according to the task plan, and generates tool call instructions based on the entities, relationships, attributes, and highly relevant text fragments.

[0068] The reflection module performs consistency, rationality, and boundary condition checks on the tool call instructions generated at each step or stage.

[0069] The remote sensing tool library is used to interpret the preprocessed remote sensing images and obtain the interpretation results;

[0070] The system output module is responsible for merging and presenting the results generated in multiple steps, and the output includes the model's final natural language response and tool call results.

[0071] The system implementation method of the present invention is the same as the method implementation method.

[0072] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A remote sensing image interpretation method based on a large language model, characterized in that, include: Acquire user query data and remote sensing images; The trained remote sensing image interpretation model is used to process user query data and remote sensing images to obtain remote sensing image interpretation results; The remote sensing image interpretation model includes a task planning module, a knowledge enhancement module, a reflection module, and a remote sensing tool module; Training the remote sensing image interpretation model includes: acquiring a sample set, which includes user query data and remote sensing images; preprocessing the samples; inputting the user query data into the task planning module for intent parsing to form a task plan, and generating initial tool call instructions based on the task plan; detecting the initial tool call instructions, and when the instructions involve domain knowledge or external information requirements, calling the knowledge enhancement module to retrieve task-related entities, relationships, attributes, and highly relevant text fragments from a structured local knowledge base or online sources, and injecting the retrieval results as knowledge supplements into the model context; updating and optimizing the tool call instructions using a reflection module; controlling the remote sensing tool module according to the updated tool call instructions and interpreting the preprocessed remote sensing images to obtain interpretation results; constructing the model's loss function based on the interpretation results, adjusting the model's parameters, and completing the model training when the loss function converges. The reflection module is used to update and optimize tool invocation instructions, including: automatically verifying the validity of tool names, task matching, parameter completeness and value range, as well as consistency and boundary conditions across steps, and combining knowledge-enhanced context to diagnose possible contradictions, omissions or unexecutable parameters; when the verification fails, the driver rewrites and corrects the tool selection or parameters and verifies them again until the tool invocation instructions that meet the consistency and executability constraints are output. The model's loss function consists of supervised learning loss and reinforcement learning loss. The supervised learning loss is used to minimize the negative log-likelihood of the reference tool call instruction sequence under given context conditions, in order to learn structured instruction generation and tool-parameter alignment. The reinforcement learning loss adopts a strategy optimization based on group relative advantage and combines the pruning policy objective to update the model. Its reward signal includes the validity of structured syntax and the semantic matching degree of tool or parameter, thereby improving the correctness and stability of the system calling tools while ensuring executability.

2. The remote sensing image interpretation method based on a large language model according to claim 1, characterized in that, Preprocessing of the samples includes: format verification and anomaly detection of user query data and remote sensing images; and standardizing remote sensing images into a unified request object.

3. The remote sensing image interpretation method based on a large language model according to claim 1, characterized in that, The process of inputting user query data into the task planning module for intent parsing includes: jointly understanding the type information of user requests and input remote sensing data; extracting task objectives, objects of interest, spatiotemporal range, accuracy / resolution requirements, and output format constraints; mapping the intent to standard remote sensing task types and decomposing the tasks to form a task plan that includes the order and dependencies of subtasks; and standardizing the task plan.

4. The remote sensing image interpretation method based on a large language model according to claim 1, characterized in that, The knowledge enhancement module retrieves entities, relationships, attributes, and highly relevant text fragments related to the task plan from a structured local knowledge base. This includes: extracting search keywords from user requests and task plans as search conditions; performing a search in the structured local knowledge base to return entities, relationships, attributes, and highly relevant text fragments that match the search conditions; and sorting and deduplicating the search results based on relevance to form knowledge enhancement content that can be directly injected into the model context.

5. The remote sensing image interpretation method based on a large language model according to claim 1, characterized in that, The interpretation of the preprocessed remote sensing image includes: matching the corresponding tool from the remote sensing tool library and loading the parameters according to the final tool call instruction; executing the remote sensing interpretation algorithm on the preprocessed remote sensing image to obtain the sub-task results; backfilling the sub-task results as intermediate states into the execution trajectory for reference by subsequent sub-tasks until the task plan is completed; and summarizing the results of multiple steps to obtain the final interpretation result.

6. A remote sensing image interpretation system based on a large language model, the system being used to execute the remote sensing image interpretation method based on a large language model as described in any one of claims 1 to 5, characterized in that, The system includes: a user input module, a task planning module, a knowledge enhancement module, a reflection module, a remote sensing tool module, and a system output module; The user input module is used to receive users' natural language queries and multimodal remote sensing data, complete format verification, anomaly detection and preprocessing, and standardize multi-source inputs into a unified request object; The task planning module is used to understand and decompose user intent to form a task plan; The knowledge enhancement module retrieves entities, relationships, attributes, and highly relevant text fragments related to the task plan from the structured local knowledge base according to the task plan, and generates tool call instructions based on the entities, relationships, attributes, and highly relevant text fragments. The reflection module performs consistency, rationality, and boundary condition checks on the tool call instructions generated at each step or stage. The remote sensing tool module is used to interpret the preprocessed remote sensing images and obtain the interpretation results; The system output module is responsible for merging and presenting the results generated in multiple steps, and the output includes the model's final natural language response and tool call results.

Images