Remote sensing large language model precision improvement method and device based on subdivision database and human feedback, and electronic equipment

Abstract

The application provides a remote sensing large language model precision improvement method and device based on a subdivision database and human feedback, and electronic equipment, and relates to the technical field of remote sensing image large language models. The method constructs an initial remote sensing large language model, and adds a preprocessing layer, an audit layer and a feedback optimization layer to the initial remote sensing large language model. Specifically, the preprocessing layer processes multi-source remote sensing image data and natural language instructions to construct a subdivision database, solves the problem that text descriptions are mostly coarse-grained and global scene descriptions, lack precise annotation of specific ground object targets, fine attributes, spatial relationships and professional connotations in images, and realizes a fine data system. In the audit layer, the accuracy of the inference result obtained by the reasoning layer is verified through a pre-constructed scenario-based professional subnetwork. If the verification is passed, the inference result that passes the verification is provided to the output layer, and the output professional, targeted and safe inference result is output by the output layer.

Description

Technical Field

[0001] This application relates to the field of remote sensing image large language model technology, and in particular to a method, device and electronic device for improving the accuracy of remote sensing large language models based on a segmented database and human feedback. Background Technology

[0002] In recent years, Large Language Models (LLMs) have achieved remarkable success in the field of natural language processing. Their powerful contextual understanding and logical reasoning capabilities have provided a new paradigm for the intelligent interpretation of remote sensing imagery. Researchers have begun to attempt to construct remote sensing large language models, aiming to deeply integrate the visual features of remote sensing imagery with natural language descriptions. This would enable the model to understand and respond to specialized remote sensing queries, such as "extract all ship targets in this area and describe their spatial distribution," or "compare and analyze the changing trends of green space area in this city over the past five years."

[0003] Existing related technical solutions mainly rely on plug-in approaches based on general-purpose large language models. These methods utilize pre-trained general-purpose LLMs (such as the GPT series and LLaMA series) as the core inference engine. Additional visual encoders (such as CLIP and ResNet) convert remotely sensed images into visual feature vectors, which are then input into the LLM as part of the prompts. The model aligns the visual and linguistic feature spaces through projection layers, ultimately completing tasks such as visual question answering and image description. Here, GPT stands for Generative Pre-trained Transformer; LLaMA stands for Large Language Model Meta; CLIP stands for Contrastive Language Image Pre-training; and ResNet stands for Residual Network.

[0004] While the aforementioned methods have demonstrated considerable potential, the training objectives of existing models primarily rely on next-word prediction or cross-entropy loss functions. Their optimization focuses on "predicting the next word" rather than "generating the most professional, useful, and safest answer." This results in model outputs that may be grammatically correct and seemingly reasonable, but lack the professionalism, relevance, and safety expected and preferred by domain experts and users. This deficiency severely limits the performance ceiling and application effectiveness of remote sensing large language models, necessitating a solution to this technical problem. Summary of the Invention

[0005] In view of the above problems, this application is proposed to provide a method, apparatus, and electronic device for improving the accuracy of remote sensing large language models based on segmented databases and human feedback, which overcomes or at least partially solves the above problems. The technical solution is as follows:

[0006] Firstly, a method for improving the accuracy of remote sensing large language models based on segmented databases and human feedback is provided, the method comprising:

[0007] Based on an open-source visual universal language model, and adapted to multi-source remote sensing image data and natural language commands, an initial remote sensing large language model is constructed. The architecture of the initial remote sensing large language model includes a data layer, a fusion layer, an inference layer, and an output layer. The data layer collects multi-source remote sensing image data and natural language commands; the fusion layer achieves accurate alignment and fusion of cross-modal features to obtain multi-modal fused features; and the inference layer performs interpretation tasks to obtain inference results.

[0008] By adding a preprocessing layer, a review layer, and a feedback optimization layer to the initial remote sensing large language model, an improved remote sensing large language model is obtained. The preprocessing layer processes multi-source remote sensing image data and natural language instructions to construct a segmented database. The review layer verifies the accuracy of the inference results obtained by the inference layer through a pre-constructed scenario-based professional sub-network. If the verification is successful, the verified inference result is provided to the output layer, which then outputs the verified inference result. The feedback optimization layer optimizes the initial remote sensing large language model based on human feedback and the scenario-based rewards for remote sensing tasks.

[0009] In one possible implementation, the segmentation database includes a segmented image database and a segmented text database. The preprocessing layer processes multi-source remote sensing image data and natural language instructions to construct the segmentation database, including:

[0010] The preprocessing layer performs data preprocessing, cross-sensor feature calibration, and multi-dimensional feature extraction on multi-source remote sensing image data to construct a subdivided image database with three dimensions: land cover type, task scene, and feature type. The constructed subdivided image database includes multiple fields such as feature ID, feature name, land cover type, scene label, feature parameters, image coordinates, and acquisition time.

[0011] The preprocessing layer normalizes natural language commands using remote sensing terminology, parses and decomposes command semantics, and constructs a segmented text database that includes task type, core object, and spatiotemporal precision constraints.

[0012] In one possible implementation, the cross-sensor feature calibration steps include:

[0013] The selection of homogeneous land cover samples involves collecting multi-source data from the same area and selecting a first preset number of homogeneous land cover samples to ensure that the samples are free from interference from mixed land cover.

[0014] Optical sensor calibration specifically involves calibrating the first and second optical sensors for the same type of wavelength bands. After obtaining the spectral response curves of the first and second optical sensors, a spectral response function fitting method is used, with the wavelength band of the second optical sensor as the reference, and a linear fitting formula is applied. ,in, To calibrate the reflectivity, , where is the original reflectance, and a and b are fitting coefficients. The reflectance of the corresponding band of the first optical sensor is mapped to the reference band of the second optical sensor, so that the reflectance error between the bands of the first optical sensor and the second optical sensor is less than a preset threshold.

[0015] The calibration of synthetic aperture radar (SAR) and optical features involves establishing a linear regression model for the backscattering coefficient of SAR and the normalized vegetation index of the optical sensor to achieve dimensional unification of SAR features and optical features.

[0016] The calibration verification process involves randomly selecting a second preset number of non-sample homogeneous material regions to verify the consistency of cross-sensor features after calibration. If the error exceeds a preset threshold, the fitting coefficients are readjusted until the requirements are met.

[0017] In one possible implementation, the method further includes:

[0018] Based on the constructed segmented image database, a dynamic association map is built, including core features, associated features, and feature constraints. Among them, core features are the key features that determine the interpretation results in a specific scene; associated features are auxiliary features that affect the core features; and feature constraints are the effective value range of the features.

[0019] The weights of core features and associated features are calculated by determining the initial weights based on remote sensing technology, then using random forest for feature importance analysis, inputting multiple sets of feature combinations and interpretation results, training and optimizing the initial weights, calculating the contribution of each associated feature to the core feature, and adjusting the weights to the optimal level.

[0020] A preset graph database is used to store the dynamic association graph. Nodes include feature ID, feature name, scene label, and feature constraints. Edges include association type, weight value, and confidence level. The weight values ​​and constraints of the dynamic association graph are updated with new data collected by the feedback optimization layer at set time intervals to ensure the timeliness of the association logic.

[0021] In one possible implementation, the inference layer performs an interpretation task to obtain an inference result, including:

[0022] The inference layer incorporates remote sensing expertise during the pre-training phase;

[0023] The inference layer performs interpretation tasks, specifically land cover classification, object detection, change analysis, and visual question answering, to obtain inference results. The land cover classification task takes multimodal fusion features as input and outputs land cover type, proportion, and confidence level. The object detection task outputs the geographic coordinates, category, confidence level, and attributes of the target. The change analysis task compares multi-temporal fusion features and outputs a description of the change and its causes. The visual question answering task generates professional answers.

[0024] In one possible implementation, the method further includes:

[0025] After the user inputs the command, the current task scenario and core requirements are determined by analyzing the command task mapping in the detailed text database.

[0026] The dynamic association graph is invoked to extract the core features and associated feature sets of the current task scenario based on the current task scenario and core requirements, and to calculate the task contribution of each core feature and each associated feature.

[0027] Based on the task contribution of each core feature and each associated feature, dynamic weights are assigned to each core feature and each associated feature.

[0028] During inference, one or more target features are selected and loaded based on the dynamic weights of each core feature and each related feature.

[0029] The ranking effect is verified by calculating the reasoning time and accuracy after the reasoning is completed. If the time is greater than the preset time threshold or the accuracy is less than the preset accuracy threshold, the calculation parameters of the task contribution are readjusted.

[0030] In one possible implementation, the review layer verifies the accuracy of the inference results obtained by the inference layer through a pre-built scenario-based professional sub-network, including:

[0031] The review layer invokes the professional sub-network of the current task scenario in the pre-built scenario-based professional sub-network;

[0032] The review layer inputs the inference result obtained by the inference layer into the professional sub-network of the current task scenario. The professional sub-network of the current task scenario calls the professional knowledge base for cross-validation. If the validation passes, the validated inference result is provided to the output layer. If the validation fails, feature data is supplemented according to the validation situation and the inference is returned to the inference layer for re-inference.

[0033] In one possible implementation, the feedback optimization layer, based on human feedback and targeting the contextual rewards of the remote sensing task, optimizes the initial remote sensing large language model, including:

[0034] The feedback optimization layer optimizes the initial remote sensing large language model through multi-dimensional human feedback collection, feedback quality control and classification labeling, feedback-based training reward model and fine-tuning strategy model;

[0035] The method further includes:

[0036] The feedback optimization layer supplements the subdivided image database with new features involved in the feedback.

[0037] The feedback optimization layer supplements the subdivided text database with new natural language instructions involved in the feedback.

[0038] Secondly, a device for improving the accuracy of remote sensing large language models based on segmented databases and human feedback is provided, the device comprising:

[0039] The building unit is used to construct an initial remote sensing large language model based on an open-source visual universal language model, adapted to multi-source remote sensing image data and natural language commands. The architecture of the initial remote sensing large language model includes a data layer, a fusion layer, an inference layer, and an output layer. The data layer collects multi-source remote sensing image data and natural language commands; the fusion layer realizes accurate alignment and fusion of cross-modal features to obtain multi-modal fused features; the inference layer performs interpretation tasks to obtain inference results.

[0040] An enhancement unit is used to add a preprocessing layer, a review layer, and a feedback optimization layer to the initial remote sensing large language model to obtain a remote sensing large language model with improved accuracy. The preprocessing layer processes multi-source remote sensing image data and natural language instructions to construct a segmented database. The review layer verifies the accuracy of the inference results obtained by the inference layer through a pre-constructed scenario-based professional sub-network. If the verification is successful, the verified inference results are provided to the output layer, which then outputs the verified inference results. The feedback optimization layer optimizes the initial remote sensing large language model based on human feedback and the scenario-based rewards for the remote sensing task.

[0041] Thirdly, an electronic device is provided, comprising a processor and a memory, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the remote sensing large language model accuracy improvement method based on segmented databases and human feedback as described above.

[0042] Using the above technical solutions, the remote sensing large language model accuracy improvement method, device, and electronic device based on a segmented database and human feedback provided in this application embodiment construct an initial remote sensing large language model and add a preprocessing layer, a review layer, and a feedback optimization layer to the initial remote sensing large language model. Specifically, the preprocessing layer processes multi-source remote sensing image data and natural language instructions to construct a segmented database, solving the problem that text descriptions are mostly coarse-grained and global scene descriptions, lacking accurate annotation of specific ground objects, fine attributes, spatial relationships, and professional connotations in the images, thus realizing a refined data system; the review layer verifies the inference layer through a pre-constructed scene-based professional sub-network. The accuracy of the obtained inference results is verified. If the verification is successful, the verified inference results are provided to the output layer, which then outputs the verified inference results, improving the professionalism, relevance, and security of the output. In addition, the feedback optimization layer optimizes the initial remote sensing large language model based on human feedback and the scenario-based rewards for remote sensing tasks. Here, human feedback is the core optimization target, and a two-dimensional evaluation system of expert feedback and user feedback is constructed. The feedback is transformed into reinforcement learning reward signals to fine-tune the model, so that the model output shifts from grammatically correct to professional and useful. Ultimately, the user satisfaction of the model output is increased to over 90%, and 85% of the results can be directly used for industry decision-making without secondary processing. Attached Figure Description

[0043] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0044] Figure 1 A flowchart is shown below illustrating the method for improving the accuracy of remote sensing large language models based on segmented databases and human feedback, as provided in an embodiment of this application.

[0045] Figure 2 This illustration shows a schematic diagram of the acquisition of multi-source input data provided in an embodiment of this application;

[0046] Figure 3 This illustration shows a schematic diagram of the specific framework for the refined preprocessing of remote sensing data provided in an embodiment of this application;

[0047] Figure 4 This illustration shows a schematic diagram of a specialized preprocessing framework for natural language instructions provided in an embodiment of this application;

[0048] Figure 5 This illustration shows a schematic diagram of a specific framework for cross-modal feature fusion and semantic association establishment provided in an embodiment of this application;

[0049] Figure 6 This illustration shows a schematic diagram of the core reasoning and task execution framework of the remote sensing big language model provided in an embodiment of this application;

[0050] Figure 7 This illustration shows a schematic diagram of the specific framework for result verification of the professional review layer provided in an embodiment of this application;

[0051] Figure 8 This paper illustrates a schematic diagram showing the various output formats of the preliminary results provided in the embodiments of this application;

[0052] Figure 9 This illustration shows a schematic diagram of a specific framework for a human feedback-driven model optimization loop provided in an embodiment of this application.

[0053] Figure 10 The diagram shows the structure of the remote sensing large language model accuracy improvement device based on a segmented database and human feedback provided in an embodiment of this application.

[0054] Figure 11 A structural diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0055] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0056] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the term "comprising" and its variations should be interpreted as open-ended terms meaning "including but not limited to."

[0057] To address the aforementioned technical problems, embodiments of this application provide a method for improving the accuracy of remote sensing large language models based on segmented databases and human feedback, such as... Figure 1 As shown, the method for improving the accuracy of remote sensing large language models based on segmented databases and human feedback may include the following steps S101 and S102:

[0058] Step S101: Based on the open-source visual universal language model, adapt to multi-source remote sensing image data and natural language commands to construct an initial remote sensing large language model. The architecture of the initial remote sensing large language model includes a data layer, a fusion layer, an inference layer, and an output layer. The data layer collects multi-source remote sensing image data and natural language commands; the fusion layer realizes accurate alignment and fusion of cross-modal features to obtain multi-modal fused features; the inference layer performs interpretation tasks to obtain inference results.

[0059] In this step, the visual general language model can be an open-source model such as VisualGLM (Visual General Language Model)-6B, and this embodiment does not limit it.

[0060] Multi-source remote sensing image data may include optical data, SAR (Synthetic Aperture Radar) data, hyperspectral data, time-series data, etc., and this embodiment does not limit this.

[0061] Natural language commands can include expert commands and user commands. Expert commands are professional commands proposed by different remote sensing experts, while user commands are business commands from users in industries such as agriculture, environmental protection, and emergency response.

[0062] Step S102 involves adding a preprocessing layer, a review layer, and a feedback optimization layer to the initial remote sensing large language model to obtain a remote sensing large language model with improved accuracy. The preprocessing layer processes multi-source remote sensing image data and natural language instructions to construct a segmented database. The review layer verifies the accuracy of the inference results obtained by the inference layer through a pre-constructed scenario-based professional sub-network. If the verification is successful, the verified inference results are provided to the output layer, which then outputs the verified inference results. The feedback optimization layer optimizes the initial remote sensing large language model based on human feedback and the scenario-based rewards for remote sensing tasks.

[0063] This embodiment constructs an initial remote sensing large language model and adds a preprocessing layer, a review layer, and a feedback optimization layer to it. Specifically, the preprocessing layer processes multi-source remote sensing image data and natural language commands, constructs a segmented database, and addresses the problem that text descriptions are mostly coarse-grained and global scene descriptions, lacking precise annotation of specific ground objects, fine attributes, spatial relationships, and professional connotations in the images, thus achieving a refined data system. In the review layer, the accuracy of the inference results obtained from the inference layer is verified through a pre-constructed scene-based professional sub-network. If the verification passes, the verified inference is... The results are provided to the output layer, which outputs the validated inference results, improving the professionalism, relevance, and security of the output. In addition, the feedback optimization layer optimizes the initial remote sensing large language model based on human feedback and the contextual rewards of remote sensing tasks. Here, human feedback is the core optimization target, and a two-dimensional evaluation system of expert feedback and user feedback is constructed. The feedback is transformed into reinforcement learning reward signals to fine-tune the model, so that the model output shifts from grammatical correctness to professional usefulness. Ultimately, the user satisfaction of the model output is increased to over 90%, and 85% of the results can be directly used for industry decision-making without secondary processing.

[0064] This application embodiment provides a possible implementation method. The segmentation database mentioned in step S102 may include a segmentation image database and a segmentation text database. The preprocessing layer in step S102 processes multi-source remote sensing image data and natural language instructions to construct the segmentation database, which may specifically include the following steps A1 and A2:

[0065] Step A1: The preprocessing layer preprocesses the multi-source remote sensing image data, performs cross-sensor feature calibration, and extracts multi-dimensional features to construct a subdivided image database that includes three dimensions: land cover type, task scene, and feature type. The constructed subdivided image database includes multiple fields such as feature ID (identifier), feature name, land cover type, scene label, feature parameters, image coordinates, and acquisition time.

[0066] Step A2: The preprocessing layer normalizes the natural language instructions using remote sensing terminology, parses and decomposes the instruction semantics, and constructs a subdivided text database that includes task type, core object, and spatiotemporal precision constraints.

[0067] This embodiment constructs a "segmented image database" and a "segmented text database," and adds cross-sensor feature calibration to map similar features from different sensors (such as optical, SAR, and hyperspectral) to a unified benchmark. This reduces cross-sensor feature errors while establishing a one-to-one mapping between visual features and fine semantics, meeting the industry's demand for fine interpretation.

[0068] This application provides a possible implementation method in which the cross-sensor feature calibration step may include the following steps B1 to B4:

[0069] Step B1, homogeneous material sample selection, specifically involves collecting multi-source data from the same area and selecting a first preset number of homogeneous material samples to ensure that the samples are free from interference from mixed ground features; the first preset number here is such as 30, etc., but this embodiment does not limit this;

[0070] Step B2, optical sensor calibration, specifically involves calibrating the first and second optical sensors for the same type of wavelength bands. After obtaining the spectral response curves of the first and second optical sensors, a spectral response function fitting method is used, with the wavelength band of the second optical sensor as the reference, and a linear fitting formula is applied. ,in, To calibrate the reflectivity, denoted as the original reflectance, and a and b as fitting coefficients, the reflectance of the corresponding band of the first optical sensor is mapped to the reference band of the second optical sensor, so that the reflectance error between the bands of the first and second optical sensors is less than a preset threshold; the preset threshold here is such as 5%, etc., but this embodiment does not limit it;

[0071] Step B3, Synthetic Aperture Radar (SAR) and Optical Feature Calibration, specifically, establishes a linear regression model for the backscattering coefficient of the SAR and the normalized vegetation index of the optical sensor to achieve dimensional unification of SAR features and optical features.

[0072] Step B4, calibration verification, specifically involves randomly selecting a second preset number of non-sample homogeneous material regions to verify the consistency of cross-sensor features after calibration. If the error is greater than a preset threshold, the fitting coefficient is readjusted until the requirements are met. Here, the second preset number is such as 10, etc., and this embodiment does not limit this.

[0073] This application provides a possible implementation method that introduces remote sensing professional corpus, such as textbook texts and journal articles, into the model pre-training stage, and adds a dynamic feature association mechanism. This involves constructing a dynamic association map of core features, associated features, and feature constraints. For example, in an agricultural scenario, NDVI (Normalized Difference Vegetation Index) is the core feature, and the associated features are soil moisture (weight 0.4), growth period (weight 0.3), and precipitation (weight 0.3). This enables the model to reason based on causal links (e.g., if NDVI decreases by 0.3 and soil moisture is <15%, then the crop is determined to be in drought), achieving an industry expert-level logical level. Specifically, this may include the following steps C1 to C3:

[0074] Step C1: Based on the constructed segmented image database, a dynamic association map is constructed, including core features, associated features, and feature constraints. Among them, core features are the key features that determine the interpretation results in a specific scene; associated features are auxiliary features that affect the core features; and feature constraints are the effective value range of the features.

[0075] Step C2: Calculate the weights of the core features and related features. Specifically, the initial weights are determined based on remote sensing technology. Then, random forest is used for feature importance analysis. Multiple sets of feature combinations and interpretation results are input to train and optimize the initial weights. The contribution of each related feature to the core feature is calculated, and the weights are adjusted to the optimal value.

[0076] Step C3 involves storing the dynamic association graph in a pre-defined graph database. Nodes include feature ID, feature name, scene label, and feature constraints, while edges include association type, weight value, and confidence level. The weight values ​​and constraints of the dynamic association graph are updated with new data collected by the feedback optimization layer at set time intervals to ensure the timeliness of the association logic.

[0077] This application embodiment provides a possible implementation method in which the inference layer performs the interpretation task in step S101 to obtain the inference result, which may specifically include the following steps D1 and D2:

[0078] Step D1: The inference layer incorporates remote sensing expertise during the pre-training phase.

[0079] Step D2: The inference layer performs interpretation tasks, specifically land cover classification, object detection, change analysis, and visual question answering, to obtain inference results. The land cover classification task takes multimodal fusion features as input and outputs land cover type, proportion, and confidence level. The object detection task outputs the geographic coordinates, category, confidence level, and attributes of the target. The change analysis task compares multi-temporal fusion features and outputs a description of the change and its causes. The visual question answering task generates a professional answer.

[0080] This embodiment significantly improves the accuracy, efficiency, automation level, and system generalization ability of remote sensing image interpretation by integrating remote sensing expertise into the pre-training stage and combining multiple interpretation tasks, realizing end-to-end intelligent analysis from raw data to semantic knowledge.

[0081] This application provides a possible implementation method, which can also adaptively prioritize features. Specifically, it can dynamically adjust feature weights according to the task scenario to avoid irrelevant features occupying resources and strengthen the contribution of key features. This can include the following steps E1 to E5:

[0082] Step E1: After the user inputs the command, the current task scenario and core requirements are determined by subdividing the command task mapping in the text database.

[0083] Step E2: Invoke the dynamic association graph, extract the core features and associated feature sets of the current task scenario based on the current task scenario and core requirements, and calculate the task contribution of each core feature and each associated feature.

[0084] Step E3: Assign dynamic weights to each core feature and each associated feature according to their task contribution.

[0085] Step E4: During inference, select one or more target features based on the dynamic weights of each core feature and each related feature, and load one or more target features.

[0086] Step E5, sorting effect verification, specifically involves calculating the inference time and accuracy after the inference is completed. If the time is greater than the preset time threshold or the accuracy is less than the preset accuracy threshold, the calculation parameters of the task contribution are readjusted.

[0087] This application embodiment provides a possible implementation method in which the review layer verifies the accuracy of the inference result obtained by the inference layer through a pre-constructed scenario-based professional sub-network in step S102. Specifically, this may include the following steps F1 and F2:

[0088] Step F1: The review layer calls the professional sub-network of the current task scenario in the pre-built scenario-based professional sub-network;

[0089] In step F2, the review layer inputs the inference result obtained by the inference layer into the professional sub-network of the current task scenario. The professional sub-network of the current task scenario calls the professional knowledge base for cross-validation. If the validation passes, the validated inference result is provided to the output layer; if the validation fails, feature data is supplemented according to the validation situation and the inference is returned to the inference layer for re-inference.

[0090] This embodiment verifies the accuracy of the inference results through a scenario-specific professional subnetwork, thus avoiding erroneous output.

[0091] This application embodiment provides a possible implementation method. In step S102, the feedback optimization layer is based on human feedback and optimizes the initial remote sensing large language model for the contextual rewards of the remote sensing task. Specifically, it may include the following steps G1 to G3:

[0092] Step G1: The feedback optimization layer optimizes the initial remote sensing large language model through multi-dimensional human feedback collection, feedback quality control and classification labeling, feedback-based training reward model and fine-tuning strategy model.

[0093] In step G2, the feedback optimization layer supplements the subdivided image database with the new features involved in the feedback;

[0094] In step G3, the feedback optimization layer supplements the segmented text database with the new natural language instructions involved in the feedback.

[0095] This embodiment achieves continuous model evolution through a feedback, reward, and training closed loop.

[0096] The above introduces Figure 1 The embodiments shown have various implementation methods for each step. The following will further explain the accuracy improvement method of remote sensing large language model based on subdivided database and human feedback in this application through specific embodiments.

[0097] This specific embodiment addresses the shortcomings of existing technologies by constructing a segmented database, introducing an RLHF (Reinforcement Learning from Human Feedback) mechanism, adding a professional review layer, and incorporating a "cross-sensor feature calibration module," a "dynamic feature association mechanism," and an "adaptive feature priority ranking." Its core advantages are as follows:

[0098] (i) A refined data system and composite cross-sensor calibration completely bridge the semantic and data error gap.

[0099] A segmented database is constructed, including a segmented image database (also known as a segmented image feature library) and a segmented text database (also known as a segmented text semantic library), and a cross-sensor feature calibration module is added: by fitting the 6S (Second Simulation of the Satellite Signal in the Solar Spectrum) radiative transfer model and spectral response function, the same type of features from different sensors (such as optical, SAR, hyperspectral) are mapped to a unified benchmark, which reduces cross-sensor feature errors and establishes a one-to-one mapping between visual features and fine semantics, meeting the industry's demand for fine interpretation.

[0100] (ii) Integrate domain knowledge with dynamic features to enhance professional reasoning and causal logic abilities.

[0101] In the model pre-training stage, remote sensing professional corpora, such as textbook texts and journal articles, are introduced, and a dynamic feature association mechanism is added. That is, a dynamic association map of core features, associated features, and feature constraints is constructed. For example, in the agricultural scenario, NDVI is the core feature, and the associated features are soil moisture (weight 0.4), growth period (weight 0.3), and precipitation (weight 0.3). This enables the model to reason based on causal links (such as if NDVI decreases by 0.3 and soil moisture is <15%, then the crop is determined to be in drought), reaching the level of industry expert-level logic.

[0102] (III) RLHF-driven optimization to align with real human preferences

[0103] With human feedback as the core optimization goal, a two-dimensional evaluation system of expert feedback and user feedback is constructed (the scoring dimensions include professionalism, accuracy, and practicality). Feedback is transformed into reinforcement learning reward signals. The model is fine-tuned through the PPO (Proximal Policy Optimization) algorithm, so that the model output shifts from grammatical correctness to professional usefulness. Ultimately, the user satisfaction of the model output is increased to over 90%, and 85% of the results can be directly used for industry decision-making without secondary processing.

[0104] (iv) Closed-loop iterative mechanism integrates adaptive feature ranking to improve the efficiency of adaptation and inference.

[0105] A closed loop of "data update - model fine-tuning - feedback collection" is established. Feedback on new scenarios or tasks is collected periodically to supplement new features (such as "spectral features of new photovoltaic power stations") into the subdivided database. The model is then fine-tuned using LoRA (Low-Rank Adaptation) (reducing the cost to 1 / 10 of the original full-scale fine-tuning) to achieve continuous model evolution and meet real-time requirements. Based on this, an adaptive feature priority ranking is added. Feature weights are dynamically adjusted according to the task scenario (such as meteorological monitoring and agricultural yield estimation). For example, slope and SAR coherence are prioritized for meteorological monitoring tasks, while NDVI and fertility period are prioritized for agricultural tasks, improving inference efficiency and the accuracy of key feature interpretation.

[0106] The specific implementation steps are as follows:

[0107] 1. Acquisition of multi-source input data

[0108] This step aims to collect raw data from multiple dimensions and scenarios, laying the foundation for subsequent construction of segmented databases, such as... Figure 2 As shown, it includes multi-source remote sensing image acquisition and natural language command acquisition, from the raw remote sensing data pool and initial command set to the data input interface.

[0109] 1.1 Multi-source remote sensing image acquisition

[0110] Optical remote sensing data: Data was acquired from WorldView-3 (0.5-meter resolution, covering visible to near-infrared bands), Sentinel-2 (10-meter resolution, 13 bands), and Landsat-8 (30-meter resolution, including thermal infrared bands), covering four major scenarios: urban, agricultural, meteorological, and ecological. Here, WorldView-3, Sentinel-2, and Landsat-8 are three Earth observation satellites with different characteristics and uses, each with its own focus in terms of resolution, band coverage, and application areas.

[0111] SAR data: Data was acquired from Sentinel-1 (C-band, dual polarization, resolution 5-20 meters) and ALOS-2 (L-band, high resolution 10 meters) for interpreting cloudy and rainy scenes and nighttime scenes. Sentinel-1 and ALOS-2 are two powerful radar remote sensing satellites, each with its own characteristics in terms of band, resolution, and application focus, together providing crucial data support for global Earth observation.

[0112] Hyperspectral data: Hyperion (242 bands, spectral resolution 10nm) data is acquired for the identification of fine-grained attributes of ground features (such as water quality and vegetation pests and diseases). Here, Hyperion is a hyperspectral imaging spectrometer that can provide a continuous, fine-grained spectral curve of up to 242 bands for each pixel on the Earth's surface.

[0113] Time series data: Collect Sentinel-2 data for a certain area for 12 consecutive months for analysis of dynamic changes in land features (such as crop growth and building construction).

[0114] 1.2 Natural Language Command Acquisition

[0115] Expert instructions: Collect professional instructions from various remote sensing experts.

[0116] User instructions: Collect business instructions from users in industries such as agriculture, environmental protection, and emergency response.

[0117] Instruction scale: The final initial instruction set will contain hundreds of thousands of instructions, covering nearly thirty subdivided task scenarios.

[0118] 2. Refined preprocessing of remote sensing data (constructing a detailed image feature library)

[0119] This step transforms the raw data into standardized features through data cleaning and correction, a cross-sensor feature calibration module, multi-dimensional feature extraction and subdivided image feature library construction, and a dynamic feature association mechanism. Figure 3 As shown.

[0120] 2.1 Data Cleaning and Correction

[0121] Standardized format: All remote sensing data were converted to GeoTIFF (Geospatial Tagged Image File Format), with the coordinate system being WGS84 (World Geodetic System 1984).

[0122] Resolution matching: Low-resolution data (such as Landsat-8) is resampled to 10-meter resolution using bilinear interpolation, consistent with Sentinel-2.

[0123] Denoising: Lee filtering (a classic adaptive filtering algorithm with a window size of 5×5) is used to suppress speckle noise in SAR images, while Gaussian filtering (standard deviation 0.5) is used to remove Gaussian noise in optical images.

[0124] Cloud and fog removal: The Fmask (Function of mask) algorithm is used to identify cloud and fog areas, and the pixel values ​​of the cloud-covered areas are repaired by the peripheral pixel interpolation method.

[0125] Radiation correction: The 6S model is used to convert the original DN values ​​(Digital Number) into atmospheric apparent reflectance and surface reflectance in sequence, eliminating the effects of atmospheric scattering and absorption.

[0126] 2.2 Cross-sensor feature calibration module

[0127] To address the issue of inconsistent characteristics among multi-source sensors, perform the following calibration steps:

[0128] Selection of homogeneous landform samples: Collect multi-source data (WorldView-3 optical data, Sentinel-1 SAR data, Hyperion hyperspectral data, etc.) from the same area (e.g., 10km×10km), and select more than 30 homogeneous landform samples (bare land, uniform farmland, calm water bodies, each sample area ≥100m×100m) to ensure that the samples are free from mixed landform interference;

[0129] Optical sensor calibration: For the same wavelength bands of WorldView-3 and Sentinel-2 (e.g., red band: 640-670nm for WorldView-3, 665nm for Sentinel-2), after obtaining the spectral response curves of the two sensors, a spectral response function fitting method was used.

[0130] Using the Sentinel-2 band as a baseline, a linear fitting formula was used. ,in, To calibrate the reflectivity, The original reflectance is given, and a and b are fitting coefficients. The reflectance of the corresponding band of WorldView-3 is mapped to the reference band of Sentinel-2, so that the reflectance error of the red band of the two sensors is ≤5%.

[0131] SAR and optical feature calibration: A linear regression model is established for the backscattering coefficient of Sentinel-1 SAR and the NDVI of Sentinel-2 to achieve dimensional unification of SAR features and optical features.

[0132] Calibration verification: Randomly select 10 non-sample homogeneous material regions to verify the consistency of cross-sensor features after calibration. If the error exceeds 5%, readjust the fitting coefficients until the requirements are met.

[0133] 2.3 Multidimensional Feature Extraction

[0134] Spatial features: The Canny edge detection algorithm (threshold 100-200) is used to extract ground feature boundaries, and texture features (contrast, entropy, and correlation) are calculated using a gray-level co-occurrence matrix (window size 3×3, stride 1). For example, the contrast of the building area is >500 and the entropy is >7, while the correlation of the vegetation area is >0.8.

[0135] Spectral characteristics: Calculate professional spectral indices, including NDVI, NDWI (Nomralized Difference Water Idnex), NDBI (Nomralized Difference Built-up Idnex), and SAVI (Soil Adjusted Vegetation Idnex), and store the numerical range of the indices and their corresponding land cover semantics (e.g., NDVI=0.6-0.8 corresponds to the period of vigorous vegetation growth).

[0136] Multi-temporal variation characteristics: CVA (Change Vector Analysis) is used to calculate the magnitude of change in time series data (e.g., "from March to September 2023, the NDVI change magnitude in a certain area > 0.4 corresponds to significant vegetation growth"). Time series data is decomposed using STL (Seasonal and Trend decomposition using Loess) to extract trend terms (e.g., year-by-year increase in building area) and seasonal terms (e.g., seasonal crop growth).

[0137] 2.4 Construction of Subdivided Image Feature Library

[0138] Classification Dimensions: Data is stored according to three dimensions: feature type, task scenario, and feature type, as shown in Table 1 below:

[0139] Table 1 is categorized and stored according to three dimensions: land cover type, task scenario, and feature type.

[0140]

[0141] Storage format: It adopts PostgreSQL (Object Relational Database Management System) and PostGIS (Spatial Database Extension) database, supports spatial indexing (such as R tree index) and fast feature retrieval. A single feature record contains fields such as feature ID, feature name, land cover type, scene label, feature parameters, image coordinates, and acquisition time.

[0142] 2.5 Dynamic Feature Association Mechanism

[0143] Based on a segmented image feature library, a dynamic correlation graph is constructed, consisting of "core features - associated features - feature constraints," with the following specific meanings:

[0144] Key features: The key features that determine the interpretation results in a specific scenario (such as NDVI for agricultural yield estimation, slope of landslides).

[0145] Related features: Auxiliary features that influence the core features (such as soil moisture and precipitation in agricultural scenarios, and SAR coherence in meteorological scenarios).

[0146] Feature constraints: the effective range of feature values ​​(e.g., NDVI=0.6-0.8 corresponds to the vigorous growth period of vegetation).

[0147] Then, the corresponding association weights are calculated, and the initial weights are determined based on remote sensing technology (e.g., in the agricultural scenario: NDVI weight 0.5, soil moisture weight 0.3, precipitation weight 0.2). Then, random forest feature importance analysis is used, and multiple sets of feature combinations and interpretation results are input to train and optimize the initial weights. The contribution of each associated feature to the core feature is calculated, and the weights are adjusted to the optimal (e.g., after optimization: NDVI weight 0.45, soil moisture weight 0.35, precipitation weight 0.2).

[0148] After calculation, the association graph is stored in the Neo4j graph database. Nodes include feature ID, feature name, scene label, and constraints. Edges include association type, weight value, and confidence level. The graph weights and constraints are updated monthly by using new data collected from the feedback optimization layer (such as the association between NDVI and biomass of new crops) to ensure the timeliness of the association logic.

[0149] 3. Specialized preprocessing of natural language instructions (building a segmented text semantic library)

[0150] This step eliminates instruction ambiguity and constructs a precise semantic system through terminology normalization, semantic parsing, and structured storage, such as... Figure 4 As shown.

[0151] 3.1 Normalization of Remote Sensing Terminology

[0152] Construct a remote sensing terminology database: Include core terms in the field of remote sensing, including conceptual terms (such as "radiometric calibration" and "coherence"), index terms (such as "NDVI" and "NDWI"), and standard terms (such as "GB / T38597-2020"). Each term is accompanied by a standardized definition (such as "NDVI: Normalized Difference Vegetation Index, used to characterize vegetation cover and growth status, with a value range of -1 to 1").

[0153] Term disambiguation: A BERT-based term recognition model is used to disambiguate ambiguous instructions. For example, "monitoring growth" is mapped to "monitoring crop biomass changes (based on NDVI time series data)" in the agricultural scenario and "monitoring tree diameter at breast height and tree height growth (based on high-resolution images)" in the forestry scenario.

[0154] 3.2 Analysis and Decomposition of Instruction Semantics

[0155] Core element extraction: Using dependency parsing and named entity recognition technology, three key elements—task type, core object, and constraints—are extracted from the instruction. An example is shown below:

[0156] Original instruction: "Identify the distribution of wheat rust in farmland in a certain region in a certain year, with a required accuracy of boundary error ≤ 3 meters";

[0157] Analysis results: Task type = "Pest and disease target detection", core object = "Wheat rust area", constraints = "Time: a certain year; Space: a certain region; Accuracy: ≤3 meters";

[0158] Ambiguity handling: Supplement default rules for instructions without explicit constraints (e.g., if no precision is specified, the default boundary error is ≤5 meters; if no time is specified, the default is the most recent year).

[0159] 3.3 Construction of a Subdivided Text Semantic Library

[0160] Structured storage: Parsed instructions are stored in JSON (JavaScript Object Notation, a lightweight, text-based, open data interchange format) format.

[0161] Instruction expansion: Generate synonymous instructions based on the semantic library (such as generating "extract traffic network" and "mark road boundaries" from "identify road"), expanding the initial instruction library to cover more semantic scenarios.

[0162] 4. Cross-modal feature fusion and semantic association establishment

[0163] This step utilizes the Transformer architecture to achieve deep fusion of remote sensing features and text semantics, such as... Figure 5As shown, it includes a feature input layer, an attention interaction layer, and a feature fusion layer.

[0164] 4.1 Feature Input

[0165] Remote sensing features: Spatial, spectral, and multi-temporal joint features are extracted from the subdivided image feature library and mapped to a 768-dimensional feature vector through a fully connected layer (consistent with the LLM word embedding dimension).

[0166] Text semantics: Extract instruction word embeddings from the segmented text semantic library (encoded using the BERT-base model) to generate a semantic vector with a dimension of 768.

[0167] 4.2 Attention Interaction

[0168] A 12-head multi-head self-attention mechanism is used to calculate the attention weights between remote sensing features and text semantics. For example, the weight value of "water body spectral features (NDWI>0.3)" is high with the semantic meaning of "identify rivers" in the instruction, indicating high correlation, while the weight value of "identify buildings" is low, indicating low correlation.

[0169] By using a cross-attention layer, remote sensing features are focused on the core objects of the command (such as increasing the weight of "NDVI<0.5" in the remote sensing features of the command "identify wheat rust").

[0170] 4.3 Feature Fusion

[0171] A Transformer Encoder-Decoder structure is adopted, where the Encoder processes remote sensing features and the Decoder processes text semantics. A fusion layer (containing layer normalization and residual connections) encodes the two types of features into a 1024-dimensional multimodal fusion feature vector. The fusion feature contains the correlation information between visual and textual information, serving as input for model inference.

[0172] 5. Core Reasoning and Task Execution of Remote Sensing Large Language Model

[0173] This step utilizes a large model incorporating specialized knowledge to perform various remote sensing tasks. Based on the VisualGLM-6B model, it adapts to multimodal fusion features and employs LoRA technology (rank 8, learning rate 1e-4) for lightweight fine-tuning, avoiding the high cost of full parameter updates. Figure 6 As shown, this includes the model infrastructure and task execution output.

[0174] 5.1 Integration of professional knowledge

[0175] Pre-training phase: Lightweight fine-tuning pre-training enhancement, adding 5 million remote sensing professional texts (including textbooks, papers, standards, etc.) to the VisualGLM-6B pre-training corpus to enable the model to master relevant remote sensing domain knowledge.

[0176] Reasoning phase: Add specialized sub-modules, for example:

[0177] Agricultural task: Embed the WOFOST crop growth model and combine it with NDVI time-series data and growth period to estimate yield. The WOFOST crop growth model here is a dynamic interpretable model specifically designed to simulate the growth and development of annual crops under specific soil and climate conditions.

[0178] 5.2 Task Execution Logic

[0179] Land feature classification task: Input multimodal fusion features, and the model outputs land feature type, proportion and confidence level, such as "In this area, cultivated land is 65% (confidence level 0.95), buildings are 20% (confidence level 0.92), and water bodies are 15% (confidence level 0.98)".

[0180] Target detection task: Output the geographic coordinates, category, confidence level, and attributes of the target.

[0181] Change analysis task: Compare multi-temporal fusion features and output a description of the changes and their causes.

[0182] Visual question answering task: Generate professional answers (e.g., "The average NDVI of the farmland is 0.72, it is in the heading stage, the estimated yield is 550 kg per mu, and the error is ≤8%").

[0183] 5.3 Adaptive Feature Priority Ranking

[0184] Dynamically adjust feature weights based on the task scenario to avoid irrelevant features consuming resources and strengthen the contribution of key features. The steps are as follows:

[0185] Task scenario recognition: After the user inputs a command, the current task scenario and core requirements are determined by subdividing the command task mapping in the text semantic library.

[0186] Feature contribution calculation: The dynamic association graph is invoked to extract the set of "core features - associated features" for the current scene. The XGBoost (eXtreme Gradient Boosting, a high-performance machine learning algorithm based on the gradient boosting framework) algorithm is used to calculate the task contribution of each feature. An example is shown in Table 2 below:

[0187] Table 2. Task contribution of each feature in different task scenarios.

[0188]

[0189] Dynamic weight allocation and reasoning: Assign feature weights based on contribution scores (e.g., for meteorological monitoring tasks: slope is 40%, SAR coherence is 35%, and NDVI is 25%).

[0190] During inference, high-priority features are loaded first (e.g., calculate slope anomaly areas first, and then filter areas that meet the SAR coherence requirements). Low-priority features are only called when high-priority features cannot determine the results (e.g., NDVI is only used to assist in verifying the vegetation damage of landslide bodies).

[0191] Ranking effect verification: After the inference is completed, the inference time and accuracy are statistically analyzed (e.g., for meteorological detection tasks: before ranking, the time was 120s and the accuracy was 92%; after ranking, the time was 84s and the accuracy was 95%). If the time exceeds the threshold or the accuracy does not meet the standard, the contribution calculation parameters are readjusted (e.g., by increasing the sample size).

[0192] 6. Verification of results by the professional review layer

[0193] This step verifies the accuracy of the inference results through scenario-specific specialized subnetworks to avoid erroneous outputs, such as... Figure 7 As shown.

[0194] 6.1 Construction of Specialized Subnetworks

[0195] For different task scenarios, construct dedicated review sub-networks, for example:

[0196] Agricultural sub-network: Integrates WOFOST crop model and NY / T3921-2021 standard, inputs "yield estimation results, NDVI data, growth period", and outputs theoretical yield and error assessment.

[0197] Meteorological monitoring sub-network: integrates building damage model and GB / T38597-2020 standard, inputs "damaged area, building texture, slope", outputs damage level (slight, moderate, severe) and verification results.

[0198] Environmental sub-network: integrates water quality evaluation models (such as the single-factor index method), inputs "chlorophyll concentration, water body spectrum", and outputs water quality grade (Class IV).

[0199] 6.2 Review Process

[0200] Step 1: Input the preliminary results from Step 5 into the corresponding professional sub-network.

[0201] Step 2: The sub-network calls upon the professional knowledge base for cross-validation. For example, the agricultural sub-network uses the WOFOST model to calculate the theoretical yield of 520 kg / mu for the estimated wheat yield of 550 kg / mu, with an error of 5.8% ≤ 10%, and the verification is approved.

[0202] Step 3: If the review is successful, proceed to step 7 for output; if the review fails (e.g., error > 10%), return to step 5 and adjust the reasoning logic (e.g., add the "soil moisture" feature).

[0203] The review criteria here can be that the result must simultaneously meet three conditions: "error ≤ 10%", "complies with the corresponding industry standard", and "no errors in professional terminology".

[0204] 7. Multiple output formats for preliminary results

[0205] This step offers three types of output to meet different user needs: text, structured data, and visualization. Figure 8 As shown:

[0206] 7.1 Natural Language Response

[0207] Generate professional yet accessible text, including results, analysis, and recommendations.

[0208] 7.2 Structured Data

[0209] Provides data in JSON or Excel format for easy secondary analysis by users.

[0210] 7.3 Visualization Results

[0211] Land feature classification map: rendered using ArcGIS, with color scheme conforming to industry conventions (farmland: light yellow, buildings: dark gray, water bodies: dark blue), and labeled with land feature types and proportions. ArcGIS here refers to a Geographic Information System platform used to collect, manage, analyze, and visualize geographic location-related data, and to support map-based decision-making and information sharing.

[0212] Change detection heatmap: The intensity of change is represented by a red-yellow-green gradient (red: new additions, green: decreases, yellow: no change), with geographic boundaries superimposed.

[0213] Vector graphics: Outputs .shp format feature boundary files, including "feature ID, type, and attribute" fields, which can be imported into GIS software for further analysis.

[0214] 8. Human Feedback-Driven Model Optimization Loop (RLHF Layer)

[0215] This step achieves continuous model evolution through feedback, rewards, and a training loop, such as... Figure 9 As shown.

[0216] 8.1 Multi-dimensional Human Feedback Collection

[0217] Expert feedback: Remote sensing experts were invited to rate the output of step 7 (1-5 points), with dimensions including "professionalism" (precision of terminology, correctness of knowledge), "accuracy" (error from field verification), and "completeness" (whether it covers the core needs), and suggestions for improvement were made (such as "small ponds with an area of ​​<0.1 hectares were missed, it is recommended to add small water body characteristics").

[0218] User feedback: Collect industry users' ratings (1-5 points) for "usability" (whether it meets business needs) and "explainability" (whether it is easy to understand), as well as suggestions for improvement (such as adding other correlation analysis).

[0219] 8.2 Feedback Quality Control and Classification

[0220] Cross-validation: Feedback is reviewed by different experts. If the difference between the scores of two or more experts is ≥2 points, a third-party expert review is initiated to filter out low-quality feedback such as "subjective assumptions" and "incorrect judgments".

[0221] Categorization and labeling: Valid feedback is labeled into four categories: "factual errors (e.g., NDVI calculation errors)," "missing details (e.g., missing small features)," "optimization suggestions (e.g., adding new features)," and "text errors (e.g., spelling errors)," forming a feedback dataset.

[0222] 8.3 Feedback-based bidirectional optimization (RLHF core)

[0223] 8.3.1 Training the reward model

[0224] Using the results and feedback from the feedback dataset as sample pairs (positive samples: score ≥ 4 points, negative samples: score ≤ 2 points), a BERT-based Reward Model (RM) is trained, enabling the RM to automatically predict reward values ​​(0-10 points, with 10 points being optimal) based on the model output. Here, BERT stands for Bidirectional Encoder Representations from Transformers, and RM stands for Reward Model.

[0225] 8.3.2 Fine-tuning strategy model

[0226] Using the large model from step 5 as the policy model, the multimodal fusion feature generation result is input, the reward value is given by RM, and the PPO algorithm is used to update the policy model parameters (learning rate 1e-5, batch size 32, discount factor 0.95). At the same time, KL divergence constraint (KL coefficient 0.01) is introduced to avoid unstable model output.

[0227] 8.3.3 Data Optimization

[0228] The "new feature requests (such as 'post-disaster building ruin textures')" mentioned in the feedback will be added to the subdivided image feature library.

[0229] Add "high-quality instructions (such as 'high-precision farmland boundary extraction, error ≤ 2 meters')" to the subdivided text semantic library.

[0230] Iteration cycle: RM update, strategy model fine-tuning and database supplementation are completed monthly to achieve bidirectional evolution of model and data.

[0231] Furthermore, the differences between existing related technologies and the solution of this embodiment are described as follows:

[0232] 1) Existing segmented image databases only achieve static classification and storage of ground objects, scenes, and features. They fail to address cross-sensor system errors in multi-source remote sensing data (such as optical data, SAR data, and hyperspectral data). (Differences in spectral responses between different sensors lead to reflectance deviations of up to 15%-20% for the same ground object.) Furthermore, they do not establish dynamic correlations between features (such as the causal link between vegetation NDVI changes and soil moisture and precipitation), resulting in poor feature reusability and low accuracy in multi-source data fusion. Existing databases only store raw features. The solution in this embodiment improves the consistency of multi-source data features through cross-sensor calibration, addressing the pain point of "feature conflicts" during multi-source data fusion.

[0233] 2) Existing segmented text databases only normalize remote sensing terminology and do not construct a semantic-to-task-to-technical-parameter association system (e.g., the "wheat yield estimation" command only maps to "agricultural scenario," without automatically associating it with WOFOST model parameters such as growth period, NDVI threshold range, and sampling density requirements). This necessitates manual addition of numerous parameters to execute the task, failing to achieve end-to-end connection between commands and execution. Existing text databases only parse semantic elements. The solution in this embodiment achieves automatic mapping from semantics to parameters, reducing manual parameter addition steps and improving task execution efficiency.

[0234] 3) Existing human feedback mechanisms, employing a general RLHF framework (such as simple expert scoring plus PPO fine-tuning), fail to optimize for the specific scenarios of remote sensing tasks (e.g., meteorological monitoring tasks prioritize "accuracy" and "timeliness," while agricultural tasks prioritize "precision" and "interpretability"). This leads to a disconnect between reward signals and task requirements, resulting in insufficient efficiency in improving model accuracy. Existing RLHF uses fixed reward weights. The solution in this embodiment uses scenario-based rewards to differentiate reward weights for different tasks. Through adaptive feature prioritization, targeted performance improvements are significantly differentiated.

[0235] It should be noted that the sequence numbers of the steps in the above embodiments do not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In practical applications, all the above possible implementation methods can be arbitrarily combined in a combined manner to form possible embodiments of this application, which will not be described in detail here.

[0236] Based on the methods for improving the accuracy of remote sensing large language models based on subdivided databases and human feedback provided in the above embodiments, and based on the same inventive concept, this application also provides a device for improving the accuracy of remote sensing large language models based on subdivided databases and human feedback.

[0237] Figure 10 This is a structural diagram of the remote sensing large language model accuracy improvement device based on a segmented database and human feedback provided in an embodiment of this application. Figure 10 As shown, the remote sensing large language model accuracy improvement device based on subdivided databases and human feedback may specifically include a construction unit 1010 and an improvement unit 1020.

[0238] Building unit 1010 is used to construct an initial remote sensing large language model based on an open-source visual universal language model, adapted to multi-source remote sensing image data and natural language commands. The architecture of the initial remote sensing large language model includes a data layer, a fusion layer, an inference layer, and an output layer. The data layer collects multi-source remote sensing image data and natural language commands; the fusion layer realizes accurate alignment and fusion of cross-modal features to obtain multi-modal fused features; the inference layer performs interpretation tasks to obtain inference results.

[0239] The enhancement unit 1020 is used to add a preprocessing layer, a review layer, and a feedback optimization layer to the initial remote sensing large language model to obtain a remote sensing large language model with improved accuracy. The preprocessing layer processes multi-source remote sensing image data and natural language instructions to construct a segmented database. The review layer verifies the accuracy of the inference results obtained by the inference layer through a pre-constructed scenario-based professional sub-network. If the verification is successful, the verified inference results are provided to the output layer, which then outputs the verified inference results. The feedback optimization layer optimizes the initial remote sensing large language model based on human feedback and the scenario-based rewards for the remote sensing task.

[0240] This application embodiment provides a possible implementation, wherein the segmentation database includes a segmentation image database and a segmentation text database, and the lifting unit 1020 is further configured to:

[0241] The preprocessing layer performs data preprocessing, cross-sensor feature calibration, and multi-dimensional feature extraction on multi-source remote sensing image data to construct a subdivided image database with three dimensions: land cover type, task scene, and feature type. The constructed subdivided image database includes multiple fields such as feature ID, feature name, land cover type, scene label, feature parameters, image coordinates, and acquisition time.

[0242] The preprocessing layer normalizes natural language commands using remote sensing terminology, parses and decomposes command semantics, and constructs a segmented text database that includes task type, core object, and spatiotemporal precision constraints.

[0243] This application embodiment provides a possible implementation, wherein the lifting unit 1020 is further configured to:

[0244] The selection of homogeneous land cover samples involves collecting multi-source data from the same area and selecting a first preset number of homogeneous land cover samples to ensure that the samples are free from interference from mixed land cover.

[0245] Optical sensor calibration specifically involves calibrating the first and second optical sensors for the same type of wavelength bands. After obtaining the spectral response curves of the first and second optical sensors, a spectral response function fitting method is used, with the wavelength band of the second optical sensor as the reference, and a linear fitting formula is applied. ,in, To calibrate the reflectivity, , where is the original reflectance, and a and b are fitting coefficients. The reflectance of the corresponding band of the first optical sensor is mapped to the reference band of the second optical sensor, so that the reflectance error between the bands of the first optical sensor and the second optical sensor is less than a preset threshold.

[0246] The calibration of synthetic aperture radar (SAR) and optical features involves establishing a linear regression model for the backscattering coefficient of SAR and the normalized vegetation index of the optical sensor to achieve dimensional unification of SAR features and optical features.

[0247] The calibration verification process involves randomly selecting a second preset number of non-sample homogeneous material regions to verify the consistency of cross-sensor features after calibration. If the error exceeds a preset threshold, the fitting coefficients are readjusted until the requirements are met.

[0248] This application embodiment provides a possible implementation, wherein the lifting unit 1020 is further configured to:

[0249] Based on the constructed segmented image database, a dynamic association map is built, including core features, associated features, and feature constraints. Among them, core features are the key features that determine the interpretation results in a specific scene; associated features are auxiliary features that affect the core features; and feature constraints are the effective value range of the features.

[0250] The weights of core features and associated features are calculated by determining the initial weights based on remote sensing technology, then using random forest for feature importance analysis, inputting multiple sets of feature combinations and interpretation results, training and optimizing the initial weights, calculating the contribution of each associated feature to the core feature, and adjusting the weights to the optimal level.

[0251] A preset graph database is used to store the dynamic association graph. Nodes include feature ID, feature name, scene label, and feature constraints. Edges include association type, weight value, and confidence level. The weight values ​​and constraints of the dynamic association graph are updated with new data collected by the feedback optimization layer at set time intervals to ensure the timeliness of the association logic.

[0252] This application embodiment provides a possible implementation, wherein the construction unit 1010 is further configured to:

[0253] The inference layer incorporates remote sensing expertise during the pre-training phase;

[0254] The inference layer performs interpretation tasks, specifically land cover classification, object detection, change analysis, and visual question answering, to obtain inference results. The land cover classification task takes multimodal fusion features as input and outputs land cover type, proportion, and confidence level. The object detection task outputs the geographic coordinates, category, confidence level, and attributes of the target. The change analysis task compares multi-temporal fusion features and outputs a description of the change and its causes. The visual question answering task generates professional answers.

[0255] This application embodiment provides a possible implementation, wherein the construction unit 1010 is further configured to:

[0256] After the user inputs the command, the current task scenario and core requirements are determined by analyzing the command task mapping in the detailed text database.

[0257] The dynamic association graph is invoked to extract the core features and associated feature sets of the current task scenario based on the current task scenario and core requirements, and to calculate the task contribution of each core feature and each associated feature.

[0258] Based on the task contribution of each core feature and each associated feature, dynamic weights are assigned to each core feature and each associated feature.

[0259] During inference, one or more target features are selected and loaded based on the dynamic weights of each core feature and each related feature.

[0260] The ranking effect is verified by calculating the reasoning time and accuracy after the reasoning is completed. If the time is greater than the preset time threshold or the accuracy is less than the preset accuracy threshold, the calculation parameters of the task contribution are readjusted.

[0261] This application embodiment provides a possible implementation, wherein the lifting unit 1020 is further configured to:

[0262] The review layer invokes the professional sub-network of the current task scenario in the pre-built scenario-based professional sub-network;

[0263] The review layer inputs the inference result obtained by the inference layer into the professional sub-network of the current task scenario. The professional sub-network of the current task scenario calls the professional knowledge base for cross-validation. If the validation passes, the validated inference result is provided to the output layer. If the validation fails, feature data is supplemented according to the validation situation and the inference is returned to the inference layer for re-inference.

[0264] This application embodiment provides a possible implementation, wherein the lifting unit 1020 is further configured to:

[0265] The feedback optimization layer optimizes the initial remote sensing large language model through multi-dimensional human feedback collection, feedback quality control and classification labeling, feedback-based training reward model and fine-tuning strategy model;

[0266] The method further includes:

[0267] The feedback optimization layer supplements the subdivided image database with new features involved in the feedback.

[0268] The feedback optimization layer supplements the subdivided text database with new natural language instructions involved in the feedback.

[0269] Based on the same inventive concept, embodiments of this application also provide an electronic device, including a processor and a memory, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the remote sensing large language model accuracy improvement method based on segmented database and human feedback of any of the above embodiments.

[0270] In an exemplary embodiment, an electronic device is provided, such as Figure 11 As shown, Figure 11 The illustrated electronic device 1100 includes a processor 1101 and a memory 1103. The processor 1101 and the memory 1103 are connected, for example, via a bus 1102. Optionally, the electronic device 1100 may also include a transceiver 1104. It should be noted that in practical applications, the transceiver 1104 is not limited to one unit, and the structure of this electronic device 1100 does not constitute a limitation on the embodiments of this application.

[0271] Processor 1101 may be a CPU (Central Processing Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 1101 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0272] Bus 1102 may include a pathway for transmitting information between the aforementioned components. Bus 1102 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 1102 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 11 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0273] The memory 1103 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0274] The memory 1103 is used to store computer program code that executes the scheme of this application, and its execution is controlled by the processor 1101. The processor 1101 is used to execute the computer program code stored in the memory 1103 to implement the content shown in the foregoing method embodiments.

[0275] Among them, electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 11 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0276] Those skilled in the art will clearly understand that the specific working process of the systems, devices, and modules described above can be referred to the corresponding process in the foregoing method embodiments. For the sake of brevity, it will not be repeated here.

[0277] Those skilled in the art will understand that the technical solution of this application, or all or part of it, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several program instructions to cause an electronic device (e.g., a personal computer, server, or network device) to execute all or part of the steps of the methods described in the embodiments of this application when running the program instructions. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0278] Alternatively, all or part of the steps of the foregoing method embodiments can be implemented by hardware (such as electronic devices like personal computers, servers, or network devices) associated with program instructions. The program instructions can be stored in a computer-readable storage medium. When the program instructions are executed by the processor of the electronic device, the electronic device executes all or part of the steps of the methods described in the embodiments of this application.

[0279] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that within the spirit and principles of this application, modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the corresponding technical solutions to leave the protection scope of this application.

Claims

1. A method for improving the accuracy of remote sensing large language models based on segmented databases and human feedback, characterized in that, The method includes: Based on an open-source visual universal language model, and adapted to multi-source remote sensing image data and natural language commands, an initial remote sensing large language model is constructed. The architecture of the initial remote sensing large language model includes a data layer, a fusion layer, an inference layer, and an output layer. The data layer collects multi-source remote sensing image data and natural language commands; the fusion layer achieves accurate alignment and fusion of cross-modal features to obtain multi-modal fused features; and the inference layer performs interpretation tasks to obtain inference results. By adding a preprocessing layer, a review layer, and a feedback optimization layer to the initial remote sensing large language model, an improved remote sensing large language model is obtained. The preprocessing layer processes multi-source remote sensing image data and natural language commands to construct a segmented database. The review layer verifies the accuracy of the inference results obtained by the inference layer through a pre-constructed scenario-based professional sub-network. If the verification is successful, the verified inference result is provided to the output layer, which then outputs the verified inference result. The feedback optimization layer optimizes the initial remote sensing large language model based on human feedback and the scenario-based rewards for remote sensing tasks. The review layer verifies the accuracy of the inference results obtained by the inference layer through a pre-constructed scenario-based professional sub-network, including: The review layer invokes the professional sub-network of the current task scenario in the pre-built scenario-based professional sub-network; wherein, the current task scenario is determined by inputting user instructions and then mapping the instructions and tasks in the subdivided text database. The review layer inputs the inference result obtained by the inference layer into the professional sub-network of the current task scenario. The professional sub-network of the current task scenario calls the professional knowledge base for cross-validation. If the validation passes, the validated inference result is provided to the output layer. If the validation fails, feature data is supplemented according to the validation situation and the inference is returned to the inference layer for re-inference.

2. The method according to claim 1, characterized in that, The segmented database includes a segmented image database and a segmented text database. The preprocessing layer processes multi-source remote sensing image data and natural language instructions to construct the segmented database, including: The preprocessing layer performs data preprocessing, cross-sensor feature calibration, and multi-dimensional feature extraction on multi-source remote sensing image data to construct a subdivided image database with three dimensions: land cover type, task scene, and feature type. The constructed subdivided image database includes multiple fields such as feature ID, feature name, land cover type, scene label, feature parameters, image coordinates, and acquisition time. The preprocessing layer normalizes natural language commands using remote sensing terminology, parses and decomposes command semantics, and constructs a segmented text database that includes task type, core object, and spatiotemporal precision constraints.

3. The method according to claim 2, characterized in that, The steps involved in cross-sensor feature calibration include: The selection of homogeneous land cover samples involves collecting multi-source data from the same area and selecting a first preset number of homogeneous land cover samples to ensure that the samples are free from interference from mixed land cover. Optical sensor calibration specifically involves calibrating the first and second optical sensors for the same type of wavelength bands. After obtaining the spectral response curves of the first and second optical sensors, a spectral response function fitting method is used, with the wavelength band of the second optical sensor as the reference, and a linear fitting formula is applied. ,in, To calibrate the reflectivity, , where is the original reflectance, and a and b are fitting coefficients. The reflectance of the corresponding band of the first optical sensor is mapped to the reference band of the second optical sensor, so that the reflectance error between the bands of the first optical sensor and the second optical sensor is less than a preset threshold. The calibration of synthetic aperture radar (SAR) and optical features involves establishing a linear regression model for the backscattering coefficient of SAR and the normalized vegetation index of the optical sensor to achieve dimensional unification of SAR features and optical features. The calibration verification process involves randomly selecting a second preset number of non-sample homogeneous material regions to verify the consistency of cross-sensor features after calibration. If the error exceeds a preset threshold, the fitting coefficients are readjusted until the requirements are met.

4. The method according to claim 2, characterized in that, The method further includes: Based on the constructed segmented image database, a dynamic association map is built, including core features, associated features, and feature constraints. Among them, core features are the key features that determine the interpretation results in a specific scene; associated features are auxiliary features that affect the core features; and feature constraints are the effective value range of the features. The weights of core features and associated features are calculated by determining the initial weights based on remote sensing technology, then using random forest for feature importance analysis, inputting multiple sets of feature combinations and interpretation results, training and optimizing the initial weights, calculating the contribution of each associated feature to the core feature, and adjusting the weights to the optimal level. A preset graph database is used to store the dynamic association graph. Nodes include feature ID, feature name, scene label, and feature constraints. Edges include association type, weight value, and confidence level. The weight values ​​and constraints of the dynamic association graph are updated with new data collected by the feedback optimization layer at set time intervals to ensure the timeliness of the association logic.

5. The method according to claim 4, characterized in that, The inference layer performs the interpretation task and obtains the inference results, including: The inference layer incorporates remote sensing expertise during the pre-training phase; The inference layer performs interpretation tasks, specifically land cover classification, object detection, change analysis, and visual question answering, to obtain inference results. The land cover classification task takes multimodal fusion features as input and outputs land cover type, proportion, and confidence level. The object detection task outputs the geographic coordinates, category, confidence level, and attributes of the target. The change analysis task compares multi-temporal fusion features and outputs a description of the change and its causes. The visual question answering task generates professional answers.

6. The method according to claim 5, characterized in that, The method further includes: After the user inputs the command, the current task scenario and core requirements are determined by analyzing the command task mapping in the detailed text database. The dynamic association graph is invoked to extract the core features and associated feature sets of the current task scenario based on the current task scenario and core requirements, and to calculate the task contribution of each core feature and each associated feature. Based on the task contribution of each core feature and each associated feature, dynamic weights are assigned to each core feature and each associated feature. During inference, one or more target features are selected and loaded based on the dynamic weights of each core feature and each related feature. The ranking effect is verified by calculating the reasoning time and accuracy after the reasoning is completed. If the time is greater than the preset time threshold or the accuracy is less than the preset accuracy threshold, the calculation parameters of the task contribution are readjusted.

7. The method according to claim 2, characterized in that, The feedback optimization layer, based on human feedback and targeting the contextual rewards of remote sensing tasks, optimizes the initial remote sensing large language model, including: The feedback optimization layer optimizes the initial remote sensing large language model through multi-dimensional human feedback collection, feedback quality control and classification labeling, feedback-based training reward model and fine-tuning strategy model; The method further includes: The feedback optimization layer supplements the subdivided image database with new features involved in the feedback. The feedback optimization layer supplements the subdivided text database with new natural language instructions involved in the feedback.

8. A device for improving the accuracy of remote sensing large language models based on segmented databases and human feedback, characterized in that, The device includes: The building unit is used to construct an initial remote sensing large language model based on an open-source visual universal language model, adapted to multi-source remote sensing image data and natural language commands. The architecture of the initial remote sensing large language model includes a data layer, a fusion layer, an inference layer, and an output layer. The data layer collects multi-source remote sensing image data and natural language commands; the fusion layer realizes accurate alignment and fusion of cross-modal features to obtain multi-modal fused features; the inference layer performs interpretation tasks to obtain inference results. An enhancement unit is used to add a preprocessing layer, a review layer, and a feedback optimization layer to the initial remote sensing large language model to obtain a remote sensing large language model with improved accuracy. The preprocessing layer processes multi-source remote sensing image data and natural language instructions to construct a segmented database. The review layer verifies the accuracy of the inference results obtained by the inference layer through a pre-constructed scenario-based professional sub-network. If the verification is successful, the verified inference results are provided to the output layer, which then outputs the verified inference results. The feedback optimization layer optimizes the initial remote sensing large language model based on human feedback and the scenario-based rewards for the remote sensing task. The lifting unit is also used for: The review layer invokes the professional sub-network of the current task scenario in the pre-built scenario-based professional sub-network; wherein, the current task scenario is determined by inputting user instructions and then mapping the instructions and tasks in the subdivided text database. The review layer inputs the inference result obtained by the inference layer into the professional sub-network of the current task scenario. The professional sub-network of the current task scenario calls the professional knowledge base for cross-validation. If the validation passes, the validated inference result is provided to the output layer. If the validation fails, feature data is supplemented according to the validation situation and the inference is returned to the inference layer for re-inference.

9. An electronic device, characterized in that, The system includes a processor and a memory, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the remote sensing large language model accuracy improvement method based on a segmented database and human feedback as described in any one of claims 1 to 7.

Images