Flood remote sensing image language understanding method and system based on priori guidance large model

Abstract

This invention relates to the field of flood remote sensing image understanding, specifically disclosing a method and system for language understanding of flood remote sensing images based on a priori-guided large-scale model. The method includes: constructing a dataset containing multi-temporal flood remote sensing images and multi-granularity linguistic descriptions; constructing and training a flood prior model to extract prior information on the spatial distribution of floods and convert it into textual prior information; constructing a large-scale language model containing a visual encoder, a projection layer, and a language module, and training it by inputting the images, textual descriptions, and textual prior information together; inputting the image to be understood and the corresponding textual prior information into the trained large-scale language model, and outputting the language description result. This invention, by introducing a flood prior model to provide explicit spatial distribution guidance, effectively alleviates the semantic understanding bias and spatial positioning error of general visual language models in flood scenarios, significantly improving the accuracy of linguistic descriptions of flood images.

Description

Technical Field

[0001] This invention relates to the field of flood remote sensing image understanding, and in particular to a method and system for language understanding of flood remote sensing images based on a priori guided large model. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Flood image understanding is of great significance during flood disasters. It can convert image information into concise language and transmit information with extremely low communication costs, so as to inform people of the location of floods in a timely manner and guide rescue efforts.

[0004] Early research primarily utilized traditional machine learning or deep learning techniques for semantic segmentation of flood images, classifying flooded and non-flooded areas at the pixel level and outputting binary image results. For example, some researchers used Transformer-based models combined with the characteristics of SAR images for flood detection; others analyzed the effectiveness of the VV channel of SAR images for flood detection in arid regions. However, these methods can only output segmentation maps and cannot generate linguistic explanations that directly describe the disaster situation. They are not user-friendly for non-experts and struggle to capture and express high-level semantic information such as the temporal changes of floods.

[0005] With the development of multimodal large-scale models, visual language models have demonstrated powerful image understanding and language generation capabilities. However, when applying general-domain visual language models directly to flood disaster image understanding, the following technical bottlenecks exist: a lack of high-quality image-text pairing corpora specifically for wide-area flood disasters; and general models struggle to accurately understand the characteristics of floods at wide-area scales, such as blurred boundaries, complex backgrounds, and drastic scale changes, leading to semantic understanding biases and insufficient spatial positioning accuracy. Summary of the Invention

[0006] This invention aims to address the technical problems of insufficient accuracy in understanding flood disaster images, lack of dedicated datasets, and lack of prior knowledge guidance in existing technologies. It proposes a method and system for language understanding of flood remote sensing images based on a prior-guided large model.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: Firstly, a method for language understanding of flood remote sensing images based on a priori guided large model is provided, including: Construct a dataset; the dataset includes multi-temporal flood remote sensing images and corresponding multi-granularity language description text; A flood prior model is constructed and trained; the flood prior model is used to extract prior information on the spatial distribution of floods from the input flood remote sensing image and convert it into textual prior information. A large language model is constructed, comprising a visual encoder, a projection layer, and a language layer connected in sequence; the flood remote sensing images and corresponding multi-granularity language description texts in the dataset, as well as the text priors converted from the flood prior model, are input into the large language model for training to obtain a trained large language model; The flood remote sensing image to be understood is input into a trained large language model, and the corresponding textual prior information for the image is generated using the flood prior model and input together; the trained large language model outputs the corresponding language description result.

[0008] Furthermore, the multi-granularity language description text includes: simple single-image description, medium-level single-image description, complex single-image description, and multi-image description.

[0009] Furthermore, the method for obtaining the medium-level single-graph description is as follows: Initial descriptions are generated using a multimodal large model with pixel-level labeled flood tags. Correct image-text description pairs are selected through manual verification, and the selected image-text description pairs are input into the CLIP model for training. The CLIP model is then used to infer all image-text pairs to obtain the consistency degree of each image-text pair. The inference results are ranked according to their consistency. The inconsistent inference results are selected according to a certain proportion and the inference is re-inferred to obtain the language description after correction. After another manual verification, a single-image description of moderate quality was obtained.

[0010] Furthermore, the specific steps for constructing and training the flood prior model are as follows: Each flood remote sensing image is divided into multiple preset sub-regions, and the pixel proportion of flood in each sub-region is statistically analyzed based on the corresponding pixel-level annotations, which serves as the training label; A lightweight convolutional neural network was used for training, with the original flood remote sensing image as input and the flood percentage of a sub-region as output. After training, the system outputs the flood prediction percentage of each sub-region for the input image and formats it into prior information in natural language.

[0011] Furthermore, the plurality of preset sub-regions include five sub-regions: upper left, lower left, upper right, lower right, and center.

[0012] Furthermore, when training the large language model, all parameters of the visual encoder are frozen, all parameters of the projection layer are fully fine-tuned, and the LoRA low-rank adaptation method is introduced into the language layer.

[0013] Furthermore, the trained large language model outputs corresponding language description results, specifically including: Based on the input prompts, the model is guided to output language description results at the corresponding granularity; the language description results at the corresponding granularity include simple single-graph descriptions, medium-level single-graph descriptions, complex single-graph descriptions, and multi-graph descriptions.

[0014] Secondly, a language understanding system for flood remote sensing images based on a priori guided large model is provided, including: A dataset construction module is used to build and train a flood prior model; the flood prior model is used to extract prior information on the spatial distribution of floods from input flood remote sensing images and convert it into textual prior information. The flood prior module is used to construct and train a flood prior model; the flood prior model is used to extract prior information on the spatial distribution of floods from the input flood remote sensing image and convert it into textual prior information. The large language model module is used to construct a large language model, which includes a visual encoder, a projection layer, and a language layer connected in sequence. The flood remote sensing images and corresponding multi-granularity language description text in the dataset, as well as the text prior converted from the flood prior model, are input into the large language model for training to obtain a trained large language model. The image understanding module is used to input the flood remote sensing image to be understood into the trained large language model, and at the same time use the flood prior model to generate corresponding text prior information for the image and input it together; the trained large language model outputs the corresponding language description result.

[0015] Thirdly, an electronic device is also provided, comprising: Memory, used for non-transitory storage of computer-readable instructions; and Processor, for executing the computer-readable instructions, When the computer-readable instructions are executed by the processor, they perform the method described in the first aspect above.

[0016] Fourthly, a storage medium is also provided for non-transitory storage of computer-readable instructions, wherein the method described in the first aspect is performed when the non-transitory computer-readable instructions are executed by a computer.

[0017] The above technical solution has the following advantages or beneficial effects: (1) A large-scale, high-quality dataset for wide-area flood image language understanding was constructed, which effectively filled the gap in the field of wide-area flood understanding datasets and provided a solid data foundation for model training.

[0018] (2) The present invention designs a flood prior knowledge guidance module. This module extracts the spatial distribution prior of flood disasters and finally guides the detection results in the model through language modality input, so that the model can more accurately identify the regional distribution of floods when generating language descriptions, and solves the problem of semantic understanding deviation and spatial positioning error of general visual language models in flood scenarios. Attached Figure Description

[0019] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0020] Figure 1 This is a flowchart of the flood remote sensing image language understanding method based on a priori guided large model in Embodiment 1 of the present invention; Figure 2 This refers to the division of different regions in a flood image in Embodiment 1 of the present invention; Figure 3 (a) is a structural diagram of language understanding of flood remote sensing images using only language models in the prior art; Figure 3 (b) is a structural diagram of flood remote sensing image language understanding using the prior-guided large model-based flood remote sensing image language understanding method in Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0021] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0022] This invention aims to provide a technical solution capable of accurately understanding flood remote sensing images and generating natural language descriptions. By introducing a prior knowledge guidance mechanism into a general large language model, it effectively solves the problems of semantic understanding bias and spatial positioning error in existing technologies. The invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are only for explaining the invention and do not constitute any limitation on the scope of protection of the invention.

[0023] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments of the invention. The terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] Those skilled in the art should understand that the actual scope of protection of this invention is not limited to the specific embodiments described below. Any equivalent substitutions or modifications based on the technical concept of this invention are also within the scope of protection of this invention. Furthermore, without conflict, the various embodiments and technical features in these embodiments can be combined with each other, and the combined technical solutions also fall within the scope of protection of this invention.

[0025] In this embodiment of the invention, "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of this invention, "multiple" refers to two or more.

[0026] Furthermore, to facilitate a clear description of the technical solutions of the embodiments of the present invention, the terms "first" and "second" are used in the embodiments of the present invention to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.

[0027] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0028] It should be noted that the data collection and processing processes involved in this invention strictly comply with relevant laws and regulations. All data are from public channels or have been legally authorized. Sensitive information has been de-identified during the processing to ensure that there is no risk of personal privacy leakage.

[0029] When training the model, the language descriptions generated by this invention are all in English. For ease of explanation, the corresponding Chinese sentences are used as examples in this method.

[0030] Terminology Explanation Large language models are a type of large-scale natural language processing model based on deep learning, which uses a huge amount of corpus data and parameters during training.

[0031] Fine-tuning involves transferring the learning patterns and features of a pre-trained model to a new task, thereby improving performance and reducing the need for training data.

[0032] Prompt instructions refer to natural language instructions provided by the user to the model during the input phase, which are used to guide the model to generate output results in a specific format or with a specific granularity.

[0033] LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique designed to address the problems of excessive computational resource consumption and huge storage costs during the full fine-tuning of large language models.

[0034] Prior knowledge refers to domain knowledge known before model training. In this invention, it specifically refers to the spatial distribution characteristics of floods, which are used to guide the model to more accurately understand flood remote sensing images.

[0035] The terms mentioned above are commonly used technical terms in the art, but they have specific meanings in the context of this invention. Those skilled in the art should understand these terms in conjunction with the overall technical solution of this invention. The following embodiments will describe the technical solution of this invention in detail to facilitate implementation by those skilled in the art.

[0036] Example 1 like Figure 1 As shown, this embodiment provides a method for language understanding of flood remote sensing images based on a priori guided large model, including: S1: Construct a dataset; the dataset includes multi-temporal flood remote sensing images and corresponding multi-granularity language description text; S2: Construct and train a flood prior model; the flood prior model is used to extract prior information on the spatial distribution of floods from the input flood remote sensing image and convert it into textual prior information; S3: Construct a large language model, which includes a visual encoder, a projection layer, and a language layer connected in sequence; input the flood remote sensing images and corresponding multi-granular language description texts in the dataset, as well as the text priors converted from the flood prior model, into the large language model for training to obtain a trained large language model; S4: Input the flood remote sensing image to be understood into the trained large language model, and at the same time use the flood prior model to generate corresponding text prior information for the image and input it together; the trained large language model outputs the corresponding language description result.

[0037] The specific steps of S1 are as follows: S1.1: Data preparation. Collect over 100,000 satellite remote sensing images of typical global flood events as source images, along with pixel-level corresponding labeled images, i.e., binary label images, where a pixel value of "1" represents a flooded area and "0" represents a non-flooded area.

[0038] S1.2: Generate a medium-level single-image description. Specifically, input the labeled image into the first multimodal large model. In this embodiment, the Qwen-32B multimodal large model is used, and the prompt is: "White areas (value 1) are flood-affected areas, black areas (value "0") are areas where no flood occurred. Please describe the location (e.g., upper left, middle, etc.) and magnitude (e.g., small amount, large amount) of the flood in concise language." Apply the same operation to each image in a loop to generate an initial description for each image.

[0039] To overcome the "illusion" and "fuzziness" problems inherent in the Qwen-32B multimodal large model, the CLIP model is used for verification. Specifically, the initial descriptions are compared with their corresponding images through manual verification, and 1000 image-text description pairs with correct descriptions are selected in this embodiment. Then, the selected image-text description pairs are input into the CLIP model for training to obtain a model that can judge the consistency between flood label images and flood language descriptions. After training, the CLIP model is used to infer all image-text pairs to obtain the consistency level of each pair. The consistency levels are ranked, and inference results with a certain percentage of inconsistencies are re-inferred to obtain a revised language description; in this embodiment, this is set to 50%. Finally, all image-text pairs are manually verified again to ensure accuracy, resulting in single-image descriptions with a moderate level of consistency.

[0040] The above-mentioned method for generating moderate-level descriptions innovatively combines the generation capabilities of multimodal large models with the cross-modal verification capabilities of CLIP models, significantly improving the quality of text-image description pairs and providing high-quality supervision signals for the subsequent training of large language models.

[0041] S1.3: Generate complex single-image descriptions. Specifically, for each flood image, the original image, label image, and the moderately complex linguistic description obtained in S1.2 are input into the second multimodal large-scale model. The model is informed that this is an accurate description; in the label image, white areas correspond to flooded areas in the original image, and black areas correspond to non-flooded areas. The model is required to output, based on the label image and accurate linguistic input, the reasons for identifying flooded areas as flooded (e.g., darker color, shape similar to submerged rivers or lakes) and the reasons for identifying non-flooded areas as non-flooded (e.g., lighter color, indicating arid plains or cities). This ultimately yields a complex description for each flood remote sensing image. In this embodiment, the second multimodal large-scale model is the Qwen-32B multimodal large-scale model.

[0042] The complexity of single-image descriptions not only includes the location and magnitude of floods but also the criteria used for judgment, providing rich material for models to learn deeper semantic information. This descriptive form is similar to an expert's explanatory report, helping models understand the intrinsic connection between the visual features of floods in remote sensing images and semantic concepts, thereby improving the model's interpretability and generalization ability.

[0043] S1.4: Generate a simple single-image description. The simplified description referred to in this invention mainly refers to the presence / absence of flood in the flood image and the amount of flood (large / small). The specific method for generating a simple single-image description is as follows: Based on the proportion of white areas in the label image, a simple description of the presence / absence of flood and the amount of flood is automatically generated using rule-based judgment. For example, a proportion greater than threshold T1 indicates "large flood," less than threshold T2 indicates "small flood," otherwise it is "moderate flood," and a proportion of 0 indicates no flood in this area, thus obtaining a simple single-image description.

[0044] Simple single-image descriptions serve as a quick entry point for understanding disaster situations. Generating these descriptions through rule-based judgment ensures efficiency and consistency. Such descriptions are suitable for scenarios requiring rapid access to macro-level information, such as emergency command, demonstrating the adaptability of the method to different application scenarios.

[0045] S1.5: Generate multi-image descriptions. In this invention, "multi-image" refers to multiple images of the same area taken at different times. The specific method for generating multi-image descriptions is as follows: several simple descriptions and moderate-level descriptions of single images of the same area and adjacent time phases are input into a third multimodal large-scale model, and a joint description prompt template is designed. This template explicitly requires the model to focus on the following three dimensions when generating multi-image descriptions: first, the location distribution of flood occurrences within each sub-image (e.g., upper left, lower middle, entire area, etc.) and the flood inundation magnitude (small, medium, large); second, the flood inundation area and the temporal trend of inundation changes among multiple images (e.g., expansion, contraction, northward shift, etc.). In this embodiment, the third multimodal large-scale model is the Qwen-32B multimodal large-scale model.

[0046] Multi-image description focuses on the spatiotemporal evolution of floods, providing dynamic information that single-image descriptions cannot offer. Through joint analysis of multi-temporal images, the model can capture key dynamic features such as the advance and retreat of floods and their migration paths, which is of great significance for disaster evolution prediction and post-disaster assessment. This step fully utilizes the long context processing capabilities of multimodal large-scale models, fusing information from multiple images into a coherent linguistic description, demonstrating the advanced nature of this invention in temporal understanding.

[0047] S1.6: Following S1.1-S1.4, a dataset containing image-multi-granularity text pairings was constructed. Multi-granularity text descriptions refer to simple single-image descriptions, medium-level single-image descriptions, complex single-image descriptions, and multi-image descriptions. S1.1 to S1.6 detail the dataset construction process, covering the complete workflow from raw data collection and generation of multiple granularity descriptions to final dataset integration. This process fully considers the characteristics of flood remote sensing image understanding tasks, ensuring the diversity and accuracy of the dataset through a combination of multimodal large models, rule-based judgment, and manual verification.

[0048] In step S1 of this embodiment, the first multimodal large model, the second multimodal large model, and the third multimodal large model are all Qwen-32B multimodal large models. They have the same structure and run independently.

[0049] The Qwen-32B model will be described in detail below: The Qwen-32B multimodal large-scale model (also known as the Qwen-32B visual language model) is a large-scale pre-trained language model developed by Alibaba's Tongyi Qianwen team. It has 32 billion parameters and belongs to the decoder-only architecture of the Transformer model. Pre-trained on massive amounts of multi-source image-text data, this model possesses powerful natural language understanding, generation, reasoning, and instruction following capabilities, and can be widely applied to tasks such as dialogue systems, content creation, question answering, and code generation.

[0050] The Qwen-32B multimodal large-scale model employs a typical decoder-only architecture, consisting of three main parts: a visual Transformer encoder, a projection layer, and a language Transformer decoder layer. The core operating principle of the model is autoregressive language modeling: given an input token sequence, the model extracts features layer by layer, ultimately outputting the probability distribution of the next token. During the generation process, the output at each step depends on all previously generated tokens, ensuring that the generated text conforms to the causal logic of natural language.

[0051] Each Transformer decoder layer contains two core sub-modules: a multi-head self-attention mechanism and a feedforward neural network. Each sub-module is followed by residual connections and layer normalization to stabilize the training process and accelerate convergence. After all decoder layers are stacked, the output is mapped to a probability distribution over the vocabulary through a linear projection layer and a softmax function.

[0052] The model employs a causal mask to ensure its autoregressive property, meaning each position can only focus on its previous positions, preventing the leakage of future information. This design makes the model naturally suitable for text generation tasks and allows it to adapt to diverse downstream applications through supervised fine-tuning and reinforcement learning with human feedback.

[0053] The Qwen-32B employs grouped query attention as its attention mechanism, configuring 64 query heads and 8 key-value heads. In traditional multi-head attention, each query head corresponds to an independent key-value head, resulting in a large KV cache usage during inference. Grouped query attention groups multiple query heads together to share the same set of key-value heads, significantly reducing memory usage while maintaining the model's expressive power through grouping. Real-world testing on NVIDIA V100 and A100 hardware shows that this design improves inference speed by approximately 2-3 times, making it particularly suitable for long sequence generation scenarios.

[0054] Qwen-32B uses rotational position embedding to inject positional information. RoPE incorporates relative position encoding into attention computation through a complex rotation matrix, enabling the model to effectively capture the relative positional relationships between tokens and possessing good length extrapolation capabilities.

[0055] To support ultra-long contexts, Qwen-32B also introduces a block-based sparse attention mechanism: it performs full attention computation on each local window to preserve detailed information, while setting a small number of global tokens to enable cross-segment communication and maintain global awareness. This hybrid strategy ensures information integration capabilities for long sequences while maintaining computational efficiency.

[0056] The feedforward network of Qwen-32B uses variants of gated linear units, such as SwiGLU or GeGLU, instead of the traditional ReLU activation function. The gated structure dynamically controls the information flow, enhancing nonlinear fitting capabilities while mitigating the vanishing gradient problem and significantly improving the expressive efficiency of each network layer. In its implementation, the feedforward network typically contains two linear transformation layers connected by a gated activation function, with the output dimension matching the input dimension.

[0057] Layer normalization and residual connections are applied after each sub-layer (attention layer and feedforward layer). Layer normalization standardizes the hidden state of each token, accelerating training convergence; residual connections allow gradients to flow directly through the sub-layers, alleviating the vanishing gradient problem in deep networks. Qwen-32B uses a pre-norm approach for layer normalization, meaning normalization is performed before inputting to the sub-layers, which helps stabilize the training of large-scale models.

[0058] The training of Qwen-32B follows a three-stage paradigm of "pre-training - fine-tuning - alignment". The pre-training stage uses a large amount of image-language data to enable the model to learn fundamental information from massive datasets. Training employs a distributed framework, supporting stable optimization of models with hundreds of billions of parameters.

[0059] During the inference phase, Qwen-32B supports technologies such as dynamic batch processing and quantized deployment (e.g., FP8, INT4), enabling efficient operation in both the cloud and edge. Its open weights and commercial licensing have facilitated widespread adoption in academic research and industrial applications.

[0060] The powerful capabilities of the Qwen-32B model provide reliable support for this invention. Its long context processing capabilities and instruction-following characteristics are particularly suitable for the multi-granularity description generation task of this invention, laying a solid technical foundation for flood remote sensing image understanding. Choosing this model as the base model allows for full utilization of its language knowledge and reasoning capabilities learned on large-scale general data, while also enabling rapid adaptation to the specific needs of the flood domain through subsequent fine-tuning, achieving an organic combination of versatility and specialization.

[0061] Next, we will provide a detailed introduction to the CLIP model in S1: This invention is based on a contrastive learning framework and performs customized training on the CLIP (Contrastive Language-Image Pre-training) model to learn the semantic alignment relationship between images and text. The overall method continues the dual-encoder architecture, mapping multimodal data to a unified feature space by constructing an image encoder and a text encoder, and optimizing it with a specific dataset (flood remote sensing data) to enable the model to have cross-modal understanding capabilities for specific scenarios.

[0062] In terms of model structure, this invention adopts the CLIP framework based on existing open-source implementations. The model contains two independent encoding branches: an image encoder and a text encoder. The image encoder uses the Convnext architecture. The image encoder receives the input image (uniformly adjusted to a fixed size, such as 224×224 or 256×256) and extracts its visual feature representation; the text encoder receives the corresponding text description and generates semantic feature representation through word segmentation and embedding. The feature vectors output by both encoders are normalized and uniformly scaled using a learnable scaling factor to ensure the stability of similarity calculation.

[0063] In terms of training data organization, this invention constructs a pairwise image-text dataset, where each image corresponds to a textual description of its content. During training, data is input in batches, with each batch containing N pairs of image-text data. To improve data quality, this invention filters and selects training samples, and controls the composition of the training data through a custom index list, thereby eliminating outlier samples and improving the stability and convergence of model training.

[0064] Regarding the training mechanism, this invention employs a contrastive learning strategy. Specifically, for each batch of N pairs of images and text, the model calculates image features and text features respectively, and constructs an N×N similarity matrix. The diagonal elements in the matrix represent matching image-text pairs (positive samples), and the off-diagonal elements represent mismatched image-text pairs (negative samples). By optimizing a symmetric contrastive loss function, the model maximizes the similarity of positive sample pairs while minimizing the similarity of negative sample pairs, thereby achieving cross-modal semantic alignment.

[0065] Regarding optimization strategies, this invention employs the AdamW optimizer to update model parameters and sets differentiated weight decay strategies for different types of parameters: no weight decay is applied to bias terms and normalization layer parameters, while weight decay is applied to other parameters to improve generalization ability. Simultaneously, a cosine learning rate scheduling strategy is introduced to dynamically adjust the learning rate during training, thereby improving training stability and final performance.

[0066] In terms of computational optimization, this invention supports mixed-precision training (such as BF16 or AMP) to reduce GPU memory usage and improve computational efficiency. It also supports gradient clipping to prevent gradient explosion and is compatible with multi-GPU environments through a distributed training interface. In single-GPU debugging mode, a distributed runtime environment is simulated by setting distributed environment variables, thereby ensuring the consistency of the training code.

[0067] Regarding the training process, this invention iterates through epochs. In each epoch, the model output and loss function are calculated via forward propagation, and the model parameters are updated via backpropagation. During training, information such as loss values, learning rates, and feature scales are recorded periodically to monitor the training status.

[0068] Regarding the evaluation mechanism, this invention designs a dedicated evaluation method for a specific application scenario (flood dataset). During the evaluation process, the model encodes the images and text in the validation set and calculates their matching scores (similarity). By extracting the diagonal elements of the matching matrix, the matching score for each image-text pair is obtained, and all samples are sorted and statistically analyzed. Finally, the evaluation results are saved as a structured file (such as Excel or CSV) for subsequent data filtering and quality analysis.

[0069] Furthermore, this invention, by introducing the CLIP training framework, achieves automatic evaluation of the consistency of multimodal data. Through quantitative analysis of the matching degree between images and text, it can effectively identify data samples with descriptive errors or semantic inconsistencies, thereby improving the overall quality of the dataset. Compared to traditional manual screening methods, this method is more efficient and objective, and is suitable for large-scale multimodal data processing scenarios.

[0070] In summary, this invention, by constructing a CLIP training method based on contrastive learning and combining it with data processing and evaluation strategies for specific application scenarios, achieves an efficient and scalable training process for image-text alignment models, providing reliable basic model support for subsequent multimodal tasks.

[0071] The specific steps for S2 to construct and train the flood prior model are as follows: S2.1: Constructing the dataset for training the flood prior model. First, each original flood remote sensing image is divided into multiple predefined sub-regions. In this embodiment, it is divided into five sub-images: upper left, lower left, upper right, lower right, and center. The division of each part of the image is as follows: Figure 2 As shown. After the splitting is complete, the label map is split in the same way; then the proportion of flooded areas in each split label map is calculated. This proportion is used as the label of the original sub-map.

[0072] S2.2: Select a lightweight convolutional neural network, such as LSNet, as the prediction network. Take the original flood remote sensing image as input and the flood proportion of five sub-regions as output for supervised training, so that the prediction network can accurately predict the flood proportion of the sub-image.

[0073] S2.3: After training, for any input original image, this module can output the flood prediction percentage of five sub-regions. Then, these values ​​are formatted into natural language text prior information, such as: "The flood percentages of each part in this image are: top left 0%, top right 10%, bottom left 5%, bottom right 80%, and center 5%..."

[0074] Step S2 achieves efficient extraction of prior knowledge of flood spatial distribution through a lightweight network and transforms numerical information into natural language form, providing easily fusionable prior knowledge for large language models. This design ensures the accuracy of prior information while avoiding excessive computational burden. It should be noted that the sub-region division method is determined based on the characteristics of flood remote sensing images and practical application requirements, achieving a good balance between spatial positioning accuracy and computational complexity. Those skilled in the art can adjust the number and division method of sub-regions according to actual conditions, which still falls within the scope of protection of this invention.

[0075] The following section provides a detailed introduction to the lightweight convolutional neural network LSNet (See Large, Focus Small Network) structure used in S2.

[0076] LSNet is a lightweight visual network paradigm proposed by a research team from Tsinghua University at CVPR 2025. Its core design concept is inspired by the dynamic heteroscale visual capabilities of the human visual system: through a two-stage mechanism of "large field of view perception" and "small field of view focusing", it achieves efficient feature modeling with extremely low computational overhead.

[0077] Existing lightweight networks primarily rely on self-attention mechanisms or convolutions for token mixing, but these mechanisms have inherent limitations in the perception and aggregation processes: self-attention employs global perception and aggregation, which can easily lead to over-focusing on weakly related regions such as the background, and the overlapping of perception and aggregation ranges results in excessively high computational complexity; while convolutions are computationally efficient, their receptive field is limited, and fixed convolutional kernel weights lack adaptability to dynamic contexts. These issues make it difficult for lightweight networks to balance performance and efficiency under limited computing power budgets.

[0078] The core innovation of LSNet lies in proposing LS convolution, which decouples large-scale context awareness and local detail aggregation into two stages through a scale separation strategy: Large-Kernel Perception (LKP): This technique mimics the "wide field of view" function of peripheral rod cells in the human retina, employing large-kernel depthwise convolution to efficiently capture a broad range of contextual information. In its implementation, input features are reduced in dimensionality through point convolution, and then spatial relationships are modeled using large-kernel depthwise convolution (default kernel size K_L=7), generating context-adaptive aggregation weights. This achieves global perception capabilities with minimal computational cost.

[0079] Small-Kernel Aggregation (SKA): This module simulates the "small field of view focusing" function of foveal cone cells, employing small-kernel dynamic convolution to perform fine-grained feature aggregation within locally highly correlated regions. Guided by weights generated by LKP, this module performs dynamic convolution operations on features through a grouping mechanism (default G=C / 8) (default kernel size K_S=3), reducing memory and computational overhead while enhancing the model's sensitivity to complex contextual changes.

[0080] The unique feature of LS convolution is that LKP utilizes rich, large-domain visual perception information to guide SKA in adaptive feature fusion within highly relevant contexts, rather than simply combining large and small kernel convolutions. This "perceptual-guided aggregation" mechanism enables the model to achieve more discriminative representations for dynamically complex visual information.

[0081] The LSNet network architecture is as follows: LSNet, built upon LS convolution, employs a four-stage hierarchical design: Input projection: Overlapping patch embedding is used for input projection to gradually reduce spatial resolution.

[0082] Stage design: The first three stages stack LS convolutional blocks, each block containing LS convolution, depthwise convolution, SE layer to enhance local inductive bias, skip connections, and feedforward network for channel mixing; the final stage uses multi-head self-attention blocks to capture long-range dependencies.

[0083] Downsampling: Spatial downsampling is achieved by combining depthwise convolution and pointwise convolution.

[0084] LSNet offers several variants to suit different computational budgets: LSNet-T (ultra-lightweight, 0.3G FLOPs), LSNet-S (lightweight, 0.5G FLOPs), and LSNet-B (basic, 1.3G FLOPs). LSNet outperforms existing lightweight models on tasks such as ImageNet classification, COCO detection and segmentation, and ADE20K semantic segmentation, demonstrating an excellent balance between performance and efficiency.

[0085] The computational cost of LS convolution mainly consists of three parts: point convolution in LKP (O(3HWC² / 4 + HWCD / 2), large kernel depth convolution in LKP (O(HWCK²_L / 2), and convolution aggregation in SKA (O(HWCK²_S). The total computational cost maintains a linear complexity relative to the input resolution (O(HWC / 4(3C + 2K²_L + (2G+4)K²_S)), ensuring the lightweight nature of the model.

[0086] Applying LSNet to the flood prior model of this invention has the following advantages: its LKP module can efficiently capture the spatial distribution context of flood areas at a wide scale, solving the problems of ambiguous flood boundaries and complex backgrounds; its SKA module enhances the sensitivity to local details through dynamic aggregation, accurately capturing the subtle distribution features of floods in each sub-region; the overall lightweight design ensures that the prior model can run in parallel with the large language model as a low-overhead module without increasing the system's inference burden.

[0087] LSNet's "large field of view perception" and "small field of view focusing" mechanism can simultaneously grasp the macroscopic distribution and microscopic features of floods, while its lightweight nature ensures the overall system's operational efficiency, making it an important component of the technical solution of this invention. It should be emphasized that LSNet is only a preferred embodiment of this invention, and those skilled in the art can use other lightweight convolutional neural networks (such as MobileNet, ShuffleNet, etc.) as alternatives, as long as they can achieve similar prior extraction functions, they do not depart from the protection scope of this invention.

[0088] The specific steps for S3 are as follows: S3.1: Construct a large language model, which consists of three sub-modules: a visual encoder, a projection layer, and a language layer. In this embodiment, the large language model adopts the Qwen-2B multimodal large model.

[0089] The visual encoder employs a ViT architecture, responsible for encoding the input image into a high-dimensional visual feature vector. In this embodiment, during the training of the large language model, all parameters of the visual encoder are frozen to preserve its visual representation capabilities learned from massive amounts of general data, avoiding overfitting and feature shift in limited samples of flood remote sensing images. Simultaneously, a full fine-tuning strategy is used to update all parameters of the projection layer, and the LoRA low-rank adaptation method is introduced to efficiently fine-tune the parameters of the large language layer, reducing training overhead while maintaining semantic generation capabilities.

[0090] After obtaining the high-dimensional visual feature vector extracted by the visual encoder, it is input into the projection layer. In this embodiment, the projection layer is a single-layer linear mapping network that aligns the visual features output by the visual encoder to the text embedding space of the large language model. In this embodiment, all parameters of the projection layer are fine-tuned during the training of the large language model to enable it to effectively convert the unique spectral, texture, and spatial structure information of the flood basin into semantic vectors that the language model can understand.

[0091] The language layer employs a Transformer decoder structure, using visual features output from the projection layer as prefixes to autoregressively generate descriptive text that conforms to the conventions of expressing flood disasters. To incorporate flood domain knowledge while controlling computational overhead, this embodiment introduces LoRA (Low-Rank Adaptation) technology during the training of the large language model. A trainable low-rank branch is added to the projection matrix of Q (query), K (key), V (value), and O (output projection matrix) in the Transformer structure, enabling efficient parameter fine-tuning, effectively mitigating catastrophic forgetting, and improving the accuracy and fluency of description generation.

[0092] S3.2: Training a large language model, specifically in the following way: The large language model is trained by using images from the dataset constructed in S1, corresponding multi-granularity descriptive text, and textual priors generated by the flood prior model in S2 as input. The training objective is to autoregressively generate sentences consistent with the real descriptive text. This process enables the model to learn to perform more accurate flood image understanding under the guidance of textual priors. The sentences consistent with the real descriptive text support multi-granularity outputs, including simple, moderate, and complex descriptions.

[0093] The training process of S3.2 organically combines prior knowledge with data-driven learning, enabling the model to fully utilize prior information to guide image understanding. Simultaneously, through learning multi-granularity descriptive text, it acquires the ability to flexibly output descriptions of different granularities based on instructions. During training, prior textual information serves as additional input cues, participating in language generation alongside visual features. This essentially provides the model with a spatial attention mechanism, guiding it to focus more on the flood-affected areas when generating descriptions, thereby effectively improving the accuracy of spatial localization.

[0094] The specific steps of S4 are as follows: like Figure 3 As shown in (b), the flood remote sensing image to be understood is simultaneously input into the visual encoder of the trained large language model and the trained flood prior model. The flood prior model generates corresponding text prior information for the image. Prior textual information, along with the flood-related questions, is input into the language layer to obtain linguistic features. These linguistic features are then fused with visual features mapped through a projection layer. The final language description at the language layer is the language description result output by the trained large language model. Based on input prompts, the model can be guided to output language description results of corresponding granularities. These granularities include simple single-image descriptions, medium-level single-image descriptions, complex single-image descriptions, and multi-image descriptions of the same region across multiple temporal phases.

[0095] The reasoning process in S4 realizes the final application of the technical solution of this invention. By combining prior guidance and prompts, it can quickly generate flood image descriptions that meet user needs, providing timely and accurate information support for flood disaster emergency response. This step fully demonstrates the flexibility and efficiency of the method in practical applications. Users only need to input images and corresponding prompts to obtain language descriptions of different granularities, greatly facilitating decision-makers at different levels to obtain disaster information. For example, emergency command centers can quickly grasp the macro-level disaster situation through simple descriptions, while water conservancy experts can gain a deeper understanding of the details of flood distribution through complex descriptions.

[0096] It should be understood that, because the model of this invention uses a multi-granularity dataset containing simple, medium, and complex single-image descriptions and multiple-image descriptions during the training phase, and has learned the ability to distinguish generation tasks based on input instructions, during inference, different prompts can be input into the model, such as: "Please describe it simply in one sentence," "Please describe the flood distribution characteristics in detail," or "Please compare the flood change trends of multiple images," to guide the model to output language description results of corresponding granularity, including but not limited to simple descriptions, medium-level descriptions, complex descriptions of a single image, and multiple-image descriptions of multiple temporal images of the same area. For example, for a single image, a simple description could be "There is a large amount of flood in the image," and a complex description could be "There is a large area of ​​flood in the upper right, lower right, and middle parts of the image, distributed in irregular patches and stripes."

[0097] Experimental verification To verify the effectiveness of the prior model proposed in this invention, a comparative experiment was conducted on the test set, and the results are shown in Table 1.

[0098] Table 1 Comparison of flood image understanding accuracy with and without prior models

[0099] As can be seen from Table 1, after adding the prior model, the model's understanding accuracy increased from 0.6758 to 0.7445, proving the technical effectiveness of the present invention.

[0100] Experimental results show that the prior guidance mechanism proposed in this invention significantly improves model performance, verifying the effectiveness and advancement of the technical solution and providing reliable experimental evidence for practical applications. It should be noted that this experiment was conducted as a fair comparison under the same test set and evaluation criteria, ensuring the experimental data is authentic and reliable, and fully reflecting the technical advantages of this invention. Those skilled in the art will understand that improved accuracy means more accurate identification of flood areas and more precise description of disaster situations in practical applications, thereby providing more reliable information support for disaster prevention and mitigation decision-making.

[0101] Example 2 This embodiment provides a language understanding system for flood remote sensing images based on a priori guided large model, including: A dataset construction module is used to build and train a flood prior model; the flood prior model is used to extract prior information on the spatial distribution of floods from input flood remote sensing images and convert it into textual prior information. The flood prior module is used to construct and train a flood prior model; the flood prior model is used to extract prior information on the spatial distribution of floods from the input flood remote sensing image and convert it into textual prior information. The large language model module is used to construct a large language model, which includes a visual encoder, a projection layer, and a language layer connected in sequence. The flood remote sensing images and corresponding multi-granularity language description text in the dataset, as well as the text prior converted from the flood prior model, are input into the large language model for training to obtain a trained large language model. The image understanding module is used to input the flood remote sensing image to be understood into the trained large language model, and at the same time use the flood prior model to generate corresponding text prior information for the image and input it together; the trained large language model outputs the corresponding language description result.

[0102] The system in this embodiment corresponds to the method in Embodiment 1. Through modular design, it enables the practical application of the method. Each module can be developed, tested, and optimized independently, demonstrating significant engineering value. This system can be used as a standalone software product or integrated into existing disaster monitoring platforms, providing intelligent support for flood disaster emergency response. It should be noted that the above module division is only a logical functional division. In actual implementation, multiple modules can be integrated into a single physical device or distributed across different devices to work collaboratively; both are within the scope of this invention.

[0103] It should be noted that the aforementioned dataset construction module, flood prior module, large language model module, and image understanding module correspond to steps S1 to S4 in Embodiment 1. The examples and application scenarios implemented by these modules and their corresponding steps are the same, but they are not limited to the content disclosed in Embodiment 1. It should also be noted that these modules, as part of the system, can be executed in a computer system, such as a set of computer-executable instructions.

[0104] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0105] The proposed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of modules described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0106] Example 3 The structure of the electronic device according to an embodiment of the present invention will be described in detail below. Figure 4 This is a schematic diagram of the composition structure of an electronic device provided in an embodiment of the present invention. It can be understood that... Figure 4 The diagram shows only an exemplary structure of the electronic device, not the entire structure. Some or all of the structures shown may be implemented as needed.

[0107] The electronic device provided in this embodiment of the invention includes: at least one processor 401, a memory 402, a user interface 403, and at least one network interface 404. The various components in the flood remote sensing image language understanding system based on a priori guided large model are coupled together via a bus system 405. It can be understood that the bus system 405 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 405 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 4 The general designated all buses as Bus System 405.

[0108] The user interface 403 may include a monitor, keyboard, mouse, trackball, click wheel, buttons, touchpad, or touch screen.

[0109] It is understood that memory 402 can be volatile memory or non-volatile memory, or both. In this embodiment of the invention, memory 402 is capable of storing data to support the operation of the terminal. Examples of this data include any computer programs used to operate on the terminal, such as operating systems and applications. The operating system includes various system programs, such as framework layers, core library layers, driver layers, etc., used to implement various basic services and handle hardware-based tasks. Applications can include various applications.

[0110] In some embodiments, the flood remote sensing image language understanding system based on a priori guided large model provided in this invention can be implemented using a combination of hardware and software. As an example, the flood remote sensing image language understanding system based on a priori guided large model provided in this invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the flood remote sensing image language understanding method based on a priori guided large model provided in this invention. For example, the processor in the form of a hardware decoding processor can employ one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0111] As an example, processor 401 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0112] As an example of the hardware implementation of the flood remote sensing image language understanding system based on a priori guided large model provided in this embodiment of the invention, the device provided in this embodiment of the invention can be directly executed by a processor 401 in the form of a hardware decoding processor. For example, it can be executed by one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components to implement the flood remote sensing image language understanding method based on a priori guided large model provided in this embodiment of the invention.

[0113] The memory 402 in this embodiment of the invention is used to store various types of data to support the operation of a flood remote sensing image language understanding system based on a priori guided large model, or to store data for execution. Figure 1 The program code for the method shown. Examples of this data include: any executable instructions for operation on a flood remote sensing image language understanding system based on a priori-guided large model, such as executable instructions that can be included in the executable instructions to implement the flood remote sensing image language understanding method based on a priori-guided large model of the present invention.

[0114] The electronic device in this embodiment, through the collaborative work of hardware and software, can efficiently run the method of this invention, providing a reliable computing platform for flood disaster emergency response, and has good practicality and promotional value. This electronic device can take various forms such as a server, workstation, personal computer, or embedded device, adapting to the needs of different scales and application scenarios. Those skilled in the art can select appropriate hardware configurations according to actual needs, such as adding a GPU accelerator card to improve model inference speed; this is a reasonable variation of this invention and should still be considered within the protection scope of this invention.

[0115] Example 4 This embodiment also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the method described in Embodiment 1.

[0116] In this embodiment, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0117] In this embodiment, the computer instructions contained on the computer-readable storage medium can be written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Java, C++, etc.) and conventional procedural programming languages ​​(such as C or similar programming languages). The computer instructions can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0118] It should be noted that the computer-readable storage media described in this embodiment include, but are not limited to, the types listed above. Any medium capable of storing computer instructions and allowing subsequent reading falls within the protection scope of this embodiment. The computer instructions may be in the form of source code, object code, or executable file, or may be compressed or encrypted instructions, as long as they can be recognized and executed by a processor.

[0119] Through the aforementioned computer-readable storage medium, the flood remote sensing image language understanding method based on a priori guided large model provided by this invention can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0120] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0121] Those skilled in the art should understand that the scope of protection of the present invention is not limited to the specific embodiments described above. Any modifications or improvements based on the technical concept of the present invention, as long as they do not depart from the spirit and principles of the present invention, should be regarded as equivalent embodiments of the present invention and fall within the scope of protection of the present invention.

[0122] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some improvements and modifications without departing from the concept of the present invention, and such improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A language understanding method for flood remote sensing images based on a priori guided large model, characterized in that, include: Build the dataset; The dataset includes multi-temporal flood remote sensing images and corresponding multi-granularity language description text; A flood prior model is constructed and trained; the flood prior model is used to extract prior information on the spatial distribution of floods from the input flood remote sensing image and convert it into textual prior information. A large language model is constructed, comprising a visual encoder, a projection layer, and a language layer connected in sequence; the flood remote sensing images and corresponding multi-granularity language description texts in the dataset, as well as the text priors converted from the flood prior model, are input into the large language model for training to obtain a trained large language model; The flood remote sensing image to be understood is input into a trained large language model, and the corresponding textual prior information for the image is generated using the flood prior model and input together; the trained large language model outputs the corresponding language description result.

2. The flood remote sensing image language understanding method based on a priori guided large model as described in claim 1, characterized in that, The multi-granularity language description text includes: simple single-image description, medium-level single-image description, complex single-image description, and multi-image description.

3. The flood remote sensing image language understanding method based on a priori guided large model as described in claim 2, characterized in that, The method for obtaining the medium-level single-graph description is as follows: Initial descriptions are generated using a multimodal large model with pixel-level labeled flood tags. Correct image-text description pairs are selected through manual verification, and the selected image-text description pairs are input into the CLIP model for training. The CLIP model is then used to infer all image-text pairs to obtain the consistency degree of each image-text pair. The inference results are ranked according to their consistency. The inconsistent inference results are selected according to a certain proportion and the inference is re-inferred to obtain the language description after correction. After another manual verification, a single-image description of moderate quality was obtained.

4. The flood remote sensing image language understanding method based on a priori guided large model as described in claim 1, characterized in that, The specific steps for constructing and training the flood prior model are as follows: Each flood remote sensing image is divided into multiple preset sub-regions, and the pixel proportion of flood in each sub-region is statistically analyzed based on the corresponding pixel-level annotations, which serves as the training label; A lightweight convolutional neural network was used for training, with the original flood remote sensing image as input and the flood percentage of a sub-region as output. After training, the system outputs the flood prediction percentage of each sub-region for the input image and formats it into prior information in natural language.

5. The flood remote sensing image language understanding method based on a priori guided large model as described in claim 4, characterized in that, The multiple preset sub-regions include five sub-regions: upper left, lower left, upper right, lower right, and center.

6. The flood remote sensing image language understanding method based on a priori guided large model as described in claim 1, characterized in that, During the training of the large language model, all parameters of the visual encoder are frozen, all parameters of the projection layer are fine-tuned, and the LoRA low-rank adaptation method is introduced into the language layer.

7. The flood remote sensing image language understanding method based on a priori guided large model as described in claim 1, characterized in that, The trained large language model outputs corresponding language description results, specifically including: Based on the input prompts, the model is guided to output language description results at the corresponding granularity; the language description results at the corresponding granularity include simple single-graph descriptions, medium-level single-graph descriptions, complex single-graph descriptions, and multi-graph descriptions.

8. A language understanding system for flood remote sensing images based on a priori guided large model, characterized in that, include: A dataset construction module is used to build and train a flood prior model; the flood prior model is used to extract prior information on the spatial distribution of floods from input flood remote sensing images and convert it into textual prior information. The flood prior module is used to construct and train a flood prior model; the flood prior model is used to extract prior information on the spatial distribution of floods from the input flood remote sensing image and convert it into textual prior information. The large language model module is used to construct a large language model, which includes a visual encoder, a projection layer, and a language layer connected in sequence. The flood remote sensing images and corresponding multi-granularity language description text in the dataset, as well as the text prior converted from the flood prior model, are input into the large language model for training to obtain a trained large language model. The image understanding module is used to input the flood remote sensing image to be understood into the trained large language model, and at the same time use the flood prior model to generate corresponding text prior information for the image and input it together; the trained large language model outputs the corresponding language description result.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the flood remote sensing image language understanding method based on a priori guided large model as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the steps in the flood remote sensing image language understanding method based on a priori guided large model as described in any one of claims 1-7.

Images