A remote sensing image water body extraction method and system based on multi-modal space-time complementarity and large model cooperation
By employing a multimodal spatiotemporal complementarity and large model collaboration approach, and utilizing lightweight networks for preliminary localization and heterogeneous feature verification, the problems of cloud and fog obstruction and terrain undulation in remote sensing water body extraction were solved, achieving high-precision and low-cost automated water body extraction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- WUHAN UNIV
- Filing Date
- 2026-03-20
- Publication Date
- 2026-06-09
Smart Images
Figure CN122176400A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of remote sensing image processing technology, specifically to a method and system for extracting water bodies from remote sensing images based on multimodal spatiotemporal complementarity and large model collaboration. Background Technology
[0002] Water body extraction is one of the core tasks of remote sensing applications. Currently, mainstream methods are mainly divided into three categories: Traditional methods based on spectral / thresholds: These rely on indices such as NDWI, require manual threshold setting, and cannot process SAR imagery. Domain-specific supervised learning methods (lightweight models): Such as using networks like U-Net, DeepLab, and FCN trained on specific data. Disadvantage 1 (optical): Severely affected by cloud cover; water bodies cannot be detected in cloud-covered areas, leading to fragmented results and missed detections. Disadvantage 2 (SAR): Greatly affected by terrain undulations; shadows on mountains are easily misidentified as water bodies (false alarms), and the edges are blurry, resulting in poor generalization. Methods based on visual foundational models (large models): Such as SAM (Segment Anything Model). Disadvantages: Lack of domain knowledge; requires precise prompts to function. Using full-map grid point prompts is computationally intensive and prone to background noise; manual bounding boxes cannot meet automation requirements. Summary of the Invention
[0003] To overcome the shortcomings of existing water body extraction methods, such as the need for strict pixel alignment, high training difficulty, and inability to eliminate the effects of cloud occlusion and false alarms, this invention provides a remote sensing image water body extraction method and system based on multimodal spatiotemporal complementarity and large model collaboration. By constructing a cascaded technical architecture of "small sample lightweight localization + logical gating strategy based on heterogeneous feature alignment and validity observation judgment + large model collaborative segmentation", it achieves sub-pixel-level fine segmentation of the land-water boundary to obtain a high-precision water body mask with a clean background and complete edges.
[0004] According to one aspect of the present invention, a method for water body extraction from remote sensing images based on multimodal spatiotemporal complementarity and large-model collaboration is provided, comprising: step S1: acquiring a first modal remote sensing image and a second modal remote sensing image of the same geographic region to be detected; step S2: using a pre-trained first lightweight semantic segmentation network and a second lightweight semantic segmentation network to perform inference on the first modal remote sensing image and the second modal remote sensing image respectively, to obtain a first preliminary mask and a second preliminary mask; step S3: performing connected component analysis and boundary extraction on the first preliminary mask to generate a first candidate target containing several potential water body targets. Step S4: Based on the logical gating strategy of heterogeneous feature alignment and validity observation judgment, the second verification target set is used to perform spatial overlap verification on each candidate water body target in the first candidate target set to obtain valid prompts; Step S5: The valid prompts and the first modal remote sensing image are input into the pre-trained visual basic model to perform fine segmentation of the first modal remote sensing image under the guidance of the valid prompts, and output the final high-precision water body extraction result.
[0005] Further, step S4 includes: traversing each candidate target in the first candidate target set, extracting the corresponding region of each candidate target in the second modal remote sensing image coordinate system; determining whether each corresponding region is a valid observation region in the second modal remote sensing image; if it is a valid observation region, then executing a heterogeneous cross-validation strategy: through the spatial consistency constraint of heterogeneous data, using each verification water body target in the second verification target set as a truth gate, performing spatial overlap verification on each corresponding region to determine whether it meets the preset confidence conditions; if it does, then determining that the corresponding candidate target is a real water body and retaining the candidate target as a valid prompt; otherwise, determining that the corresponding candidate target is a false alarm and removing the candidate target; if it is an invalid observation region, then executing a first modal confidence strategy: retaining the candidate target as a valid prompt, so as to use the penetration of the first modal remote sensing image to repair the cloud blind area of the second modal remote sensing image.
[0006] Further, determining whether the corresponding region is a valid observation region in the second modality remote sensing image includes: generating a confidence mask for the second modality remote sensing image using a valid observation region determination strategy; calculating the cloud coverage ratio within the corresponding region based on the confidence mask; determining it as a valid observation region when the cloud coverage ratio within the corresponding region is lower than a preset threshold; and determining it as an invalid observation region when the cloud coverage ratio within the corresponding region is higher than a preset threshold.
[0007] Furthermore, using each verification water body target in the second verification target set as a truth gate, spatial overlap verification is performed on each corresponding region to determine whether the preset confidence condition is met. This includes: extracting the intersection region between the corresponding region and each verification target in the second verification target set; calculating the number of second-mode water body pixels in the intersection region; if the number of second-mode water body pixels in the intersection region is greater than zero, it is determined that the preset confidence condition is met; if the number of second-mode water body pixels in the intersection region is equal to zero, it is determined that the preset confidence condition is not met.
[0008] Furthermore, the effective prompt is generated by extracting the minimum bounding rectangle of the retained candidate target as the effective prompt.
[0009] Furthermore, both the first lightweight semantic segmentation network and the second lightweight semantic segmentation network are semantic segmentation networks or object detection networks with encoder-decoder structures. The semantic segmentation networks with encoder-decoder structures include U-Net models, DeepLab series models, or FCN models, and the object detection networks include YOLO models or SSD series models.
[0010] Furthermore, the visual foundation model is a general-purpose large model with cue-driven segmentation capability, and the visual foundation model includes the SAM model or the Fast SAM model.
[0011] According to one aspect of the present invention, a remote sensing image water body extraction system based on multimodal spatiotemporal complementarity and large model collaboration is provided, comprising: a data acquisition module for acquiring a first modal remote sensing image and a second modal remote sensing image of the same geographic area to be detected; a preliminary extraction module for using a pre-trained first lightweight semantic segmentation network and a second lightweight semantic segmentation network to perform inference on the first modal remote sensing image and the second modal remote sensing image respectively, to obtain a first preliminary mask and a second preliminary mask; and a target set generation module for performing connected component analysis and boundary extraction on the first preliminary mask to generate a first target set containing several potential water body targets. A candidate target set; performing connected component analysis on the second preliminary mask to generate a second verification target set containing several verification water body targets; a logic fusion module, used to perform spatial overlap verification on each candidate water body target in the first candidate target set using a logic gating strategy based on heterogeneous feature alignment and validity observation judgment, to obtain valid prompts; a large model inference module, used to input the valid prompts and the first modal remote sensing image into a pre-trained visual basic model, so as to perform fine segmentation of the first modal remote sensing image under the guidance of the valid prompts, and output the final high-precision water body extraction result.
[0012] According to one aspect of the present invention, an electronic device is provided, including a memory and a processor, the memory storing program instructions that are executed by the processor, the processor invoking the program instructions to execute the aforementioned method for extracting water bodies from remote sensing images based on multimodal spatiotemporal complementarity and large model collaboration.
[0013] According to one aspect of the present invention, a non-transitory computer-readable storage medium is provided, the non-transitory computer-readable storage medium storing computer instructions that cause the computer to execute the aforementioned method for extracting water bodies from remote sensing images based on multimodal spatiotemporal complementarity and large model collaboration.
[0014] The above technical solution constructs a cascaded technical architecture of "small-sample lightweight localization + logical gating strategy based on heterogeneous feature alignment and validity observation judgment + large-model collaborative segmentation": Small-sample lightweight localization: Two lightweight target coarse localization networks are used to perform rapid inference on the first modality remote sensing image with all-weather penetration characteristics and the second modality remote sensing image with rich spectral semantic features, respectively. This stage only requires a very small number of samples for training to meet the coarse localization requirements. Logical gating strategy based on heterogeneous feature alignment and validity observation judgment: To address the physical contradiction between "cloud occlusion in the second modality remote sensing image" and "false alarms in the first modality remote sensing image", the second modality remote sensing image acts as a "fake detector", that is, the validity of the second modality remote sensing image is used to verify the false alarms in the first modality remote sensing image, so as to eliminate the false alarms in the first modality remote sensing image; at the same time, the first modality remote sensing image acts as a "completor", using the penetration of the first modality remote sensing image to repair the cloud blind spots of the second modality remote sensing image. Large-scale collaborative segmentation: Utilizing effective cues obtained through screening to guide a pre-trained visual base model for sub-pixel-level fine segmentation, achieving high-precision extraction at low cost.
[0015] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0016] (1) Extremely low data dependence and super generalization ability: Traditional deep learning methods often require hundreds of labeled images to converge. This invention adopts a decoupling strategy of "lightweight localization + large model segmentation", which only requires a very small number of samples to train a lightweight target coarse localization network to achieve coarse localization of candidate regions, greatly reducing the data labeling cost for engineering implementation.
[0017] (2) Automatic False Alarm Suppression Mechanism Based on Heterogeneous Feature Verification: To address the common problem of false alarms in first-mode remote sensing images being misidentified as water bodies, this invention utilizes the spectral features of second-mode remote sensing images as a "truth verifier." Through logical filtering, it can accurately eliminate false alarm targets that appear as bright or dark areas in first-mode remote sensing images but lack optical water body features, effectively solving the problem of high false alarm rates in first-mode remote sensing images.
[0018] (3) Strong robustness to multimodal registration errors: Traditional multimodal fusion methods (such as channel overlay and feature-level fusion) require strict pixel-level alignment between the first and second modal remote sensing images. Any tiny registration deviation will lead to severe performance degradation. This invention adopts a "box-level prompt" strategy. Even if there is a nonlinear offset of several pixels between the two modal images, as long as the prompt box generated by the first modal remote sensing image can roughly cover the target area, the visual basic model can adaptively attach to the correct physical boundary on the original first modal remote sensing image with its powerful zero-sample segmentation capability. This "loosely coupled" architecture greatly reduces the requirements for data preprocessing accuracy, enhances the engineering practicality of the system, and solves the pain point of difficult image alignment.
[0019] (4) Achieved low-cost, high-efficiency, automated, high-precision segmentation: This invention eliminates the need for fine-tuning of the massive visual base model, saving expensive computing resources. A lightweight target coarse localization network automatically generates prompts, replacing manual interaction and achieving fully automated end-to-end processing. While maintaining industrial-grade accuracy, it achieves a dual reduction in computing and annotation costs. Attached Figure Description
[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0021] Figure 1 The first flowchart of a remote sensing image water body extraction method based on multimodal spatiotemporal complementarity and large model collaboration is provided for an embodiment of the present invention.
[0022] Figure 2 The second flowchart of a remote sensing image water body extraction method based on multimodal spatiotemporal complementarity and large model collaboration is provided for an embodiment of the present invention.
[0023] Figure 3 This is a schematic diagram illustrating the heterogeneous feature verification and false alarm suppression effect provided in an embodiment of the present invention.
[0024] Figure 4 This is a comparison chart of the accuracy of water extraction results provided in an embodiment of the present invention. Detailed Implementation
[0025] It should be noted that:
[0026] The terms “comprising” and “having”, and any variations thereof, in the specification, claims, and accompanying drawings of this invention are intended to cover a non-exclusive inclusion, such as a process, method, system, product, or apparatus that includes a series of steps or units, not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0027] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices. The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be decomposed, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.
[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined to form new technical solutions. Such combinations are not bound by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.
[0029] Please refer to Figure 1 and Figure 2 This invention provides a method for extracting water bodies from remote sensing images based on multimodal spatiotemporal complementarity and large model synergy, specifically including the following steps:
[0030] Step S1: Acquire the first modal remote sensing image and the second modal remote sensing image of the same geographic area to be detected.
[0031] In step S1, paired remote sensing image data of the same geographic area to be detected are first acquired, including a first modal remote sensing image (such as SAR, long-wave infrared) with all-weather penetration characteristics and a second modal remote sensing image (such as visible light, multispectral) with hyperspectral semantic features. In addition, the data sources of the paired remote sensing image data include heterogeneous data acquired by satellite platforms or UAV platforms, which are not limited here.
[0032] It should be noted that this invention incorporates three pre-trained deep learning models: a first lightweight semantic segmentation network (small in scale, used for quickly extracting potential water body features from the first modality of remote sensing imagery), and a second lightweight semantic segmentation network (also lightweight, used for extracting high-confidence water body features from the second modality of remote sensing imagery). The Visual Foundation Model is a general-purpose large model with cue-driven segmentation capabilities, used for subsequent refined segmentation. Both the first and second lightweight semantic segmentation networks can employ encoder-decoder structures for semantic segmentation or object detection. Encoder-decoder semantic segmentation networks include, but are not limited to, U-Net, DeepLab series models, or FCN models, while object detection networks include, but are not limited to, YOLO or SSD series models. The Visual Foundation Model includes, but is not limited to, SAM or Fast SAM models.
[0033] In this embodiment, the first modality remote sensing image is a SAR image, and the second modality remote sensing image is an optical image. The first lightweight semantic segmentation network is a U-Net model pre-trained for the characteristics of SAR images, used to quickly extract potential water body features from SAR images; the second lightweight semantic segmentation network is a U-Net model trained for optical characteristics, used to extract high-confidence water body features from optical images; and the visual base model is a SAM model. Understandably, the first lightweight semantic segmentation network, the second lightweight semantic segmentation network, and the visual base model can all be selected according to actual needs, and no limitation is imposed here.
[0034] Step S2: Use the pre-trained first lightweight semantic segmentation network and the second lightweight semantic segmentation network to perform inference on the first modality remote sensing image and the second modality remote sensing image respectively to obtain the first preliminary mask and the second preliminary mask.
[0035] Step S2 processes two data streams in parallel: ① SAR stream processing: The first modality remote sensing image is input into the first lightweight semantic segmentation network, and a first preliminary mask is output. ② Optical stream processing: The second modality remote sensing image is input into the second lightweight semantic segmentation network, and a second preliminary mask is output. The first preliminary mask refers to the binarized segmented image output after the first modality remote sensing image is coarsely extracted by the lightweight semantic segmentation network. This image not only contains real water bodies but also typically includes false alarm noise such as terrain shadows. The second preliminary mask refers to the binarized segmented image output after the second modality remote sensing image is extracted by the lightweight semantic segmentation network. This mask represents a high-confidence water body area, but it is easily obscured by clouds and may appear fragmented, broken, or concave in shape. It is used as a verification base map.
[0036] Step S3: Perform connected component analysis and boundary extraction on the first preliminary mask to generate a first candidate target set containing several potential water targets; perform connected component analysis on the second preliminary mask to generate a second verification target set containing several verification water targets.
[0037] In step S3, a connected component is an independent geometric block in the masked image composed of spatially adjacent pixels with the same pixel value (i.e., all predicted as water bodies). A boundary is the geometric outline surrounding the connected component, specifically the smallest bounding rectangle of the connected region. Potential water targets are each independent connected component block extracted from the first preliminary mask. Because the first modality remote sensing image is susceptible to false alarms due to terrain undulations, these targets are only candidates and need to be verified for their authenticity through subsequent logic. Verifying water targets involves high-confidence water connected components extracted from the second preliminary mask, which are used as a verification base map to verify the authenticity of potential water targets extracted from the first modality remote sensing image.
[0038] Step S4: Based on the logical gating strategy of heterogeneous feature alignment and validity observation judgment, the second verification target set is used to perform spatial overlap verification on each candidate water body target in the first candidate target set to obtain a valid prompt.
[0039] In step S4, heterogeneous feature alignment refers to the spatial consistency constraint between heterogeneous data (i.e., the first modal remote sensing image and the second modal remote sensing image). Therefore, the high semantic accuracy of each verification water body target in the second verification target set can be used as a truth gate to perform spatial overlap verification on each candidate water body target in the first candidate target set. Validity observation determination refers to using a valid observation area determination strategy to determine whether the corresponding area of each candidate target in the second modal remote sensing image coordinate system is a valid observation area in the second modal remote sensing image. Logical gating strategy refers to a set of gating control rules based on heterogeneous feature alignment and validity observation determination, used to filter and obtain valid prompts.
[0040] Specifically, each candidate target in the first candidate target set is traversed, and the corresponding region of each candidate target in the second modal remote sensing image coordinate system is extracted; it is then determined whether each corresponding region is a valid observation area in the second modal remote sensing image.
[0041] ① If the observation area is valid (i.e., not obscured by clouds or fog), a heterogeneous cross-validation strategy is implemented. False alarms from the first modality of remote sensing imagery are eliminated based on the accuracy of the second modality. Specifically: through spatial consistency constraints of heterogeneous data, the high semantic accuracy of each water body target in the second validation target set is used as a truth gate. Spatial overlap validation is performed on each corresponding area to determine if pre-set confidence conditions are met. If the pre-set confidence conditions are met, the corresponding candidate target is determined to be a real water body and retained as a valid cue. Otherwise, the corresponding candidate target is determined to be a false alarm specific to the first modality (such as mountain shadows or noise) and is eliminated. The valid cue is generated by extracting the minimum bounding box of the retained candidate targets as the valid cue.
[0042] ② If the observation area is invalid (i.e., completely obscured by clouds), the first modality trust strategy is executed: the candidate target is directly accepted and retained as a valid cue. This leverages the penetration of the first modality remote sensing image to fill the cloud blind spots in the second modality remote sensing image, thus logically repairing the missing information in the second modality remote sensing image. It should be noted that the second modality remote sensing image may miss water bodies in cloud-obscured areas (resulting in cloud blind spots), while the first modality remote sensing image possesses the physical characteristic of penetrating clouds and can extract complete water body contours. Therefore, when encountering an invalid observation area obscured by clouds, the system will skip optical verification, directly extract and retain the minimum bounding rectangle of the candidate target in the first modality remote sensing image. This minimum bounding rectangle will serve as a high-confidence spatial cue to guide the visual base model to segment the complete water body, thereby logically repairing the geometric morphological information missing from the second modality remote sensing image due to cloud obscuration.
[0043] It should be noted that existing technologies typically attempt to fuse multimodal features through end-to-end network training, which often leads to the network struggling to converge between two contradictory features: "cloud occlusion in second-modality remote sensing images" and "false alarms in first-modality remote sensing images". To address the physical contradiction between "cloud occlusion in second-modality remote sensing imagery" and "false alarms in first-modality remote sensing imagery," this invention constructs a "decision-level logic repair pipeline based on complementary physical characteristics." Through decoupling design, the first-modality and second-modality remote sensing images act as "quality inspectors" for each other. Specifically, the second-modality remote sensing imagery acts as a "fake detector," using its validity to verify false alarms in the first-modality remote sensing imagery caused by terrain undulations, radar speckle, or sensor noise, thus eliminating false alarms in the first-modality remote sensing imagery. Simultaneously, the first-modality remote sensing imagery acts as a "completor," using its penetration to repair cloud blind spots in the second-modality remote sensing imagery. Finally, a large model (i.e., the visual base model) acts as a "refiner," performing fine segmentation on the first-modality remote sensing imagery.
[0044] Further, determining whether the corresponding region is a valid observation area in the second modality remote sensing image includes: generating a confidence mask for the second modality remote sensing image using a valid observation area determination strategy (e.g., cloud detection algorithm or brightness / texture threshold); calculating the cloud coverage ratio within the corresponding region based on the confidence mask; determining a valid observation area when the cloud coverage ratio within the corresponding region is lower than a preset threshold; and determining an invalid observation area when the cloud coverage ratio within the corresponding region is higher than the preset threshold. It should be noted that the confidence mask and cloud coverage ratio are inversely proportional. A higher cloud coverage ratio indicates a more severe cloud obstruction of the second modality remote sensing image, resulting in a lower confidence level for the corresponding region in the confidence mask (when it exceeds the preset threshold, it is directly determined as an invalid observation area). Conversely, a lower cloud coverage ratio indicates a clearer and more reliable second modality remote sensing image, resulting in a higher confidence level for the corresponding region in the confidence mask (when it is lower than the preset threshold, it is determined as a valid observation area).
[0045] Furthermore, the pre-set confidence condition is: the number of second-modality water body pixels in the intersection region of the corresponding region and the second verification target set is greater than zero. That is, as long as the second-modality remote sensing image has a water body semantic response in the corresponding region, regardless of whether the response is complete, it is considered that the corresponding candidate target has passed the spatial overlap verification. Based on this, each verification water body target in the second verification target set is used as a truth gate to perform spatial overlap verification on each corresponding region to determine whether the pre-set confidence condition is met. This includes: extracting the intersection region of the corresponding region and each verification water body target in the second verification target set; calculating the number of second-modality water body pixels in the intersection region; if the number of second-modality water body pixels in the intersection region is greater than zero (i.e., the second-modality remote sensing image confirms that there is water in the corresponding region), it is determined that the pre-set confidence condition is met; if the number of second-modality water body pixels in the intersection region is equal to 0 (i.e., the second-modality remote sensing image confirms that the area is background), it is determined that the pre-set confidence condition is not met.
[0046] Step S5: Input the selected valid prompts and the first modality remote sensing image into the pre-trained visual base model, so as to perform fine segmentation of the first modality remote sensing image under the guidance of the valid prompts, and output the final high-precision water body extraction result.
[0047] In step S5, a valid cue refers to the smallest bounding rectangle extracted from the candidate target (SAR candidate box in this embodiment). Therefore, a valid cue is actually a cue box. The coordinate information of the original first-modality remote sensing image and the cue box are input into the pre-trained visual base model. Under the strong spatial constraints of the cue box, the visual base model uses its powerful texture understanding and edge perception capabilities to perform sub-pixel-level fine segmentation of the land and water boundaries within the cue box, and finally outputs a high-precision water mask with a clean background and complete edges (i.e., high-precision water extraction result).
[0048] Please refer to the following: Figure 3 and Figure 4 , Figure 3 The diagram illustrates the effects of heterogeneous feature verification and false alarm suppression (top row: left image: SAR image extraction result prompt box, including real water bodies and misjudged shadow false alarms, a total of 14 boxes; middle image: optical verification base map, only responding at real water bodies, the upper part of this image is farmland with water accumulation; right image: optical image prompt box overlaps with SAR image prompt box, only the green box part is judged as a real target; bottom row: left image: result after logical fusion, SAR image shadow false alarms are successfully removed, and real water body prompt boxes are retained; middle image: SAM segmentation result; right image: final water body segmentation result, white represents water bodies, black represents non-water bodies). Figure 4The diagram illustrates the accuracy comparison results of water body extraction (left: real label | middle: extraction result of this method | right: real label and mask superimposed) (F1: 0.978 | Precision: 0.979 | Recall: 0.977 | IoU: 0.957 | OA: 0.979, compared with the real mask: green = correct, red water body = false, yellow detection = missed detection).
[0049] Based on the same technical concept as the aforementioned embodiments, this invention also provides a remote sensing image water body extraction system based on multimodal spatiotemporal complementarity and large model collaboration, comprising: a data acquisition module for acquiring a first modal remote sensing image and a second modal remote sensing image of the same geographic area to be detected; a preliminary extraction module for using a pre-trained first lightweight semantic segmentation network and a second lightweight semantic segmentation network to perform inference on the first modal remote sensing image and the second modal remote sensing image respectively, to obtain a first preliminary mask and a second preliminary mask; and a target set generation module for performing connected component analysis and boundary extraction on the first preliminary mask to generate a first candidate target containing several potential water body targets. The system comprises: a set of verification targets; a second verification target set containing several verification water targets; a logic fusion module, which uses a logic gating strategy based on heterogeneous feature alignment and validity observation judgment to perform spatial overlap verification on each candidate water target in the first candidate target set using the second verification target set, and then filters valid prompts based on the spatial overlap verification results; and a large model inference module, which inputs the filtered valid prompts and the first modal remote sensing image into a pre-trained visual basic model, so as to perform fine segmentation of the first modal remote sensing image under the guidance of the valid prompts, and output the final high-precision water extraction result.
[0050] Based on the same technical concept as the foregoing embodiments, the present invention also provides an electronic device, including a memory and a processor, wherein the memory stores program instructions that are executed by the processor, and the processor calls the program instructions to execute the aforementioned method for extracting water bodies from remote sensing images based on multimodal spatiotemporal complementarity and large model collaboration.
[0051] Based on the same technical concept as the foregoing embodiments, the present invention also provides a non-transitory computer-readable storage medium storing computer instructions that cause the computer to execute the aforementioned method for extracting water bodies from remote sensing images based on multimodal spatiotemporal complementarity and large model collaboration.
[0052] In summary, this invention constructs a cascaded technical architecture of "small-sample lightweight localization + logical gating strategy based on heterogeneous feature alignment and validity observation judgment + large-model collaborative segmentation": Small-sample lightweight localization: Two lightweight target coarse localization networks are used to perform rapid inference on a first modality remote sensing image with all-weather penetration characteristics and a second modality remote sensing image with rich spectral semantic features, respectively. This stage only requires a very small number of samples for training to meet the coarse localization requirements. Logical gating strategy based on heterogeneous feature alignment and validity observation judgment: To address the physical contradiction between "cloud occlusion in the second modality remote sensing image" and "false alarms in the first modality remote sensing image", the second modality remote sensing image acts as a "fake detector", that is, the validity of the second modality remote sensing image is used to verify the false alarms in the first modality remote sensing image, so as to eliminate the false alarms in the first modality remote sensing image; at the same time, the first modality remote sensing image acts as a "completor", using the penetration of the first modality remote sensing image to repair the cloud blind spots of the second modality remote sensing image. Large-scale collaborative segmentation: Utilizing effective cues obtained through screening to guide a pre-trained visual base model for sub-pixel-level fine segmentation, achieving high-precision extraction at low cost.
[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.
Claims
1. A method for extracting water bodies from remote sensing images based on multimodal spatiotemporal complementarity and large-model synergy, characterized in that, include: Step S1: Acquire the first modal remote sensing image and the second modal remote sensing image of the same geographic area to be detected; Step S2: Use the pre-trained first lightweight semantic segmentation network and the second lightweight semantic segmentation network to perform inference on the first modality remote sensing image and the second modality remote sensing image respectively to obtain the first preliminary mask and the second preliminary mask. Step S3: Perform connected component analysis and boundary extraction on the first preliminary mask to generate a first candidate target set containing several potential water targets; Perform connected component analysis on the second preliminary mask to generate a second set of verification targets containing several verification water targets; Step S4: Based on the logical gating strategy of heterogeneous feature alignment and validity observation judgment, the second verification target set is used to perform spatial overlap verification on each candidate water body target in the first candidate target set to obtain a valid prompt; Step S5: Input the effective prompt and the first modal remote sensing image into the pre-trained visual base model, so as to perform fine segmentation of the first modal remote sensing image under the guidance of the effective prompt, and output the final high-precision water body extraction result.
2. The method for extracting water bodies from remote sensing images based on multimodal spatiotemporal complementarity and large-model synergy as described in claim 1, characterized in that, Step S4 includes: Traverse each candidate target in the first candidate target set and extract the corresponding region of each candidate target in the second modal remote sensing image coordinate system; Determine whether each corresponding region is a valid observation area in the second modality remote sensing image; If the observation area is valid, a heterogeneous cross-validation strategy is executed: through the spatial consistency constraint of heterogeneous data, each water body target in the second set of validation targets is used as a truth gate to perform spatial overlap validation on each corresponding area to determine whether the preset confidence conditions are met. If they are met, the corresponding candidate target is determined to be a real water body and the candidate target is retained as a valid alert; otherwise, the corresponding candidate target is determined to be a false alarm and the candidate target is removed. If the observation area is invalid, the first modality trust strategy is executed: the candidate target is retained as a valid cue to utilize the penetration of the first modality remote sensing image to repair the cloud blind spots of the second modality remote sensing image.
3. The method for extracting water bodies from remote sensing images based on multimodal spatiotemporal complementarity and large-model synergy as described in claim 2, characterized in that, Determining whether the corresponding region is a valid observation region in the second modality remote sensing image includes: A confidence mask for the second modality of remote sensing imagery is generated using an effective observation area determination strategy. The cloud coverage ratio within the corresponding area is statistically calculated based on the confidence mask. When the cloud coverage ratio in the corresponding area is lower than a preset threshold, it is determined to be a valid observation area; When the cloud coverage ratio in the corresponding area is higher than the preset threshold, it is determined to be an invalid observation area.
4. The method for extracting water bodies from remote sensing images based on multimodal spatiotemporal complementarity and large-model synergy as described in claim 2, characterized in that, Using each water body target in the second set of verification targets as a truth gate, spatial overlap verification is performed on each corresponding region to determine whether the preset confidence conditions are met, including: Extract the intersection region between the corresponding region and each verification target in the second verification target set; Calculate the number of second-modal water pixels within the intersection region; If the number of second-modal water body pixels within the intersection area is greater than zero, it is determined that the preset information condition is met; If the number of second-modal water body pixels in the intersection area is zero, it is determined that the preset information condition is not met.
5. The method for extracting water bodies from remote sensing images based on multimodal spatiotemporal complementarity and large-model synergy as described in claim 2, characterized in that, The effective prompt is generated in the following way: Extract the minimum bounding rectangle of the retained candidate targets as a valid cue.
6. The method for extracting water bodies from remote sensing images based on multimodal spatiotemporal complementarity and large-model synergy as described in claim 1, characterized in that, Both the first and second lightweight semantic segmentation networks are semantic segmentation networks or object detection networks with encoder-decoder structures. The semantic segmentation networks with encoder-decoder structures include U-Net models, DeepLab series models, or FCN models, and the object detection networks include YOLO models or SSD series models.
7. The method for extracting water bodies from remote sensing images based on multimodal spatiotemporal complementarity and large-model synergy as described in claim 1, characterized in that, The visual foundation model is a general-purpose large model with cue-driven segmentation capability, including the SAM model or the Fast SAM model.
8. A remote sensing image water body extraction system based on multimodal spatiotemporal complementarity and large model synergy, characterized in that, include: The data acquisition module is used to acquire first-modal remote sensing images and second-modal remote sensing images of the same geographic area to be detected. The preliminary extraction module is used to perform reasoning on the first modal remote sensing image and the second modal remote sensing image using a pre-trained first lightweight semantic segmentation network and a second lightweight semantic segmentation network, respectively, to obtain a first preliminary mask and a second preliminary mask. The target set generation module is used to perform connected component analysis and boundary extraction on the first preliminary mask to generate a first candidate target set containing several potential water targets. Perform connected component analysis on the second preliminary mask to generate a second set of verification targets containing several verification water targets; The logic fusion module is used for a logic gating strategy based on heterogeneous feature alignment and validity observation judgment. It uses the second verification target set to perform spatial overlap verification on each candidate water body target in the first candidate target set to obtain a valid prompt. The large model inference module is used to input the effective prompts and the first modal remote sensing image into a pre-trained visual base model, so as to perform fine segmentation of the first modal remote sensing image under the guidance of the effective prompts and output the final high-precision water body extraction result.
9. An electronic device, characterized in that, The device includes a memory and a processor, wherein the memory stores program instructions that are executed by the processor, and the processor invokes the program instructions to execute the remote sensing image water body extraction method based on multimodal spatiotemporal complementarity and large model collaboration as described in any one of claims 1 to 7.
10. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that cause the computer to execute the remote sensing image water body extraction method based on multimodal spatiotemporal complementarity and large model collaboration as described in any one of claims 1 to 7.