Semantic segmentation method and device for remote sensing image

Abstract

The application provides a semantic segmentation method and device for remote sensing images, and relates to the technical field of image processing. The semantic segmentation method for remote sensing images comprises the following steps: training an encoder by using a plurality of first remote sensing image samples to obtain a remote sensing visual encoder; fine-tuning the remote sensing visual encoder and updating the fine-tuning result; constructing a basic remote sensing segmentation model based on the updated remote sensing visual encoder, a mask decoder and a segmentation head; training the segmentation head of the basic remote sensing segmentation model based on knowledge transfer according to a plurality of pre-trained segmentation models by using second remote sensing image samples and segmentation true values, obtaining a target segmentation head, and further obtaining a target remote sensing semantic segmentation model; and inputting to-be-segmented remote sensing image data into the target remote sensing semantic segmentation model to obtain a remote sensing semantic segmentation result. The application realizes high-precision, strong-generalization and low-cost remote sensing semantic segmentation through the pre-training + teacher-student model knowledge transfer paradigm.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and specifically to a semantic segmentation method and apparatus for remote sensing images. Background Technology

[0002] Currently, with the rapid development of remote sensing satellite and UAV observation technologies, both the spatial and temporal resolutions of surface information acquisition have achieved a qualitative leap. High-resolution remote sensing imagery can provide rich textures, spectral and geometric features of ground features, offering crucial data support for fields such as land surveys, ecological monitoring, agricultural yield estimation, disaster assessment, and urban planning. However, with the exponential growth of remote sensing data, how to efficiently and accurately extract semantic information from massive amounts of imagery has become an important research direction for intelligent remote sensing analysis.

[0003] In the field of semantic segmentation of remote sensing images, the existing mainstream methods can be mainly divided into the following categories: traditional machine learning methods, deep learning methods based on convolutional neural networks (CNN), global modeling methods based on the Transformer structure, and basic models in the field of remote sensing.

[0004] However, most existing models rely on labeled data, which is inefficient and costly. At the same time, existing CNN or Transformer models have poor generalization and are difficult to transfer stably across different scenarios. Although large remote sensing models are versatile, they lack task-level accuracy and are difficult to deploy in a lightweight manner. Summary of the Invention

[0005] Based on this, this application provides a semantic segmentation method and apparatus for remote sensing images, which achieves high-precision, strong generalization, and low-cost remote sensing semantic segmentation, and can effectively solve the dual problems of insufficient accuracy of large models and poor generalization of small models.

[0006] According to one aspect of this application, a semantic segmentation method for remote sensing images is proposed, comprising: training a pre-constructed encoder using multiple pre-set first remote sensing image samples to obtain a remote sensing visual encoder; fine-tuning the remote sensing visual encoder and updating it using the fine-tuning results; constructing a basic remote sensing segmentation model based on the updated remote sensing visual encoder, a pre-constructed mask decoder, and a pre-constructed segmentation head; using pre-set second remote sensing image samples and their segmentation ground truth, and according to multiple pre-trained segmentation models, performing knowledge transfer-based training on the segmentation head of the basic remote sensing segmentation model to obtain a target segmentation head, thereby obtaining a target remote sensing semantic segmentation model; and inputting the remote sensing image data to be segmented into the target remote sensing semantic segmentation model to obtain the remote sensing semantic segmentation result.

[0007] According to some embodiments, a pre-constructed encoder is trained using a plurality of preset first remote sensing image samples to obtain a remote sensing visual encoder, including: performing self-supervised training on a pre-constructed first visual encoder using a plurality of first remote sensing image samples to obtain a first training result as a remote sensing visual encoder, and / or performing image-text comparison training on a pre-constructed second visual encoder using a plurality of first remote sensing image samples to obtain a second training result as a remote sensing visual encoder.

[0008] According to some embodiments, a first visual encoder pre-constructed is self-supervised and trained using multiple first remote sensing image samples to obtain a first training result as the remote sensing visual encoder. This includes: dividing each first remote sensing image sample into multiple image blocks, and randomly masking the multiple image blocks according to a preset ratio to obtain visible image blocks and mask labels; inputting the visible image blocks into the pre-constructed first visual encoder to obtain feature representations of the visible image blocks; inputting the feature representations and mask labels of the visible image blocks into a pre-constructed decoder to obtain reconstructed image samples; calculating the MAE reconstruction loss based on the difference between the first remote sensing image samples and the reconstructed image samples, and training the first visual encoder based on the MAE reconstruction loss to obtain a first training result as the remote sensing visual encoder.

[0009] According to some embodiments, using multiple first remote sensing image samples, a pre-constructed second visual encoder is trained using image-text contrast to obtain a second training result as the remote sensing visual encoder. This includes: inputting the first remote sensing image samples into the pre-constructed second visual encoder to obtain first sample image features; inputting the text description of the first remote sensing image samples into a pre-constructed text encoder to obtain first sample text features; calculating the similarity matrix between the first sample image features and the first sample text features; calculating the image-text contrast loss based on the similarity matrix; and training the second visual encoder based on the image-text contrast loss to obtain a second training result as the remote sensing visual encoder.

[0010] According to some embodiments, the first visual encoder is loaded with parameters from the second training result.

[0011] According to some embodiments, the second visual encoder is loaded with parameters from the first training result.

[0012] According to some embodiments, a pre-constructed encoder is trained using a plurality of preset first remote sensing image samples to obtain a remote sensing visual encoder, including: dividing each first remote sensing image sample into multiple image blocks, and randomly masking the multiple image blocks according to a preset ratio to obtain visible image blocks and mask markers; inputting the visible image blocks into a pre-constructed third visual encoder to obtain feature representations of the visible image blocks; inputting the feature representations of the visible image blocks and mask markers into a pre-constructed decoder to obtain reconstructed image samples; calculating the MAE reconstruction loss based on the difference between the first remote sensing image samples and the reconstructed image samples; inputting the first remote sensing image samples into the third visual encoder to obtain second sample image features, and inputting the text description of the first remote sensing image samples into a pre-constructed text encoder to obtain second sample text features; calculating the similarity matrix between the second sample image features and the second sample text features; calculating the image-text contrast loss based on the similarity matrix; calculating the joint optimization loss based on the MAE reconstruction loss and the image-text contrast loss; and training the third visual encoder based on the joint optimization loss to obtain a third training result as the remote sensing visual encoder.

[0013] According to some embodiments, the remote sensing visual encoder is fine-tuned, and the result of the fine-tuning is used to update the remote sensing visual encoder, including: freezing the parameters of the remote sensing visual encoder and inserting a first low-rank matrix in parallel next to the attention module of the remote sensing visual encoder; updating the gradient of the first low-rank matrix using the LoRA low-rank fine-tuning strategy; merging the updated first low-rank matrix with the parameters of the remote sensing visual encoder, and using the merged result to update the parameters of the remote sensing visual encoder.

[0014] According to some embodiments, using a preset second remote sensing image sample and its segmentation ground value, and based on multiple pre-trained segmentation models, the segmentation head of the basic remote sensing segmentation model is trained based on knowledge transfer to obtain a target segmentation head, and then a target remote sensing semantic segmentation model is obtained. This includes: inputting the second remote sensing image sample into an updated remote sensing visual encoder to obtain third sample image features; inputting the second remote sensing image sample into multiple pre-trained segmentation models to obtain multiple segmentation results; inputting the third sample image features into a mask decoder to obtain sample semantic information; inputting the sample semantic information into a pre-constructed segmentation head to obtain predicted segmentation results; and training the segmentation head using the multiple segmentation results, the segmentation ground value corresponding to the second remote sensing image sample, and the difference between the predicted segmentation results and the actual segmentation results to obtain the target segmentation head.

[0015] According to some embodiments, the mask decoder is pre-trained through the following steps: a second low-rank matrix is ​​inserted in parallel next to the attention module of the pre-built decoder to obtain the decoder to be fine-tuned; the gradient of the second low-rank matrix is ​​updated using the LoRA low-rank fine-tuning strategy; and the updated second low-rank matrix is ​​merged with the parameters of the decoder to obtain the mask decoder.

[0016] According to some embodiments, the method further includes: performing a hierarchical structured reorganization of the remote sensing semantic segmentation results to obtain a hierarchical segmentation result; performing geometric optimization based on spatial consistency constraints on the hierarchical segmentation results to obtain a topology optimization result; converting the topology optimization result into a preset data structure format to obtain a standardized remote sensing semantic segmentation result; and overlaying the standardized remote sensing semantic segmentation result onto the remote sensing image data to be segmented.

[0017] According to some embodiments, the remote sensing image data to be segmented is input into the target remote sensing semantic segmentation model to obtain the remote sensing semantic segmentation result, including: inputting the remote sensing image data to be segmented into the updated remote sensing visual encoder to obtain target image features; inputting the target image features into the mask decoder to obtain target semantic information; and inputting the target semantic information into the target segmentation head to obtain the remote sensing semantic segmentation result.

[0018] According to one aspect of this application, a semantic segmentation apparatus for remote sensing images includes: an encoder training unit, used to train a pre-constructed encoder using a plurality of preset first remote sensing image samples to obtain a remote sensing visual encoder; an encoder fine-tuning unit, used to fine-tune the remote sensing visual encoder and update the remote sensing visual encoder using the fine-tuning results; a base model unit, used to construct a base remote sensing segmentation model based on the updated remote sensing visual encoder, a pre-constructed mask decoder, and a pre-constructed segmentation head; a segmentation head training unit, used to perform knowledge transfer-based training on the segmentation head of the base remote sensing segmentation model using preset second remote sensing image samples and their segmentation ground truth values, according to a plurality of pre-trained segmentation models, to obtain a target segmentation head, and thus obtain a target remote sensing semantic segmentation model; and a model inference unit, used to input remote sensing image data to be segmented into the target remote sensing semantic segmentation model to obtain a remote sensing semantic segmentation result.

[0019] According to one aspect of this application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method as described above.

[0020] According to one aspect of this application, a computer-readable medium is provided that stores a computer program or instructions thereon, which, when executed by a processor, implement the method as described above.

[0021] Through the embodiments provided in this application, during the training phase, a remote sensing visual encoder is constructed using a large-scale first remote sensing image sample, and the obtained remote sensing visual encoder is fine-tuned to obtain a remote sensing visual encoder adapted to remote sensing semantics, fully exploiting the value of unlabeled data while significantly reducing the cost of manual annotation; based on the updated remote sensing visual encoder, a pre-built mask decoder, and a pre-built segmentation head, a basic remote sensing segmentation model is constructed; the basic remote sensing segmentation model is used as a student model, and simultaneously, using labeled small-scale second remote sensing image samples, through teacher-student knowledge transfer, the segmentation capabilities of multiple specialized segmentation models in specific domains are effectively transferred to the segmentation head of the student model, enabling the student model to achieve high-precision segmentation under small sample conditions, while maintaining robustness and adaptability across regions and task scenarios, thus balancing accuracy and generalization performance; the trained target segmentation head, the updated remote sensing visual encoder, and the pre-built mask decoder together constitute the target remote sensing semantic segmentation model; during the inference phase, the remote sensing image data to be segmented is input into the target remote sensing semantic segmentation model to obtain the remote sensing semantic segmentation result, achieving high-precision, strong generalization, and low-cost remote sensing semantic segmentation. Attached Figure Description

[0022] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application.

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings, without exceeding the scope of protection claimed by this application.

[0024] Figure 1 One of the flowcharts for a semantic segmentation method for remote sensing images provided in an embodiment of this application;

[0025] Figure 2 The flowchart of the semantic segmentation method for remote sensing images provided in this application embodiment utilizes multiple first remote sensing image samples to perform self-supervised training on a pre-constructed first visual encoder, and obtains the first training result as a remote sensing visual encoder.

[0026] Figure 3 The flowchart of the semantic segmentation method for remote sensing images provided in this application embodiment utilizes multiple first remote sensing image samples to perform image-text comparison training on a pre-constructed second visual encoder, and obtains the second training result as a remote sensing visual encoder.

[0027] Figure 4The flowchart of the semantic segmentation method for remote sensing images provided in this application embodiment is as follows: a pre-constructed encoder is trained using multiple preset first remote sensing image samples to obtain a remote sensing visual encoder.

[0028] Figure 5 A schematic diagram illustrating the iterative training process of a third visual encoder for the semantic segmentation method of remote sensing images provided in this application embodiment;

[0029] Figure 6 The flowchart illustrates the process of fine-tuning the remote sensing visual encoder and updating the remote sensing visual encoder using the fine-tuning results in the semantic segmentation method for remote sensing images provided in this application embodiment.

[0030] Figure 7 The semantic segmentation method for remote sensing images provided in this application utilizes a preset second remote sensing image sample and its segmentation ground value. Based on multiple pre-trained segmentation models, the segmentation head of the basic remote sensing segmentation model is trained based on knowledge transfer to obtain the target segmentation head, and thus the flowchart of the target remote sensing semantic segmentation model is obtained.

[0031] Figure 8 This is a schematic diagram of the architecture of the teacher model for the semantic segmentation method of remote sensing images provided in the embodiments of this application;

[0032] Figure 9 A flowchart of the mask decoder pre-training steps of the semantic segmentation method for remote sensing images provided in the embodiments of this application;

[0033] Figure 10 A schematic diagram of the architecture of the target remote sensing semantic segmentation model in knowledge transfer training for the semantic segmentation method of remote sensing images provided in the embodiments of this application;

[0034] Figure 11 A second schematic flowchart of the semantic segmentation method for remote sensing images provided in this application embodiment;

[0035] Figure 12 The flowchart illustrates the process of inputting remote sensing image data to be segmented into a target remote sensing semantic segmentation model to obtain the remote sensing semantic segmentation result in the semantic segmentation method for remote sensing images provided in this application embodiment.

[0036] Figure 13 One of the schematic diagrams of the remote sensing image data to be segmented, target semantic information, and remote sensing semantic segmentation results in the inference stage of the semantic segmentation method for remote sensing images provided in the embodiments of this application;

[0037] Figure 14 This is the second schematic diagram of the remote sensing image data to be segmented, target semantic information, and remote sensing semantic segmentation results in the inference stage of the semantic segmentation method for remote sensing images provided in this application embodiment.

[0038] Figure 15 The third schematic diagram shows the remote sensing image data to be segmented, target semantic information, and remote sensing semantic segmentation results in the inference stage of the semantic segmentation method for remote sensing images provided in the embodiments of this application.

[0039] Figure 16 The fourth schematic diagram shows the remote sensing image data to be segmented, target semantic information, and remote sensing semantic segmentation results in the inference stage of the semantic segmentation method for remote sensing images provided in the embodiments of this application.

[0040] Figure 17 A block diagram of a semantic segmentation apparatus for remote sensing images provided in an embodiment of this application;

[0041] Figure 18 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0042] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0044] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0045] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily need to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0046] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of this application. As used herein, the term "and / or" includes all combinations of any one and more of the associated listed items.

[0047] In the field of semantic segmentation of remote sensing images, existing mainstream methods can be mainly divided into the following categories:

[0048] The first category is traditional machine learning methods, such as Support Vector Machines (SVM) and Random Forest. These methods rely on manually designed features (texture, color, spectral indices, etc.), which perform reasonably well in low-resolution images or simple scenes, but their performance is severely limited when faced with complex terrain features, shadow interference, or multi-scale objects. In addition, manually designed features lack universality and are difficult to adapt to data differences in different regions, seasons, and sensor conditions.

[0049] The second category comprises deep learning methods based on convolutional neural networks (CNNs), such as FCN, U-Net, DeepLab, and SegNet. These methods achieve automatic feature extraction within an end-to-end framework, significantly improving segmentation accuracy. However, they heavily rely on large-scale, high-quality labeled samples, while the labeling of remote sensing images is often costly, inefficient, and heavily influenced by expert subjectivity. Furthermore, CNN structures primarily focus on local receptive fields, limiting their ability to model wide-area spatial relationships in remote sensing images, resulting in poor cross-regional generalization and instability under geographical migration or multi-source data conditions.

[0050] The third category comprises global modeling methods based on the Transformer architecture, such as Swing Transformer, SegFormer, and ViT-Adapter. These methods can capture long-range dependencies and enhance semantic hierarchical representation capabilities, achieving significant progress in high-resolution scenes. However, Transformer models have a large parameter scale, making training from scratch extremely costly. Even when using publicly available pre-trained models for transfer learning, the "domain shift" problem often occurs, meaning the model performs well on general image data but its accuracy drops significantly in remote sensing scenes. Furthermore, Transformer models are prone to overfitting under small sample conditions and are sensitive to the number and distribution of annotations.

[0051] The fourth category is Remote Sensing Foundation Models. These models typically borrow from large-scale model paradigms in the natural image domain (such as CLIP, SAM, DINOv2, and DINOv3), and are pre-trained using self-supervised or contrastive learning on large-scale unlabeled remote sensing data to form a remote sensing foundation model with general feature representation capabilities. Although these large-scale remote sensing models have shown good potential in transfer learning and multi-task scenarios, they still have problems such as insufficient accuracy for specific tasks, limited domain adaptability, and high resource consumption.

[0052] In summary, current remote sensing semantic segmentation technology faces the following main bottlenecks:

[0053] (1) Insufficient high-quality labeled data and high cost;

[0054] (2) Existing CNN or Transformer models have poor generalization ability and are difficult to transfer stably across different scenarios;

[0055] (3) Although the remote sensing large model has universality, it lacks mission-level accuracy and is difficult to deploy in a lightweight manner.

[0056] Based on this, this application proposes a semantic segmentation method and apparatus for remote sensing images.

[0057] For specific implementation details, please refer to the following examples.

[0058] Figure 1 This is one of the flowcharts for a semantic segmentation method for remote sensing images provided in an embodiment of this application. For example... Figure 1 As shown, the method includes steps S110-S150.

[0059] In step S110, the pre-constructed encoder is trained using a plurality of preset first remote sensing image samples to obtain a remote sensing visual encoder.

[0060] This application utilizes large-scale unlabeled remote sensing data to enable a pre-built encoder to learn feature representations, thereby obtaining a remote sensing visual encoder. This application does not restrict the acquisition method of the large-scale unlabeled remote sensing data; it can use publicly available datasets or collected data.

[0061] To distinguish them, the data used to train the remote sensing visual encoder is denoted as the first remote sensing image sample.

[0062] Pre-built encoders can use publicly available encoder models, and this application does not impose any restrictions on this. According to the example embodiment, DINOv3 is selected as the encoder.

[0063] In step S120, the remote sensing visual encoder is fine-tuned, and the result of the fine-tuning is used to update the remote sensing visual encoder.

[0064] The fine-tuning method can be selected according to the actual situation. For example, the Parameter Efficient Fine-Tuning (PEFT) method can be selected to fine-tune the remote sensing visual encoder.

[0065] According to the example implementation, the Parameter Efficient Tuning (PEFT) method includes tuning methods such as Adapter, PromptTuning, and LoRA.

[0066] After fine-tuning, the fine-tuning result is used as the new remote sensing visual encoder.

[0067] In step S130, a basic remote sensing segmentation model is constructed based on the updated remote sensing visual encoder, the pre-built mask decoder, and the pre-built segmentation head.

[0068] The mask decoder and segmentation head are selected according to the actual situation, and this application does not impose any restrictions on them.

[0069] In step S140, using a preset second remote sensing image sample and its segmentation ground truth, the segmentation head of the basic remote sensing segmentation model is trained based on knowledge transfer according to multiple pre-trained segmentation models to obtain the target segmentation head, and then the target remote sensing semantic segmentation model is obtained.

[0070] A basic remote sensing segmentation model is trained using small-scale labeled remote sensing image data.

[0071] During the training process, multiple specialized segmentation models in specific domains are used as teacher models, and basic remote sensing segmentation models are used as student models. The main approach is to use a knowledge transfer technique based on the teacher-student paradigm to effectively transfer the segmentation capabilities of multiple specialized segmentation models in specific domains to the student model.

[0072] During the training of the student model, the weights of the remote sensing visual encoder were kept frozen to maximize its strong generalization ability and reduce computational overhead. The training primarily focused on the segmentation head to achieve knowledge transfer.

[0073] To distinguish them, the data used for knowledge transfer training is designated as the second remote sensing image sample.

[0074] Furthermore, to achieve efficient and low-cost domain adaptation, the parameters of the mask decoder can also be updated in a fine-tuning manner during the pre-training stage.

[0075] The trained segmentation head is denoted as the target segmentation head. The target segmentation head, together with the remote sensing visual encoder and the mask decoder, constitutes the target remote sensing semantic segmentation model.

[0076] In step S150, the remote sensing image data to be segmented is input into the target remote sensing semantic segmentation model to obtain the remote sensing semantic segmentation result.

[0077] During the inference process, the remote sensing image data to be segmented is input into the target remote sensing semantic segmentation model, and the remote sensing semantic segmentation result is directly output.

[0078] In the training phase, this application constructs a remote sensing visual encoder using a large-scale set of first remote sensing image samples and fine-tunes the resulting encoder to obtain one adapted to remote sensing semantics. This fully leverages the value of unlabeled data while significantly reducing manual annotation costs. A basic remote sensing segmentation model is built based on the updated remote sensing visual encoder, a pre-built mask decoder, and a pre-built segmentation head. This basic model serves as a student model. Simultaneously, using labeled small-scale second remote sensing image samples, teacher-student knowledge transfer effectively transfers the segmentation capabilities of multiple specialized segmentation models in specific domains to the student model's segmentation head. This enables the student model to achieve high-precision segmentation under small sample conditions while maintaining robustness and adaptability across regions and task scenarios, thus balancing accuracy and generalization performance. The trained target segmentation head, the updated remote sensing visual encoder, and the pre-built mask decoder together form the target remote sensing semantic segmentation model. In the inference phase, the remote sensing image data to be segmented is input into the target remote sensing semantic segmentation model to obtain the remote sensing semantic segmentation result, achieving high-precision, strong generalization, and low-cost remote sensing semantic segmentation.

[0079] According to some embodiments, in step S110, a pre-constructed encoder is trained using a plurality of preset first remote sensing image samples to obtain a remote sensing visual encoder, which can be specifically implemented through step S111 and / or step S112.

[0080] In step S111, a pre-constructed first visual encoder is trained using multiple first remote sensing image samples to obtain a first training result as the remote sensing visual encoder.

[0081] To address the issues of poor generalization of remote sensing image segmentation under different scenarios (such as climate, region, and lighting) and the fact that traditional segmentation models rely on a large amount of labeled data and are difficult to cover diverse scenarios, a self-supervised learning pre-training strategy is adopted to perform large-scale pre-training on a pre-constructed visual encoder, and the first training result is used as the remote sensing visual encoder.

[0082] To achieve global semantic learning and local detail capture of remote sensing images, self-supervised learning pre-training strategies include self-supervised learning based on the mask autoencoder (MAE) paradigm, SimCLR (A Simple Framework for Contrastive Learning of Visual Representations), BYOL (Bootstrap Your Own Latent), MoCo (Momentum Contrast), or rotation prediction tasks. This application does not impose any specific limitations on these strategies.

[0083] This application uses the Mask Autoencoder (MAE) paradigm as an example for illustration.

[0084] Understandably, the Masked Autoencoder (MAE) is an efficient self-supervised learning framework that draws inspiration from BERT in natural language processing. It drives the model to learn the global and local structural relationships by reconstructing the masked parts of the image.

[0085] Furthermore, for the purpose of differentiation, the visual encoder based on self-supervised training is referred to as the first visual encoder.

[0086] In step S112, multiple first remote sensing image samples are used to train the pre-constructed second visual encoder using image-text contrast, and the second training result is used as the remote sensing visual encoder.

[0087] To enhance semantic understanding capabilities, this application introduces a pre-training paradigm based on contrastive learning to train the visual encoder and obtain a second training result as the remote sensing visual encoder.

[0088] Understandably, the core idea of ​​image-text contrastive training is to establish a connection between the visual representation of a remote sensing image and its corresponding natural language description within the same high-dimensional feature space, thereby enabling the model to learn to "understand" the deeper meaning of the image. This architecture includes a visual encoder and a text encoder. This paradigm aims to enable the model to learn deeper and more generalized semantic representations by aligning visual information with natural language descriptions.

[0089] Furthermore, for the purpose of differentiation, the visual encoder trained based on image-text contrast is referred to as the second visual encoder.

[0090] It should be noted that before performing image-text comparison training, the first remote sensing image sample also needs to be annotated with natural language descriptions.

[0091] According to an example embodiment, a natural language description of the first remote sensing image sample is generated based on prompt engineering.

[0092] The following provides a further explanation of the specific steps involved in generating a natural language description of the first remote sensing image sample based on prompt engineering, as mentioned in the embodiments of this application.

[0093] The Vision Language Model (VLM) is used as the engine for generating natural language descriptions.

[0094] When generating text descriptions, the input to VLM is not limited to the original remote sensing images but also incorporates multimodal structured annotation information from the source dataset. For example, segmentation masks, bounding boxes, scene category labels, and other land cover attributes are all input as part of the prompts. This approach allows the model to focus on specific regions or targets in the image when generating descriptions, providing more accurate and fine-grained semantic annotations. For instance, for urban building clusters, the model can distinguish different building types or functional areas; for road networks, it can describe their morphology, connectivity, and surrounding environmental features.

[0095] Furthermore, this application's embodiments introduce a hierarchical and conditional strategy in the prompt design to enhance the diversity and interpretability of text descriptions. Specifically, prompts can be divided into three levels: global scene description, local target description, and detailed attribute description. The generation strategy can be dynamically adjusted according to different task requirements. For example, when constructing a traffic scene dataset, detailed descriptions of roads, vehicles, and traffic facilities can be prioritized, while in urban planning-related tasks, high-level information such as building layout and green space coverage can be emphasized. Through this multi-layered and controllable prompt generation method, VLM can generate text tags that are rich in content, clearly structured, and highly matched to visual features.

[0096] Ultimately, through a text generation strategy based on prior knowledge and structured prompts, VLM served as a highly efficient "semantic enhancement engine" in this task. It not only rapidly processes massive amounts of remote sensing image data but also generates high-quality, refined, and semantically complete text annotations, providing reliable data support for CLIP-style contrastive learning. This significantly improves the visual encoder's semantic understanding and multi-task generalization capabilities in urban remote sensing scenarios.

[0097] Furthermore, self-supervised training and image-text contrast training can be performed simultaneously, including iterative training and joint training.

[0098] According to some embodiments, refer to Figure 2 In step S111, a pre-constructed first visual encoder is trained using multiple first remote sensing image samples to obtain the first training result as the remote sensing visual encoder. This can be achieved through steps S210-S240.

[0099] In step S210, each first remote sensing image sample is divided into multiple image blocks, and the multiple image blocks are randomly masked according to a preset ratio to obtain visible image blocks and mask markers.

[0100] For the first input remote sensing image sample, it is divided into a series of non-overlapping image patches. Then, a large portion of these image patches are randomly "discarded" (i.e., masked) according to a preset ratio.

[0101] According to the example embodiment, the input first remote sensing image sample is divided into non-overlapping 16×16 pixel blocks, and the preset ratio of random occlusion is 75%.

[0102] The first input remote sensing image sample is masked so that the model cannot rely on simple interpolation based on local neighbor information, but must understand the deep semantic information such as the global structure and local distribution of the image in order to complete the image reconstruction task.

[0103] In step S220, the visible image block is input to a pre-built first visual encoder to obtain the feature representation of the visible image block.

[0104] The feature representation of visible image patches is extracted using a pre-built first visual encoder.

[0105] In step S230, the feature representation and mask mark of the visible image block are input into the pre-built decoder to obtain the reconstructed image sample.

[0106] In this application, the decoder's task is to use the contextual information of the visible blocks to predict and reconstruct the original pixel values ​​of the occluded area, thereby obtaining a reconstructed image sample.

[0107] The decoder's input includes feature representations of the visible blocks generated by the encoder, as well as "mask markers" representing the occluded locations.

[0108] Pre-built decoders can be selected as needed.

[0109] According to the example embodiment, a simple cascaded multi-layer Transformer is used as the decoder.

[0110] In step S240, the MAE reconstruction loss is calculated based on the difference between the first remote sensing image sample and the reconstructed image sample, and the first visual encoder is trained based on the MAE reconstruction loss to obtain the first training result as the remote sensing visual encoder.

[0111] The optimization objective is achieved by setting a loss function.

[0112] In this embodiment, the optimization objective of the first visual encoder model is to minimize the difference between the reconstructed image and the original image in the occluded area.

[0113] Based on this, the embodiments of the present invention use mean squared error (MSE) as the loss function to calculate the difference between the predicted pixel value and the true pixel value. Through backpropagation of this reconstruction loss, the encoder is driven to learn a powerful feature representation that can capture the inherent regularity of remote sensing images. The training result obtained is recorded as the first training result, which is the remote sensing visual encoder.

[0114] The embodiments of this application are based on the self-supervised pre-training mechanism of mask autoencoder (MAE), which realizes pixel-level reconstruction of remote sensing images. This enables the model to fully capture local detailed features during the learning process, avoid feature bias caused by local overfitting, and thus build a more robust basic feature representation.

[0115] According to some embodiments, refer to Figure 3 In step S112, multiple first remote sensing image samples are used to train the pre-constructed second visual encoder by image-text comparison, and the second training result is used as the remote sensing visual encoder. This can be achieved through steps S310-S330.

[0116] In step S310, the first remote sensing image sample is input to the pre-built second visual encoder to obtain the first sample image features, and the text description of the first remote sensing image sample is input to the pre-built text encoder to obtain the first sample text features.

[0117] This embodiment employs the multimodal contrastive learning framework CLIP (Contrastive Language-Image Pre-training), which includes a visual encoder and a text encoder.

[0118] A batch of first remote sensing image samples and their paired text descriptions are fed into the second visual encoder and the text encoder respectively to extract image feature vectors and text feature vectors. The resulting image feature vectors are denoted as the first sample image features (I1, I2, ...), and the resulting text feature vectors are denoted as the first sample text features (T1, T2, ...).

[0119] In step S320, the similarity matrix between the first sample image features and the first sample text features is calculated.

[0120] Calculate the similarity matrix of the two feature vector sets (I1, I2, ...) of the first sample image features and (T1, T2, ...) of the first sample text features.

[0121] In step S330, the image-text contrast loss is calculated based on the similarity matrix, and the second visual encoder is trained based on the image-text contrast loss to obtain the second training result as the remote sensing visual encoder.

[0122] Based on the similarity matrix, a contrastive loss is applied to drive the optimization of the model, maximizing the similarity of matched image-text pairs and minimizing the similarity of non-matching pairs, thereby improving semantic understanding. The training result obtained is denoted as the second training result, which is the remote sensing visual encoder.

[0123] The embodiments of this application employ image-text comparison training, enabling the model to form a holistic understanding of the entire scene in multimodal association, thereby deepening the understanding and expression of the semantic relationships within the scene.

[0124] According to some embodiments, the first visual encoder is loaded with parameters from the second training result.

[0125] In this embodiment, self-supervised training and image-text contrast training are performed simultaneously in an iterative manner, so that the remote sensing visual encoder obtained after training has both fine-grained structural perception capability and global semantic understanding capability.

[0126] Based on this, the order of self-supervised training and image-text contrast training is based on the iterative process. When image-text contrast training is performed first, the first visual encoder is loaded with the parameters of the second training result.

[0127] According to some embodiments, the second visual encoder is loaded with parameters from the first training result.

[0128] In this embodiment, self-supervised training and image-text contrast training are performed simultaneously in an iterative manner, so that the remote sensing visual encoder obtained after training has both fine-grained structural perception capability and global semantic understanding capability.

[0129] Based on this, the order of self-supervised training and image-text contrast training is based on the iterative process. When self-supervised training is performed first, the second visual encoder is loaded with the parameters of the first training result.

[0130] According to some embodiments, refer to Figure 4In step S110, the pre-constructed encoder is trained using multiple preset first remote sensing image samples to obtain a remote sensing visual encoder, which can be specifically implemented through steps S410-S490.

[0131] In step S410, each first remote sensing image sample is divided into multiple image blocks, and the multiple image blocks are randomly masked according to a preset ratio to obtain visible image blocks and mask markers.

[0132] In step S420, the visible image patch is input to the pre-built third visual encoder to obtain the feature representation of the visible image patch.

[0133] In step S430, the feature representation and mask mark of the visible image block are input into the pre-built decoder to obtain the reconstructed image sample.

[0134] In step S440, the MAE reconstruction loss is calculated based on the difference between the first remote sensing image sample and the reconstructed image sample.

[0135] In this embodiment, self-supervised training and image-text contrast training are performed simultaneously in a joint training manner. For distinction, the visual encoder in the joint training is referred to as the third visual encoder.

[0136] During self-supervised training, the steps of image block partitioning, masking, feature representation, image reconstruction, and MAE reconstruction loss calculation are performed. For details, please refer to steps S210-S240.

[0137] In step S450, the first remote sensing image sample is input to the third visual encoder to obtain the second sample image features, and the text description of the first remote sensing image sample is input to the pre-built text encoder to obtain the second sample text features.

[0138] During the image-text comparison training process, image feature vectors and text feature vectors are first extracted. The resulting image feature vectors are denoted as the second sample image features, and the resulting text feature vectors are denoted as the second sample text features. For specific details, refer to step S310.

[0139] This application embodiment employs self-supervised training and image-text contrast training to jointly train the third visual encoder. See the flowchart below. Figure 5 In the diagram, the visual encoder refers to the third visual encoder.

[0140] In step S460, the similarity matrix between the second sample image features and the second sample text features is calculated.

[0141] Calculate the similarity matrix between the two feature vector sets: the image features of the second sample and the text features of the second sample.

[0142] In step S470, the image-text contrast loss is calculated based on the similarity matrix.

[0143] Based on the similarity matrix, the image-text contrast loss is calculated.

[0144] In step S480, the joint optimization loss is calculated based on the MAE reconstruction loss and the image-text contrast loss.

[0145] Based on the MAE reconstruction loss and the image-text contrast loss, a joint optimization loss is calculated.

[0146] According to the example implementation, the MAE reconstruction loss and the image-text contrast loss are weighted and summed together as the overall loss function of the model, denoted as the joint optimization loss.

[0147] In step S490, the third visual encoder is trained according to the joint optimization loss, and the third training result is used as the remote sensing visual encoder.

[0148] Based on the joint optimization loss, the third visual encoder is trained, and the training result is recorded as the third training result, which is the remote sensing visual encoder.

[0149] This application's embodiments achieve pixel-level reconstruction of remote sensing images by introducing a self-supervised pre-training mechanism based on masked autoencoders (MAE). This allows the model to fully capture local detailed features during the learning process, avoiding feature bias caused by local overfitting, thereby constructing a more robust basic feature representation. Simultaneously, by combining an image-text contrastive training strategy, the model develops a holistic understanding of the entire scene in multimodal association, deepening its understanding and expression of scene semantic relationships. The MAE reconstruction loss and image-text contrastive loss are jointly optimized during the pre-training stage, simultaneously constraining the model's learning in pixel-level reconstruction accuracy and cross-modal high-level semantic alignment capability. This allows the local and global feature capabilities obtained through pixel-level reconstruction using MAE to be combined with the scene-level holistic understanding provided by image-text contrastive training. The trained remote sensing visual encoder possesses both fine-grained structural perception capabilities and global semantic understanding capabilities, enabling it to balance detailed local information and overall semantic understanding in complex remote sensing scenes, achieving higher generalization ability, and providing more semantically consistent and expressive basic features for subsequent few-sample fine-tuning and task-specific segmentation.

[0150] According to some embodiments, refer to Figure 6 In step S120, the remote sensing visual encoder is fine-tuned, and the remote sensing visual encoder is updated using the fine-tuning results. This can be achieved through steps S610-S630.

[0151] In step S610, the parameters of the remote sensing visual encoder are frozen, and a first low-rank matrix is ​​inserted in parallel next to the attention module of the remote sensing visual encoder.

[0152] This application embodiment uses a high-performance ViT model (i.e., remote sensing visual encoder) that has been pre-trained on massive general image data as a starting point for fine-tuning.

[0153] Understandably, training a large-scale ViT (Vision Transformer) model directly from scratch is extremely costly. Therefore, performing full fine-tuning on the entire model not only consumes enormous computational resources but may also cause the model to forget valuable knowledge learned in general domains when learning new remote sensing domain knowledge, i.e., "catastrophic forgetting." To avoid the "catastrophic forgetting" and high computational cost caused by full fine-tuning, this embodiment employs low-rank adaptation (LoRA) technology, updating only the low-rank matrices (A and B) in the Transformer attention module.

[0154] Specifically, the weights of the original remote sensing visual encoder are frozen, and a low-rank matrix (ΔW = BA) for fine-tuning is inserted next to the attention module of the remote sensing visual encoder, denoted as the first low-rank matrix. Understandably, this first low-rank matrix is ​​inserted via an adapter.

[0155] In step S620, the gradient of the first low-rank matrix is ​​updated using the LoRA low-rank fine-tuning strategy.

[0156] The gradient of the first low-rank matrix is ​​updated using the LoRA low-rank fine-tuning strategy.

[0157] During the fine-tuning process, unlabeled first remote sensing image samples can be used.

[0158] According to the example embodiment, in this embodiment, the LoRA rank is set to 8, the training is performed for 100 epochs, and the learning rate is 1e-4.

[0159] In step S630, the updated first low-rank matrix is ​​merged with the parameters of the remote sensing visual encoder, and the parameters of the remote sensing visual encoder are updated using the merged result.

[0160] After training, the first low-rank matrix is ​​merged with the original weights of the remote sensing visual encoder without introducing inference delay, ensuring efficient adaptation while retaining general feature extraction capabilities.

[0161] Furthermore, to further optimize the fine-tuning effect, in some embodiments, a task-specific regularization strategy is introduced in LoRA fine-tuning.

[0162] For example, by adding an L2 regularization term to the loss function, the update magnitude of the low-rank matrix can be limited, preventing the model from deviating from the general features.

[0163] Furthermore, considering the spatial characteristics of remote sensing images, in some embodiments, dynamic adjustments to location encoding are added to the attention module to enhance the model's ability to perceive spatial relationships in the data.

[0164] The embodiments of this application employ a fine-tuning approach to construct a remote sensing visual encoder adapted to remote sensing semantics, fully leveraging the value of unlabeled data while significantly reducing the cost of manual annotation.

[0165] According to some embodiments, refer to Figure 7 In step S140, using the preset second remote sensing image sample and its segmentation ground value, the segmentation head of the basic remote sensing segmentation model is trained based on knowledge transfer according to multiple pre-trained segmentation models to obtain the target segmentation head, and then the target remote sensing semantic segmentation model is obtained. This can be achieved through steps S710-S740.

[0166] In step S710, the second remote sensing image sample is input into the updated remote sensing visual encoder to obtain the features of the third sample image, and the second remote sensing image sample is input into multiple pre-trained segmentation models to obtain multiple segmentation results.

[0167] A remote sensing visual encoder pre-trained on large-scale data is used as the backbone for feature extraction. To maximize its strong generalization ability and reduce computational overhead, the weights of the remote sensing visual encoder are frozen during the fine-tuning stage. The remote sensing visual encoder transforms the input second remote sensing image sample into a series of high-dimensional image features, denoted as the third sample image features.

[0168] In this embodiment, multiple parallel dedicated segmentation models are also introduced. Each segmentation model can directly process the second remote sensing image sample to generate a preliminary segmentation prediction and / or a segmentation prediction with specific advantages in certain aspects (such as boundary clarity), denoted as the segmentation result.

[0169] In step S720, the features of the third sample image are input into the mask decoder to obtain the sample semantic information.

[0170] The features of the third sample image are then fed into a mask decoder to integrate the features and obtain the sample semantic information.

[0171] The mask decoder can be selected as needed, or it can be fine-tuned based on the existing mask decoder.

[0172] In step S730, the semantic information of the sample is input into the pre-constructed segmentation head to obtain the predicted segmentation result.

[0173] The deep features (i.e., sample semantic information) from the mask decoder and the segmentation results from the dedicated segmentation model are fed together into a segmentation head for effective fusion. The segmentation head integrates the two sets of information to make a final pixel-level classification decision.

[0174] In the specific implementation process, the features (i.e. sample semantic information) decoded by the mask decoder are sent to the pre-constructed segmentation head to complete pixel-level semantic prediction and obtain the predicted segmentation result.

[0175] According to the example implementation, the PSPNet Head structure is selected to form the segmentation head. It is understood that, given the characteristics of large differences in the scale of ground features and complex spatial structures in remote sensing images, the PSPNet Head uses a pyramid pooling module to perform contextual modeling of feature maps at multiple scales, effectively fusing local detail information with global semantic information, and enhancing the model's overall understanding of large-scale scenes.

[0176] In step S740, the segmentation head is trained using multiple segmentation results and the ground truth segmentation values ​​corresponding to the second remote sensing image samples, as well as the differences between the ground truth and the predicted segmentation results, to obtain the target segmentation head.

[0177] Based on the predicted segmentation results, the segmentation head is optimized end-to-end by combining the ground truth of the second remote sensing image sample and multiple segmentation results output by the dedicated segmentation model. This allows for efficient learning of the segmentation capabilities for specific remote sensing tasks while fully utilizing the knowledge of the pre-trained model.

[0178] Among them, the ground truth serves as a supervision signal, while multiple segmentation results output by the dedicated segmentation model serve as complementary perspectives and features, jointly guiding the training of the segmentation head.

[0179] Furthermore, regarding the selection of dedicated segmentation models, in some embodiments, DeepLabv3+ and PSPNet are chosen as dedicated segmentation models for remote sensing semantic segmentation, used to construct the teacher network and provide high-quality supervision signals, referring to... Figure 8 . Figure 8 In this context, conv represents convolution, rate represents dilation, image pooling represents image pooling, and POOL represents pooling.

[0180] Understandably, both models belong to classic semantic segmentation architectures, suitable for refined processing of high-resolution remote sensing images. DeepLabv3+, based on atrous convolution and atrous spatial pyramid pooling (ASPP) modules, can expand the receptive field while maintaining feature resolution, exhibiting strong modeling capabilities for complex ground boundary and multi-scale targets; its encoder-decoder structure excels in fusing high-level semantic information with low-level spatial details, making it suitable for refined remote sensing segmentation tasks. PSPNet, on the other hand, aggregates global contextual information through multi-scale pooling operations, demonstrating significant advantages in addressing common problems in remote sensing images such as scale variations, class imbalance, and large-scene semantic consistency modeling. Both types of models have been widely validated in the field of remote sensing segmentation, exhibiting high maturity and stability.

[0181] It is important to emphasize that, in order to overcome the problems that dedicated segmentation models rely on a large amount of labeled data for training in specific domains and have limited generalization ability, while using only remote sensing semantic base models for segmentation makes it difficult to guarantee high accuracy, this application adopts a "pre-training + teacher-student model" paradigm. It designs an advanced parameter-efficient fine-tuning (PEFT) technique based on pre-trained models, which makes full use of large-scale unlabeled remote sensing data for feature representation learning to generate high-quality base features. The results of dedicated segmentation models are used as pseudo-labels to supervise the domain adaptation segmentation training of remote sensing base models. This organically integrates the segmentation experience and performance advantages accumulated by multiple dedicated segmentation models (teacher models) in their respective domains and passes them to a single student model (base remote sensing segmentation model), enabling it to achieve high-precision segmentation under small sample conditions and reducing the dependence on manually labeled data. Meanwhile, this method not only ensures the adaptability of the student model to complex remote sensing scenarios and its robustness across tasks and regions, but also balances accuracy and generalization performance. While ensuring segmentation accuracy, it significantly reduces training costs and manpower investment, and significantly enhances the practical value and technological innovation of the model in diverse application environments.

[0182] According to some embodiments, refer to Figure 9 The mask decoder can be pre-trained through steps S910-S930.

[0183] In step S910, a second low-rank matrix is ​​inserted in parallel next to the attention module of the pre-built decoder to obtain the decoder to be fine-tuned.

[0184] In step S920, the gradient of the second low-rank matrix is ​​updated using the LoRA low-rank fine-tuning strategy.

[0185] In step S930, the updated second low-rank matrix is ​​merged with the parameters of the decoder to obtain the mask decoder.

[0186] For the mask decoder, this embodiment uses the same method as for fine-tuning the remote sensing visual encoder. The fine-tuning method can be selected according to the actual situation, for example, the Parameter Efficient Fine-Tune (PEFT) method can be selected to fine-tune the remote sensing visual encoder.

[0187] According to the example implementation, the Parameter Efficient Tuning (PEFT) method includes tuning methods such as Adapter, PromptTuning, and LoRA.

[0188] This application uses low-rank adaptation (LoRA) technology as an example for illustration, referring to... Figure 10 , Figure 10 The diagram illustrates the process of training the segmentation head by keeping the weights of the remote sensing visual encoder frozen during knowledge transfer training, fine-tuning the mask decoder based on low-rank adaptation (LoRA) technology.

[0189] For detailed steps, please refer to steps S610-S630. The LoRA adaptation strategy introduces low-rank learnable components into the pre-trained weight matrix, updating only a small number of parameters. This allows the model to quickly complete transfer and fine-tuning under different remote sensing datasets and application scenarios, avoiding the high computational cost and overfitting risk of traditional full-parameter fine-tuning.

[0190] During the fine-tuning process, only a small number of injected LoRA parameters need to be updated, rather than the weights of the entire decoder, thus achieving "lightweight" fine-tuning.

[0191] According to some embodiments, the method further includes steps S160-S190.

[0192] In step S160, the remote sensing semantic segmentation results are structurally reorganized at the category level to obtain hierarchical segmentation results.

[0193] Considering that remote sensing segmentation results often have geometric problems such as jagged edges, breaks, and self-intersections, this application's embodiments designed a set of result optimization and standardized output processes for engineering applications after the remote sensing semantic segmentation results are generated, realizing a high-quality conversion from pixel-level segmentation results to topologically consistent vectorized results.

[0194] Specifically, for the remote sensing semantic segmentation results output by the target remote sensing semantic segmentation model, a hierarchical structured reorganization of categories is first performed, independently identifying and hierarchically storing different semantic categories in the segmentation mask. This step enables hierarchical management of target categories and differentiation of object instances, laying the foundation for subsequent spatial topology analysis.

[0195] According to the example implementation, the pixel-level label map of the remote sensing semantic segmentation result is used as input. The category label of each pixel is read, and the background region (category value of 0) is removed. Each foreground category is processed independently. To ensure that multiple discrete targets in the same category can be accurately separated, the Connected Component Labeling algorithm is used to divide the instance region based on pixel connectivity, thereby realizing instance-level target recognition and hierarchical classification management.

[0196] Meanwhile, the embodiments of this application support the synchronous processing of multiple categories and scales of objects to ensure the continuity of object boundaries in complex terrain scenes.

[0197] In step S170, the hierarchical segmentation result is geometrically optimized based on spatial consistency constraints to obtain the topology optimization result.

[0198] After obtaining the hierarchical segmentation results, this application proposes a geometric optimization approach based on spatial consistency constraints. By comprehensively adjusting the connectivity, smoothness, and adjacency of object boundaries, it corrects broken edges, eliminates pseudo-small blocks, and avoids polygon self-intersections, thereby ensuring that the results meet the topological requirements of closure and non-overlapping in spatial geometry.

[0199] According to the example implementation, based on the hierarchical segmentation results, the OpenCV findContours algorithm is used for outer contour detection. The RETR_EXTERNAL mode (detect only the outer contour) is used to extract only the outermost boundary, avoiding interference from holes. The CHAIN_APPROX_SIMPLE compression method is combined to reduce redundant boundary points, effectively reducing storage and computational overhead. For issues such as jagged edges, breaks, and self-intersections, the Shapely library's Polygon and buffer(0) techniques (polygon with zero-distance buffer repair) are used for geometric legalization, automatically correcting invalid polygons. When encountering complex terrain features that generate MultiPolygons, they are decomposed into multiple sub-polygons and recorded separately to ensure result integrity and topological rationality, thus obtaining topology optimization results. Optional steps include filtering or merging small-area noise or anomalous shapes to further optimize the robustness of the segmentation results.

[0200] Furthermore, step S170 is implemented using a modular design, which can flexibly adapt to the accuracy or complexity requirements of different scenarios.

[0201] In step S180, the topology optimization results are converted into a preset data structure format to obtain standardized remote sensing semantic segmentation results.

[0202] After completing the topology optimization, the topology optimization results are output in a unified preset data structure format, which is denoted as the standardized remote sensing semantic segmentation result.

[0203] According to the example embodiment, the preset data structure format includes JSON or GeoJSON format.

[0204] In addition to boundary coordinates, the standardized remote sensing semantic segmentation results can also include attribute fields such as category number, area, and perimeter in the output file, which facilitates subsequent statistical analysis and map making.

[0205] In step S190, the standardized remote sensing semantic segmentation results are overlaid on the remote sensing image data to be segmented.

[0206] To facilitate user viewing, the segmentation polygons from the standardized remote sensing semantic segmentation results are overlaid on the remote sensing image data to be segmented, enabling visual verification and facilitating the evaluation of the consistency between the vectorized boundaries and the original image.

[0207] According to the example embodiment, the segmented polygons are overlaid in a semi-transparent manner on the base image (i.e., the remote sensing image data to be segmented) that has undergone histogram equalization.

[0208] According to the example embodiment, ViT (Vision Transformer) is selected as the base model of the remote sensing visual encoder, and LoRA (Low Rank Adaptation) technology is selected for fine-tuning. Based on this, steps S110-S190 are performed. The overall flowchart is shown below. Figure 11 .

[0209] According to some embodiments, refer to Figure 12 In step S150, the remote sensing image data to be segmented is input into the target remote sensing semantic segmentation model to obtain the remote sensing semantic segmentation result, which can be achieved through steps S1210-S1230.

[0210] In step S1210, the remote sensing image data to be segmented is input into the updated remote sensing visual encoder to obtain the target image features.

[0211] Input a single remote sensing image and input a self-supervised pre-trained and fine-tuned remote sensing visual encoder to obtain a series of high-dimensional image features, denoted as target image features.

[0212] According to the example embodiment, the input remote sensing image data to be segmented can be referenced. Figure 13 (a) Figure 14 (a) Figure 15 (a) and Figure 16 (a).

[0213] In step S1220, the target image features are input to the mask decoder to obtain the target semantic information.

[0214] The target image features are fed into a mask decoder, and the extracted high-dimensional features are mapped back to a pixel-level semantic segmentation map, which is denoted as the target semantic information.

[0215] According to the example embodiment, the target semantic information can be referred to Figure 13 (b) Figure 14 (b) Figure 15 (b) and Figure 16 (b).

[0216] In step S1230, the target semantic information is input into the target segmentation head to obtain the remote sensing semantic segmentation result.

[0217] The target semantic information is fed into the target segmentation head to generate the final pixel-level classification decision, which is denoted as the remote sensing semantic segmentation result.

[0218] Based on the example implementation, the remote sensing semantic segmentation results can be referenced. Figure 13 (c) Figure 14 (c) Figure 15 (c) and Figure 16 (c).

[0219] The following describes an apparatus embodiment of this application, which can be used to perform the method embodiment of this application. For details not disclosed in the apparatus embodiment of this application, please refer to the method embodiment of this application.

[0220] Figure 17 A block diagram of a semantic segmentation apparatus for remotely sensed images according to an exemplary embodiment is shown.

[0221] Figure 17 The apparatus shown can perform the semantic segmentation method for remote sensing images according to the embodiments of this application.

[0222] like Figure 17 As shown, a semantic segmentation device for remote sensing images may include:

[0223] See Figure 17 Referring to the preceding description, the encoder training unit 1710 is used to train a pre-constructed encoder using a plurality of preset first remote sensing image samples to obtain a remote sensing visual encoder.

[0224] The encoder fine-tuning unit 1720 is used to fine-tune the remote sensing visual encoder and update the remote sensing visual encoder using the fine-tuning results.

[0225] The basic model unit 1730 is used to build a basic remote sensing segmentation model based on the updated remote sensing visual encoder, a pre-built mask decoder, and a pre-built segmentation head.

[0226] The segmentation head training unit 1740 is used to train the segmentation head of the basic remote sensing segmentation model based on knowledge transfer using a preset second remote sensing image sample and its segmentation ground value, according to multiple pre-trained segmentation models, to obtain the target segmentation head, and then obtain the target remote sensing semantic segmentation model.

[0227] The model inference unit 1750 is used to input the remote sensing image data to be segmented into the target remote sensing semantic segmentation model to obtain the remote sensing semantic segmentation result.

[0228] The device performs functions similar to those described above; other functions are described in the preceding descriptions and will not be repeated here.

[0229] This application discloses an electronic device, including: a processor; and a memory storing a computer program, which, when executed by the processor, causes the processor to execute the above-described instruction generation method.

[0230] For example, refer to Figure 18 , Figure 18 The illustrated electronic device 1800 includes a processor 1801 and a memory 1803. The processor 1801 and the memory 1803 are connected, for example, via a bus 1802. Optionally, the electronic device 1800 may also include a transceiver 1804. It should be noted that in practical applications, the transceiver 1804 is not limited to one type, and the structure of this electronic device 1800 does not constitute a limitation on the embodiments of this application.

[0231] Processor 1801 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in this application. Processor 1801 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0232] Bus 1802 may include a pathway for transmitting information between the aforementioned components. Bus 1802 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 1802 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 18 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0233] The memory 1803 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other storage medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0234] The memory 1803 is used to store application code that executes the scheme of this application, and its execution is controlled by the processor 1801. The processor 1801 is used to execute the application code stored in the memory 1803 to implement the content shown in the foregoing method embodiments.

[0235] Figure 18 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0236] This application discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, causes the processor to execute an instruction generation method.

[0237] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0238] The above are only some embodiments of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for semantic segmentation of a remote sensing image, characterized in that, include: Using multiple pre-set first remote sensing image samples, a pre-constructed encoder is trained to obtain a remote sensing visual encoder; The remote sensing visual encoder is fine-tuned, and the results of the fine-tuning are used to update the remote sensing visual encoder. Based on the updated remote sensing visual encoder, the pre-built mask decoder, and the pre-built segmentation head, a basic remote sensing segmentation model is constructed. Using a preset second remote sensing image sample and its segmentation ground value, and based on multiple pre-trained segmentation models, the segmentation head of the basic remote sensing segmentation model is trained based on knowledge transfer to obtain the target segmentation head, and then the target remote sensing semantic segmentation model is obtained. The remote sensing image data to be segmented is input into the target remote sensing semantic segmentation model to obtain the remote sensing semantic segmentation result; The process of fine-tuning the remote sensing visual encoder and updating it using the fine-tuning results includes: Freeze the parameters of the remote sensing visual encoder, and insert a first low-rank matrix in parallel next to the attention module of the remote sensing visual encoder; The gradient of the first low-rank matrix is ​​updated using the LoRA low-rank fine-tuning strategy. The updated first low-rank matrix is ​​merged with the parameters of the remote sensing visual encoder, and the merged result is used to update the parameters of the remote sensing visual encoder.

2. The method of claim 1, wherein, Using a set of pre-defined first remote sensing image samples, a pre-constructed encoder is trained to obtain a remote sensing visual encoder, including: Using the multiple first remote sensing image samples, a pre-constructed first visual encoder is trained under self-supervised supervision to obtain a first training result; Using the multiple first remote sensing image samples, a pre-constructed second visual encoder is trained by image-text comparison to obtain a second training result; The first training result or the second training result is used as the remote sensing visual encoder.

3. The method of claim 2, wherein, Using the plurality of first remote sensing image samples, a pre-constructed first visual encoder is trained under self-supervised conditions to obtain a first training result as the remote sensing visual encoder, including: Each of the first remote sensing image samples is divided into multiple image blocks, and the multiple image blocks are randomly masked according to a preset ratio to obtain visible image blocks and mask markers; The visible image patch is input into a pre-constructed first visual encoder to obtain the feature representation of the visible image patch; The feature representation of the visible image patch and the mask mark are input into a pre-built decoder to obtain a reconstructed image sample; Based on the difference between the first remote sensing image sample and the reconstructed image sample, the MAE reconstruction loss is calculated, and the first visual encoder is trained based on the MAE reconstruction loss to obtain the first training result as the remote sensing visual encoder.

4. The method of claim 2, wherein, Using the plurality of first remote sensing image samples, a pre-constructed second visual encoder is trained using image-text contrast, and the second training result is used as the remote sensing visual encoder, including: The first remote sensing image sample is input into a pre-built second visual encoder to obtain the first sample image features, and the text description of the first remote sensing image sample is input into a pre-built text encoder to obtain the first sample text features; Calculate the similarity matrix between the first sample image features and the first sample text features; Based on the similarity matrix, the image-text contrast loss is calculated, and the second visual encoder is trained based on the image-text contrast loss to obtain the second training result as the remote sensing visual encoder.

5. The method of claim 2, wherein, The first visual encoder is loaded with parameters from the second training result.

6. The method of claim 2, wherein, The second visual encoder is loaded with the parameters of the first training result.

7. The method of claim 1, wherein, Using a set of pre-defined first remote sensing image samples, a pre-constructed encoder is trained to obtain a remote sensing visual encoder, including: Each of the first remote sensing image samples is divided into multiple image blocks, and the multiple image blocks are randomly masked according to a preset ratio to obtain visible image blocks and mask markers; The visible image patch is input into a pre-built third visual encoder to obtain the feature representation of the visible image patch; The feature representation of the visible image patch and the mask mark are input into a pre-built decoder to obtain a reconstructed image sample; The MAE reconstruction loss is calculated based on the difference between the first remote sensing image sample and the reconstructed image sample. The first remote sensing image sample is input into the third visual encoder to obtain the second sample image features, and the text description of the first remote sensing image sample is input into the pre-built text encoder to obtain the second sample text features; Calculate the similarity matrix between the second sample image features and the second sample text features; Based on the similarity matrix, calculate the image-text contrast loss; Calculate the joint optimization loss based on the MAE reconstruction loss and the image-text contrast loss; The third visual encoder is trained based on the joint optimization loss, and the third training result is used as the remote sensing visual encoder.

8. The method of claim 1, wherein, Using a pre-set second remote sensing image sample and its segmentation ground truth, and based on multiple pre-trained segmentation models, the segmentation head of the basic remote sensing segmentation model is trained using knowledge transfer to obtain a target segmentation head, thereby obtaining a target remote sensing semantic segmentation model, including: The second remote sensing image sample is input into the updated remote sensing visual encoder to obtain the third sample image features, and the second remote sensing image sample is input into multiple pre-trained segmentation models to obtain multiple segmentation results; The features of the third sample image are input into the mask decoder to obtain the sample semantic information; The sample semantic information is input into the pre-constructed segmentation head to obtain the predicted segmentation result; The segmentation head is trained using the multiple segmentation results, the ground truth segmentation values ​​corresponding to the second remote sensing image samples, and the differences between these ground truth values ​​and the predicted segmentation results to obtain the target segmentation head.

9. The method according to claim 1 or 8, characterized in that, The mask decoder is pre-trained through the following steps: A second low-rank matrix is ​​inserted in parallel next to the attention module of the pre-built decoder to obtain the decoder to be fine-tuned; The gradient of the second low-rank matrix is ​​updated using the LoRA low-rank fine-tuning strategy. The updated second low-rank matrix is ​​combined with the parameters of the decoder to obtain the mask decoder.

10. The method of claim 1, wherein, The method further includes: The remote sensing semantic segmentation results are restructured at the category level to obtain hierarchical segmentation results; The hierarchical segmentation results are subjected to geometric optimization based on spatial consistency constraints to obtain topology optimization results; The topology optimization results are converted into a preset data structure format to obtain standardized remote sensing semantic segmentation results; The standardized remote sensing semantic segmentation results are overlaid on the remote sensing image data to be segmented.

11. The method of claim 1, wherein, The remote sensing image data to be segmented is input into the target remote sensing semantic segmentation model to obtain the remote sensing semantic segmentation result, including: The remote sensing image data to be segmented is input into the updated remote sensing visual encoder to obtain the target image features; The target image features are input into the mask decoder to obtain the target semantic information; The target semantic information is input into the target segmentation head to obtain the remote sensing semantic segmentation result.

12. A device for semantic segmentation of a remote sensing image, characterized in that, include: The encoder training unit is used to train a pre-constructed encoder using multiple preset first remote sensing image samples to obtain a remote sensing visual encoder. The encoder fine-tuning unit is used to fine-tune the remote sensing visual encoder and update the remote sensing visual encoder using the fine-tuning results; The basic model unit is used to construct a basic remote sensing segmentation model based on the updated remote sensing visual encoder, the pre-built mask decoder, and the pre-built segmentation head. The segmentation head training unit is used to train the segmentation head of the basic remote sensing segmentation model based on knowledge transfer using a preset second remote sensing image sample and its segmentation ground value, according to multiple pre-trained segmentation models, to obtain the target segmentation head, and then to obtain the target remote sensing semantic segmentation model. The model inference unit is used to input the remote sensing image data to be segmented into the target remote sensing semantic segmentation model to obtain the remote sensing semantic segmentation result; The encoder fine-tuning unit is also used for: Freeze the parameters of the remote sensing visual encoder, and insert a first low-rank matrix in parallel next to the attention module of the remote sensing visual encoder; The gradient of the first low-rank matrix is ​​updated using the LoRA low-rank fine-tuning strategy. The updated first low-rank matrix is ​​merged with the parameters of the remote sensing visual encoder, and the merged result is used to update the parameters of the remote sensing visual encoder.

13. An electronic device, comprising: include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-11.

14. A computer readable storage medium having stored thereon a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the method as described in any one of claims 1-11.

Images