A large model-based semi-supervised remote sensing image semantic segmentation method and device

Abstract

The application discloses a large model-based semi-supervised remote sensing image semantic segmentation method and device, wherein the method comprises the following steps: obtaining an unlabeled remote sensing image, performing zero-shot dense prompt segmentation on the unlabeled remote sensing image through a pre-trained general visual large model to obtain a third mask; obtaining a labeled remote sensing image, fine-tuning a lightweight classifier according to the labeled remote sensing image; performing semantic prediction on the third mask, filtering the semantic prediction result based on a classification confidence threshold, and obtaining pseudo-labels of the unlabeled remote sensing image; combining the pseudo-labels of the unlabeled remote sensing image, constructing a model training data set and a joint loss function, and building a remote sensing image semantic segmentation model; training the remote sensing image semantic segmentation model in combination with the model training data set and the joint loss function; and adjusting the weights of each part of the joint loss function by using dynamic curriculum learning to obtain a trained remote sensing image semantic segmentation model. The application effectively alleviates the noise pseudo-label and error accumulation problems generated by the prior art.

Description

Technical Field

[0001] This application relates to the field of remote sensing image technology, and in particular to a semi-supervised remote sensing image semantic segmentation method and apparatus based on a large model. Background Technology

[0002] With the rapid development of remote sensing technology, the acquisition of high spatial resolution remote sensing images has become increasingly convenient, providing a massive data foundation for applications such as detailed interpretation of ground features, dynamic urban monitoring, and disaster emergency response. However, achieving high-precision semantic segmentation usually relies on a large amount of pixel-level fine-annotation data, while the manual annotation of remote sensing images is extremely costly, time-consuming, and requires a high level of professional knowledge, which seriously restricts the deployment and promotion of deep learning models in practical scenarios.

[0003] To alleviate the reliance on fully supervised annotation, semi-supervised learning methods have been introduced into the field of remote sensing segmentation. Existing techniques mainly employ pseudo-label self-training and consistency regularization strategies. Pseudo-label self-training utilizes the model's high-confidence predictions of unlabeled images to generate pseudo-labels, which are then used as supervisory signals to iteratively optimize the model. Consistency regularization applies different perturbations to the same unlabeled image, forcing the model output to remain consistent. Furthermore, some hybrid methods, by combining the advantages of pseudo-labels and consistency regularization, adopt a paradigm of "weak enhancement to generate pseudo-labels + strong enhancement to calculate consistency loss," which has also achieved advanced capabilities. However, these methods all have drawbacks: the initial model has weak performance due to extremely limited training data; the generated pseudo-labels are noisy and incomplete, easily leading to error accumulation and model collapse, especially in remote sensing images where the scale of ground features varies, boundaries are blurred, and inter-class similarity is high.

[0004] In recent years, general-purpose large-scale vision models, represented by Segment Anything Models (SAMs), have demonstrated powerful zero-shot segmentation capabilities, enabling high-quality instance segmentation of natural images based on cues such as points and bounding boxes without task-specific training. Some studies have attempted to use SAMs for remote sensing image preprocessing, but their output is merely a binary mask without semantic categories, and they require manual interaction by default, making them difficult to use directly for end-to-end semantic segmentation training. Summary of the Invention

[0005] This application provides a semi-supervised remote sensing image semantic segmentation method and apparatus based on a large model, which can effectively alleviate the problems of noise pseudo-labels and error accumulation caused by the reliance on weak initial models in existing technologies.

[0006] Firstly, this application provides a semi-supervised semantic segmentation method for remote sensing images based on a large model. This method is executed by a computing device, which can be understood as a computer or server, etc., and is not limited thereto in this application. It includes:

[0007] Unlabeled remote sensing images are obtained, and zero-shot dense cue segmentation is performed on these images using a pre-trained general visual model to obtain a third mask. Labeled remote sensing images are then obtained, and a lightweight classifier is fine-tuned based on these images. Semantic prediction is performed on the third mask using the lightweight classifier, and the semantic prediction results are filtered based on a classification confidence threshold to obtain pseudo-labels for the unlabeled remote sensing images. A model training dataset and a joint loss function are constructed using these pseudo-labels, and a remote sensing image semantic segmentation model is built. The remote sensing image semantic segmentation model is then trained using the model training dataset and the joint loss function. During training, dynamic curriculum learning is used to adjust the weights of each part of the joint loss function to obtain the trained remote sensing image semantic segmentation model.

[0008] Through the above methods, this application introduces a pre-trained general visual large model (such as SAM) to automatically generate high-quality instance masks, and combines a lightweight classifier and confidence filtering mechanism to construct reliable pseudo-labels, effectively alleviating the problems of noisy pseudo-labels and error accumulation caused by the reliance on weak initial models in existing technologies. At the same time, a dynamic course learning strategy is adopted, which focuses on external prior guidance in the early stage of training and shifts to model self-mining in the later stage, thereby improving the accuracy and robustness of semantic segmentation of remote sensing images.

[0009] In the aforementioned semi-supervised remote sensing image semantic segmentation method based on a large model, obtaining the third mask specifically includes:

[0010] An unlabeled remote sensing image is obtained and divided into several sub-image patches. The size of each sub-image patch is adapted to the input requirements of a pre-trained general-purpose visual model. For each sub-image patch, a dense set of spatial cue points is generated. The point labels corresponding to the cue point set are used as input cues and input into the pre-trained general-purpose visual model to generate a first mask for each cue point. The first mask is a candidate instance mask. Redundancy removal is performed on all first masks within the same sub-image patch to obtain a second mask. The second mask is a set of classless instance masks covering the entire unlabeled remote sensing image. Noise patches are filtered from the second mask to obtain a third mask.

[0011] In this application, unlabeled remote sensing images are divided into sub-image blocks, and a dense set of spatial cue points is generated for each sub-image block. The point labels corresponding to the cue point set are used as input cues to train a large-scale visual model. The large-scale visual model generates a first mask for each cue point, and then performs redundancy removal processing on the first mask to obtain a second mask. The second mask is then filtered for noise patches to obtain a third mask. In this way, this application uses a pre-trained general-purpose visual model (SAM) to perform zero-sample dense cue segmentation on unlabeled remote sensing images, generating high-recall instance masks and improving the reliability of instance masks.

[0012] In the aforementioned semi-supervised remote sensing image semantic segmentation method based on a large model, labeled remote sensing images are obtained, and a lightweight classifier is fine-tuned based on the labeled remote sensing images. Semantic prediction is then performed on a third mask using the lightweight classifier, and the semantic prediction results are filtered based on a classification confidence threshold to obtain pseudo-labels for unlabeled remote sensing images, including:

[0013] Paired samples are constructed based on labeled remote sensing image data. A lightweight convolutional neural network is used as the backbone network, and the last fully connected layer of the lightweight convolutional neural network is replaced with a new classification head whose output dimension is equal to the number of predefined remote sensing land cover categories. In the replaced lightweight convolutional neural network, the cross-entropy loss function is used as the target, and fine-tuning is performed on the paired samples to obtain a lightweight instance classifier for remote sensing land cover recognition. The minimum bounding box of the third mask is calculated, and the first region corresponding to the minimum bounding box is cropped from the unlabeled remote sensing image. The first region is input into the lightweight instance classifier to obtain the probability distribution value of the first region. The regional semantic label of the first region is obtained based on the classification confidence threshold. All regional semantic labels are concatenated to generate pseudo-labels for the unlabeled remote sensing image.

[0014] In this way, this application performs semantic prediction on instance masks by finely tuning a lightweight classifier ResNet on a small amount of labeled data, and filters unreliable results based on classification confidence thresholds to generate high-quality pixel-level pseudo-labels.

[0015] The aforementioned semi-supervised remote sensing image semantic segmentation method based on a large model, combined with the unlabeled remote sensing image pseudo-labels, constructs a model training dataset and a joint loss function, and builds a remote sensing image semantic segmentation model, including:

[0016] Unlabeled remote sensing images and their corresponding unlabeled pseudo-labels are denoted as the unlabeled set; weakly enhanced and strongly enhanced views are generated for each unlabeled sample in the unlabeled set; the predicted probability and maximum response confidence of the weakly enhanced output are calculated based on the weakly enhanced view, and weakly enhanced pseudo-labels are generated; consistency loss is calculated based on the strongly enhanced view and the weakly enhanced pseudo-labels; the standard cross-entropy loss of the labeled set data and the pseudo-label supervision loss of the unlabeled set data are calculated, and the joint loss function is calculated based on the consistency loss, standard cross-entropy loss, and pseudo-label supervision loss; a remote sensing image semantic segmentation model is built based on the model training dataset.

[0017] Through the above methods, the initial training of this application mainly uses SAM pseudo-label supervision, and the consistency regularization weight under strong-weak data augmentation is gradually enhanced in the later stage, so as to achieve a smooth transition from external prior guidance to model self-mining.

[0018] In the aforementioned semi-supervised remote sensing image semantic segmentation method based on a large model, the trained remote sensing image semantic segmentation model specifically includes:

[0019] Stochastic gradient descent or AdamW optimizer is used with the joint loss function as the objective function to train the semantic segmentation model end-to-end. During training, a monotonically decreasing function is used to control the pseudo-label loss weights, while a monotonically increasing function is used to increase the consistency loss weights to achieve dynamic course learning. In each training batch, labeled and unlabeled samples are sampled simultaneously, and the gradient is calculated to update the parameters according to the adjusted joint loss function to obtain the final model.

[0020] Through the above method, this application realizes the training of a remote sensing image semantic segmentation model using a model training dataset and a joint loss function. At the same time, during the training process, the weights of each part of the joint loss function are adjusted to obtain the trained remote sensing image semantic segmentation model, thereby improving the reliability of the trained remote sensing image semantic segmentation model.

[0021] In the aforementioned semi-supervised remote sensing image semantic segmentation method based on a large model, a dense set of spatial cue points is generated for each sub-image module, including:

[0022] A uniform grid is constructed on the image plane with a fixed step size, and a foreground cue is generated at the center of each grid cell; a set of dense spatial cue point coordinates is formed based on the foreground cue.

[0023] The set of coordinates of dense spatial cue points is represented as

[0024] ;

[0025] Where N is the total number of prompts.

[0026] In the aforementioned remote sensing semantic segmentation method, generating a first mask for each cue point also includes:

[0027] A quality score is generated for the first mask, which is the predicted crossover ratio (CLORD) output by the model.

[0028] In the aforementioned semi-supervised remote sensing image semantic segmentation method based on a large model, redundancy removal processing is performed on all first masks within the same sub-image module, including:

[0029] Sort the quality scores of the first masks from highest to lowest, and then iterate through each first mask in turn; if the intersection-union ratio of the first mask with the already retained masks is greater than a preset threshold... If a certain value is found to be a duplicate, it is considered a duplicate detection and is removed; otherwise, it is retained to obtain the retained mask after redundancy removal. The retained mask after redundancy removal is mapped to the remote sensing image coordinate system, the mask results from different sub-image modules are merged, and the overlapping areas at the cross-block boundaries are fused to generate a second mask.

[0030] The second mask is represented as

[0031] ;

[0032] Each of them Represents a binary instance region.

[0033] Secondly, this application provides a semi-supervised remote sensing image semantic segmentation device based on a large model, including: a mask acquisition module, a classifier fine-tuning module, a pseudo-label generation module, a data construction module, and a model training module;

[0034] The system comprises the following modules: a mask acquisition module for obtaining unlabeled remote sensing images, performing zero-shot dense cue segmentation on these images using a pre-trained general visual model to obtain a third mask; a classifier fine-tuning module for obtaining labeled remote sensing images and fine-tuning a lightweight classifier based on these images; a pseudo-label generation module for performing semantic prediction on the third mask using the lightweight classifier, filtering the semantic prediction results based on a classification confidence threshold to obtain pseudo-labels for the unlabeled remote sensing images; a data construction module for building a model training dataset and a joint loss function using the pseudo-labels for the unlabeled remote sensing images, and constructing a remote sensing image semantic segmentation model; and a model training module for training the remote sensing image semantic segmentation model using the model training dataset and the joint loss function. During training, dynamic curriculum learning is used to adjust the weights of each part of the joint loss function to obtain the trained remote sensing image semantic segmentation model.

[0035] Thirdly, the present invention also provides a computing device, comprising: a memory for storing program instructions; and a processor for calling the program instructions stored in the memory and executing the method described in the first aspect according to the obtained program instructions.

[0036] Fourthly, the present invention also provides a computer-readable storage medium storing computer-readable instructions, which, when read and executed by a computer, implement the method of the first aspect described above.

[0037] Fifthly, the present invention provides a computer program product comprising a computer program executable by a computer device, wherein when the program is run on the computer device, the computer device performs the method described in the first aspect.

[0038] Beneficial effects: Through the above methods, this invention automatically generates high-quality instance masks by introducing a pre-trained general-purpose visual large model (such as SAM), and constructs reliable pseudo-labels by combining a lightweight classifier and a confidence filtering mechanism. This effectively alleviates the problems of noisy pseudo-labels and error accumulation caused by existing technologies that rely on weak initial models. At the same time, by adopting a dynamic course learning strategy, it focuses on external prior guidance in the early stage of training and shifts to model self-mining in the later stage, which improves the accuracy and robustness of semi-supervised remote sensing image semantic segmentation based on large models and has high engineering practical value. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 A flowchart illustrating a semi-supervised remote sensing image semantic segmentation method based on a large model provided in this application;

[0041] Figure 2 A schematic diagram of the third mask acquisition process for a semi-supervised remote sensing image semantic segmentation method based on a large model provided in this application;

[0042] Figure 3 A schematic diagram illustrating a semi-supervised remote sensing image semantic segmentation method based on a large model provided in this application;

[0043] Figure 4 A schematic diagram of a semi-supervised remote sensing image semantic segmentation device based on a large model provided in this application;

[0044] Figure 5 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0046] In the embodiments described hereinafter of the present application, "and / or" describes the association relationship between associated objects, which means that three relationships may exist. For example, A and / or B may represent: the case where A exists alone, both A and B exist, and B exists alone, wherein A and B may be singular or plural. The character " / " generally indicates that the preceding and following associated objects are in an "or" relationship. The expression "at least one of the following" or similar expressions thereof refers to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b or c can represent: a, b, c, a-b, a-c, b-c, or a-b-c, wherein a, b and c can be single or multiple. Singular expressions "a", "an", "the foregoing", "the" and "this one" are intended to also include expressions such as "one or more" unless the context clearly indicates otherwise. In addition, unless otherwise stated, ordinal numbers such as "first" and "second" mentioned in the embodiments of the present application are used to distinguish a plurality of objects, and are not used to limit the order, timing, priority or importance of the plurality of objects.

[0047] References to "one embodiment" or "some embodiments" described in the specification of the present application mean that one or more embodiments of the present application include the specific feature, structure, or characteristic described in connection with the embodiment. Thus, the appearances of the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in additional embodiments" and the like in different places in this specification are not necessarily all referring to the same embodiment, but instead mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "comprising", "including", "having" and their variants all mean "including but not limited to", unless otherwise specifically emphasized.

[0048] Example 1

[0049] Example 1 of the present invention provides a semi-supervised semantic segmentation method for remote sensing images based on a large model. Before introducing the method, the nouns of related concepts are first explained:

[0050] 1. High-recall instance masks

[0051] The "high recall" in the present invention refers to driving the SAM model to perform full-coverage segmentation on remote sensing images through an automated dense prompting mechanism (such as uniform point grids and multi-scale sliding windows), so that most significant feature regions in the image (including small-scale or weakly textured targets) can be responded to by at least one candidate mask, thereby preferentially ensuring detection completeness rather than accuracy or uniqueness in the mask generation stage. Subsequent steps will improve accuracy through confidence filtering and topology correction, forming a two-stage pseudo-label generation paradigm of "high recall first, high precision later".

[0052] II. Error Cumulative Effect

[0053] Because only a small number of labeled samples are available in the early stages of training, the model's generalization ability is limited, and the generated pseudo-labels often contain a large number of misclassifications or boundary biases. More seriously, such methods usually equate the high confidence of the model's output with the accuracy of the prediction, causing high-confidence but incorrect pseudo-labels to be mistakenly identified as reliable supervision signals. This reinforces the erroneous decision-making patterns in subsequent training, forming a cumulative effect of errors that is difficult to correct. This problem is often more prominent in remote sensing images, where the high similarity between land cover types and large scale variations make it extremely easy to cause systematic missegmentation, severely restricting the performance and stability of semi-supervised models.

[0054] The specific steps of Embodiment 1 of this application are as follows: Figure 1 As shown. This method is executed by a computing device, which can be understood as a computer or other similar device; however, this invention does not limit the scope of the invention. The steps of this method are as follows:

[0055] Step 101: Obtain unlabeled remote sensing images, and perform zero-sample dense cue segmentation on the unlabeled remote sensing images using a pre-trained general vision model to obtain a third mask.

[0056] The third mask is the initial instance segmentation result with high recall, low redundancy, and a reasonable structure, serving as the input basis for subsequent pseudo-label mapping. The specific method for obtaining the third mask is as follows: Figure 2 As shown, it includes the following steps:

[0057] Step 201: Obtain an unlabeled remote sensing image and divide the unlabeled remote sensing image into several sub-image patches; wherein the size of the sub-image patches is adapted to the input requirements of the pre-trained general vision large model;

[0058] Step 202: For each sub-image module, generate a set of dense spatial cue points;

[0059] A uniform grid is constructed on the image plane with a fixed step size, and a foreground cue is generated at the center of each grid cell; a set of dense spatial cue point coordinates is formed based on the foreground cue.

[0060] The set of coordinates of dense spatial cue points is represented as

[0061]

[0062] Where N is the total number of prompts.

[0063] Step 203: Use the point labels corresponding to the set of prompt points as input prompts, input them into the pre-trained general vision model, and generate a first mask for each prompt point; wherein, the first mask is a candidate instance mask;

[0064] A quality score is generated for the first mask, which is the predicted crossover ratio (CLORD) output by the model.

[0065] Step 204: Perform redundancy removal processing on all first masks within the same sub-image module to obtain the second mask; the second mask is a set of classless instance masks that cover the entire unlabeled remote sensing image.

[0066] Sort the quality scores of the first mask from high to low, and then iterate through each first mask in turn;

[0067] If the intersection-union ratio of the first mask and the already preserved mask is greater than a preset threshold If the result is a duplicate detection, it is considered a duplicate detection and is removed; otherwise, it is retained, and the retained mask after redundancy removal is obtained.

[0068] The redundant-free retained mask is mapped to the remote sensing image coordinate system, the mask results from different sub-image modules are merged, and the overlapping areas at the cross-block boundaries are fused to generate a second mask;

[0069] The second mask is represented as

[0070] ;

[0071] Each of them Represents a binary instance region.

[0072] Step 205: Perform noise patch filtering on the second mask to obtain the third mask.

[0073] For each second mask, the number of effective pixels is calculated. If the area is less than the preset minimum region threshold (typically 64 or 100 pixels), it is considered a noise patch and is removed, thereby obtaining an initial instance segmentation result with high recall, low redundancy and reasonable structure, which serves as the input basis for subsequent pseudo-label category mapping.

[0074] After obtaining the third mask through the above method, continue to execute the content of step 102 of this application.

[0075] Step 102: Obtain the labeled remote sensing image and fine-tune the lightweight classifier based on the labeled remote sensing image.

[0076] Specifically, paired samples are constructed based on labeled remote sensing image data; a lightweight convolutional neural network is used as the backbone network, and the last fully connected layer of the lightweight convolutional neural network is replaced with a new classification head whose output dimension is equal to the number of predefined remote sensing land cover categories.

[0077] Step 103: Use a lightweight classifier to perform semantic prediction on the third mask, and filter the semantic prediction results based on the classification confidence threshold to obtain unlabeled remote sensing image pseudo-labels.

[0078] Specifically, in the replaced lightweight convolutional neural network, the cross-entropy loss function is used as the target, and fine-tuning is performed on paired samples to obtain a lightweight instance classifier for remote sensing ground object recognition. The minimum bounding box of the third mask is calculated, and the first region corresponding to the minimum bounding box is cropped from the unlabeled remote sensing image. The first region is input into the lightweight instance classifier to obtain the probability distribution value of the first region. The semantic label of the first region is obtained based on the classification confidence threshold. All region semantic labels are concatenated to generate pseudo-labels for the unlabeled remote sensing image.

[0079] Step 104: Combine unlabeled remote sensing image pseudo-labels to construct the model training dataset and joint loss function, and build a remote sensing image semantic segmentation model.

[0080] Specifically, unlabeled remote sensing images and their corresponding unlabeled pseudo-labels are denoted as the unlabeled set; weakly enhanced and strongly enhanced views are generated for each unlabeled sample in the unlabeled set; the predicted probability and maximum response confidence of the weakly enhanced output are calculated based on the weakly enhanced view, and weakly enhanced pseudo-labels are generated; consistency loss is calculated based on the strongly enhanced view and the weakly enhanced pseudo-labels; the standard cross-entropy loss of the labeled set data and the pseudo-label supervision loss of the unlabeled set data are calculated, and the joint loss function is calculated based on the consistency loss, standard cross-entropy loss, and pseudo-label supervision loss; wherein, the labeled set is the labeled set of remote sensing images with a small number of pixel-level annotations; a remote sensing image semantic segmentation model is built based on the model training dataset.

[0081] Step 105: Train the remote sensing image semantic segmentation model by combining the model training dataset and the joint loss function; during the training process, dynamic course learning is used to adjust the weights of each part in the joint loss function to obtain the trained remote sensing image semantic segmentation model.

[0082] Specifically, stochastic gradient descent or AdamW optimizer is used with the joint loss function as the objective function to train the semantic segmentation model end-to-end. During training, a monotonically decreasing function is used to control the pseudo-label loss weights, while a monotonically increasing function is used to increase the consistency loss weights, thus achieving dynamic course learning. In each training batch, labeled and unlabeled samples are sampled simultaneously, and the gradient is calculated to update the parameters based on the adjusted joint loss function to obtain the final model.

[0083] A monotonically decreasing function is used to control the pseudo-label loss weight, while a monotonically increasing function is used to increase the consistency loss weight, thus achieving a course-style transition from "initially relying on high-quality SAM pseudo-labels to later shifting to consistency learning through model self-mining".

[0084] Using stochastic gradient descent or the AdamW optimizer, with the joint loss function as the objective function, the main segmentation network is optimized. (e.g., U-Net) performs end-to-end training; in each training batch, labeled and unlabeled samples are sampled simultaneously, and gradients are calculated and parameters are updated according to the dynamic weights mentioned above; as training progresses, the model gradually transitions from being guided by prior knowledge from a large external model to utilizing its own robustness to mine structural consistency information in unlabeled data, effectively mitigating the problem of error accumulation caused by early pseudo-label bias.

[0085] Through the above methods, this application introduces a pre-trained general visual model (such as SAM) to automatically generate high-quality instance masks, and combines a lightweight classifier and confidence filtering mechanism to construct reliable pseudo-labels, effectively alleviating the problems of noisy pseudo-labels and error accumulation caused by the reliance on weak initial models in existing technologies. At the same time, a dynamic course learning strategy is adopted, which focuses on external prior guidance in the early stage of training and shifts to model self-mining in the later stage, improving the accuracy and robustness of semantic segmentation of remote sensing images, and has high engineering practical value.

[0086] Example 2

[0087] Embodiment 2 of this application provides a semi-supervised remote sensing image semantic segmentation method based on a large model, building upon Embodiment 1. The specific steps are as follows: Figure 3 As shown. This method is executed by a computing device, which can be understood as a computer or other similar device; however, this invention does not limit the scope of the invention. The steps of this method are as follows:

[0088] Step 1: Large model-guided instance segmentation: Use the pre-trained general visual large model SAM to perform zero-shot dense cue segmentation on unlabeled remote sensing images to generate high-recall instance masks.

[0089] Specifically, step 1 includes:

[0090] Step 1-1: Obtain the unlabeled remote sensing image to be processed and divide it into several overlapping or non-overlapping sub-image blocks; the size of each sub-image block is adapted to the input requirements of the pre-trained general visual large model (SAM), with a typical size of 1024×1024 pixels. If the original image size is smaller than this size, it is directly input.

[0091] For each sub-image patch, a set of dense spatial cue points is automatically generated. Specifically, a uniform grid is constructed on the image plane with a fixed step size d (typically 16, 32, or 64 pixels), and a foreground cue point is generated at the center of each grid, forming a set of dense spatial cue points. , where N is the total number of hints.

[0092] Steps 1-2: Take the set of dense spatial cue points and their corresponding point labels (all set as foreground label 1) as input cue points and send them to the inference module of the pre-trained SAM model. Call its mask decoder to generate multiple candidate instance masks for each cue point as the first mask and obtain its corresponding quality score; the quality score is the predicted intersection-union ratio output by the model.

[0093] Steps 1-3: Perform redundancy removal on all generated first masks within the same sub-image block to obtain second masks: First, sort them from highest to lowest quality score, then iterate through each mask sequentially. If the intersection-union ratio (IUU) of the mask with the already retained mask is greater than a preset threshold... If the value is typically 0.85, it is considered a duplicate detection and is removed; otherwise, it is retained.

[0094] All retained masks after redundancy removal are mapped back to the original remote sensing image coordinate system. Mask results from different sub-image patches are merged, and overlapping regions at patch boundaries are fused using the principle of prioritizing the highest quality score. Finally, a second mask is generated, which is a set of classless instance masks covering the entire unlabeled remote sensing image. Each of them Represents a binary instance region;

[0095] Steps 1-4: For each mask in the second mask Calculate the number of effective pixels;

[0096] If the calculated area is less than the preset minimum region threshold If the value is 64 or 100 pixels, it is considered a noise patch and removed to obtain a third mask, generating an labeled remote sensing image. In this way, a high-recall, low-redundancy and well-structured initial instance segmentation result is obtained, which serves as the input basis for subsequent pseudo-label category mapping.

[0097] Step 1 above is a preferred embodiment of step 101 in Embodiment 1 of this application, and this application does not limit this embodiment.

[0098] Step 2: Confidence-aware pseudo-label category mapping: Semantic prediction of instance masks is performed by a lightweight classifier ResNet fine-tuned on a small amount of labeled data, and unreliable results are filtered based on the classification confidence threshold to generate high-quality pixel-level pseudo-labels.

[0099] Specifically, step 2 includes:

[0100] Step 2-1: Construct supervised training samples for semantic classification of ground features on a small dataset of labeled remote sensing images;

[0101] For each labeled image, the connected components of the real semantic region are extracted as instances, and the corresponding image patches are cropped to form "image patch-category label" paired samples;

[0102] A lightweight convolutional neural network (ResNet-18) pre-trained on ImageNet is used as the backbone network, and its last fully connected layer is replaced with an output dimension equal to the predefined number of remote sensing land cover categories. The new category header;

[0103] In the replaced lightweight convolutional neural network, the network is fine-tuned end-to-end on paired samples with the cross-entropy loss function as the target. During the fine-tuning process, the parameters of the first two residual stages of the network are frozen, and only the high-level feature extraction module and the classification head are optimized to alleviate overfitting on small samples. Finally, a lightweight instance classifier for remote sensing ground object recognition is obtained.

[0104] Step 2-2: For each third mask output during the large model guided instance segmentation stage Calculate its minimum bounding rectangle and crop the corresponding region from the original unlabeled remote sensing image;

[0105] The cropped image is scaled to a fixed input size, and optionally a third mask is used. Zero-value masking is applied to the background region of the cropped image to enhance foreground focus;

[0106] The preprocessed image is then fed into a finely tuned lightweight classifier. The classifier output is converted into a probability distribution using the softmax function. ;

[0107] in and .

[0108] Steps 2-3: Confidence assessment and threshold filtering.

[0109] Calculate probability distribution Maximum response value This serves as the confidence score for the classification result of this instance; if If the prediction is reliable, then the corresponding category index is used. As a semantic label for this instance; if If the classification of the instance is unreliable, its semantic label will not be assigned for the time being.

[0110] Steps 2-4: Pseudo-label graph construction.

[0111] Initialize an array of all zeros with the same size as the original remote sensing image. And set the ignore index value to ignore=255;

[0112] Traverse all third masks For reliable instances, assign them semantic category labels. Fill to The corresponding mask area (i.e.) );

[0113] For unreliable instances and pixel locations not covered by any mask, a value of "ignore" is uniformly assigned; finally, a spatially aligned pixel-level pseudo-label image containing ignored regions is generated. , serving as a supervisory signal for unlabeled samples during the semi-supervised training phase.

[0114] Step 2 above is a preferred embodiment of steps 102 to 103 in Embodiment 1 of this application, and this application does not limit this embodiment.

[0115] Step 3: Dynamic course semi-supervised training: In the early stage of training, SAM pseudo-label supervision is the main method. In the later stage, the consistency regularization weight under strong-weak data augmentation is gradually increased to achieve a smooth transition from external prior guidance to model self-mining.

[0116] Specifically, step 3 includes:

[0117] Step 3-1: Construct a hybrid training dataset.

[0118] A small number of remote sensing images with pixel-level annotations are recorded as a labeled set.

[0119] ;

[0120] A large number of unlabeled remote sensing images and their corresponding large models are generated into pseudo-labeled images and denoted as the unlabeled set.

[0121] ;

[0122] in Includes valid category labels and ignored areas.

[0123] Step 3-2: Design a dynamic weighted joint loss function.

[0124] For each unlabeled sample Generate a weakly enhanced view

[0125] ;

[0126] and enhanced view

[0127] ;

[0128] The weakly enhanced view includes random horizontal flipping and zooming; the strongly enhanced view includes RandAugment, CutMix, and color perturbation.

[0129] Calculate the prediction probability of weakly enhanced output

[0130] ;

[0131] And generate its pseudo-tags

[0132] ;

[0133] Simultaneously calculate the maximum response confidence.

[0134] ;

[0135] Among them, only when (Typical value 0.95) will Considered a reliable target for monitoring strongly enhanced branches;

[0136] Calculate consistency loss

[0137] ;

[0138] in .

[0139] Calculate the standard cross-entropy loss on labeled data.

[0140] ;

[0141] in This indicates the positions of all valid pixels.

[0142] Finally, the pseudo-label supervision loss for the unlabeled dataset is calculated.

[0143] ;

[0144] in .

[0145] The joint loss function is calculated based on the consistency loss, standard cross-entropy loss, and pseudo-label supervision loss. The joint loss function is:

[0146] .

[0147] Step 3-3: Design a dynamic course weight scheduling strategy.

[0148] Number of training iterations ( At the position representing the total number of training steps, a normalization factor for training progress is introduced. ;

[0149] A monotonically decreasing function is used to control the weight of the pseudo-label loss, while a monotonically increasing function is used to increase the weight of the consistency loss. Specifically:

[0150]

[0151] ;

[0152] in and The initial maximum weights are (typical values ​​are 1 and 0.5 respectively). This is the scheduling index (typically 2).

[0153] The above method enables a course-based transition from "initially relying on high-quality SAM pseudo-labels to later shifting to consistent learning through model self-mining".

[0154] Steps 3-4: End-to-end optimization of the main segmentation network.

[0155] Using stochastic gradient descent or the AdamW optimizer, For the objective function, the main segmentation network (e.g., U-Net) performs end-to-end training; in each training batch, labeled and unlabeled samples are sampled simultaneously, and gradients are calculated and parameters are updated according to the dynamic weights mentioned above; as training progresses, the model gradually transitions from being guided by prior knowledge from a large external model to utilizing its own robustness to mine structural consistency information in unlabeled data, effectively mitigating the problem of error accumulation caused by early pseudo-label bias.

[0156] Step 3 above is a preferred embodiment of steps 104 to 105 in Embodiment 1 of this application, and this application does not limit this embodiment.

[0157] This invention introduces a pre-trained general-purpose visual model (such as SAM) to automatically generate high-quality instance masks, and combines a lightweight classifier and confidence filtering mechanism to construct reliable pseudo-labels, effectively alleviating the problems of noisy pseudo-labels and error accumulation caused by the reliance on weak initial models in existing technologies. At the same time, it adopts a dynamic course learning strategy, focusing on external prior guidance in the early stage of training and shifting to model self-mining in the later stage, which improves the accuracy and robustness of semantic segmentation of remote sensing images and has high engineering practical value.

[0158] Example 3

[0159] Embodiment 3 of the present invention provides a semi-supervised remote sensing image semantic segmentation device based on a large model, based on Embodiment 1. A schematic diagram of the device is shown below. Figure 4 As shown, the relevant content for implementing the semi-supervised remote sensing image semantic segmentation method based on a large model provided by the present invention includes:

[0160] The module includes a mask acquisition module, a classifier fine-tuning module, a pseudo-label generation module, a data construction module, and a model construction module.

[0161] The mask acquisition module is used to obtain unlabeled remote sensing images. It performs zero-sample dense cue segmentation on the unlabeled remote sensing images using a pre-trained general vision model to obtain a third mask.

[0162] The classifier fine-tuning module is used to obtain labeled remote sensing images and fine-tune the lightweight classifier based on the labeled remote sensing images.

[0163] The pseudo-label generation module is used to perform semantic prediction on the third mask using a lightweight classifier, and to filter the semantic prediction results based on the classification confidence threshold to obtain pseudo-labels for unlabeled remote sensing images.

[0164] The data construction module is used to combine unlabeled remote sensing image pseudo-labels to build a model training dataset and joint loss function, and to build a remote sensing image semantic segmentation model.

[0165] The model training module is used to train a remote sensing image semantic segmentation model by combining the model training dataset and the joint loss function. During the training process, dynamic course learning is used to adjust the weights of each part in the joint loss function to obtain the trained remote sensing image semantic segmentation model.

[0166] Training is performed based on a joint loss function, which dynamically changes during training according to a weight adjustment strategy.

[0167] Example 4

[0168] Having introduced a semi-supervised remote sensing image semantic segmentation apparatus based on a large model according to an exemplary embodiment of this application, the following describes a computing device according to another exemplary embodiment of this application.

[0169] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0170] In some possible implementations, the computing device according to this application may include at least one processor and at least one memory. The memory stores a computer program that, when executed by the processor, causes the processor to perform the steps in the large-model-based semi-supervised remote sensing image semantic segmentation method according to various exemplary embodiments of this application described above.

[0171] The following reference Figure 5 To describe a computing device 130 according to this embodiment of the present application. Figure 5 The computing device 130 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this application. Figure 5 As shown, the computing device 130 is presented in the form of a general-purpose smart terminal (or Bluetooth headset). The components of the computing device 130 may include, but are not limited to: at least one processor 131, at least one memory 132, and a bus 133 connecting different system components (including memory 132 and processor 131).

[0172] Bus 133 represents one or more of several bus architectures, including a memory bus or memory controller, peripheral bus, processor, or local bus using any of the various bus architectures. Memory 132 may include readable media in the form of volatile memory, such as random access memory (RAM) 1321 and / or cache memory 1322, and may further include read-only memory (ROM) 1323. Memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0173] The computing device 130 can also communicate with one or more external devices 134 (e.g., keyboard, pointing device, etc.), and / or with any device that enables the computing device 130 to communicate with one or more other smart terminals (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 135. Furthermore, the computing device 130 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 136. As shown, network adapter 136 communicates with other modules used in the computing device 130 via bus 133. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the computing device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0174] In some possible implementations, various aspects of the large-model-based semi-supervised remote sensing image semantic segmentation method provided in this application can also be implemented as a program product comprising a computer program that, when run on a computer device, causes the computer device to perform the steps in the large-model-based semi-supervised remote sensing image semantic segmentation method according to various exemplary embodiments of this application as described above.

[0175] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0176] The program product for time-domain noise processing according to the embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include a computer program, and may run on a smart terminal. However, the program product of this application is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0177] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a readable computer program. Such propagated data signals may take various forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0178] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0179] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0180] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable access frequency prediction device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable access frequency prediction device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0181] These computer program instructions may also be stored in a computer-readable storage medium capable of directing a computer or other programmable access predictive device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0182] These computer program instructions can also be loaded onto a computer or other programmable access predictive device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0183] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0184] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A semi-supervised semantic segmentation method for remote sensing images based on a large model, characterized in that, include: An unlabeled remote sensing image is obtained, and zero-shot dense cue segmentation is performed on the unlabeled remote sensing image using a pre-trained general-purpose visual model to obtain a third mask. Obtaining the third mask includes obtaining the unlabeled remote sensing image and dividing it into several sub-image blocks. The size of each sub-image block is adapted to the input requirements of the pre-trained general-purpose visual model. For each sub-image block, a set of dense spatial cue points is generated. Generating the set of dense spatial cue points includes constructing a uniform grid on the image plane with a fixed step size, generating a foreground cue at the center of each grid cell, and forming a set of coordinates for the dense spatial cue points based on the foreground cue points. The set of coordinates for the dense spatial cue points is represented as... ； Wherein, N is the total number of prompt points, and the and stated The pixel coordinates of the foreground cue point are represented, where i is an index ranging from 1 to N. The point labels corresponding to the cue point set are used as input cue points and input into the pre-trained general visual model to generate a first mask for each cue point. The first mask is a candidate instance mask. Redundancy removal is performed on all first masks within the same sub-image module to obtain a second mask. The second mask is a set of classless instance masks covering the entire unlabeled remote sensing image. Redundancy removal on all first masks within the same sub-image module includes sorting the quality scores of the first masks from high to low, and then sequentially traversing each first mask. If the intersection-union ratio (IUU) of the first mask and the retained masks is greater than a preset threshold... If a value is found to be duplicated, it is considered a duplicate detection and removed; otherwise, it is retained, resulting in a redundancy-free retained mask. This retained mask is then mapped to the remote sensing image coordinate system, and the mask results from different sub-image modules are merged. Overlapping regions at cross-block boundaries are then fused to generate a second mask. The second mask is represented as... ； Among them, the This represents an instance mask, where k is an index ranging from 1 to N, and each... Let H represent a binary instance region, where H represents the height and W represents the width; perform noise patch filtering on the second mask to obtain the third mask; Obtain labeled remote sensing images, and fine-tune a lightweight classifier based on the labeled remote sensing images; The lightweight classifier is used to perform semantic prediction on the third mask, and the semantic prediction results are filtered based on the classification confidence threshold to obtain unlabeled remote sensing image pseudo-labels. By combining the unlabeled remote sensing image pseudo-labels, a model training dataset and joint loss function are constructed, and a remote sensing image semantic segmentation model is built. The remote sensing image semantic segmentation model is trained by combining the model training dataset and the joint loss function; during the training process, dynamic course learning is used to adjust the weights of each part of the joint loss function to obtain the trained remote sensing image semantic segmentation model.

2. The method according to claim 1, characterized in that, Calculate the minimum bounding rectangle of the third mask, and crop the first region corresponding to the minimum bounding rectangle in the unlabeled remote sensing image; input the first region into the lightweight instance classifier to obtain the probability distribution value of the first region; The semantic label of the first region is obtained based on the classification confidence threshold; Concatenate all region semantic labels to generate pseudo-labels for unlabeled remote sensing images, including: Construct paired samples based on labeled remote sensing image data; A lightweight convolutional neural network is used as the backbone network, and the last fully connected layer of the lightweight convolutional neural network is replaced with a new classification head whose output dimension is equal to the number of predefined remote sensing land cover categories; In the replaced lightweight convolutional neural network, the cross-entropy loss function is used as the target and fine-tuned in the paired samples to obtain a lightweight instance classifier for remote sensing ground object identification. Calculate the minimum bounding rectangle of the third mask, and crop the first region corresponding to the minimum bounding rectangle in the unlabeled remote sensing image; The first region is input into the lightweight instance classifier to obtain the probability distribution value of the first region; the region semantic label of the first region is obtained based on the classification confidence threshold; By concatenating all the semantic labels of the aforementioned regions, pseudo-labels for the unlabeled remote sensing images are generated.

3. The method according to claim 1, characterized in that, The process of combining the unlabeled remote sensing image pseudo-labels to construct a model training dataset and a joint loss function, and building a remote sensing image semantic segmentation model, includes: The unlabeled remote sensing images and their corresponding unlabeled remote sensing image pseudo-labels are denoted as the unlabeled set; For each unlabeled sample in the unlabeled set, generate a weakly enhanced view and a strongly enhanced view; The predicted probability and maximum response confidence of the weakly enhanced output are calculated based on the weakly enhanced view, and weakly enhanced pseudo-labels are generated. Calculate the consistency loss based on the strongly enhanced view and the weakly enhanced pseudo-label; Calculate the standard cross-entropy loss of the labeled set of data and the pseudo-label supervision loss of the unlabeled set of data, and calculate the joint loss function based on the consistency loss, standard cross-entropy loss and pseudo-label supervision loss; A remote sensing image semantic segmentation model is built based on the training dataset of the model.

4. The method according to claim 1, characterized in that, The obtained trained remote sensing image semantic segmentation model specifically includes: The semantic segmentation model is trained end-to-end using stochastic gradient descent or AdamW optimizer with the joint loss function as the objective function. During the training process, a monotonically decreasing function is used to control the pseudo-label loss weight, while a monotonically increasing function is used to increase the consistency loss weight, thereby achieving dynamic course learning. In each training batch, labeled and unlabeled samples are sampled simultaneously, and the gradient is calculated based on the adjusted joint loss function to update the parameters, thus obtaining the final model.

5. The method according to claim 1, characterized in that, The step of generating a first mask for each cue point also includes: A quality score is generated for the first mask, where the quality score is the predicted intersection-union ratio output by the model.

6. A semi-supervised remote sensing image semantic segmentation device based on a large model, characterized in that, include: A mask acquisition module is used to obtain an unlabeled remote sensing image, and to perform zero-sample dense cue segmentation on the unlabeled remote sensing image using a pre-trained general-purpose visual model to obtain a third mask. Obtaining the third mask includes obtaining the unlabeled remote sensing image and dividing it into several sub-image blocks. The size of each sub-image block is adapted to the input requirements of the pre-trained general-purpose visual model. For each sub-image block, a dense spatial cue point set is generated. Generating the dense spatial cue point set includes constructing a uniform grid on the image plane with a fixed step size, generating a foreground cue at the center of each grid cell, and forming a dense spatial cue point coordinate set based on the foreground cue. The dense spatial cue point coordinate set is represented as... ； Wherein, N is the total number of prompt points, and the and stated The pixel coordinates of the foreground cue point are represented, where i is an index ranging from 1 to N. The point labels corresponding to the cue point set are used as input cue points and input into the pre-trained general visual model to generate a first mask for each cue point. The first mask is a candidate instance mask. Redundancy removal is performed on all first masks within the same sub-image module to obtain a second mask. The second mask is a set of classless instance masks covering the entire unlabeled remote sensing image. Redundancy removal on all first masks within the same sub-image module includes sorting the quality scores of the first masks from high to low, and then sequentially traversing each first mask. If the intersection-union ratio (IUU) of the first mask and the retained masks is greater than a preset threshold... If a certain value is found to be a duplicate, it is considered a duplicate detection and is removed; otherwise, it is retained to obtain the retained mask after redundancy removal. The retained mask after redundancy removal is mapped to the remote sensing image coordinate system, the mask results from different sub-image modules are merged, and the overlapping areas at the cross-block boundaries are fused to generate a second mask. The second mask is represented as ； Among them, the This represents an instance mask, where k is an index ranging from 1 to N, and each... Let H represent a binary instance region, where H represents the height and W represents the width; perform noise patch filtering on the second mask to obtain the third mask; A classifier fine-tuning module is used to obtain labeled remote sensing images and fine-tune a lightweight classifier based on the labeled remote sensing images. The pseudo-label generation module is used to perform semantic prediction on the third mask through the lightweight classifier, filter the semantic prediction results based on the classification confidence threshold, and obtain pseudo-labels for unlabeled remote sensing images. The data construction module is used to combine the unlabeled remote sensing image pseudo-labels to construct the model training dataset and joint loss function, and to build a remote sensing image semantic segmentation model. The model training module is used to train the remote sensing image semantic segmentation model by combining the model training dataset and the joint loss function. During the training process, dynamic course learning is used to adjust the weights of each part of the joint loss function to obtain the trained remote sensing image semantic segmentation model.

7. A computing device, characterized in that, include: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the method as described in any one of claims 1-5 according to the obtained program instructions.

Images