A method and device for remote sensing image segmentation based on a SAM model

Abstract

The present application relates to a kind of remote sensing image segmentation method based on SAM model and its device, compared with prior art, solve the defect that when the point prompt quality is not good or the position distribution is not reasonable, resulting in target missing detection and cross-instance leakage when the application of prompt segmentation basic model is applied to multi-instance remote sensing image segmentation.The present application includes the following steps: remote sensing image acquisition and preprocessing;Set the point prompt space refining framework based on the prompt segmentation basic model;Training of point prompt space refining framework;Acquire and preprocess the remote sensing image to be segmented;The remote sensing image to be segmented and the calibrated point prompt set are input into the prompt segmentation basic model, and the remote sensing image ground feature segmentation mask is output.The present application realizes the closed-loop optimization of query front point prompt calibration and query back feedback type iterative prompt refining combination, improves the accuracy of dense multi-instance remote sensing image segmentation result.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, specifically to a remote sensing image segmentation method and apparatus based on the SAM model. Background Technology

[0002] High-resolution remote sensing imagery segmentation is a crucial foundation for tasks such as urban mapping, agricultural monitoring, and disaster assessment. Compared to natural imagery, remote sensing imagery typically features dense targets and significant scale variations, leading to greater segmentation challenges. Cue-based segmentation models, such as the Segment Anything Model (SAM), can generate corresponding segmentation masks based on cue information such as points and bounding boxes. Point cueing offers a simple and flexible target guidance method, but its segmentation performance is highly dependent on the quality and spatial distribution of the cue points.

[0003] Existing methods typically achieve remote sensing scene adaptation by fine-tuning segmentation model parameters, leading to increased training and deployment costs, or by post-processing the segmentation results, making it difficult to correct errors at the source. Meanwhile, existing point suggestion optimization methods are limited by preset candidate point sets or global spatial constraints, making it difficult to effectively supplement missing target suggestions and adapt to dense, multi-instance remote sensing scenes.

[0004] Therefore, how to automatically generate high-quality prompts and further optimize them based on segmentation feedback while keeping the basic segmentation model parameters unchanged remains an urgent problem to be solved in prompt-based segmentation of remote sensing images. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies where point hints are of poor quality or have unreasonable location distribution when applied to multi-instance remote sensing image segmentation, leading to target omissions and cross-instance leakage. This invention provides a remote sensing image segmentation method and apparatus based on the SAM model to solve the above problems.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] A remote sensing image segmentation method based on the SAM model includes the following steps:

[0008] Acquisition and preprocessing of remote sensing images;

[0009] A point cue spatial refinement framework based on the cue-enabled segmentation model is established: The point cue spatial refinement framework includes an initial point cue generation module, a boundary-aware scale-adaptive cue routing module, and a feedback-based iterative cue refinement module. Under the condition of keeping the parameters of the cue-enabled segmentation model fixed, remote sensing image segmentation is achieved by optimizing the input point cue.

[0010] The training of the point cue space refinement framework, while keeping the parameters of the cueable segmentation base model fixed, utilizes training remote sensing images and their ground truth masks to train the boundary-aware scale-adaptive cue routing module and the feedback-based iterative cue refinement module.

[0011] The remote sensing image to be segmented is acquired and preprocessed, its dense semantic features are extracted, initial positive and negative point prompts are generated by multi-reference prototype matching and voting fusion, and the pre-query point prompts are calibrated using the trained boundary-aware scale-adaptive prompt routing module to obtain the calibrated point prompt set.

[0012] Input the remote sensing image to be segmented and the calibrated set of point cues into the basic model for cue segmentation to obtain the current segmentation mask; form target clusters based on the spatial proximity of the point cues; use the trained feedback iterative cue refinement module to iteratively edit the point cues and update the segmentation mask in the local area of ​​each target cluster until the action is stopped or the preset maximum number of iterations is reached, and output the remote sensing image ground feature segmentation mask.

[0013] The acquisition and preprocessing of the remote sensing images include the following steps:

[0014] Adjust the reference remote sensing image and the remote sensing image to be segmented to the preset size;

[0015] Initial semantic features of the reference remote sensing image and the remote sensing image to be segmented are extracted using a self-supervised visual feature extractor.

[0016] The initial semantic features are upsampled in the feature space using a feature upsampler to obtain a dense semantic feature map corresponding to the spatial location of the remote sensing image.

[0017] The framework for refining the point suggestion space based on the suggestion segmentation model includes:

[0018] An initial point prompt generation module is set up to generate a target prototype based on the reference remote sensing image and its ground truth mask, and to generate initial positive and negative point prompts through multi-reference prototype matching and voting fusion.

[0019] A boundary-aware scale-adaptive cue routing module is set up, including a feature projection layer, a multi-scale expert, a global context expert, a spatial routing module, a positive point prediction branch, and a boundary negative point prediction branch. The initial point cue is calibrated to obtain a calibrated set of point cue.

[0020] A feedback-based iterative prompt refinement module is set up, including a status encoding unit, a point prompt editing action header, and a stop action header. It is used to select a prompt editing action based on the current segmentation state, update the point prompt set based on the selected prompt editing action, and repeat the prompt update and segmentation until the stop condition is met to obtain the final segmentation mask.

[0021] The training of the point cue space refinement framework includes the following steps:

[0022] Based on the ground truth mask of the training remote sensing image, construct positive point supervision regions and boundary negative point supervision regions, and then apply the ground truth mask... Perform an erosion operation to obtain the internal region of the target as the monitoring region, and then use it to mask the ground truth. Perform dilation and calculate the difference region between the dilated mask and the original mask to obtain the outer boundary region of the target as the boundary negative point supervision region, as shown in the following formula:

[0023] , ,

[0024] in, This indicates a corrosion operation. This indicates an expansion operation. and Let represent the size of the structuring element corresponding to the operation. The positive cue supervision region and the boundary negative cue supervision region supervise the positive cue response map and the boundary negative cue response map respectively, enabling them to learn the positive point distribution inside the target and the negative point distribution at the target boundary. The formula is as follows:

[0025] ,

[0026] in, and These represent the predicted positive cue response map and the boundary negative cue response map, respectively. and These represent the positive alert monitoring area and the boundary negative alert monitoring area, respectively. This represents the weighted binary cross-entropy loss function, used to alleviate the problem of an imbalance in the number of positive and negative cue supervision regions. and These represent the positive cue loss weight and the boundary negative cue loss weight, respectively. This represents the weighted loss between the positive and negative cue response graphs and their supervisory labels;

[0027] Based on the error region between the current segmentation mask output by the basic segmentation model during the training phase and the ground truth mask, an expert action trajectory is generated. The direction of prompt editing is determined based on the error region. When there is a missed detection region, a prompt action to add a positive point is generated. When there is a false detection region, a prompt action to add a negative point is generated. When there is a positive point in the false detection region, a prompt action to delete the wrong positive point is generated. When the stop action is selected, the iterative editing stops.

[0028] The feedback-based iterative prompt refinement module predicts prompt editing actions based on state information. By optimizing the matching relationship between the action prediction results and expert actions, it learns prompt editing strategies under different segmentation error states, as shown in the following formula:

[0029] ,

[0030] in, Representing the state-action pair sample set The number of samples in Indicates the first This prompt displays status information during the editing process. This represents the expert action generated based on the current segmentation error. This represents the probability that the strategy will choose the expert action. The parameters represent the policy network. This represents the negative log-likelihood loss for optimizing the suggestion editing strategy;

[0031] During training, the current cue set is perturbed, including randomly deleting existing positive and negative cue points and adding interference cue points, to construct different cue error states. Expert action trajectories are regenerated for the perturbed cue states to improve the adaptability of the feedback iterative cue refinement module to different cue quality states.

[0032] The initial point setting prompt generation module includes the following steps:

[0033] Multiple reference remote sensing images and their ground truth masks are acquired. The initial semantic features of each reference remote sensing image are extracted using a pre-trained visual feature extractor. The initial semantic features are then spatially upsampled using a feature upsampler to obtain a dense semantic feature map corresponding to the spatial location of the reference remote sensing image.

[0034] The target region location set is determined based on the ground truth mask. Features corresponding to the target region are extracted from the dense semantic feature map and then subjected to average pooling and normalization to obtain the target prototype features corresponding to each reference remote sensing image. The formula is as follows:

[0035] ,

[0036] in, Indicates the first Zhang references the target prototype features corresponding to the remote sensing image. Indicates the first Location in Zhang reference remote sensing image Dense semantic features at the location, Indicates the target area The number of spatial locations included This indicates a feature normalization operation;

[0037] Using the same feature extractor and feature upsampling unit as the reference remote sensing image, a dense semantic feature map corresponding to the remote sensing image to be segmented is obtained. The cosine similarity between the dense semantic features of each spatial location in the remote sensing image to be segmented and the prototype features of each target is calculated to obtain multiple target-related response maps, as shown in the following formula:

[0038] ,

[0039] in, Indicates the first The target prototype is located at the position in the remote sensing image to be segmented. The target-related response value at the location, Indicates the location of the remote sensing image to be segmented Dense semantic features at the location, Indicates the first Zhang references the target prototype features corresponding to the remote sensing image. This indicates the calculation of cosine similarity.

[0040] Multiple target-related response maps are fused to obtain a comprehensive target-related response map. Candidate positive point hints are selected from the target-related response regions, and candidate negative point hints are determined in combination with boundary constraints.

[0041] The number of times each candidate location is supported as a positive or negative point by multiple reference remote sensing images is counted, and candidate locations whose number of support reaches a preset voting threshold are retained, as shown in the following formula:

[0042] ,

[0043] in, For reference, the number of figures, For the first A candidate point set generated from Zhang's reference remote sensing imagery. To combine the set of negative points selected from the candidate region with the target boundary constraints, the two constitute the first... Candidate hints provided by the reference image , Indicates spatial location, Indicates an indicator function, and Candidate positions The number of times each positive and negative cue point is marked by multiple reference remote sensing images is used, and overly dense candidate locations are removed by spatial sparsity constraints to automatically generate an initial cue point set.

[0044] The set boundary-aware scale adaptive prompting routing module includes the following steps:

[0045] Based on the dense semantic feature map corresponding to the remote sensing image to be segmented, the dense semantic feature map is input into the feature projection layer. Channel dimension transformation and feature mapping are performed on the dense semantic features at each spatial location to obtain the projected input features, as shown in the following formula:

[0046] ,

[0047] in, This indicates the feature dimension transformation and mapping operations performed by the feature projection layer. Indicates spatial location Dense semantic features at the location, This represents the input features after feature projection;

[0048] The input features are fed into multiple multi-scale experts and global context experts with different receptive fields. Local semantic features and scene-level context features at different spatial scales are extracted respectively. The spatial routing module assigns normalized fusion weights to each expert based on the feature responses at different spatial locations, and performs position-wise weighted fusion of the features from each expert to obtain spatially adaptive fusion features, as shown in the following formula:

[0049]

[0050] in, Indicates position The feature representation after spatial adaptive fusion. Indicates the number of experts. Indicates the first An expert network analyzes input features Location extracted The characteristics of the experts in the field The spatial routing module is the first An expert in position The normalized fusion weights assigned at each location;

[0051] The spatial adaptive fusion features are input into the positive point prediction branch and the boundary negative point prediction branch respectively to generate positive point score maps and boundary negative point score maps. The positive point score map is used to characterize the confidence of each spatial location as a positive point cue, and the boundary negative point score map is used to characterize the confidence of the target boundary related locations as negative point cue.

[0052] Based on the punctuality score map, the initial punctuality prompts are calibrated, low-confidence punctuality prompts are filtered out, and missed punctuality prompts are supplemented. Based on the boundary negative point score map, boundary negative point prompts are filtered in the relevant area of ​​the target boundary, and the boundary negative point prompts are merged with the initial negative point prompts. At the same time, the processed punctuality prompts and negative point prompts are merged to obtain the calibrated point prompt set.

[0053] The setting of the feedback-based iterative prompting refinement module includes the following steps:

[0054] Target clusters are formed based on the spatial proximity of the point prompts in the calibrated point prompt set, and local processing regions are constructed for each target cluster;

[0055] A local state is constructed based on the current segmentation mask, historical on-time alert information, historical off-time alert information, remote sensing image patches within the local processing area, and semantic features. This local state information is then... Recorded as:

[0056] ,

[0057] in, This is the current segmentation mask. and These are historical on-time alert messages and historical off-time alert messages, respectively. For local remote sensing image patches, For the corresponding semantic features;

[0058] Local state information is input into the state encoding unit to obtain state features. The point cue editing action head and the stop action head then select a cue editing action based on these state features. Cue editing actions include adding a positive cue, adding a negative cue, deleting an incorrect positive cue, and a stop action. When a non-stop action is selected, the point cue set is updated based on the cue editing action, and the updated point cue set is re-input into the cue-enabled segmentation base model, as shown in the following formula:

[0059] ,

[0060] in, Indicates the first The set of point hints at the next iteration This indicates the prompt to edit the action based on the current state. This indicates an operation that updates the set of prompts based on the edit action. This represents the updated set of dot hints. This indicates that the basic model can be segmented. This indicates the input of remote sensing images. This represents a new segmentation mask generated using the updated set of dot hints;

[0061] When the stop action is selected or the preset maximum number of iterations is reached, the stop point prompts for editing and outputs the current segmentation mask.

[0062] A remote sensing image segmentation device based on the SAM model includes:

[0063] The data processing module is used to acquire and preprocess remote sensing images and extract dense semantic features;

[0064] The dot hint generation and calibration module is used to generate initial positive and negative dot hints through multi-reference prototype matching and voting fusion, and to perform pre-query calibration using boundary-aware scale-adaptive hint routing;

[0065] The mask generation module is used to generate the current segmentation mask based on the remote sensing image to be segmented and the calibrated set of point cues.

[0066] The feedback-based iterative prompting and refining module is used to iteratively edit the prompting points and update the current segmentation mask within the local processing area corresponding to the target cluster, based on the current segmentation mask.

[0067] The result output module is used to output the remote sensing image feature segmentation mask when the stopping condition is met.

[0068] A computer-readable storage medium storing a computer program that, when executed by a processor, enables a remote sensing image segmentation method based on the SAM model.

[0069] A computer device is characterized by comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a remote sensing image segmentation method based on the SAM model can be implemented.

[0070] Beneficial effects

[0071] This invention discloses a remote sensing image segmentation method and apparatus based on the SAM model. Compared with existing technologies, this method reduces the impact of differences in the appearance of reference remote sensing images and the quality and spatial configuration of input point prompts on the segmentation results while keeping the model parameters unchanged. It improves the reliability of initial positive and negative point prompts, calibrates the point prompt positions based on the feature responses and boundary information of targets at different scales, reduces the problems of positive point positioning deviation, missed detection of targets at different scales, and insufficient negative points at the boundary, and uses segmentation mask feedback to iteratively adjust the point prompts, gradually repairing the problems of mask fragmentation and cross-instance leakage. It achieves closed-loop optimization by combining pre-query point prompt calibration and post-query feedback iterative prompt refinement, thus improving the accuracy of segmentation results for dense multi-instance remote sensing images. Attached Figure Description

[0072] Figure 1 This is the overall flowchart of the present invention;

[0073] Figure 2 This is a flowchart of the remote sensing image preprocessing and dense semantic feature extraction involved in this invention;

[0074] Figure 3This is a schematic diagram of the device structure of the present invention;

[0075] Figure 4 This is a schematic diagram of the boundary-aware scale-adaptive prompting routing module involved in this invention;

[0076] Figure 5 This is a schematic diagram of the feedback-based iterative prompting refining module involved in the present invention;

[0077] Figure 6 This is a flowchart illustrating the feedback-based iterative refinement process involved in this invention.

[0078] Figure 7 This is a schematic diagram illustrating the point prompt editing optimization based on segmentation error involved in this invention;

[0079] Figure 8 This is a schematic diagram illustrating the point hints and segmentation mask updates in the feedback-based iterative hint refinement process involved in this invention. Detailed Implementation

[0080] To provide a better understanding of the structural features and effects achieved by the present invention, a detailed description is provided below, accompanied by preferred embodiments and accompanying drawings:

[0081] like Figure 1 As shown, the remote sensing image segmentation method based on the SAM model of the present invention includes the following steps:

[0082] The first step is the acquisition and preprocessing of remote sensing images. For example... Figure 2 The diagram shown is a flowchart of remote sensing image preprocessing and dense semantic feature extraction, which includes the following steps:

[0083] (1) Adjust the reference remote sensing image and the remote sensing image to be segmented to the preset size.

[0084] (2) Use a self-supervised visual feature extractor to extract the initial semantic features of the reference remote sensing image and the remote sensing image to be segmented.

[0085] (3) Use a feature upsampler to upsample the initial semantic features in the feature space to obtain a dense semantic feature map corresponding to the spatial location of the remote sensing image.

[0086] The second step is to define a point suggestion space refinement framework based on the suggestion segmentation model: such as... Figure 3 As shown, the point cue spatial refinement framework includes an initial point cue generation module, a boundary-aware scale-adaptive cue routing module, and a feedback-based iterative cue refinement module. Under the condition of keeping the basic parameters of the cueable segmentation model fixed, it achieves remote sensing image segmentation by optimizing the input point cue.

[0087] The above design was adopted because pixel-level annotation of remote sensing images is costly, while point annotation is simple, flexible, and suitable as input for the cue-based segmentation model. The initial point cue generation module constructs a target prototype using reference remote sensing images and their ground truth masks. Through multi-reference prototype matching and voting fusion, it automatically generates initial positive and negative point cuees, providing a foundation for subsequent cue refinement. Point cue errors in remote sensing images exhibit different forms. After initial point cue generation, issues such as positional offsets, unreasonable scale adaptations, and insufficient boundary constraints can be identified and calibrated before calling the model using dense semantic features, multi-scale responses, global context information, and boundary information. However, even if the position of a single point cue is relatively reasonable, when multiple point cuees are combined and input into the cue-based segmentation model, problems such as segmentation masks crossing adjacent instances, fragmented local regions, and non-target responses may still occur. These defects usually require further evaluation in conjunction with the current segmentation mask output by the model. Therefore, the boundary-aware scale-adaptive cue routing module is mainly used to calibrate the spatial location, scale adaptation relationship, and boundary constraints of the initial point cue, while the feedback-based iterative cue refining module adds, deletes, and corrects point cue within the local region of the target cluster based on the error revealed by the current segmentation mask.

[0088] Through the synergistic effect of the three modules, point hint generation, hint calibration, mask feedback, and hint update can be integrated into a continuous processing flow. Without changing the parameters of the basic model for hint segmentation, the quality of point hints and the accuracy of segmentation results can be gradually improved, which is especially helpful in improving the instance differentiation effect in dense target and complex boundary scenes.

[0089] In practice, while the current segmentation mask provides a basis for subsequent hint refinement, it may also contain errors caused by initial point hint deviations or unreasonable point hint combinations. Therefore, identifying defects such as instance clustering, local fragmentation, and non-target responses based on the not entirely accurate current segmentation mask, and determining corresponding point hint adjustment methods, is a key issue that feedback-based iterative hint refinement needs to address. Simultaneously, it is also necessary to reasonably limit the scope of hint updates and iteration stopping conditions to avoid the accumulation of errors during iteration or instability in the segmentation results due to repeated adjustments.

[0090] (1) Set an initial point prompt generation module to generate target prototypes based on reference remote sensing images and their ground truth masks, and generate initial positive and negative point prompts through multi-reference prototype matching and voting fusion.

[0091] Generating initial point cues requires addressing issues such as target localization, representativeness of reference information, and differentiation between positive and negative points. Targets in remote sensing images are typically small-scale and densely distributed. When directly using pre-trained visual features for matching, the features of small-scale targets are easily diluted by surrounding background information, leading to unclear target responses or inaccurate localization. Furthermore, the target prototype formed by a single reference remote sensing image and its ground truth mask is insufficient to fully cover the variations in scale, shape, texture, and imaging conditions of targets across different images, easily causing the matching results to be influenced by a single reference sample.

[0092] Furthermore, the target area and complex background may have high similarity in local texture, making it difficult to reliably distinguish between positive and negative points based on a single feature matching result; different reference remote sensing images may also have different responses to the same spatial location. When candidate points are too dense, location duplication or mutual interference may also occur. Therefore, achieving a balance between target coverage integrity, candidate point reliability, and spatial distribution of cue points is a key issue that needs to be addressed in the initial point cue generation process.

[0093] A1) Obtain multiple reference remote sensing images and their ground truth masks, use a pre-trained visual feature extractor to extract the initial semantic features of each reference remote sensing image, and use a feature upsampler to spatially upsample the initial semantic features to obtain a dense semantic feature map corresponding to the spatial location of the reference remote sensing image.

[0094] The target region location set is determined based on the ground truth mask. Features corresponding to the target region are extracted from the dense semantic feature map and then subjected to average pooling and normalization to obtain the target prototype features corresponding to each reference remote sensing image. The formula is as follows:

[0095] ,

[0096] in, Indicates the first Zhang references the target prototype features corresponding to the remote sensing image. Indicates the first Location in Zhang reference remote sensing image Dense semantic features at the location, Indicates the target area The number of spatial locations included This indicates a feature normalization operation;

[0097] A2) Using the same feature extractor and feature upsampling unit as the reference remote sensing image, obtain the dense semantic feature map corresponding to the remote sensing image to be segmented. Calculate the cosine similarity between the dense semantic features at each spatial location in the remote sensing image to be segmented and the prototype features of each target to obtain multiple target-related response maps. The formula is as follows:

[0098] ,

[0099] in, Indicates the first The target prototype is located at the position in the remote sensing image to be segmented. The target-related response value at the location, Indicates the location of the remote sensing image to be segmented Dense semantic features at the location, Indicates the first Zhang references the target prototype features corresponding to the remote sensing image. This indicates the calculation of cosine similarity.

[0100] Multiple target-related response maps are fused to obtain a comprehensive target-related response map. Candidate positive point hints are selected from the target-related response regions, and candidate negative point hints are determined in combination with boundary constraints.

[0101] A3) Count the number of times each candidate location is supported as a positive or negative point by multiple reference remote sensing images, and retain the candidate locations whose number of support reaches a preset voting threshold, as shown in the following formula:

[0102] ,

[0103] in, For reference, the number of figures, For the first A candidate point set generated from Zhang's reference remote sensing imagery. To combine the set of negative points selected from the candidate region with the target boundary constraints, the two constitute the first... Candidate hints provided by the reference image , Indicates spatial location, Indicates an indicator function, and Candidate positions The number of times each positive and negative cue point is marked by multiple reference remote sensing images is used, and overly dense candidate locations are removed by spatial sparsity constraints to automatically generate an initial cue point set.

[0104] The above processing reduces the impact of local mismatches in a single reference image on initial point cues. In actual fusion, the voting threshold should not be set too low or too high. If the threshold is too low, erroneous candidate points lacking sufficient reference support may be retained; if the threshold is too high, small-scale targets or targets with significant appearance differences may be missed. Furthermore, overly concentrated candidate points may cause redundant constraints from multiple closely spaced cues, affecting the response of the subsequent cue-providing segmentation model. Therefore, it is necessary to comprehensively consider the reference consistency, target coverage, and spatial distribution of candidate points to form an appropriate number and reasonable placement of initial positive and negative cues, providing a stable input source for subsequent cue calibration.

[0105] (2) Set up a boundary-aware scale-adaptive prompting routing module, including a feature projection layer, a multi-scale expert, a global context expert, a spatial routing module, a positive point prediction branch and a boundary negative point prediction branch.

[0106] When calibrating initial point cues, significant differences exist in target scale, local structure, and background environment at different spatial locations. Adaptively selecting and fusing expert features from different receptive fields based on spatial location is a key challenge this module needs to address. Fixed-scale feature extraction struggles to adapt to targets of varying sizes simultaneously. When the receptive field is large, features of small-scale targets are easily diluted by surrounding background information, resulting in indistinct target responses or even difficulty in effective perception. Conversely, when the receptive field is small, it's difficult to fully represent larger targets and their neighborhood structures. Relying solely on local features is insufficient to adequately reflect target density, background complexity, and the overall relationship between the target and its surrounding area. Furthermore, positive point prediction and boundary negative point prediction need to work together. Offsets in positive point positions may lead to insufficient target area coverage, while inaccurate boundary negative point positions may over-suppress the target or fail to prevent the segmentation results from expanding into adjacent instances and the background region.

[0107] like Figure 4 As shown, this is a schematic diagram of the boundary-aware scale-adaptive prompting routing module, which includes a feature projection layer, a multi-scale expert, a global context expert, a spatial routing module, a positive point prediction branch, and a boundary negative point prediction branch. This module is used to perform pre-query calibration of the initial point prompts, specifically including the following steps:

[0108] B1) Based on the dense semantic feature map corresponding to the remote sensing image to be segmented, the dense semantic feature map is input into the feature projection layer. Channel dimension transformation and feature mapping are performed on the dense semantic features at each spatial location to obtain the projected input features, as shown in the following formula:

[0109] ,

[0110] in, This indicates the feature dimension transformation and mapping operations performed by the feature projection layer. Indicates spatial location Dense semantic features at the location, This represents the input features after feature projection;

[0111] B2) Input features are fed into multiple multi-scale experts and global context experts with different receptive fields. Local semantic features and scene-level context features at different spatial scales are extracted respectively. The spatial routing module assigns normalized fusion weights to each expert based on the feature responses at different spatial locations, and performs position-wise weighted fusion of the features of each expert to obtain spatial adaptive fusion features, as shown in the following formula:

[0112]

[0113] in, Indicates position The feature representation after spatial adaptive fusion. Indicates the number of experts. Indicates the first An expert network analyzes input features Location extracted The characteristics of the experts in the field The spatial routing module is the first An expert in position The normalized fusion weights assigned at each location;

[0114] B3) Input the spatial adaptive fusion features into the positive point prediction branch and the boundary negative point prediction branch respectively to generate a positive point score map and a boundary negative point score map; wherein, the positive point score map is used to characterize the confidence of each spatial location as a positive point indication, and the boundary negative point score map is used to characterize the confidence of the target boundary related locations as negative point indications;

[0115] B4) Based on the punctuality score map, the initial punctuality prompts are calibrated, low-confidence punctuality prompts are filtered out, and missing punctuality prompts are supplemented. Based on the boundary negative point score map, boundary negative point prompts are filtered in the relevant area of ​​the target boundary, and the boundary negative point prompts are merged with the initial negative point prompts. At the same time, the processed punctuality prompts and negative point prompts are merged to obtain the calibrated point prompt set.

[0116] After the above processing, the positive point prediction branch can calibrate the initial positive point position based on the high-response region inside the target, and reduce positive point offset and target missed detection by filtering out low-confidence points and supplementing with high-response peaks. The boundary negative point prediction branch supplements negative point hints based on the response results of the target boundary neighborhood, providing separation constraints between adjacent instances and between the target and the background. Fusing the processed positive point hints and boundary negative point hints with the initial point hints respectively can supplement new hint information while retaining the prototype matching results, resulting in a calibrated set of point hints. This process is completed only based on the dense semantic features and boundary responses of the remote sensing image to be segmented, without relying on the segmentation mask output by the hintable segmentation base model. Therefore, it can improve the spatial location, target coverage, and boundary constraints of the input point hints before the first call to the hintable segmentation base model, providing a more reliable input for subsequent feedback-based iterative hint refinement.

[0117] (3) Set up a feedback-based iterative prompting refinement module, such as Figure 5 As shown, it specifically includes a status encoding unit, a point cue editing action header, and a stop action header. It is used to select a cue editing action based on the current segmentation status, update the point cue set based on the selected cue editing action, and repeat the cue update and segmentation until the stop condition is met to obtain the final segmentation mask.

[0118] In the feedback-based iterative refinement process, the current segmentation mask not only provides a basis for subsequent editing actions but may also contain errors caused by initial point hint deviations or unreasonable point hint combinations. Therefore, the module needs to solve the problem of determining which type of problem a local region belongs to based on the current segmentation result and deciding whether to add positive points, add negative points, delete erroneous positive points, or stop the iteration.

[0119] Meanwhile, each addition or deletion of point hints alters the subsequent output of the basic segmentation model. While a single edit can improve a local area, it may also cause other areas to be oversuppressed or introduce new false responses, leading to fluctuations in the iteration results. Therefore, it is necessary to comprehensively consider the current segmentation mask, historical positive and negative point hints, local remote sensing imagery, and semantic features to determine the editing location, editing type, and stopping timing, and to reasonably control the iteration range and number of iterations, balancing segmentation effectiveness with model call overhead while reducing error accumulation.

[0120] like Figure 6 The diagram shown is a flowchart of the feedback-based iterative refinement process, and the specific steps are as follows:

[0121] C1) Form target clusters based on the spatial proximity of the positive point prompts in the calibrated point prompt set, and construct local processing regions for each target cluster;

[0122] C2) Construct a local state based on the current segmentation mask, historical positive and negative alert information, remote sensing image blocks within the local processing area, and semantic features, and then process the local state information. Recorded as:

[0123] ,

[0124] in, This is the current segmentation mask. and These are historical on-time alert messages and historical off-time alert messages, respectively. For local remote sensing image patches, For the corresponding semantic features;

[0125] C3) Local state information is input into the state encoding unit to obtain state features. The point cue editing action head and the stop action head select cue editing actions based on the state features. Cue editing actions include adding positive cue, adding negative cue, deleting incorrect positive cue, and stopping. When a non-stop action is selected, the point cue set is updated according to the cue editing action, and the updated point cue set is re-input into the cue-enabled segmentation base model, as shown in the following formula:

[0126] ,

[0127] in, Indicates the first The set of point hints at the next iteration This indicates the prompt to edit the action based on the current state. This indicates an operation that updates the set of prompts based on the edit action. This represents the updated set of dot hints. This indicates that the basic model can be segmented. This indicates the input of remote sensing images. This represents a new segmentation mask generated using the updated set of dot hints;

[0128] C4) When the stop action is selected or the preset maximum number of iterations is reached, the stop point prompts for editing and outputs the current segmentation mask.

[0129] The third step involves training the point-based cue space refinement framework. While keeping the parameters of the cueable segmentation base model fixed, the training remote sensing images and their ground truth masks are used to train the boundary-aware scale-adaptive cue routing module and the feedback-based iterative cue refinement module.

[0130] The above training method is adopted because the boundary-aware scale-adaptive cue routing module and the feedback-based iterative cue refining module handle different types of errors. The former uses the truth mask to generate cue supervision regions inside the target and outside the boundary, learning the spatial distribution of positive points and negative points on the boundary. The latter generates expert action trajectories based on the error between the current segmentation mask and the truth mask, learning cue editing strategies such as adding positive points, adding negative points, deleting erroneous positive points, and stopping iteration.

[0131] This training method, while keeping the parameters of the basic segmentation model fixed, enables two modules to acquire the ability to calibrate prompts before model invocation and the ability to correct errors after segmentation result feedback, respectively, while avoiding reliance on manual interaction. The challenge lies in the fact that the supervised regions corresponding to positive prompts and boundary negative prompts are different, and the quantity distribution of the two types of prompts may differ significantly. It is necessary to reasonably construct the supervised regions and balance the training effects of different prompt signals. Furthermore, each prompt edit changes the subsequent segmentation state, potentially introducing new misclassifications or omissions. Therefore, it is necessary to reasonably construct the training state and stopping conditions so that the feedback-based iterative prompt refinement module can adapt to prompts and segmentation results of varying quality, reducing error accumulation.

[0132] (1) Construct positive point supervision regions and boundary negative point supervision regions based on the ground truth mask of the training remote sensing image, and process the ground truth mask. Perform an erosion operation to obtain the internal region of the target as the monitoring region, and then use it to mask the ground truth. Perform dilation and calculate the difference region between the dilated mask and the original mask to obtain the outer boundary region of the target as the boundary negative point supervision region, as shown in the following formula:

[0133] , ,

[0134] in, This indicates a corrosion operation. This indicates an expansion operation. and Let represent the size of the structuring element corresponding to the operation. The positive cue supervision region and the boundary negative cue supervision region supervise the positive cue response map and the boundary negative cue response map respectively, enabling them to learn the positive point distribution inside the target and the negative point distribution at the target boundary. The formula is as follows:

[0135] ,

[0136] in, and These represent the predicted positive cue response map and the boundary negative cue response map, respectively. and These represent the positive alert monitoring area and the boundary negative alert monitoring area, respectively. This represents the weighted binary cross-entropy loss function, used to alleviate the problem of an imbalance in the number of positive and negative cue supervision regions. and These represent the positive cue loss weight and the boundary negative cue loss weight, respectively. This represents the weighted loss between the positive and negative cue response graphs and their supervisory labels.

[0137] (2) Generate expert action trajectory based on the error region between the current segmentation mask and the ground truth mask output by the basic model that can prompt the segmentation during the training phase. Determine the prompt editing direction based on the error region. When there is a missed detection region, generate a prompt action to add positive points. When there is a false detection region, generate a prompt action to add negative points. When there is a positive point in the false detection region, generate a prompt action to delete the wrong positive point. Stop iterative editing when the stop action is selected.

[0138] The feedback-based iterative prompt refinement module predicts prompt editing actions based on state information. By optimizing the matching relationship between the action prediction results and expert actions, it learns prompt editing strategies under different segmentation error states, as shown in the following formula:

[0139] ,

[0140] in, Representing the state-action pair sample set The number of samples in Indicates the first This prompt displays status information during the editing process. This represents the expert action generated based on the current segmentation error. This represents the probability that the strategy will choose the expert action. The parameters of the policy network are represented. This represents the negative log-likelihood loss for optimizing the suggestion editing strategy;

[0141] During training, the current cue set is perturbed, including randomly deleting existing positive and negative cue points and adding interference cue points, to construct different cue error states. Expert action trajectories are regenerated for the perturbed cue states to improve the adaptability of the feedback iterative cue refinement module to different cue quality states.

[0142] The fourth step involves acquiring and preprocessing the remote sensing image to be segmented, extracting its dense semantic features, generating initial positive and negative point prompts through multi-reference prototype matching and voting fusion, and using the trained boundary-aware scale-adaptive prompt routing module to calibrate the point prompts before the query, resulting in a calibrated set of point prompts.

[0143] The fifth step involves inputting the remote sensing image to be segmented and the calibrated set of point cues into the basic model for cue segmentation to obtain the current segmentation mask. Target clusters are formed based on the spatial proximity of the point cues. The trained feedback iterative cue refinement module is used to iteratively edit the point cues and update the segmentation mask in the local area of ​​each target cluster until the action is stopped or the preset maximum number of iterations is reached, and then the remote sensing image ground feature segmentation mask is output.

[0144] like Figure 7 As shown, this is a schematic diagram of point hint optimization based on segmentation error. While the initial calibrated point hints can guide the basic model for segmentation to locate the target, and the spatial positions of the hint points appear reasonable, the correctness of the combination relationship between the hint points can only be determined after generating the segmentation mask. When there are insufficient negative separating points between adjacent targets, the segmentation mask is prone to crossing instance boundaries and causing cross-instance leakage. Adding negative point hints at the gaps between adjacent instances can suppress the mask from expanding into adjacent targets, achieving effective separation of different instances. When the target region is under-segmented due to insufficient positive point hints, supplementing positive point hints in the target region not covered by the mask can guide the model to recover the missing region. Therefore, this invention iteratively adjusts positive and negative point hints based on the error reflected by the segmentation mask, correcting the hint configuration defect that only appears after model inference, and improving the accuracy of the segmentation results.

[0145] like Figure 8 As shown, this is a schematic diagram of the round-by-round update of point hints and segmentation masks in the feedback-based iterative hint refinement process. The initial point hint input provides a basic segmentation model, resulting in an initial segmentation mask. The feedback-based iterative hint refinement module adds or removes positive and negative point hints round by round based on the error reflected by the current segmentation mask, adjusting the hint configuration and updating the segmentation mask accordingly. In the illustrated process, adding negative point hints is used to suppress local missegmentation; however, excessively strong negative point constraints can cause some target areas to be missed. In this case, the module supplements the missed areas with positive point hints based on the updated mask feedback, restoring the over-suppressed parts. As can be seen from the results of each iteration, a single hint edit may introduce new undersegmentation while correcting local missegmentation. Therefore, the segmentation mask does not strictly improve monotonically with each iteration. However, by continuously adjusting the positive and negative point configuration using the latest mask feedback, the local deviations caused by the previous round of hints can be gradually corrected, making the segmentation mask gradually approach the true mask, thereby improving the integrity and boundary accuracy of the target area.

[0146] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A remote sensing image segmentation method based on the SAM model, characterized in that, Includes the following steps: 11) Acquisition and preprocessing of remote sensing images; 12) Define a point cue spatial refinement framework based on the cue-based segmentation model: The point cue spatial refinement framework includes an initial point cue generation module, a boundary-aware scale-adaptive cue routing module, and a feedback-based iterative cue refinement module. Under the condition of keeping the parameters of the cue-based segmentation model fixed, remote sensing image segmentation is achieved by optimizing the input point cue. 13) Training of the point cue space refinement framework: While keeping the parameters of the cueable segmentation base model fixed, the boundary-aware scale adaptive cue routing module and the feedback iterative cue refinement module are trained using training remote sensing images and their ground truth masks. 14) Acquire and preprocess the remote sensing image to be segmented, extract its dense semantic features, generate initial positive and negative point prompts through multi-reference prototype matching and voting fusion, and use the trained boundary-aware scale adaptive prompt routing module to calibrate the point prompts before the query, and obtain the calibrated point prompt set. 15) Input the remote sensing image to be segmented and the calibrated set of point hints into the basic model for hint segmentation to obtain the current segmentation mask; Based on the spatial proximity relationships indicated by the point prompts, target clusters are formed. The trained feedback iterative prompting refinement module is used to iteratively edit point prompts and update the segmentation mask in the local area of ​​each target cluster until the action is stopped or the preset maximum number of iterations is reached, and then the remote sensing image land feature segmentation mask is output.

2. The remote sensing image segmentation method based on the SAM model according to claim 1, characterized in that, The acquisition and preprocessing of the remote sensing images include the following steps: 21) Adjust the reference remote sensing image and the remote sensing image to be segmented to the preset size; 22) Use a self-supervised visual feature extractor to extract the initial semantic features of the reference remote sensing image and the remote sensing image to be segmented, respectively; 23) Use a feature upsampler to upsample the initial semantic features in the feature space to obtain a dense semantic feature map corresponding to the spatial location of the remote sensing image.

3. The remote sensing image segmentation method based on the SAM model according to claim 1, characterized in that, Setting up a point suggestion space refinement framework based on a suggestion segmentation base model includes the following steps: 31) Set up an initial point prompt generation module to generate target prototypes based on reference remote sensing images and their ground truth masks, and generate initial positive and negative point prompts through multi-reference prototype matching and voting fusion; 32) Set up a boundary-aware scale-adaptive prompting routing module, including a feature projection layer, a multi-scale expert, a global context expert, a spatial routing module, a positive point prediction branch and a boundary negative point prediction branch, and calibrate the initial point prompts to obtain a calibrated set of point prompts; 33) Set up a feedback-based iterative prompt refinement module, including a status encoding unit, a point prompt editing action head, and a stop action head, which is used to select a prompt editing action according to the current segmentation state, update the point prompt set according to the selected prompt editing action, repeat the prompt update and segmentation until the stop condition is met, and obtain the final segmentation mask.

4. The remote sensing image segmentation method based on the SAM model according to claim 1, characterized in that, The training of the point cue space refinement framework includes the following steps: 41) Construct positive point supervision regions and boundary negative point supervision regions based on the ground truth mask of the training remote sensing image, and process the ground truth mask. Perform an erosion operation to obtain the internal region of the target as the monitoring region, and then use it to mask the ground truth. Perform dilation and calculate the difference region between the dilated mask and the original mask to obtain the outer boundary region of the target as the boundary negative point supervision region, as shown in the following formula: ， ， in, This indicates a corrosion operation. This indicates an expansion operation. and Let represent the size of the structuring element corresponding to the operation. The positive cue supervision region and the boundary negative cue supervision region supervise the positive cue response map and the boundary negative cue response map respectively, enabling them to learn the positive point distribution inside the target and the negative point distribution at the target boundary. The formula is as follows: ， in, and These represent the predicted positive cue response map and the boundary negative cue response map, respectively. and These represent the positive alert monitoring area and the boundary negative alert monitoring area, respectively. This represents the weighted binary cross-entropy loss function, used to alleviate the problem of an imbalance in the number of positive and negative cue supervision regions. and These represent the positive cue loss weight and the boundary negative cue loss weight, respectively. This represents the weighted loss between the positive and negative cue response graphs and their supervisory labels; 42) Generate expert action trajectories based on the error region between the current segmentation mask and the ground truth mask output by the basic segmentation model during the training phase. Determine the prompt editing direction based on the error region. When there is a missed detection region, generate a prompt action to add positive points. When there is a false detection region, generate a prompt action to add negative points. When there is a positive point in the false detection region, generate a prompt action to delete the wrong positive point. Stop iterative editing when the stop action is selected. The feedback-based iterative prompt refinement module predicts prompt editing actions based on state information. By optimizing the matching relationship between the action prediction results and expert actions, it learns prompt editing strategies under different segmentation error states, as shown in the following formula: ， in, Representing the state-action pair sample set The number of samples in Indicates the first This prompt displays status information during the editing process. This represents the expert action generated based on the current segmentation error. This represents the probability that the strategy will choose the expert action. The parameters represent the policy network. This represents the negative log-likelihood loss for optimizing the suggestion editing strategy; During training, the current cue set is perturbed, including randomly deleting existing positive and negative cue points and adding interference cue points, to construct different cue error states. Expert action trajectories are regenerated for the perturbed cue states to improve the adaptability of the feedback iterative cue refinement module to different cue quality states.

5. The remote sensing image segmentation method based on the SAM model according to claim 3, characterized in that, The initial point setting prompt generation module includes the following steps: 51) Obtain multiple reference remote sensing images and their ground truth masks, use a pre-trained visual feature extractor to extract the initial semantic features of each reference remote sensing image, and use a feature upsampler to spatially upsample the initial semantic features to obtain a dense semantic feature map corresponding to the spatial location of the reference remote sensing image. The target region location set is determined based on the ground truth mask. Features corresponding to the target region are extracted from the dense semantic feature map and then subjected to average pooling and normalization to obtain the target prototype features corresponding to each reference remote sensing image. The formula is as follows: ， in, Indicates the first Zhang references the target prototype features corresponding to the remote sensing image. Indicates the first Location in Zhang reference remote sensing image Dense semantic features at the location, Indicates the target area The number of spatial locations included This indicates a feature normalization operation; 52) Using the same feature extractor and feature upsampling unit as the reference remote sensing image, obtain the dense semantic feature map corresponding to the remote sensing image to be segmented. Calculate the cosine similarity between the dense semantic features of each spatial location in the remote sensing image to be segmented and the prototype features of each target, and obtain multiple target-related response maps. The formula is as follows: ， in, Indicates the first The target prototype is located at the position in the remote sensing image to be segmented. The target-related response value at the location, Indicates the location of the remote sensing image to be segmented Dense semantic features at the location, Indicates the first Zhang references the target prototype features corresponding to the remote sensing image. This indicates the calculation of cosine similarity. Multiple target-related response maps are fused to obtain a comprehensive target-related response map. Candidate positive point hints are selected from the target-related response regions, and candidate negative point hints are determined in combination with boundary constraints. 53) Count the number of times each candidate location is supported as a positive or negative point by multiple reference remote sensing images, and retain the candidate locations whose number of support reaches the preset voting threshold, as shown in the following formula: ， in, For reference, the number of figures, For the first A candidate point set generated from Zhang's reference remote sensing imagery. To combine the set of negative points selected from the candidate region with the target boundary constraints, the two constitute the first... Candidate hints provided by the reference image , Indicates spatial location, Indicates an indicator function, and Candidate positions The number of times each positive and negative cue point is marked by multiple reference remote sensing images is used, and overly dense candidate locations are removed by spatial sparsity constraints to automatically generate an initial cue point set.

6. The remote sensing image segmentation method based on the SAM model according to claim 3, characterized in that, The set boundary-aware scale adaptive prompting routing module includes the following steps: 61) Based on the dense semantic feature map corresponding to the remote sensing image to be segmented, the dense semantic feature map is input into the feature projection layer. Channel dimension transformation and feature mapping are performed on the dense semantic features at each spatial location to obtain the projected input features, as shown in the following formula: ， in, This indicates the feature dimension transformation and mapping operations performed by the feature projection layer. Indicates spatial location Dense semantic features at the location, This represents the input features after feature projection; 62) Input features are fed into multiple multi-scale experts and global context experts with different receptive fields. Local semantic features and scene-level context features at different spatial scales are extracted respectively. The spatial routing module assigns normalized fusion weights to each expert based on the feature responses at different spatial locations. The features of each expert are then weighted and fused position by position to obtain spatial adaptive fusion features, as shown in the following formula: ， in, Indicates position The feature representation after spatial adaptive fusion. Indicates the number of experts. Indicates the first An expert network analyzes input features Location extracted The characteristics of the experts in the field The spatial routing module is the first An expert in position The normalized fusion weights assigned at each location; 63) Input the spatial adaptive fusion features into the positive point prediction branch and the boundary negative point prediction branch respectively to generate a positive point score map and a boundary negative point score map; wherein, the positive point score map is used to characterize the confidence of each spatial location as a positive point indication, and the boundary negative point score map is used to characterize the confidence of the target boundary related locations as negative point indications; 64) Based on the punctuality score map, the initial punctuality prompts are calibrated, low-confidence punctuality prompts are filtered out, and missing punctuality prompts are supplemented. Based on the boundary negative point score map, boundary negative point prompts are filtered in the relevant area of ​​the target boundary, and the boundary negative point prompts are merged with the initial negative point prompts. At the same time, the processed punctuality prompts and negative point prompts are merged to obtain the calibrated point prompt set.

7. The remote sensing image segmentation method based on the SAM model according to claim 3, characterized in that, The setting of the feedback-based iterative prompting refinement module includes the following steps: 71) Form target clusters based on the spatial proximity relationships between the point prompts in the calibrated point prompt set, and construct local processing regions for each target cluster; 72) Construct a local state based on the current segmentation mask, historical positive and negative alert information, remote sensing image patches within the local processing area, and semantic features, and then transfer the local state information. Recorded as: ， in, This is the current segmentation mask. and These are historical on-time alert messages and historical off-time alert messages, respectively. For local remote sensing image patches, For the corresponding semantic features; 73) Input the local state information into the state encoding unit to obtain state features, and select the prompt editing action based on the state features by the point prompt editing action head and the stop action head; the prompt editing actions include adding positive point prompts, adding negative point prompts, deleting incorrect positive point prompts, and the stop action; when a non-stop action is selected, the point prompt set is updated according to the prompt editing action, and the updated point prompt set is re-inputted into the promptable segmentation base model, as shown in the following formula: ， in, Indicates the first The set of point hints at the next iteration This indicates the prompt to edit the action based on the current state. This indicates an operation that updates the set of prompts based on the edit action. This represents the updated set of dot hints. This indicates that the basic model can be segmented. This indicates the input of remote sensing images. This represents a new segmentation mask generated using the updated set of dot hints; 74) When the stop action is selected or the preset maximum number of iterations is reached, the stop point prompts for editing and outputs the current segmentation mask.

8. A remote sensing image segmentation device based on the SAM model, characterized in that, include: The data processing module is used to acquire and preprocess remote sensing images and extract dense semantic features; The dot hint generation and calibration module is used to generate initial positive and negative dot hints through multi-reference prototype matching and voting fusion, and to perform pre-query calibration using boundary-aware scale-adaptive hint routing; The mask generation module is used to generate the current segmentation mask based on the remote sensing image to be segmented and the calibrated set of point cues. The feedback-based iterative prompting and refining module is used to iteratively edit the prompting points and update the current segmentation mask within the local processing area corresponding to the target cluster, based on the current segmentation mask. The result output module is used to output the remote sensing image feature segmentation mask when the stopping condition is met.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the remote sensing image segmentation method based on the SAM model as described in any one of claims 1-7.

10. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can implement the remote sensing image segmentation method based on the SAM model as described in any one of claims 1-7.

Images