Sam-based remote sensing image semantic segmentation method and system

Abstract

The application discloses a kind of based on SAM's remote sensing image semantic segmentation method and system, solve the technical problem that existing technology adopts SAM model to carry out remote sensing image semantic segmentation exists feature adaptability is poor, boundary details fuzzy etc..The method comprises: the remote sensing image is preprocessed, obtains original input image;The original input image is encoded using the SAM encoder of embedding frequency modulation adapter, obtains encoded feature;The encoded feature is decoded using SAM decoder, obtains preliminary segmentation feature;The preliminary segmentation feature is processed using edge guide refiner, obtains final segmentation result.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically, it relates to remote sensing image processing, and more specifically, it relates to a method and system for semantic segmentation of remote sensing images based on SAM. Background Technology

[0002] Semantic segmentation of remote sensing images, a core technology in the field of intelligent interpretation of remote sensing images, aims to accurately assign a predefined semantic category label, such as building, road, vegetation, or water body, to each pixel in an image. This technology plays an indispensable role in macro-planning and refined management, providing fundamental data support for key applications such as dynamic land cover mapping, modern urban planning, sustainable management of natural resources, and monitoring of major natural disasters.

[0003] In recent years, deep learning-driven automated interpretation paradigms have become mainstream in this field. Architectures such as Convolutional Neural Networks (CNNs) and Transformers have achieved significant success in remote sensing image segmentation tasks due to their powerful feature representation capabilities. However, these models typically follow a supervised learning paradigm, and their high performance heavily relies on training with massive amounts of pixel-level precisely labeled data, exhibiting a "data-hungry" characteristic. Simultaneously, their generalization ability is often limited by the diversity of training data, leading to performance degradation when facing new regions or data from different sensors.

[0004] Against this backdrop, the emergence of the large-scale visual foundational model, the Segment Anything Model (SAM), has brought a paradigm-breaking breakthrough to the field of computer vision. By pre-training on massive datasets of billions of images and segmentation masks, SAM has achieved unprecedented zero-shot segmentation capabilities and outstanding generalization performance, providing new possibilities for solving data sparsity problems in specific domains.

[0005] Despite the powerful versatility of the SAM model, its direct application to specialized remote sensing image semantic segmentation tasks still faces significant challenges stemming from the "modal gap" between natural and remote sensing images. A core obstacle lies in insufficient feature adaptability. SAM's training data primarily consists of natural images, while remote sensing images differ fundamentally from natural images in imaging patterns (e.g., orthophoto viewpoint), spectral characteristics, spatial resolution, and the complexity and heterogeneity of ground feature distribution. For example, in remote sensing scenarios, urban built-up areas are rich in high-frequency information due to their dense edges and textures, while vast farmlands or water bodies are dominated by smooth, continuous low-frequency information. SAM's relatively fixed feature extraction pattern, designed for natural images, struggles to adaptively handle these frequency feature differences arising from the diversity of ground feature types, thus limiting its feature representation capabilities in remote sensing applications.

[0006] Meanwhile, the inherent architecture of SAM also presents another key challenge: semantic ambiguity of object boundaries. On the one hand, man-made features (such as building roofs) and natural backgrounds (such as bare land) in remote sensing images may have similar textures or patterns, easily leading to boundary confusion. On the other hand, the patch embedding strategy employed by the SAM encoder inevitably sacrifices the fine spatial details of the image while constructing high-level semantic representations. Its decoder struggles to accurately reconstruct the sharp boundaries of objects during subsequent upsampling to restore resolution, often resulting in blurred edges and rounded corners of objects such as buildings in the segmentation results. This fails to maintain the clear geometric integrity of man-made features, severely impacting segmentation accuracy and practical value.

[0007] Therefore, optimizing the SAM model to effectively bridge the modal gap and addressing its core problems of poor feature adaptability and blurred boundary details in remote sensing image segmentation tasks, thereby comprehensively improving the accuracy and reliability of segmentation, has become a key technical challenge that urgently needs to be solved in this field. Summary of the Invention

[0008] The purpose of this invention is to propose a semantic segmentation method and system for remote sensing images based on SAM, which solves the technical problems of poor feature adaptability and blurred boundary details in the existing technology of semantic segmentation of remote sensing images using the SAM model.

[0009] To achieve the above-mentioned objectives, the SAM-based remote sensing image semantic segmentation method provided by this invention employs the following technical solution:

[0010] A semantic segmentation method for remote sensing images based on SAM includes:

[0011] The remote sensing images are preprocessed to obtain the original input images;

[0012] The original input image is encoded using a SAM encoder with an embedded frequency modulation adapter to obtain encoded features as follows: Layer normalization and multi-head attention are applied to the original input image to output intermediate features; Fourier transform is used to transform the intermediate features to the frequency domain to obtain frequency domain features; the amplitude spectrum of the frequency domain features is obtained, and the total energy of the amplitude spectrum is calculated; a dynamic partitioning radius is determined based on a preset energy percentage threshold and the total energy of the amplitude spectrum by radially accumulating energy from the center of the amplitude spectrum; the frequency domain features are separated into low-frequency and high-frequency components based on the dynamic partitioning radius; the low-frequency and high-frequency components are modulated respectively to obtain modulated low-frequency and modulated high-frequency components; the modulated low-frequency and high-frequency components are merged and transformed to the spatial domain using inverse Fourier transform to obtain enhanced features; the enhanced features are concatenated with the residual of the original input image to obtain the encoded features.

[0013] The encoded features are decoded using a SAM decoder to obtain preliminary segmentation features;

[0014] The preliminary segmentation features are processed using an edge-guided refiner to obtain the final segmentation result: the preliminary segmentation features are processed to obtain semantic features; the original input image is processed to obtain multi-scale edge features; the semantic features and the multi-scale edge features are concatenated and processed by a multilayer perceptron to obtain the final segmentation result.

[0015] In some embodiments of this application, the dynamic division radius is determined by accumulating energy radially from the center of the amplitude spectrum based on a preset energy percentage threshold and the total energy of the amplitude spectrum. Specifically, this includes:

[0016] Get radius The energy accumulated radially from the center of the amplitude spectrum within the circle ;

[0017] The conditions will be met. The radius is determined as the dynamic partitioning radius;

[0018] in, The total energy of the amplitude spectrum; The energy percentage threshold is a known value.

[0019] In some embodiments of this application, the low-frequency component and the high-frequency component are modulated respectively to obtain modulated low-frequency component and modulated high-frequency component, specifically including:

[0020] Calculate the low-frequency similarity map between the low-frequency component and the adaptive low-frequency prototype, and perform linear projection and residual connection processing on the low-frequency similarity map and the low-frequency component to obtain the modulated low-frequency component; the adaptive low-frequency prototype includes a low-frequency weight matrix, which is adaptively corrected based on the loss function using training data;

[0021] Calculate the high-frequency similarity map between the high-frequency component and the adaptive high-frequency prototype, and perform linear projection and residual connection processing on the high-frequency similarity map and the high-frequency component to obtain the modulated high-frequency component; the adaptive high-frequency prototype includes a high-frequency weight matrix, which is adaptively corrected based on the loss function using training data.

[0022] In some embodiments of this application, the preliminary segmentation features are processed to obtain semantic features, specifically including:

[0023] The initial segmentation features are processed through cascaded convolutional layers and upsampling layers to obtain semantic features with the same spatial resolution as the original input image.

[0024] In some embodiments of this application, the original input image is processed to obtain multi-scale edge features, specifically including:

[0025] The original input image is processed using an initial convolutional layer to obtain shallow features;

[0026] The shallow features are processed using at least two depth-separable convolutional layers with different receptive fields to obtain multi-scale features;

[0027] Residual edge enhancement is performed on each scale feature in the multi-scale features to obtain multiple edge enhancement features;

[0028] All the edge enhancement features are concatenated along the channel dimension and then fused using a fusion convolutional layer to obtain the multi-scale edge features.

[0029] In some embodiments of this application, the remote sensing image is preprocessed to obtain the original input image, specifically including:

[0030] The remote sensing image is cropped into image blocks of uniform size;

[0031] The pixel values ​​of the image block are normalized to obtain the original input image.

[0032] In some embodiments of this application, the method further includes:

[0033] When training the SAM encoder, SAM decoder and edge-guided refiner based on the training data, a composite and dual-supervised loss function is constructed to optimize the model parameters.

[0034] The process of constructing the composite and dual-supervised loss function includes:

[0035] The basic composite loss function is obtained by linearly combining the binary cross-entropy loss, the difference loss, and the L1 loss.

[0036] The loss between the preliminary segmentation result and the true result is calculated based on the basic composite loss function and used as the first loss; the preliminary segmentation result is obtained by performing a convolution operation on the preliminary segmentation features;

[0037] The loss between the final segmentation result and the true result is calculated based on the basic composite loss function, and this loss is used as the second loss.

[0038] The first loss and the second loss are weighted and summed to obtain the composite and dual-supervised loss function.

[0039] In some embodiments of this application, the method further includes:

[0040] When training the SAM encoder, SAM decoder, and edge-guided refiner based on the training data, the parameters of the SAM encoder backbone network and the SAM decoder are kept frozen, and the parameters of the frequency modulation adapter in the SAM encoder and the parameters of the edge-guided refiner are iteratively optimized.

[0041] To achieve the aforementioned objectives, the remote sensing semantic segmentation system based on SAM provided in this application employs the following technical solution:

[0042] A semantic segmentation system for remote sensing images based on SAM, comprising:

[0043] The raw input image acquisition unit is used to preprocess remote sensing images to obtain the raw input image;

[0044] A SAM encoder with an embedded frequency modulation adapter is used to encode the original input image as follows to obtain encoded features: The original input image is subjected to layer normalization and multi-head attention processing to output intermediate features; the intermediate features are transformed to the frequency domain using Fourier transform to obtain frequency domain features; the amplitude spectrum of the frequency domain features is obtained, and the total energy of the amplitude spectrum is calculated; a dynamic partitioning radius is determined based on a preset energy percentage threshold and the total energy of the amplitude spectrum by radially accumulating energy from the center of the amplitude spectrum; the frequency domain features are separated into low-frequency and high-frequency components according to the dynamic partitioning radius; the low-frequency and high-frequency components are modulated respectively to obtain modulated low-frequency and modulated high-frequency components; the modulated low-frequency and high-frequency components are merged and transformed to the spatial domain using inverse Fourier transform to obtain enhanced features; the enhanced features are concatenated with the residual of the original input image to obtain the encoded features.

[0045] The SAM decoder is used to decode the encoded features to obtain preliminary segmentation features;

[0046] The edge-guided refiner unit is used to process the preliminary segmentation features as follows to obtain the final segmentation result: process the preliminary segmentation features to obtain semantic features; process the original input image to obtain multi-scale edge features; concatenate the semantic features and the multi-scale edge features, and process them through a multilayer perceptron to obtain the final segmentation result.

[0047] Another object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described remote sensing image semantic segmentation method.

[0048] Compared with the prior art, the advantages and positive effects of the present invention are:

[0049] The present invention provides a SAM-based remote sensing image semantic segmentation method and system. By embedding a frequency modulation adapter in the SAM encoder, the frequency modulation adapter maps features to the frequency domain through Fourier transform and dynamically and adaptively determines the division boundaries of high and low frequency components based on the energy distribution of the frequency domain features. Then, the divided low-frequency and high-frequency components are modulated separately to obtain enhanced encoded features. This encoding mechanism significantly enhances the flexibility and adaptability of feature extraction to the diversity of remote sensing image content. Compared with fixed feature extraction strategies, it can more accurately capture and enhance frequency information that is crucial to different land cover types, thereby improving segmentation accuracy. By introducing an edge-guided refiner after the decoder, in addition to obtaining semantic features from the decoder output, it also directly extracts high-fidelity multi-scale edge information from the original input image, realizing the effective fusion of precise spatial details and deep semantic information. This can significantly sharpen object contours, correct information loss caused by the encoding-decoding process, and make the boundaries of the final segmentation result clearer and the geometry more complete. This invention achieves high-precision semantic segmentation of remote sensing images through the collaboration of a frequency modulation adapter and an edge-guided refiner, thereby comprehensively improving the accuracy and reliability of semantic segmentation.

[0050] Other features and advantages of the present invention will become clearer after reading the detailed embodiments of the invention in conjunction with the accompanying drawings. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 Flowcharts of some embodiments of the SAM-based remote sensing image semantic segmentation method provided by the present invention;

[0053] Figure 2 Structural block diagrams of some embodiments of the SAM-based remote sensing image semantic segmentation system provided by the present invention;

[0054] Figure 3 This is a comparison chart showing the segmentation results of the first dataset using various methods, including the SAM-based remote sensing image semantic segmentation method provided in this invention.

[0055] Figure 4 This is a comparison chart showing the segmentation results of using various methods, including the SAM-based remote sensing image semantic segmentation method provided in this invention, to segment the second dataset.

[0056] Figure 5 This is a comparison chart showing the segmentation results of using various methods, including the SAM-based remote sensing image semantic segmentation method provided in this invention, to segment the third dataset. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0058] It should be noted that the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they can be implemented by those skilled in the art. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0059] To address the technical problems of insufficient feature adaptability and blurred object boundaries in existing technologies using the SAM model for semantic segmentation of remote sensing images, this invention creatively proposes a SAM-based semantic segmentation method for remote sensing images. This method employs frequency adaptation and edge guidance to improve the accuracy and performance of semantic segmentation of remote sensing images.

[0060] Figure 1 The diagram shows flowcharts of some embodiments of the SAM-based semantic segmentation method for remote sensing images proposed in this invention.

[0061] like Figure 1 As shown, this embodiment uses the following process to achieve semantic segmentation of remote sensing images.

[0062] S11: Preprocess the remote sensing image to obtain the original input image.

[0063] The specific implementation of this process can employ existing technologies, and this embodiment does not limit it in this regard.

[0064] In some embodiments, preprocessing the remote sensing image to obtain the original input image specifically includes: cropping the remote sensing image into image blocks of uniform size; and normalizing the pixel values ​​of the image blocks to obtain the original input image.

[0065] In other embodiments, the remote sensing image is preprocessed to obtain the original input image, specifically including: cropping the remote sensing image into image blocks of uniform size; performing data augmentation on the image blocks; and normalizing the pixel values ​​of the augmented image blocks to obtain the original input image.

[0066] S12: The original input image is encoded using a SAM encoder with an embedded frequency modulation adapter to obtain the encoded features.

[0067] The SAM encoder with embedded frequency modulation adapter uses the SAM image encoder as the backbone network and the frequency modulation adapter as a lightweight module, embedded in the image encoder's transformer as an adapter. The SAM encoder with embedded frequency modulation adapter has its parameters pre-trained and optimized using a training dataset. The purpose of embedding the frequency modulation adapter in the SAM encoder is to address the problem of insufficient adaptability of traditional SAM encoders to remote sensing image features in existing technologies.

[0068] The SAM encoder with an embedded frequency modulation adapter encodes the original input image to obtain encoded features. The specific process includes:

[0069] The original input image is processed by layer normalization and multi-head attention mechanism to output intermediate features.

[0070] First, the intermediate features are transformed to the frequency domain using Fourier transform to obtain the frequency domain features; then, the amplitude spectrum of the frequency domain features is obtained, and the total energy of the amplitude spectrum is calculated. The amplitude spectrum is denoted as... Total energy of the amplitude spectrum The calculation formula is: .in, and These are the height and width of the amplitude spectrum, respectively. These are the coordinates of a point in the frequency domain.

[0071] Then, based on the preset energy percentage threshold and the total energy of the amplitude spectrum, the dynamic partition radius is determined by accumulating energy radially from the center of the amplitude spectrum; and based on the dynamic partition radius, the frequency domain features are separated into low-frequency components and high-frequency components.

[0072] In some embodiments, the radius is obtained as The energy accumulated radially from the center of the amplitude spectrum within the circle The conditions will be met. The radius is determined as the dynamic partition radius. Wherein, The energy percentage threshold is a known value. The radius is... The energy accumulated radially from the center of the amplitude spectrum within the circle The following method is used for calculation: .in, This indicates the amplitude spectrum after centering at the frequency point. The amplitude at that point; , Indicates the center coordinates of the amplitude spectrum; This is an indicator function used to determine the frequency point. With the center point Is the Euclidean distance within the radius? Within the range.

[0073] After determining the dynamic segmentation radius, the frequency domain features within the circle formed by the dynamic segmentation radius are identified as low-frequency components, and the frequency domain features outside the circle are identified as high-frequency components. The dynamic segmentation radius is determined based on an energy percentage threshold and the total energy of the amplitude spectrum. This dynamic segmentation radius is used to dynamically separate low-frequency and high-frequency components, ensuring that the frequency segmentation boundary can be dynamically adjusted according to the content of each input image, thus improving the adaptability and accuracy of frequency component segmentation.

[0074] Then, the separated low-frequency and high-frequency components are modulated separately to obtain modulated low-frequency and high-frequency components. The specific modulation method is not limited and can be implemented using existing technologies.

[0075] Then, the modulated low-frequency components and modulated high-frequency components are combined in the frequency domain and transformed to the spatial domain through inverse Fourier transform to obtain the enhanced features.

[0076] Finally, the enhanced features are concatenated with the residuals of the original input image to achieve feature reconstruction, obtain the encoded features, and output them.

[0077] In other embodiments, the low-frequency component and the high-frequency component are modulated separately to obtain modulated low-frequency components and modulated high-frequency components, specifically including:

[0078] The low-frequency similarity map between the low-frequency components and the adaptive low-frequency prototype is calculated. Linear projection and residual connection processing are then performed on the low-frequency similarity map and the low-frequency components to obtain the modulated low-frequency components. The adaptive low-frequency prototype includes a low-frequency weight matrix, which is adaptively corrected using training data and a loss function.

[0079] A high-frequency similarity map between the high-frequency components and the adaptive high-frequency prototype is calculated. Linear projection and residual connection processing are then performed on the high-frequency similarity map and the high-frequency components to obtain the modulated high-frequency components. The adaptive high-frequency prototype includes a high-frequency weight matrix, which is adaptively corrected using training data and a loss function.

[0080] Using the above approach, adaptive and learnable low-frequency and high-frequency prototypes are utilized, and attention mechanisms are used to modulate the low-frequency and high-frequency components, thereby refining and enhancing the frequency components that are most informative for the current segmentation task.

[0081] Taking high-frequency components as an example, high-frequency components are defined as follows: The adaptive high-frequency prototype is Modulated high-frequency components Represented as: .in, and There are two linear projection layers; This is element-wise multiplication; express transpose; High frequency components The dimension of the feature channels; This is the normalization function.

[0082] For the low-frequency weight matrix of the adaptive low-frequency prototype and the high-frequency weight matrix of the adaptive high-frequency prototype, initial values ​​are assigned by random initialization or given values ​​during initialization. During model training, the weight matrix is ​​continuously corrected by using the training data and the model loss function through backpropagation. After training, the weight matrix converges to a fixed weight matrix, which represents a general template learned from the training data for matching the frequency features of specific ground features.

[0083] S13: The encoded features are decoded using the SAM decoder to obtain preliminary segmentation features.

[0084] The decoding method and process are implemented using existing technologies, and this embodiment will not describe them in detail.

[0085] S14: The initial segmentation features are processed using an edge-guided refiner to obtain the final segmentation result.

[0086] The edge-guided refiner fuses multi-scale edge features extracted from the original image with preliminary segmentation features to sharpen and correct object boundaries, resulting in a refined final semantic segmentation result.

[0087] Specifically, the edge-guided refiner processes the initial segmentation features to obtain semantic features. The specific implementation of this process utilizes existing technologies, and this embodiment is not limited thereto. Simultaneously, the edge-guided refiner also processes the original input image to obtain multi-scale edge features, thereby extracting and enhancing edge features. The specific implementation of this process can also employ existing technologies, and this embodiment is not limited thereto. Finally, the edge-guided refiner concatenates the semantic features with the multi-scale edge features and processes them through a multilayer perceptron to utilize precise edge feature information to correct and sharpen relatively blurred semantic information, obtaining a pixel-level final segmentation result.

[0088] In some embodiments, the edge-guided refiner processes the initial segmentation features to obtain semantic features. Specifically, this includes processing the initial segmentation features through cascaded convolutional layers and upsampling layers to obtain semantic features with the same spatial resolution as the original input image. These semantic features preserve rich semantic information but have relatively blurred boundaries.

[0089] In other embodiments, the initial segmentation features are: The semantic features obtained by applying upsampling and convolution operations are: : .in, The kernel size is convolutional layers, This is an upsampling operation.

[0090] In other embodiments, the original input image is processed to obtain multi-scale edge features. Specifically, this includes: processing the original input image using an initial convolutional layer to obtain shallow features; then processing the shallow features using at least two depthwise separable convolutional layers with different receptive fields to obtain multi-scale features; then performing residual edge enhancement on each scale feature in the multi-scale features to highlight the edges, obtaining multiple edge-enhanced features; finally, concatenating all edge-enhanced features along the channel dimension and then fusing them through a fusion convolutional layer to obtain the multi-scale edge features.

[0091] In some other embodiments, each scale feature in the multi-scale features is represented as Residual edge enhancement is performed on each scale feature, and the corresponding edge enhancement feature is represented as follows: : .in, The index representing the scale feature. For average pooling operation, The kernel size is The convolutional layer.

[0092] The method described in the above embodiments for semantic segmentation of remote sensing images significantly enhances the flexibility and adaptability of feature extraction to the diversity of remote sensing image content by embedding a frequency modulation adapter in the SAM encoder and mapping features to the frequency domain through Fourier transform. It dynamically and adaptively determines the division boundaries of high- and low-frequency components based on the energy distribution of the frequency domain features, thereby improving segmentation accuracy. Compared to fixed feature extraction strategies, it can more accurately capture and enhance frequency information crucial to different land cover types, thus improving segmentation accuracy. By introducing an edge-guided refiner after the decoder, in addition to obtaining semantic features from the decoder output, it directly extracts high-fidelity multi-scale edge information from the original input image. Compared to traditional methods that rely solely on deep, low-resolution features for upsampling and boundary recovery, this method effectively fuses precise spatial details with deep semantic information, significantly sharpening object contours and correcting information loss caused by the encoding-decoding process, resulting in clearer boundaries and more complete geometric shapes in the final segmentation result. This invention achieves high-precision semantic segmentation of remote sensing images through the synergy of the frequency modulation adapter and the edge-guided refiner, comprehensively improving the accuracy and reliability of semantic segmentation. The method in this embodiment is applicable to various high-resolution remote sensing images, such as land cover mapping, refined urban management, and natural resource monitoring. It can effectively meet the needs of automated interpretation of complex surface information and provides important technical support for the application of large-scale visual basic models in the field of remote sensing.

[0093] The remote sensing image semantic segmentation model, consisting of a SAM encoder with an embedded frequency modulation adapter, a SAM decoder, and an edge-guided refiner, needs to be trained in advance using a training set.

[0094] In some embodiments, when training the model using training data, a parameter fine-tuning training method is adopted, aiming to adapt the trained model to the remote sensing image semantic segmentation task with minimal computational cost and memory overhead. Specifically, during training, the parameters of the SAM encoder backbone network and SAM decoder are kept frozen, while the parameters of the frequency modulation adapter and edge-guided refiner in the SAM encoder are iteratively optimized. The parameters of the SAM encoder backbone network and SAM decoder are kept frozen and do not participate in gradient updates; specifically, the core weight matrices of the image encoder (ViT) and mask decoder are frozen. Simultaneously, only the newly added lightweight modules introduced in this invention are trained. Specifically, the trainable parameters include all parameters of the frequency modulation adapter and all parameters of the edge-guided refiner, as well as the parameters of specific layer normalization layers in the SAM backbone network that are unfrozen to promote model adaptation. This parameter fine-tuning training method significantly reduces the number of parameters that need to be trained, effectively preventing overfitting of large models on relatively limited remote sensing datasets, and fully preserving the powerful general feature extraction capabilities of the pre-trained model. This design retains the powerful general feature representation capabilities of large-scale visual foundation models while significantly reducing the training costs and computational resources required to adapt models for remote sensing tasks. This approach greatly improves performance while avoiding the overfitting risks associated with full fine-tuning, providing a more efficient and accurate solution for applying large-scale foundation models in specific domains.

[0095] In some other embodiments, when training the model using training data, a composite and dual-supervised loss function is constructed to optimize the model parameters in order to effectively guide the model's learning. The process of constructing the composite and dual-supervised loss function includes:

[0096] First, the binary cross-entropy loss, dissimilarity loss, and L1 loss are linearly combined to obtain a basic composite loss function, which is a linear combination of the losses of the three different objectives. The corresponding function expression is: .in, Binary cross-entropy loss The weighting coefficients, For difference loss The weighting coefficients, L1 loss The weighting coefficients, The basic composite loss function is binary cross-entropy loss. This measure, at the pixel level, discrepancy loss, differentiates the model's predicted probability distribution from the true label, guiding the model to learn the overall pixel distribution. The model is optimized by calculating the overlap between the predicted segmented region and the real region, which is robust to handling the common foreground-background class imbalance problem in remote sensing images. L1 loss Optimizing the model by penalizing the absolute difference between the predicted and the true values ​​helps generate clearer and sharper object boundaries.

[0097] Then, a composite and dual-supervised loss function is constructed based on the basic composite loss function to supervise the initial segmentation results of the decoder and the final segmentation results of the edge-guided refiner. This provides more direct gradient signals to the shallower layers of the model, thereby achieving stable training. Specifically, the loss between the initial segmentation result and the true result is calculated based on the basic composite loss function, serving as the first loss; the initial segmentation result is obtained by performing a convolution operation on the initial segmentation features. The loss between the final segmentation result and the true result is also calculated based on the basic composite loss function, serving as the second loss. Finally, the first and second losses are weighted and summed to obtain the composite and dual-supervised loss function.

[0098] In some embodiments, based on the basic composite loss function The final composite and dual-supervised loss function obtained using the above dual-supervision method is denoted as: Its expression is: .in, For the final segmentation result, For the true result, This is the preliminary segmentation result. , which is a weighting coefficient used to balance the supervision intensity between the final segmentation result and the initial segmentation result; To be based on the basic composite loss function The loss between the calculated final segmentation result and the actual result, also known as the second loss; To be based on the basic composite loss function The loss between the calculated preliminary segmentation result and the actual result is also known as the first loss.

[0099] In other embodiments, when training the model using training data, the AdamW optimizer is employed to achieve stable and efficient optimization of the trainable parameters. The parameters of the AdamW optimizer are carefully selected to meet the specific needs of efficient fine-tuning. Specifically, since only lightweight modules such as the frequency modulation adapter and edge-guided refiner need to be trained, the core of the optimization strategy is to enable these new modules to quickly adapt to the feature distribution of the remote sensing data while preventing overfitting on limited data. To this end, an initial learning rate is set and combined with a learning rate scheduler (such as a cosine annealing scheduler) to achieve an optimization path of rapid exploration in the early stages of training and fine-tuning in the later stages. More importantly, to effectively control the parameter complexity of the frequency modulation adapter and edge-guided refiner, which are trained from scratch, a significant weight decay term is set in the optimizer. This term, by applying decoupled L2 regularization penalties to the trainable parameters in the model, is a key technical means to ensure the model's generalization ability and prevent overfitting to the training data. Furthermore, by introducing momentum by setting the exponential decay rates of the first and second moments of the optimizer, the optimizer can more smoothly traverse complex loss terrains and accelerate convergence when training new modules; and a numerical stability parameter is set to ensure the numerical stability of the training process.

[0100] In some other embodiments, when training the model using training data, in each training iteration, a forward propagation process is first performed: a training sample image patch is input into the model, and then sequentially passed through a SAM encoder and a SAM decoder with an embedded frequency modulation adapter to generate a preliminary segmentation result. Then, the final segmentation result is generated through the edge-guided refiner. Then, loss calculation is performed: based on the composite and dual-supervised loss functions. Calculate separately and Compared with the actual results The loss is calculated by weighting and summing the values ​​to obtain the total loss. Finally, backpropagation and parameter updates are performed: the gradient of the total loss with respect to all trainable parameters is calculated using the backpropagation algorithm, and the AdamW optimizer updates the trainable parameters based on the gradient.

[0101] In some other embodiments, to enhance the intuitiveness and readability of the segmentation results, the obtained segmentation results are also mapped and visualized. The final segmentation result output by the edge-guided refiner is a single-channel label map, where the value of each pixel is an integer index of the corresponding category (e.g., 0 represents background, 1 represents building, etc.). To facilitate intuitive qualitative analysis and performance evaluation, this label map is visualized. Specifically, this is achieved through a predefined color mapping table, where a unique and easily distinguishable RGB color value is pre-assigned to each category index. By replacing the category index with the corresponding RGB color value pixel by pixel, an intuitive, colored semantic segmentation result map is finally generated. Furthermore, performance metrics can be set and validation detection results can be used to verify the performance of the trained semantic segmentation model. Performance metrics include, but are not limited to, mean intersection-over-union ratio, overall accuracy, and F1 score. Among them, the average intersection-union ratio (AUC) is the average ratio of the intersection and union between the predicted segmentation result and the actual result, and is the core indicator for evaluating semantic segmentation performance; overall precision measures the proportion of all pixels that are correctly classified; and the F1 score is the harmonic mean of precision and recall, which can comprehensively reflect the model's precision and recall capabilities.

[0102] Figure 2 The diagram shown is a structural block diagram of some embodiments of the SAM-based remote sensing image semantic segmentation system provided by the present invention.

[0103] like Figure 2 As shown, the segmentation system of this embodiment includes structural units, the functions of the structural units, and the relationships between them, as detailed below:

[0104] The segmentation system includes:

[0105] The original input image acquisition unit 21 is used to preprocess the remote sensing image to obtain the original input image.

[0106] The SAM encoder 22 with an embedded frequency modulation adapter is used to encode the original input image acquired and output by the original input image acquisition unit 21 and obtain the encoded features.

[0107] SAM decoder 23 is used to decode the encoded features output by SAM encoder 22 with embedded frequency modulation adapter to obtain preliminary segmentation features.

[0108] The edge-guided refiner unit 24 is used to process the preliminary segmentation features output by the SAM decoder 23 and the original input image output by the original input image acquisition unit 21 to obtain the final segmentation result.

[0109] The segmentation system described above runs the corresponding software program and performs the corresponding functions, according to... Figure 1The process of remote sensing image semantic segmentation method embodiments and other embodiments performs semantic segmentation of remote sensing images to achieve the same result as... Figure 1 The corresponding technical effects of the embodiments and other embodiments.

[0110] Other embodiments of the present invention also provide a computer storage medium on which a computer program is stored, and when the computer program is executed by a processor, it implements... Figure 1 The embodiments and other embodiments provide a remote sensing image semantic segmentation method, and achieve the technical effects of the corresponding embodiments.

[0111] The aforementioned computer storage media can be implemented using any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The computer storage media can be any available storage medium accessible to general-purpose or special-purpose computers.

[0112] In some embodiments, a computer storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in application-specific integrated circuits (ASICs). Of course, the processor and storage medium can also exist as discrete components in the device.

[0113] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0114] The technical effects of the remote sensing image semantic segmentation method proposed in this invention will be further explained below with reference to simulation experiments.

[0115] Three datasets from the relevant datasets were used for simulation. Specifically, aerial imagery data was used as the first dataset, which includes multiple high-resolution aerial images, each containing three multispectral bands (near-infrared, red, and green) and a digital surface model. This first dataset covers five foreground cover categories. Aerial imagery data was used as the second dataset, which also includes multiple high-resolution aerial images, each containing four multispectral bands (near-infrared, red, green, and blue) and a digital surface model. This second dataset also covers five foreground cover categories. The third dataset contains multiple high-resolution remote sensing images, covering seven land cover types, including buildings, roads, water bodies, and woodlands. The scenes include both urban and rural areas, exhibiting high complexity and diversity.

[0116] Semantic segmentation of the three datasets was performed using the remote sensing image semantic segmentation method described in the above embodiments (hereinafter referred to as the "method of this invention"). In addition, semantic segmentation of the three datasets was also performed using various existing methods, including: FCN8s, U-Net, DeepLabv3+, MA-Net, A2-FPN, UNetFormer, CMTFNet, SFFNet, SparseFormer, SAM, SAM-Adapter, SAMed, RSPrompter, and Semantic-SAM.

[0117] Figure 3 The diagram shows a comparison of the segmentation visualizations of the first dataset, including remote sensing images, ground truth (GT) segmentation results, and segmentation visualizations of the first dataset using the aforementioned method from the prior art and the method of the present invention. In the segmentation results, white represents impermeable surfaces, dark blue represents buildings, light blue represents low vegetation, green represents trees, yellow represents cars, and red represents cluttered backgrounds.

[0118] from Figure 3 As can be seen, the segmentation map generated by the method of this invention closely matches the actual segmentation result (GT). Moreover, when processing buildings with clear geometric outlines in the first dataset, compared with the frequent boundary blurring and corner rounding problems of other SAM-based methods (such as SAM-Adapter, SAMed, etc.), the method of this invention demonstrates superior boundary sharpening capabilities and can generate segmentation results that are highly consistent with the outlines of real ground features. This fully demonstrates the effectiveness of the edge-guided refiner in the method of this invention in restoring spatial details.

[0119] Table 1 below presents a quantitative comparison of the method of this invention with other methods on various key performance indicators on the first dataset.

[0120] Table 1 Performance metrics of different methods on the first dataset (unit: %)

[0121]

[0122] As shown in Table 1, the method of this invention achieves optimal performance in three key metrics: overall accuracy (OA), mean intersection-over-union ratio (mIoU), and mean F1 score (mF1) for the first dataset. Specifically, the mIoU of the method of this invention reaches 85.24%, significantly higher than all other methods. Especially in the categories of "construction" and "automobiles," which have extremely high requirements for boundary accuracy, the method of this invention achieves the best mF1 score, which further verifies the superiority of the edge-guided refiner in the method of this invention.

[0123] Figure 4 The diagram shows a comparison of the segmentation visualizations of the second dataset, including remote sensing images, ground truth (GT) segmentation results, and segmentation of the second dataset using the aforementioned methods from the prior art and the method of this invention. In the segmentation results, white represents impermeable surfaces, dark blue represents buildings, light blue represents low vegetation, green represents trees, yellow represents cars, and red represents cluttered backgrounds.

[0124] from Figure 4 As can be seen, the segmentation map generated by the method of this invention closely matches the actual segmentation result (GT). Furthermore, when dealing with regions in the second dataset where spectral features are easily confused (such as low vegetation and impermeable surfaces), the method of this invention achieves more accurate semantic distinction, effectively reducing the probability of misclassification and omission. This is due to the frequency modulation adapter in the method of this invention, which can adaptively enhance features according to image content, thereby improving the model's ability to identify complex land cover types.

[0125] Table 2 below presents a quantitative comparison of the method of this invention with other methods on various key performance indicators on the second dataset.

[0126] Table 2 Performance metrics of different methods on the second dataset (unit: %)

[0127]

[0128] As shown in Table 2, the method of this invention achieved optimal performance in three key metrics—overall accuracy (OA), mean intersection-over-union ratio (mIoU), and mean F1 score (mF1)—on the second dataset. Specifically, the method of this invention also performed excellently on the two easily confused categories, "low vegetation" and "trees," demonstrating the outstanding ability of the frequency modulation adapter to resolve semantic ambiguity, and ultimately achieving the best overall performance with an mIoU of 86.73%.

[0129] Figure 5 The diagram shows a comparison of the segmentation visualizations of a third dataset, including remote sensing images, ground truth (GT) segmentation results, and segmentation visualizations of the third dataset using the aforementioned methods from existing technologies and the method of this invention. In the segmentation results, white represents the background, red represents buildings, yellow represents roads, dark blue represents water, purple represents barren land, green represents forests, and orange represents agriculture.

[0130] from Figure 5 As can be seen, the segmentation map generated by the method of this invention is highly consistent with the actual segmentation result (GT). Moreover, when applied to the complex scene of the urban-rural fringe in the third dataset, the method of this invention can simultaneously maintain the continuity of large-scale structures such as roads and accurately preserve the independent outlines of small-scale targets such as buildings, significantly improving the clarity and accuracy of the overall results, and providing a higher quality data foundation and decision-making basis for the analysis and application of this dataset.

[0131] Table 3 below presents a quantitative comparison of the method of this invention with other methods on various key performance indicators on the third dataset.

[0132] Table 3 Performance metrics of different methods on the third dataset (unit: %)

[0133]

[0134] As shown in Table 3, on the third dataset with more diverse scenes and categories, the method of this invention achieved leading mIoU in multiple categories such as "buildings," "roads," and "water," surpassing all comparative methods with an mIoU of 55.66%. This fully demonstrates the robustness and excellent generalization ability of the method of this invention in complex and variable scenes. These comprehensive quantitative results further highlight the advantages and value of the method of this invention in complex remote sensing image segmentation tasks.

[0135] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions claimed by the present invention.

Claims

1. A semantic segmentation method for remote sensing images based on SAM, characterized in that, The method includes: The remote sensing images are preprocessed to obtain the original input images; The original input image is encoded using a SAM encoder with an embedded frequency modulation adapter to obtain encoded features as follows: Layer normalization and multi-head attention are applied to the original input image to output intermediate features; Fourier transform is used to transform the intermediate features to the frequency domain to obtain frequency domain features; the amplitude spectrum of the frequency domain features is obtained, and the total energy of the amplitude spectrum is calculated; a dynamic partitioning radius is determined based on a preset energy percentage threshold and the total energy of the amplitude spectrum by radially accumulating energy from the center of the amplitude spectrum; the frequency domain features are separated into low-frequency and high-frequency components based on the dynamic partitioning radius; the low-frequency and high-frequency components are modulated respectively to obtain modulated low-frequency and modulated high-frequency components; the modulated low-frequency and high-frequency components are merged and transformed to the spatial domain using inverse Fourier transform to obtain enhanced features; the enhanced features are concatenated with the residual of the original input image to obtain the encoded features. The encoded features are decoded using a SAM decoder to obtain preliminary segmentation features; The preliminary segmentation features are processed using an edge-guided refiner to obtain the final segmentation result: the preliminary segmentation features are processed to obtain semantic features; the original input image is processed to obtain multi-scale edge features; the semantic features and the multi-scale edge features are concatenated and processed by a multilayer perceptron to obtain the final segmentation result.

2. The SAM-based semantic segmentation method for remote sensing images according to claim 1, characterized in that, Based on a preset energy percentage threshold and the total energy of the amplitude spectrum, the dynamic division radius is determined by accumulating energy radially from the center of the amplitude spectrum, specifically including: Get radius The energy accumulated radially from the center of the amplitude spectrum within the circle ; The conditions will be met. The radius is determined as the dynamic partition radius; in, The total energy of the amplitude spectrum; The energy percentage threshold is a known value.

3. The remote sensing image semantic segmentation method according to claim 1, characterized in that, Modulating the low-frequency component and the high-frequency component respectively to obtain modulated low-frequency component and modulated high-frequency component, specifically including: Calculate the low-frequency similarity map between the low-frequency component and the adaptive low-frequency prototype, and perform linear projection and residual connection processing on the low-frequency similarity map and the low-frequency component to obtain the modulated low-frequency component; the adaptive low-frequency prototype includes a low-frequency weight matrix, which is adaptively corrected based on the loss function using training data; Calculate the high-frequency similarity map between the high-frequency component and the adaptive high-frequency prototype, and perform linear projection and residual connection processing on the high-frequency similarity map and the high-frequency component to obtain the modulated high-frequency component; the adaptive high-frequency prototype includes a high-frequency weight matrix, which is adaptively corrected based on the loss function using training data.

4. The SAM-based semantic segmentation method for remote sensing images according to claim 1, characterized in that, The initial segmentation features are processed to obtain semantic features, specifically including: The initial segmentation features are processed through cascaded convolutional layers and upsampling layers to obtain semantic features with the same spatial resolution as the original input image.

5. The SAM-based semantic segmentation method for remote sensing images according to claim 1, characterized in that, The original input image is processed to obtain multi-scale edge features, specifically including: The original input image is processed using an initial convolutional layer to obtain shallow features; The shallow features are processed using at least two depth-separable convolutional layers with different receptive fields to obtain multi-scale features; Residual edge enhancement is performed on each scale feature in the multi-scale features to obtain multiple edge enhancement features; All the edge enhancement features are concatenated along the channel dimension and then fused using a fusion convolutional layer to obtain the multi-scale edge features.

6. The remote sensing image semantic segmentation method based on SAM according to claim 1, characterized in that, Preprocessing remote sensing images to obtain the original input images specifically includes: The remote sensing image is cropped into image blocks of uniform size; The pixel values ​​of the image block are normalized to obtain the original input image.

7. The SAM-based semantic segmentation method for remote sensing images according to any one of claims 1 to 6, characterized in that, The method further includes: When training the SAM encoder, SAM decoder and edge-guided refiner based on the training data, a composite and dual-supervised loss function is constructed to optimize the model parameters. The process of constructing the composite and dual-supervised loss function includes: The basic composite loss function is obtained by linearly combining the binary cross-entropy loss, the difference loss, and the L1 loss. The loss between the preliminary segmentation result and the true result is calculated based on the basic composite loss function and used as the first loss; the preliminary segmentation result is obtained by performing a convolution operation on the preliminary segmentation features; The loss between the final segmentation result and the true result is calculated based on the basic composite loss function, and this loss is used as the second loss. The first loss and the second loss are weighted and summed to obtain the composite and dual-supervised loss function.

8. The SAM-based semantic segmentation method for remote sensing images according to any one of claims 1 to 6, characterized in that, The method further includes: When training the SAM encoder, SAM decoder, and edge-guided refiner based on the training data, the parameters of the SAM encoder backbone network and the SAM decoder are kept frozen, and the parameters of the frequency modulation adapter in the SAM encoder and the parameters of the edge-guided refiner are iteratively optimized.

9. A semantic segmentation system for remote sensing images based on SAM, characterized in that, The system includes: The raw input image acquisition unit is used to preprocess remote sensing images to obtain the raw input image; A SAM encoder with an embedded frequency modulation adapter is used to encode the original input image as follows to obtain encoded features: The original input image is subjected to layer normalization and multi-head attention processing to output intermediate features; the intermediate features are transformed to the frequency domain using Fourier transform to obtain frequency domain features; the amplitude spectrum of the frequency domain features is obtained, and the total energy of the amplitude spectrum is calculated; a dynamic partitioning radius is determined based on a preset energy percentage threshold and the total energy of the amplitude spectrum by radially accumulating energy from the center of the amplitude spectrum; the frequency domain features are separated into low-frequency and high-frequency components according to the dynamic partitioning radius; the low-frequency and high-frequency components are modulated respectively to obtain modulated low-frequency and modulated high-frequency components; the modulated low-frequency and high-frequency components are merged and transformed to the spatial domain using inverse Fourier transform to obtain enhanced features; the enhanced features are concatenated with the residual of the original input image to obtain the encoded features. The SAM decoder is used to decode the encoded features to obtain preliminary segmentation features; The edge-guided refiner unit is used to process the preliminary segmentation features as follows to obtain the final segmentation result: process the preliminary segmentation features to obtain semantic features; process the original input image to obtain multi-scale edge features; concatenate the semantic features and the multi-scale edge features, and process them through a multilayer perceptron to obtain the final segmentation result.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the SAM-based remote sensing image semantic segmentation method as described in any one of claims 1 to 8.

Images