A remote sensing image multi-scale feature fusion semantic segmentation method based on a SAM large model

Abstract

The application discloses a kind of remote sensing image multiscale feature fusion semantic segmentation methods based on SAM large model, comprising: remote sensing image data is input after pre-processing improved SAM image encoder, and global image embedding of global semantic and high-frequency details is generated and fused;Sparse prompt point information is generated to target area mark, and is coded into prompt embedding containing position and semantic information by SAM prompt encoder;Image embedding and prompt embedding are input into SAM mask decoder to generate preliminary segmentation mask;A small amount of parameter fine-tuning is carried out using Rein method to relieve SAM model remote sensing field deviation, and original mask decoder is replaced by MACU-Net decoder, multiscale feature fusion and target boundary recovery are realized, and high-precision high-robustness semantic segmentation result suitable for complex remote sensing scene is generated.The application solves the problems of high-frequency information loss, multiscale feature boundary blur and regional deviation in the original SAM model in remote sensing image segmentation.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and in particular to a semantic segmentation method for multi-scale feature fusion of remote sensing images based on the SAM large model. Background Technology

[0002] Semantic segmentation of remote sensing images is one of the core tasks in the field of remote sensing image processing. It aims to assign precise land cover category labels (such as buildings, roads, vegetation, and water bodies) to each pixel in an image. Compared to computer vision tasks such as object detection and image classification, semantic segmentation not only focuses on the types of land covers in an image but also requires accurate identification of land cover boundaries and category distribution. Especially in remote sensing image data, where diverse and complex land cover features are present, semantic segmentation is extremely challenging. Accurate semantic segmentation has significant engineering application value in key areas such as Geographic Information System (GIS) construction, refined urban planning, dynamic ecological environment monitoring, and natural disaster emergency response.

[0003] In recent years, with the rapid development of remote sensing technology, the acquisition capability and spatial resolution of remote sensing data have continuously improved. However, this has also brought new technical challenges: remote sensing images contain a wide variety and complexity of ground features, and are affected by factors such as topography, lighting conditions, and atmospheric interference, resulting in large differences in feature scale, blurred boundaries, and a large amount of redundant background information, posing significant challenges to semantic segmentation tasks. Traditional remote sensing image segmentation methods (such as threshold-based segmentation, region growing, and edge detection methods) mainly rely on manually designed features and can only handle remote sensing images with simple scenes and single ground features. When faced with complex ground feature scenes and their intersecting areas, these methods have limited segmentation capabilities and low segmentation accuracy. Furthermore, the labeling of ground features usually relies on manual annotation, leading to high annotation costs, low efficiency, and susceptibility to subjective factors, resulting in unstable annotation quality that fails to meet practical application needs.

[0004] With the rise of deep learning technology, models such as Convolutional Neural Networks (CNN) and Transformers have gradually become the mainstream technologies in the field of semantic segmentation of remote sensing images. Deep learning significantly improves the accuracy and efficiency of segmentation by automatically learning image features. However, traditional deep learning methods still have obvious limitations: on the one hand, model training usually requires high-quality labeled data, but manual labeling of remote sensing images is costly, time-consuming, and the labeling quality is easily affected by factors such as terrain complexity and similarity of ground features, making it difficult to obtain on a large scale; on the other hand, existing deep learning models are insufficient in capturing multi-scale ground feature characteristics of remote sensing images, and still suffer from problems such as category confusion and inaccurate boundary segmentation when faced with complex backgrounds and blurred boundaries.

[0005] The Segment Anything Model (SAM), proposed by Mata AI, is a general-purpose arbitrary segmentation model. With its powerful learning capabilities for few or even zero-shot images, it can efficiently extract image features and generate accurate segmentation results, greatly advancing the development of image segmentation technology. The core advantage of the SAM model lies in its ability to perform segmentation tasks in general scenarios without requiring a large amount of labeled data, providing a new solution for the field of remote sensing image segmentation. Through a "large model + transfer learning" approach, it is expected to address the pain points of scarce labeled data and insufficient generalization ability in remote sensing images.

[0006] However, the pre-training data for the SAM model is mainly based on natural images. Directly transferring this data to remote sensing image segmentation presents significant domain adaptation problems: First, the distribution of ground features, scale characteristics, and semantic information in remote sensing images differ greatly from those in natural images, making it difficult for the pre-trained features of the SAM model to accurately match the features of remote sensing ground features. Second, remote sensing images often contain multiple scales of ground features, and the SAM model's ability to capture features of small-scale ground features is insufficient, easily leading to missed detections and false detections. These problems make it difficult for the SAM model to meet the pixel-level segmentation accuracy requirements of practical applications when directly applied to remote sensing image segmentation tasks. Therefore, targeted optimization based on the characteristics of remote sensing scenarios is urgently needed to improve the model's adaptability and segmentation performance, enabling the SAM model to better adapt to the field of remote sensing image segmentation. Summary of the Invention

[0007] The problem this invention aims to solve is to propose a multi-scale feature fusion semantic segmentation method for remote sensing images based on the SAM large model. By adding the IFormer module to enhance high-frequency features and adapting the MACU-Net decoder for remote sensing, the accuracy of feature extraction and segmentation is improved, thus solving the problem of low segmentation accuracy of traditional methods in complex remote sensing scenarios and improving the effect of semantic segmentation of remote sensing images.

[0008] This invention adopts the following technical solution: a semantic segmentation method for multi-scale feature fusion of remote sensing images based on the SAM large model, comprising the following steps:

[0009] Step 1, Data Preprocessing and Augmentation: Acquire raw remote sensing image data and perform preprocessing, including image normalization, and simultaneously perform data augmentation using geometric transformation and color perturbation to expand the training dataset;

[0010] Step 2, Improvement of SAM Image Encoder and Generation of Image Embedding: Improve the VisionTransformer (ViT) visual self-attention mechanism in the SAM image encoder, introduce the IFormer module to reconstruct the image encoder, and enhance the model's ability to extract high-frequency information; input the preprocessed remote sensing image into the improved SAM image encoder, extract image features through the ViT visual self-attention mechanism, and generate global image embedding.

[0011] Step 3, cue information generation and encoding: Foreground and background markings are performed on the target area of ​​the remote sensing image to generate a set of sparse cue point information, which is then fed into the cue encoder of SAM and encoded into a cue embedding containing location and semantic information;

[0012] Step 4: Preliminary segmentation based on the SAM pre-trained model: The global image embedding generated in Step 2 and the cue embedding generated in Step 3 are input into the mask decoder of SAM. The image features and cue information are fused through the cross attention mechanism to perform preliminary segmentation of the remote sensing image, locate the target region and generate a preliminary semantic segmentation mask.

[0013] Step 5: Fine-tuning of minor parameters: Using the Rein method, the improved SAM model is fine-tuned as a whole to improve the accuracy of the initial segmentation mask;

[0014] Step 6, MACU-Net Decoder Optimization and Final Segmentation: Replace the original mask decoder of SAM with the MACU-Net decoder, perform multi-scale feature fusion and boundary restoration on the segmentation features fine-tuned in Step 5, and output high-precision remote sensing image semantic segmentation results.

[0015] As a preferred embodiment, the data augmentation in step 1 includes random horizontal flipping, random rotation, and random color perturbation; by calculating the sample frequency of each type of land cover, and using Gaussian noise perturbation mask features whose variance is inversely proportional to the sample frequency of each type, data augmentation is performed on land cover with fewer samples to solve the problem of unbalanced label distribution.

[0016] As a preferred embodiment, the prompt embedding in step 3 is obtained as follows:

[0017] Step 3.1: Obtain sparse cue point information, which refers to a small number of discrete cue points manually marked in the target area of ​​the remote sensing image, including foreground and background cue points (single pixel level) and target bounding box (rectangular coordinate level).

[0018] Step 3.2: Take the sparse cue point information as input and pass it into the cue encoder of SAM. Through position encoding, each cue point and target bounding box is transformed into a specific high-dimensional vector, and the sparse cue embedding is generated.

[0019] Step 3.3: Obtain dense cue information, which refers to the low-dimensional semantic feature map extracted pixel by pixel from the remote sensing image. Specifically, it is the feature tensor obtained by downsampling the preprocessed remote sensing image through a 3×3 convolutional layer.

[0020] Step 3.4: Input the sparse cue embedding and dense cue embedding into the fusion layer of the SAM cue encoder, and perform feature fusion by channel-wise weighted summation to obtain the fused cue embedding. This fused cue embedding matches the dimension of the global image embedding generated by the SAM image encoder in Step 2, laying the foundation for feature fusion in the subsequent mask decoder.

[0021] As a preferred embodiment, the SAM pre-trained segmentation model described in step 4 has three core components: an image encoder, a cue encoder, and a mask decoder.

[0022] The image encoder is based on ViT and pre-trained using the masked autoencoder (MAE) method to adaptively learn the global semantic features and local detail features of remote sensing images, thus adapting to the input requirements of high-resolution remote sensing images.

[0023] The prompt encoder is used to process prompt information input by the user or system, convert it into a standardized feature representation, and guide the segmentation model to accurately segment the target region.

[0024] The mask decoder employs a cue self-attention and cross-attention fusion mechanism to update and optimize the input image embedding features and cue embedding features. After feature processing is completed through two decoder blocks, the image embedding features are upsampled to a size that matches the preprocessed remote sensing image. A linear classifier is used to calculate the mask foreground probability at each pixel location to generate a preliminary semantic segmentation mask.

[0025] As a preferred option, the Rein method (an efficient parameter fine-tuning method for domain-specific generalized semantic segmentation) described in step 5 is as follows:

[0026] Step 5.1: Input the preprocessed remote sensing image from Step 1 into the improved SAM model, and extract features through the backbone network to obtain the initial feature map. ;

[0027] Step 5.2: Introduce a learnable token, denoted as... Each token has the following dimensions: , For the number of tokens, The number of token channels, and Consistent with the number of channels in the initial feature map; achieved through matrix multiplication. The relationship between the token and the initial feature map is calculated to obtain the correlation matrix, thereby achieving adaptive alignment of the token with the remote sensing image features.

[0028] Step 5.3: Construct a similarity map This is used to measure the similarity between each token and each region in the initial feature map; it maps the similarity. The input is processed by a multilayer perceptron (MLP) to perform a nonlinear transformation, generating feature weighting coefficients that match the dimensions of the initial feature map. Then, the initial feature map is processed using these feature weighting coefficients. Pixel-wise weighted optimization is performed to obtain the updated enhanced feature map. As the core input feature for subsequent module feature enhancement, it improves the model's segmentation accuracy for complex remote sensing scenes.

[0029] As a preferred embodiment, the IFormer module in step 2 improves the SAM image encoder by modifying the multi-head seif-attention (MSA) layer, specifically including... and The branch feature processing structure is adapted to the multi-scale feature extraction requirements of remote sensing images. The specific process is as follows:

[0030] Input features It is split into a low-frequency component along the channel dimension. and a high-frequency component Low-frequency components are used to capture global semantic information of ground features in remotely sensed images, while high-frequency components are used to extract edge and texture-related detail features of ground features; among them, For the number of feature tokens, This represents the total number of feature channels. , The number of low-frequency and high-frequency component channels are respectively and satisfy the following conditions: + = ;

[0031] The low-frequency components are processed by the original multi-head self-attention layer of SAM to obtain low-frequency features. ;

[0032] The high-frequency component is further divided into two sub-components. and ,right Perform maximum filtering and point term convolution to obtain sub-features. ,right Perform pointwise convolution and depthwise convolution to obtain sub-features. ;

[0033] Then, through the initial mixer... , and The three components are combined to obtain the fusion feature. After the final feedforward network (FFN), the initial transformer finally outputs the characteristic. .

[0034] As a preferred embodiment, the MACU-Net decoder in step 6 optimizes segmentation performance by fusing asymmetric convolution blocks (ACB), multi-scale skip connections, and channel attention blocks (CAB), addressing the challenges of multi-scale and blurred boundaries of ground features in remote sensing images. Specific processing includes:

[0035] Step 6.1, Asymmetric convolutional block feature extraction: ACB captures land cover features of different dimensions in remote sensing images through three parallel convolutional branches and multi-dimensional cross receptive fields; it improves the numerical stability and non-linear expressive ability of features through batch normalization (BN) and ReLU activation function, and outputs enhanced feature maps;

[0036] The three parallel convolution branches are as follows: convolution, Horizontal convolution and Vertical convolution captures ground feature features in different dimensions of remote sensing images through multi-dimensional cross-receptive fields; the 3×3 convolution is used to capture global semantic features of ground features, and the 1×3 horizontal convolution and 3×1 vertical convolution are used to enhance the edge features of ground features in the horizontal and vertical directions, respectively.

[0037] Step 6.2, Multi-scale skip connection feature fusion: A three-level skip connection mechanism is designed to address the characteristics of large-scale terrain and small-scale features in remote sensing images;

[0038] The three-level hop connection mechanism includes:

[0039] Features of encoders and decoders at the same level are directly connected, preserving fine-grained edge details;

[0040] Low-level encoder features are upsampled by transposed convolution and processed by ACB to transmit high-precision position information;

[0041] The high-level encoder features are downsampled by max pooling and processed by ACB to convey global semantic information.

[0042] Step 6.3, Channel Attention Block Feature Reweighting: Based on the CAB improved by the CBAM module, the channel features are reweighted by multiplying the weight vector with the fused feature map channel by channel, suppressing background noise channels and improving the segmentation feature discrimination ability;

[0043] CAB module learns one-dimensional weights Realign the input feature maps Then through the weight vector With input feature map Pixel-wise multiplication enhances the weights of the feature enrichment channels, suppresses redundant background noise channels, and outputs a channel-attention-optimized feature map; where... The number of feature map channels. For feature map height, This represents the width of the feature map.

[0044] Step 6.4 Output Results: Finally, the image is restored to its original size using the upsampling module of the MACU-Net decoder, and a semantic segmentation result with clear ground feature boundaries and accurate category differentiation is output.

[0045] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0046] 1. This invention relies on the powerful few-shot / zero-shot learning capability of the SAM large model and combines it with sparse cue point encoding design to achieve efficient segmentation without relying on massive amounts of manually labeled remote sensing image data, significantly reducing the need for labeled data.

[0047] 2. By using a Rein parameter-efficient fine-tuning method, this invention significantly reduces the need for large amounts of training data and high-performance computing equipment while ensuring segmentation results. It avoids the large amount of computing resources required for full model fine-tuning, reduces training time and cost, and can effectively adapt to different remote sensing image datasets, effectively alleviating the "neighborhood shift" problem in remote sensing images.

[0048] 3. This invention combines the IFormer and MACU-Net decoder modules to effectively perform multi-scale feature fusion, improving the model's ability to process complex features and boundaries in remote sensing images. It can capture detailed information at different scales, especially for targets with different sizes and shapes in remote sensing images. It improves the model's extraction of different scales and textures, and provides better segmentation results for key areas such as object boundaries, providing high-precision pixel-level support for subsequent remote sensing data analysis. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of the overall architecture of the network model of the present invention;

[0050] Figure 2 This is a schematic diagram of the IFormer module network model provided in an embodiment of the present invention;

[0051] Figure 3 This is an example diagram showing the semantic segmentation comparison results of the fine-tuning method used in this invention;

[0052] Figure 4 These are example diagrams showing semantic segmentation results of different models in embodiments of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the application will be further described in detail below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments involved in this invention. All non-innovative embodiments based on these embodiments by other researchers in the art are within the protection scope of this invention. Furthermore, the step numbers in the embodiments of this invention are only set for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0054] In one embodiment of the present invention, a semantic segmentation method for remote sensing images based on the SAM large model is proposed. By utilizing the powerful segmentation capability of the SAM model, the method solves the problem of low segmentation accuracy of remote sensing images and obtains higher segmentation accuracy for various types of land features.

[0055] In this embodiment, the SAM-based general segmentation model mainly serves the following functions:

[0056] First, the SAM model, based on the ability to use few or even zero samples, greatly reduces the time and manpower costs of labeling remote sensing image data, and significantly shortens the data preparation time.

[0057] Secondly, we use the arbitrary segmentation model SAM proposed in the field of computer vision. This model performs excellently in image segmentation, allowing researchers to devote more energy to improving and optimizing the network architecture of the model algorithm.

[0058] Specifically, such as Figure 1 As shown, the semantic segmentation method for remote sensing images based on the SAM large model includes the following steps:

[0059] S1. Obtain the LoveDA dataset

[0060] This includes two different scene types: urban and rural. Image enhancement of the original remote sensing images commonly employs geometric transformation methods such as random horizontal flipping, random rotation, and random color perturbation (brightness / contrast / saturation). The main purposes of data augmentation are:

[0061] 1) Increase the amount of training sample data to avoid overfitting and improve the model's generalization ability.

[0062] 2) Avoid imbalanced samples: Remote sensing images have complex ground features and uneven ground feature categories, making it easy for model training to ignore ground features with fewer categories.

[0063] Furthermore, this embodiment calculates the sample frequency of each type of land cover and uses Gaussian noise perturbation mask features whose variance is inversely proportional to the sample frequency of each type to perform data augmentation on land covers with few samples, thereby solving the problem of unbalanced label distribution.

[0064] The specific calculation formula is as follows:

[0065] ;

[0066] ;

[0067] in, It is a normalized input mask feature. These are batch size, number of channels, feature map height, and width, respectively. It is adjusted to the same size as the actual ground value. It is additional Gaussian noise. It has a fixed variance. For the first The percentage of pixels representing the class in the actual mask. , It is an adjustable hyperparameter.

[0068] Improvements to S2 and SAM image encoders and image embedding generation

[0069] The ViT visual self-attention mechanism in the SAM image encoder is improved by introducing the IFormer module to reconstruct the image encoder, thereby enhancing the model's ability to extract high-frequency information.

[0070] This embodiment proposes an improved encoder network method for the SAM model to enhance high-frequency information extraction. Given that remote sensing images, compared to natural images, exhibit complex features and significant differences in semantic and spatial scales, the initial segmentation results of the original SAM encoder are insufficient to completely segment the boundary contours of various features, easily losing crucial high-frequency information such as edges and textures. Therefore, the IFormer module is incorporated into the SAM image encoder to reconstruct the multi-head self-attention layer of ViT, improving the model's ability to extract high-frequency information.

[0071] Specifically, the steps include the following:

[0072] The multi-head self-attention layer of the original SAM image encoder is improved by introducing the Inception Transformer, which contains 3×3 and 5×5 multi-branch feature processing structures. The encoder structure is as follows: Figure 2 As shown.

[0073] Input features First, it is split into a low-frequency component along the channel dimension. and a high-frequency component Low-frequency components are used to capture global semantic information of ground features in remote sensing images, while high-frequency components are used to extract detailed features such as ground feature edges and textures.

[0074] in, For the number of feature tokens, This represents the total number of feature channels. , The number of low-frequency / high-frequency component channels are respectively and satisfy the following conditions: + = .

[0075] The low-frequency component is obtained after SAM's original multi-head self-attention layer processing. The description is as follows:

[0076] ;

[0077] in, The function represents a multi-head self-attention mechanism.

[0078] The high-frequency component is further divided into two sub-components. and ,right Perform maximum filtering (i.e.) ) and point term convolution (i.e. Processing yields sub-features ,right Perform pointwise convolution (i.e. ) and depthwise convolution (i.e. Sub-features are obtained after processing. Then the initial mixer will , and The three components are combined to obtain :

[0079] ;

[0080] ;

[0081] ;

[0082] in, The function is a pointwise convolution function, which is essentially... Convolution operations are used to adjust the channel dimensions of feature maps, enabling cross-channel fusion of feature information while reducing subsequent computational complexity. The function is a maximum filtering function, used for high-frequency component quantum features. Local maximum extraction is performed to preserve the peak features of high-frequency details such as the edges and textures of remotely sensed images, thereby enhancing the distinction between the target area and the background. The function is a deep convolution function. It performs spatial convolution on each input channel individually, and then fuses the channel features through pointwise convolution. This accurately captures the spatial distribution features of ground objects in remote sensing images while reducing computational consumption, improving the efficiency and relevance of feature extraction. The Fusion(·) function implemented by the deep convolutional layer refers to the step of further merging all features, and Concat(·) is the concatenation operator. This is a feature of fusion.

[0083] Final output characteristics of the initial transformer It can be represented as:

[0084] ;

[0085] in, The function represents the processing of a multilayer perceptron.

[0086] S3. Generation and encoding of prompt messages

[0087] The SAM model transforms sparse cue point information into cue embedding features that the model can process through a cue encoder. The cue information is input in the form of location encoding, which includes foreground cue points, background cue points, and target bounding boxes. The location encoding includes the pixel coordinates of each cue point / box and the corresponding land feature category information.

[0088] Sparse cue information is a discrete cue (point, rectangular bounding box) for marking target areas in remote sensing images. The points and boxes are represented by position encoding. Each cue point (or box) is converted into a high-dimensional vector that matches the dimension of the embedding channel of the SAM image through position encoding. Then, the vector is passed into the cue encoder of SAM to generate sparse cue embedding.

[0089] The dense cue information is a fine-grained semantic feature extracted from the preprocessed remote sensing image, which is encoded by convolutional embedding: low-dimensional features are obtained by 3×3 convolution downsampling operation, and dimensionality adaptation is completed by 1×1 convolution.

[0090] Subsequently, in the cue encoder fusion layer, sparse and dense cue information are fused by channel-wise weighted summation to obtain the fused cue embedding, which supplements the detailed features of remote sensing image features for subsequent segmentation.

[0091] S4 and SAM models are initially segmented.

[0092] The image generated by the SAM image encoder and the fusion cue generated by S3 are embedded and input into the core segmentation component of SAM. The remote sensing image is initially segmented by the fusion of the ViT visual self-attention mechanism and the cross-attention mechanism, and an initial semantic segmentation mask is generated.

[0093] The image encoder is based on the Vision Transformer (ViT) architecture and is pre-trained using the Masked Autoencoder (MAE) method. It can adaptively learn the global semantic features and local detail features of remote sensing images, and adapt to the input requirements of high-resolution remote sensing images.

[0094] The prompt encoder is used to process prompt information input by the user or system, convert it into a standardized feature representation, and guide the segmentation model to accurately segment the target region.

[0095] The mask decoder employs a cue self-attention and cross-attention fusion mechanism to update and optimize the input image embedding features and cue embedding features. After feature processing is completed through two decoder blocks, the image embedding features are upsampled to a size that matches the preprocessed remote sensing image. Finally, a linear classifier is used to calculate the mask foreground probability at each pixel location to generate a preliminary semantic segmentation mask.

[0096] S5, minor parameter adjustments

[0097] Fine-tuning methods with a small number of parameters include LoRA, Adapter, Mona, and Rein.

[0098] LoRA, a low-rank adaptation matrix fine-tuning method, introduces a low-rank adaptation matrix into the weights of the original model. During the fine-tuning phase, only the parameters of the low-rank matrix are trained, thereby achieving fine-tuning of the model. Specifically, it introduces a low-rank adaptation matrix into the self-attention layer of the Transformer encoder.

[0099] The Adapter, as described, is a method for fine-tuning an adapter by inserting a small feedforward neural network as an adapter into a pre-trained model. Only a small number of parameters of the adapter are trained to achieve adaptation for a specific task. Specifically, it involves inserting a small feedforward network as an adapter into the Transformer layer.

[0100] Mona is a multi-cognitive visual adapter fine-tuning method that introduces multi-cognitive visual filters and optimizes the input feature distribution, adjusting only 5% of the parameters of the model backbone network, and enhancing the model's generalization ability through shared weights and mutual adaptation between tasks.

[0101] Rein is a parameter-efficient fine-tuning method designed specifically for domain-specific semantic segmentation tasks. It improves the model's adaptability to unknown domains by freezing the pre-trained parameters of the visual base model and introducing learnable low-rank token sequences and dynamic linking mechanisms. At the same time, it reduces parameter redundancy by sharing MLP weights.

[0102] As a preferred option, this embodiment selects to use the Rein small parameter fine-tuning method for fine-tuning.

[0103] First, the global image embedding feature map generated in the initial segmentation stage of the SAM model is used as input and fed into the pre-trained backbone network of the SAM model. The image undergoes preliminary processing through the backbone network of the pre-trained model to obtain the initial feature map. To further enhance the model's generalization ability, learnable tokens are introduced, each token having a dimension of [missing information]. , For the number of tokens, The number of token channels, and With the initial feature map The number of channels is the same.

[0104] Secondly, through matrix multiplication The relationship between the token and the initial feature map is calculated to obtain the correlation matrix, enabling adaptive alignment of the token with remote sensing image features and enhancing feature focus on target land cover regions. A similarity mapping is then constructed based on this correlation matrix. This mapping quantifies the similarity between each learnable token and each pixel region of the initial feature map, assigning higher attention weights to target land cover regions in remote sensing images to measure the similarity between each token and image features.

[0105] Finally, the obtained similarity map is input into a multilayer perceptron (MLP) for nonlinear feature transformation, generating feature weighting coefficients that match the dimensions of the initial feature map. The initial feature map is then processed using these feature weighting coefficients. Pixel-wise weighted optimization is performed to obtain the updated enhanced feature map. As the core input feature for subsequent module feature enhancement, it improves the model's segmentation accuracy for complex remote sensing scenes.

[0106] S6 and MACU-Net decoder optimization

[0107] The SAM model decoder is improved by using the MACU-Net segmentation head for feature fusion and restoration, achieving high-precision target boundary restoration.

[0108] Specifically, based on the results of the improved SAM encoder described above, a multi-scale skip connection and asymmetric convolution-based network, MACU-Net, is designed, with the following structure: Figure 4 As shown, remote sensing element segmentation is performed using the following method:

[0109] S6.1 Feature Extraction from Asymmetric Convolutional Blocks (ACB)

[0110] In this embodiment, the ACB module contains three parallel convolutional branches, namely... convolution, Horizontal convolution and Vertical convolution captures ground feature features in different dimensions of remote sensing images through multi-dimensional cross-receptive fields. Among them, 3×3 convolution captures global semantic features of ground features through a relatively large receptive field, while 1×3 horizontal convolution and 3×1 vertical convolution enhance the edge features of ground features in the horizontal and vertical directions, respectively.

[0111] Subsequently, batch normalization (BN) and the ReLU activation function are used to improve the numerical stability and nonlinear expressive power of the features, and the enhanced feature map is output. The calculation formula is as follows:

[0112] ;

[0113] ;

[0114] in, and These are the inputs and outputs of the asymmetric convolutional block (ACB). This indicates that the asymmetric convolutional block (ACB) is applied to the input features. The enhanced feature map is obtained after processing with multi-branch convolutional fusion, batch normalization (BN), and ReLU activation function. , , They represent Convolution operation Horizontal convolution operations and Vertical convolution operation. and Let these represent the variance function and expected value of the input, respectively. These are tiny constants used to maintain numerical stability. and These are two trainable parameters of the BN layer, and the normalized result is obtained through... Scaling, via Translation.

[0115] S6.2, Multi-scale jump connection

[0116] Given that traditional U-Net conventional skip connections cannot fully utilize the multi-scale features of remote sensing images, this embodiment designs a multi-scale skip connection mechanism to capture cross-level feature interactions between the encoder and decoder, while extracting fine-grained edge detail information and coarse-grained global semantic information.

[0117] 1) Direct stitching of features between encoders and decoders at the same level preserves fine-grained edge details of ground features in remote sensing images.

[0118] 2) The low-level encoder features are processed by transposed convolution upsampling + ACB module and then passed to the decoder to enhance high-precision position information.

[0119] 3) The features of the high-level encoder are processed by max pooling downsampling + ACB module and then passed to the decoder to enhance global semantic information.

[0120] It can be described as:

[0121] ;

[0122] in, The channel attention features represent the realigned channels. Represents an asymmetric convolution block. and These represent downsampling using a max pooling layer and upsampling using a transposed convolution, respectively. This indicates a splicing operation. This refers to the encoder, specifically the improved image encoder from SAM. This represents the feature layer index of the encoder, used to distinguish the feature outputs of different levels of the encoder. This represents the total number of feature layers in the encoder, i.e., the total number of layers involved in the encoder to complete feature extraction. Indicates the encoder's first... The feature map output by the layer has a dimension of . , , The first The height and width of the layer feature map This represents the number of feature channels, which includes local details or global semantic features of the ground features in the remote sensing image corresponding to this level.

[0123] S6.3, Channel Attention Module

[0124] Based on the CBAM module, this embodiment further designs a Channel Attention Block (CAB). By multiplying the weight vector with the fused feature map channel by channel, the features of each channel are reweighted, background noise channels are suppressed, and the discriminative power of the segmentation features is improved. The CAB module learns one-dimensional weights. Realign the input feature maps Then through the weight vector With input feature map Pixel-by-pixel multiplication enhances the weights of the feature enrichment channels, suppresses redundant background noise channels, and finally outputs a feature map with channel attention optimization. The number of feature map channels. For feature map height, This represents the width of the feature map.

[0125] The effectiveness of the remote sensing image semantic segmentation results in this embodiment is verified by both the accuracy of the evaluation index and the visualization results. The visualization results are as follows: Figure 3 and Figure 4 As shown. Among them, Figure 3 (a), (b), (c), and (d) in the table correspond to the LoRA, Adapter, Mona, and Rein fine-tuning methods, respectively. The first column is the original remote sensing image, the second column is the ground truth (GT) label, and the third column is the semantic segmentation result of each fine-tuning method. Figure 4 The first column is the original remote sensing image, the second to fourth columns are the comparison model results, the fifth column is the semantic segmentation result of the improved SAM model of this invention, and the sixth column is the ground truth label (GT) of the land cover category.

[0126] Furthermore, to objectively and accurately evaluate the segmentation accuracy of the model, this invention adopts the core evaluation metrics commonly used in the field of semantic segmentation of remote sensing images to quantitatively compare different fine-tuning methods and different segmentation models. In this embodiment, the accuracy evaluation of each fine-tuning method is shown in Table 1 below. The accuracy comparison of the IoU experimental results of the method of this invention on each class on the LoveDA dataset is shown in Table 2 below. The comparison of the experimental results of different models is shown in Table 3 below.

[0127] Table 1: Accuracy Results of Four Fine-Tuning Methods

[0128]

[0129] Table 2: IoU (%) for each class of different models on the LoveDA dataset

[0130]

[0131] Table 3: Comparison of experimental results for different models

[0132]

[0133] As can be seen, this invention enhances the high-frequency detail feature extraction capability by introducing the IFormer module into the SAM image encoder and strengthens multi-scale feature fusion and boundary restoration capabilities by replacing the MACU-Net decoder. Compared with the original SAM model and traditional U-Net-like remote sensing segmentation methods, it significantly improves the feature extraction accuracy and semantic segmentation effect in complex remote sensing scenarios, overcomes the limitations of traditional methods in complex remote sensing scenarios, and fully verifies the effectiveness and advancement of the technical solution of this invention.

[0134] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A semantic segmentation method for remote sensing images based on SAM large model with multi-scale feature fusion, characterized in that, Includes the following steps: Step 1, Data Preprocessing and Augmentation: Acquire raw remote sensing image data and perform preprocessing, including image normalization, and simultaneously perform data augmentation using geometric transformation and color perturbation to expand the training dataset; Step 2, Improvement of SAM Image Encoder and Generation of Image Embedding: Improve the ViT visual self-attention mechanism in the SAM image encoder, introduce the IFormer module to reconstruct the image encoder, and enhance the model's ability to extract high-frequency information; input the preprocessed remote sensing image into the improved SAM image encoder, extract image features through the ViT visual self-attention mechanism, and generate global image embedding. The IFormer module improves the SAM image encoder by modifying the multi-head self-attention layer, specifically including... and The branch feature processing structure is adapted to the needs of multi-scale ground feature extraction from remote sensing images. The process is as follows: Input features It is split into a low-frequency component along the channel dimension. and a high-frequency component Low-frequency components are used to capture global semantic information of ground features in remotely sensed images, while high-frequency components are used to extract edge and texture-related detail features of ground features; among them, For the number of feature tokens, This represents the total number of feature channels. , The number of low-frequency and high-frequency component channels are respectively and satisfy the following conditions: + = ; The low-frequency components are processed by the original multi-head self-attention layer of SAM to obtain low-frequency features. : ； in, The function represents a multi-head self-attention mechanism; The high-frequency component is divided into two sub-components. and ,right Perform maximum filtering and point term convolution to obtain sub-features. ,right Perform point-wise and depthwise convolution processing to obtain sub-features. Through the initial mixer , and The three components are combined to obtain the fusion feature. ; After passing through the feedforward network, the final output feature Y of the initial transformer is expressed as: ； in, Functions represent multilayer perceptron processing; Step 3, cue information generation and encoding: Foreground and background markings are performed on the target area of ​​the remote sensing image to generate a set of sparse cue point information, which is then fed into the cue encoder of SAM and encoded into a cue embedding containing location and semantic information; The embedded prompt is obtained as follows: Step 3.1: Obtain sparse cue point information, which refers to a small number of discrete cue points manually marked in the target area of ​​the remote sensing image, including foreground and background cue points and target bounding boxes; Step 3.2: Take the sparse cue point information as input and pass it to the cue encoder of SAM. Through position encoding, each cue point and target bounding box is converted into a high-dimensional vector that matches the embedding channel dimension of the SAM image, and the sparse cue embedding is generated. Step 3.3: Obtain dense cue information, which refers to the low-dimensional semantic feature map extracted pixel by pixel from the remote sensing image. Specifically, it is the feature tensor obtained by downsampling the preprocessed remote sensing image through a 3×3 convolutional layer. Step 3.4: Input the sparse cue embedding and dense cue embedding into the fusion layer of the SAM cue encoder, and perform feature fusion by channel-wise weighted summation to obtain the fused cue embedding. Match the dimension of the global image embedding generated by the SAM image encoder in step 2 for subsequent feature fusion in the mask decoder. Step 4: Preliminary segmentation based on the SAM pre-trained model: The global image embedding generated in Step 2 and the cue embedding generated in Step 3 are input into the mask decoder of SAM. The image features and cue information are fused through the cross attention mechanism to perform preliminary segmentation of the remote sensing image, locate the target region and generate a preliminary semantic segmentation mask. Step 5, Fine-tuning with a small number of parameters: Fine-tuning with a small number of parameters is used to fine-tune the overall improved SAM model to alleviate the remote sensing domain offset problem of the SAM model and improve segmentation accuracy; Step 6, MACU-Net Decoder Optimization and Final Segmentation: Replace the original mask decoder of SAM with the MACU-Net decoder, perform multi-scale feature fusion and boundary restoration on the segmentation features fine-tuned in Step 5, and output high-precision remote sensing image semantic segmentation results.

2. The remote sensing image multi-scale feature fusion semantic segmentation method according to claim 1, characterized in that, The data augmentation described in step 1 includes random horizontal flipping, random rotation, and random color perturbation; By calculating the sample frequency of each land cover class, and using a Gaussian noise perturbation mask feature whose variance is inversely proportional to the sample frequency of each class, data augmentation is performed on land covers with few samples to address the problem of imbalanced label distribution. The calculation formula is as follows: ； ； in, It is a normalized input mask feature. These are batch size, number of channels, feature map height, and width, respectively. It is adjusted to the same size as the actual ground value. It is additional Gaussian noise. For the first The percentage of pixels representing the class in the actual mask. It has a fixed variance. , It is an adjustable hyperparameter.

3. The remote sensing image multi-scale feature fusion semantic segmentation method according to claim 1, characterized in that, The SAM pre-trained segmentation model described in step 4 has core components including an image encoder, a cue encoder, and a mask decoder. The image encoder is based on ViT and is pre-trained using a masked autoencoder method to adaptively learn the global semantic features and local detail features of remote sensing images, thus adapting to the input requirements of high-resolution remote sensing images. The prompt encoder is used to process prompt information input by the user or system, convert it into a standardized feature representation, and guide the segmentation model to accurately segment the target region; The mask decoder employs a fusion mechanism of cue self-attention and cross-attention to update and optimize the input image embedding features and cue embedding features. After feature processing is completed by two decoder blocks, the image embedding features are upsampled to the same size as the preprocessed remote sensing image. Then, a linear classifier is used to calculate the mask foreground probability of each pixel position to generate a preliminary semantic segmentation mask.

4. The remote sensing image multi-scale feature fusion semantic segmentation method according to claim 1, characterized in that, Step 5 involves fine-tuning a small number of parameters using the Rein method, as detailed below: Step 5.1: Input the preprocessed remote sensing image from Step 1 into the improved SAM model, and extract features through the backbone network to obtain the initial feature map. ; Step 5.2: Introduce a learnable token, denoted as... Each token has the following dimensions: , For the number of tokens, The number of token channels, and Consistent with the number of channels in the initial feature map; achieved through matrix multiplication. The relationship between the token and the initial feature map is calculated to obtain the correlation matrix, thereby achieving adaptive alignment of the token with the remote sensing image features. Step 5.3: Construct a similarity map This is used to measure the similarity between each token and each region in the initial feature map; it maps the similarity. The input is processed by a multilayer perceptron to perform a nonlinear transformation, generating feature weighting coefficients that match the dimensions of the initial feature map. The initial feature map is adjusted using feature weighting coefficients. Pixel-wise weighted optimization is performed to obtain the updated enhanced feature map. .

5. The remote sensing image multi-scale feature fusion semantic segmentation method according to claim 4, characterized in that, Step 6, the MACU-Net decoder, includes the following processing: Step 6.1, Asymmetric convolution block feature extraction: The asymmetric convolution block captures land cover features of different dimensions in the remote sensing image through three parallel convolution branches and multi-dimensional cross receptive fields; the feature numerical stability and non-linear expression ability are improved by batch normalization and ReLU activation function, and the enhanced feature map is output. Step 6.2, Multi-scale skip connection feature fusion: A three-level skip connection mechanism is designed to address the characteristics of large-scale terrain and small-scale features in remote sensing images; Step 6.3, Channel Attention Block Feature Reweighting: Based on the CBAM module, the channel attention block is improved by multiplying the weight vector with the fused feature map channel by channel to reweight each feature of the channel, suppressing background noise channels and improving the segmentation feature discrimination ability; Step 6.4 Output Results: The original remote sensing image size is restored through the upsampling module of the MACU-Net decoder, and semantic segmentation results with clear ground feature boundaries and accurate category distinction are output.

6. The remote sensing image multi-scale feature fusion semantic segmentation method according to claim 5, characterized in that, In step 6.1, the three parallel convolutional branches are as follows: convolution, Horizontal convolution and Vertical convolution captures ground features in different dimensions of remote sensing images through multi-dimensional cross receptive fields; The 3×3 convolution is used to capture global semantic features of ground features, and the 1×3 horizontal convolution and 3×1 vertical convolution are used to enhance the edge features of ground features in the horizontal and vertical directions, respectively. The asymmetric convolutional block outputs an enhanced feature map, calculated using the following formula: ； ； in, and These are the input and output features of asymmetric convolutional blocks. This represents the asymmetric convolution block's response to input features. The enhanced feature map is obtained after multi-branch convolution fusion, batch normalization, and ReLU activation function processing; , , They represent Convolution operation Horizontal convolution operation and Vertical convolution operation; and Let these represent the variance function and expected value of the input, respectively. These are tiny constants used to maintain numerical stability. and These are two trainable parameters of the batch normalization layer. The normalized result is obtained through... Scaling, via Translation.

7. The remote sensing image multi-scale feature fusion semantic segmentation method according to claim 5, characterized in that, In step 6.2, the three-level hop connection mechanism includes: Features of encoders and decoders at the same level are directly connected, preserving fine-grained edge details; Low-level encoder features are upsampled by transposed convolution and processed by ACB to transmit high-precision position information; The high-level encoder features are processed by max pooling downsampling and ACB to convey global semantic information; Specifically described as follows: ； in, The channel attention features represent the realigned channels. Represents an asymmetric convolution block. and These represent downsampling using a max pooling layer and upsampling using a transposed convolution, respectively. This indicates a splicing operation. This indicates the improved image encoder from SAM. Indicates the feature layer index of the encoder. This represents the total number of feature layers in the encoder. Indicates the encoder's first... Feature maps output by the layer This represents the output characteristics of a three-stage skip connection mechanism.

8. The remote sensing image multi-scale feature fusion semantic segmentation method according to claim 5, characterized in that, In step 6.3, the channel attention block learns one-dimensional weights. Realign the input feature maps Then through the weight vector With input feature map Pixel-wise multiplication enhances the weights of the feature enrichment channels, suppresses redundant background noise channels, and outputs a channel-attention-optimized feature map; where... The number of feature map channels. For feature map height, This represents the width of the feature map.

Images