A remote sensing semantic segmentation method, system, terminal and storage medium based on SAM and multi-scale feature association

Abstract

The present invention relates to the field of image semantic segmentation technology, and discloses a remote sensing semantic segmentation method, system, terminal, and storage medium based on SAM and multi-scale feature association. The method comprises: establishing and training a remote sensing semantic segmentation model based on SAM with high- and low-resolution feature association; acquiring a remote sensing RGB image, inputting it into the trained remote sensing semantic segmentation model based on SAM with high- and low-resolution feature association; extracting low-resolution features with global receptive field and contour perception and high-resolution features with semantic perception through the low-resolution branch and high-resolution branch of an encoder, respectively; inputting the low-resolution features and high-resolution features into a decoder, completing feature fusion through layer-by-layer upsampling and fusion, and ultimately outputting a semantic segmentation result at the original image size. The present invention significantly reduces the model computational complexity, achieves better segmentation performance, and improves the accuracy of the semantic segmentation results.

Description

Technical Field

[0001] The present invention relates to the field of image semantic segmentation technology, and in particular to a remote sensing semantic segmentation method, system, terminal and computer-readable storage medium based on SAM and multi-scale feature association. Background Art

[0002] Image semantic segmentation based on deep learning is an advanced technology that obtains pixel-level classification results by inferring input images. This technology has been applied to various remote sensing tasks, such as object classification, change detection, and vegetation cover detection. With the accumulation of image data and the improvement of device computing power in recent years, the Segment Anything Model (SAM) has been proposed. SAM is trained on massive amounts of data, and its backbone network has good generalization performance and is capable of zero-shot image segmentation in common scenarios. The existing method for using SAM to perform semantic segmentation is to extract and fuse features from different layers of SAM, then obtain multi-scale features through a Feature Pyramid Network (FPN), and finally fuse them to obtain high-resolution segmentation results.

[0003] Disadvantages of existing technologies: SAM's ViT (Vision Transformer) backbone is encoded with blocks of size 16, which to a certain extent loses the original spatial detail information of the image, such as contours. Image detail information is important for remote sensing images. Therefore, directly migrating SAM to remote sensing semantic segmentation tasks is not enough. Furthermore, the method of extracting and fusing features from different layers of SAM and then obtaining multi-scale features through FPN does not fundamentally solve the problem of loss of detailed features. Using simple convolution to fuse features of different scales is not enough, as it is difficult to establish connections between these features. If ordinary cross-attention is used for feature fusion, it will result in excessive computation and lead to serious problems such as severe video memory consumption.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0005] The main purpose of the present invention is to provide a remote sensing semantic segmentation method, system, terminal and computer-readable storage medium based on SAM and multi-scale feature association, aiming to solve the problems of detail feature loss, large computational complexity and poor segmentation performance in the existing technology of image semantic segmentation based on deep learning.

[0006] To achieve the above object, the present invention provides a remote sensing semantic segmentation method based on the association of SAM and multi-scale features, the remote sensing semantic segmentation method based on the association of SAM and multi-scale features comprising the following steps:

[0007] Establishing a SAM-based high- and low-resolution feature-correlated remote sensing semantic segmentation model, wherein the SAM-based high- and low-resolution feature-correlated remote sensing semantic segmentation model includes an encoder and a decoder, and training the SAM-based high- and low-resolution feature-correlated remote sensing semantic segmentation model;

[0008] Obtain a remote sensing RGB image to be segmented, input the remote sensing RGB image into a trained SAM-based high- and low-resolution feature association remote sensing semantic segmentation model, and extract low-resolution features with global receptive field and contour perception and high-resolution features with semantic perception through the low-resolution branch and high-resolution branch of the encoder respectively;

[0009] The low-resolution features with global receptive field and contour perception, as well as the high-resolution features with semantic perception are input into the decoder, and feature fusion is completed through layer-by-layer upsampling fusion, and finally the semantic segmentation result of the original image size is output.

[0010] Optionally, in the remote sensing semantic segmentation method based on SAM and multi-scale feature association, the encoder is used to extract image features, and the decoder is used to decode the extracted image features into a target task and output it.

[0011] Optionally, the remote sensing semantic segmentation method based on SAM and multi-scale feature association, wherein the training of the SAM-based high-resolution and low-resolution feature association remote sensing semantic segmentation model specifically includes:

[0012] Acquire a remote sensing RGB image dataset, gradually extract image samples from the remote sensing RGB image dataset, and input them into the SAM-based high- and low-resolution feature association remote sensing semantic segmentation model;

[0013] Extract low-resolution features and high-resolution features respectively through the low-resolution branch and the high-resolution branch of the SAM-based high- and low-resolution feature association remote sensing semantic segmentation model;

[0014] Inputting the low-resolution features and the high-resolution features into the decoder, completing feature fusion through layer-by-layer upsampling fusion, and outputting a semantic segmentation result of the original image size;

[0015] The loss calculation from the semantic segmentation result to the true label is completed through the true label and cross entropy loss function. The model training is completed through back propagation and parameter update to obtain a trained SAM-based high- and low-resolution feature association remote sensing semantic segmentation model.

[0016] Optionally, the remote sensing semantic segmentation method based on SAM and multi-scale feature association, wherein the low-resolution branch is based on SAM ViT-B, and adaptive features generated by adding an adapter before each Transformer layer to achieve adaptation of downstream tasks, and inserting an adapter module in each Transformer layer to enhance the adaptation of features to tasks, and finally outputting low-resolution features with semantics.

[0017] Optionally, the remote sensing semantic segmentation method based on SAM and multi-scale feature association, wherein the high-resolution branch uses a lightweight detail edge enhancement module to complete the extraction of high-resolution features; the detail edge enhancement module includes a detail feature extraction module and an edge enhancement module; the detail feature extraction module consists of a convolution operation, layer normalization, and ReLU activation, and extracts shallow detail features based on the input features; the edge enhancement module performs average pooling on the input features to weaken the detail representation, subtracts the features with weakened details from the input features, and then adds the details to the input feature map to enhance the details of the original features; finally, the output of the detail feature extraction module is spliced ​​with the output of the edge enhancement module, and the two are fused using convolution, and the detail edge enhancement features are output as high-resolution features.

[0018] Optionally, the remote sensing semantic segmentation method based on SAM and multi-scale feature association, wherein the low-resolution branch and the high-resolution branch of the encoder respectively extract low-resolution features with global receptive field and contour perception and high-resolution features with semantic perception, specifically include:

[0019] Designing a high-resolution and low-resolution feature association and update module, receiving the high-resolution features and the low-resolution features through the high-resolution and low-resolution feature association and update module, using the high-resolution features and the low-resolution features as query vectors, and using the corresponding high-resolution neighborhood and low-resolution neighborhood as key and value vectors, respectively, to perform cross-attention calculations to obtain contour-aware low-resolution features and semantic-aware high-resolution features;

[0020] The low-resolution branch outputs low-resolution features with global receptive field and contour perception;

[0021] The high-resolution branch outputs semantically-aware high-resolution features at three scales.

[0022] Optionally, the remote sensing semantic segmentation method based on SAM and multi-scale feature association, wherein the low-resolution features with global receptive field and contour perception and the high-resolution features with semantic perception are input into the decoder, and feature fusion is completed by upsampling and fusion layer by layer, and finally the semantic segmentation result of the original image size is output, specifically includes:

[0023] The low-resolution features with global receptive field and contour perception, as well as the high-resolution features with semantic perception at three scales are input into the decoder;

[0024] The decoder performs layer-by-layer upsampling fusion to complete low-resolution features with global receptive field and contour perception, as well as high-resolution feature fusion with semantic perception at three scales, and finally outputs a semantic segmentation result with the same size as the remote sensing RGB image.

[0025] In addition, to achieve the above-mentioned object, the present invention further provides a remote sensing semantic segmentation system based on SAM and multi-scale feature association, wherein the remote sensing semantic segmentation system based on SAM and multi-scale feature association includes:

[0026] A model building and training module is used to build a high- and low-resolution feature-correlated remote sensing semantic segmentation model based on SAM, wherein the high- and low-resolution feature-correlated remote sensing semantic segmentation model based on SAM includes an encoder and a decoder, and the high- and low-resolution feature-correlated remote sensing semantic segmentation model based on SAM is trained;

[0027] A feature extraction module is used to obtain a remote sensing RGB image to be segmented, input the remote sensing RGB image into a trained SAM-based high- and low-resolution feature association remote sensing semantic segmentation model, and extract low-resolution features with global receptive field and contour perception and high-resolution features with semantic perception through the low-resolution branch and high-resolution branch of the encoder respectively;

[0028] The semantic segmentation module is used to input low-resolution features with global receptive field and contour perception, as well as high-resolution features with semantic perception, into the decoder, complete feature fusion through layer-by-layer upsampling fusion, and finally output the semantic segmentation result of the original image size.

[0029] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a remote sensing semantic segmentation program based on the association of SAM and multi-scale features stored in the memory and runnable on the processor. When the remote sensing semantic segmentation program based on the association of SAM and multi-scale features is executed by the processor, the steps of the remote sensing semantic segmentation method based on the association of SAM and multi-scale features as described above are implemented.

[0030] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a remote sensing semantic segmentation program based on the association of SAM and multi-scale features, and when the remote sensing semantic segmentation program based on the association of SAM and multi-scale features is executed by the processor, the steps of the remote sensing semantic segmentation method based on the association of SAM and multi-scale features as described above are implemented.

[0031] The present invention establishes a SAM-based high- and low-resolution feature-association remote sensing semantic segmentation model. The SAM-based high- and low-resolution feature-association remote sensing semantic segmentation model includes an encoder and a decoder. The SAM-based high- and low-resolution feature-association remote sensing semantic segmentation model is trained. A remote sensing RGB image to be segmented is obtained and input into the trained SAM-based high- and low-resolution feature-association remote sensing semantic segmentation model. Low-resolution features with global receptive field and contour perception and high-resolution features with semantic perception are extracted through the low-resolution branch and high-resolution branch of the encoder, respectively. The low-resolution features with global receptive field and contour perception and the high-resolution features with semantic perception are input into the decoder. Feature fusion is achieved through layer-by-layer upsampling and fusion, and finally a semantic segmentation result of the original image size is output. The present invention significantly reduces the model computational complexity, achieves better segmentation performance, and improves the accuracy of the semantic segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flow chart of a preferred embodiment of the remote sensing semantic segmentation method based on SAM and multi-scale feature association of the present invention;

[0033] Figure 2 Schematic diagram of the architecture and training of a remote sensing semantic segmentation model based on high- and low-resolution feature association of SAM in a preferred embodiment of the remote sensing semantic segmentation method based on SAM and multi-scale feature association of the present invention;

[0034] Figure 3 This is a comparison diagram of the changes of high-resolution features and low-resolution features before and after passing through the HL module in a preferred embodiment of the remote sensing semantic segmentation method based on SAM and multi-scale feature association of the present invention;

[0035] Figure 4 1 is a structural diagram of a preferred embodiment of a remote sensing semantic segmentation system based on SAM and multi-scale feature association according to the present invention;

[0036] Figure 5 FIG. 4 is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0038] The remote sensing semantic segmentation method based on SAM and multi-scale feature association described in the preferred embodiment of the present invention is as follows: Figure 1 and Figure 2 As shown, the remote sensing semantic segmentation method based on SAM and multi-scale feature association includes the following steps:

[0039] Step S10: Establish a SAM-based high- and low-resolution feature-associated remote sensing semantic segmentation model, wherein the SAM-based high- and low-resolution feature-associated remote sensing semantic segmentation model includes an encoder and a decoder, and train the SAM-based high- and low-resolution feature-associated remote sensing semantic segmentation model.

[0040] Specifically, a Remote Sensing Semantic with Segmentation Complementary High- and Low-resolution Features based on SAM (HL-SAM-Seg) model is established. HL-SAM-Seg includes an encoder and a decoder. The encoder is used to extract image features, and the decoder is used to decode the extracted features into target task outputs. The combination of the two constitutes a complete data flow, from the input image to the output semantic segmentation mask, to achieve the prediction of the category of each pixel in the image.

[0041] The SAM-based high-low resolution feature association remote sensing semantic segmentation model (HL-SAM-Seg) is trained. The training process is as follows: Figure 2 As shown, obtain the remote sensing RGB image dataset (i.e. Figure 2RGB images in the image), gradually extracting image samples from the remote sensing RGB image dataset and inputting them into the SAM-based remote sensing semantic segmentation model with high- and low-resolution feature association. The encoder consists of a low-resolution branch and a high-resolution branch. The low-resolution branch is based on SAM's VisionTransformer-Base (a visual transformer backbone of SAM, abbreviated as ViT-B). Adaptive features generated by adapters are added before each Transformer layer to achieve adaptation to downstream tasks. Adapter modules are inserted into each Transformer layer to further enhance feature adaptation to tasks, and finally output semantically qualified low-resolution features. The high-resolution branch uses a lightweight Detail Edge Enhancement Module (DEEM) to extract high-resolution features. The Detail Edge Enhancement Module (DEEM) consists of a parallel connection between a Detail Feature Extraction Module and an Edge Enhancement Module. The Detail Feature Extraction Module, consisting of convolution operations, layer normalization, and Reluctant Unit (ReLU) activation, extracts shallow detail features from the input features. The Edge Enhancement Module first performs average pooling on the input features to reduce detail, subtracts the detail-reducing features from the input features to highlight them, and then adds these details to the input feature map to enhance the details of the original features. Finally, the output of the Detail Feature Extraction Module is concatenated with the output of the Edge Enhancement Module and fused using convolution, outputting the Detail Edge Enhanced features as high-resolution features.

[0042] Because high-resolution features and low-resolution features are exponentially related in spatial size, a pixel on a low-resolution feature corresponds to an area on a high-resolution feature. However, in reality, the semantic meaning of a pixel in a low-resolution feature often corresponds to an irregularly shaped area within the high-resolution feature, and this irregularly shaped area may be larger or smaller than the area originally corresponding to the pixel. Therefore, it is most appropriate to consider the 3×3 neighborhood of the original corresponding area. The correspondence between high-resolution and low-resolution features is similar: a high-resolution pixel can be associated with a corresponding low-resolution 3×3 neighborhood. These two correspondences facilitate bidirectional matching between high-resolution and low-resolution features.

[0043] In addition, in order to complement the semantic and spatial information of low-resolution and high-resolution features and complete the fusion and update of the two through a cross-attention mechanism, the HL module (High and Low Resolution Features Association and Updating Module) was designed. Specifically, the HL module is inserted into the 3rd, 6th, and 9th Transformer layers after the global attention in the low-resolution branch. It receives two inputs, high-resolution and low-resolution features, and uses the low-resolution and high-resolution features as query vectors, respectively, and the corresponding high-resolution and low-resolution neighborhoods as key and value vectors, respectively. Cross-attention calculations are performed on each, thereby obtaining contour-aware low-resolution features and semantic-aware high-resolution features. The features are then sent back to their respective branches for subsequent calculations.

[0044] Finally, the low-resolution branch outputs low-resolution features with a global receptive field and contour awareness, while the high-resolution branch outputs semantically aware high-resolution features at three scales: H×W, H / 2×W / 2, and H / 4×W / 4 (assuming the input size is H×W, where H represents height and W represents width). The features from these two branches are fed into the decoder, where they are fused through layer-by-layer upsampling, ultimately outputting a semantic segmentation result at the original image size. The loss from the semantic segmentation result (i.e., the predicted result) to the true label is calculated using the true label and cross-entropy loss function. Model training is completed through backpropagation and parameter updates, resulting in a trained SAM-based high- and low-resolution feature association remote sensing semantic segmentation model.

[0045] Step S20: Obtain a remote sensing RGB image to be segmented, input the remote sensing RGB image into a trained SAM-based high- and low-resolution feature association remote sensing semantic segmentation model, and extract low-resolution features with global receptive field and contour perception and high-resolution features with semantic perception through the low-resolution branch and high-resolution branch of the encoder, respectively.

[0046] Specifically, after obtaining a trained SAM-based high- and low-resolution feature association remote sensing semantic segmentation model, the trained SAM-based high- and low-resolution feature association remote sensing semantic segmentation model can be directly used to perform image semantic segmentation on the remote sensing RGB image to be segmented, and the remote sensing RGB image to be segmented is input into the encoder of the trained SAM-based high- and low-resolution feature association remote sensing semantic segmentation model. Low-resolution features and high-resolution features are extracted respectively through the low-resolution branch and high-resolution branch of the encoder, and then the high-resolution features and the low-resolution features are received by the high- and low-resolution feature association and update module (HL module). The high-resolution features and the low-resolution features are respectively used as query vectors, and the corresponding high-resolution neighborhood and low-resolution neighborhood are respectively used as key and value vectors, and cross-attention calculations are performed respectively to obtain contour-aware low-resolution features and semantic-aware high-resolution features; finally, the low-resolution branch outputs low-resolution features with global receptive field and contour awareness to the decoder, and the high-resolution branch outputs semantic-aware high-resolution features of three scales to the decoder.

[0047] In remote sensing images, specific pixel regions correspond to certain semantic categories. High-resolution features have higher clarity and more obvious edge features, while low-resolution features have lower clarity and stronger semantics. A point on a low-resolution feature can correspond to a pixel region of a certain shape on a high-resolution feature. The characteristics of these two features are complementary. Therefore, the present invention proposes to establish an association between low-resolution and high-resolution features through neighborhood cross-attention, and to update the features by using the features as query vectors, thereby achieving a complementary effect to compensate for the shortcomings of feature expression.

[0048] For the features extracted by the dual branches, the present invention completes the extraction of features of different resolutions through the DEEM module and the SAM ViT-B model, and realizes the interaction of the two features through the HL module, supplementing spatial information for low-resolution features and semantic information for high-resolution features, so that the updated features have more coordinated and consistent representation capabilities, which provides more effective features for the decoding stage of the decoder. Finally, through an effective feature fusion method, the features projected to each pixel have stronger semantics, thereby improving the accuracy of the semantic segmentation results. Compared with the existing dual-branch method, the HL module of the present invention uses more efficient cross-attention, which is suitable for matching and updating between high-resolution features and low-resolution features, and improves the effect of the dual-branch model combining convolution and Transformer with a small amount of computation and parameters.

[0049] like Figure 3As shown in the figure, it shows the comparison of the changes of high-resolution features and low-resolution features before and after the HL module. Assuming that the input image size is (H, W), for high-resolution features, the feature values ​​are more consistent after the HL module. This effect will gradually accumulate and is more obvious at the (H / 4, W / 4) scale; for low-resolution features, the contour performance of the features is enhanced. After being updated by the HL module, the low-resolution features have better contour feature performance.

[0050] Step S30: input the low-resolution features with global receptive field and contour perception, and the high-resolution features with semantic perception into the decoder, complete feature fusion through layer-by-layer upsampling fusion, and finally output the semantic segmentation result of the original image size.

[0051] Specifically, after the low-resolution features with global receptive field and contour perception, and the high-resolution features with semantic perception at three scales are input into the decoder, the decoder performs layer-by-layer upsampling fusion to complete the fusion of low-resolution features with global receptive field and contour perception, and high-resolution features with semantic perception at three scales, and finally outputs a semantic segmentation result with the same size as the remote sensing RGB image.

[0052] The HL-SAM-Seg proposed in this paper is based on the SAM backbone and adapted for the remote sensing domain. It uses a dual-branch structure to extract low-resolution features from ViT and high-resolution features from a custom convolutional layer. A self-developed high- and low-resolution feature association and update module (HL module) is embedded to establish a connection between high and low resolution features, enabling matching and updating between the two. This semantic segmentation head then completes the remote sensing semantic segmentation task. Compared to traditional deep learning methods, this method builds on the well-established parameter foundation and feature extraction capabilities of the visual base model backbone and further leverages the inherent generalization capabilities of the backbone. The HL module uses cross-attention across neighborhoods to significantly reduce model computation while matching and updating high- and low-resolution features, achieving better segmentation performance. This method achieves semantic segmentation by adaptive training on an existing dataset and then using it for inference. Furthermore, the HL module improves segmentation performance overall by establishing semantic and contour associations between low- and high-resolution features.

[0053] The present invention uses the DEEM module to provide high-resolution convolution features for SAM and constructs a dual-branch model; the features of the dual branches are associated through the HL module to achieve bidirectional matching and updating of high-resolution and low-resolution features; the HL module uses efficient neighborhood cross-attention to significantly reduce the amount of computation while improving performance.

[0054] Furthermore, the present invention can also use the backbone of the HL-SAM-Seg model for different remote sensing tasks, such as change detection in remote sensing images; this method can also be applied to a variety of existing models for feature improvement, thereby improving the feature extraction capability of the dual-branch model; in image tasks where edge contours are relatively sensitive, the HL module can provide good guidance.

[0055] Further, if Figure 4 As shown, based on the above-mentioned remote sensing semantic segmentation method based on the association of SAM and multi-scale features, the present invention also provides a remote sensing semantic segmentation system based on the association of SAM and multi-scale features, wherein the remote sensing semantic segmentation system based on the association of SAM and multi-scale features includes:

[0056] A model building and training module 51 is used to build a high- and low-resolution feature-correlated remote sensing semantic segmentation model based on SAM, wherein the high- and low-resolution feature-correlated remote sensing semantic segmentation model based on SAM includes an encoder and a decoder, and the high- and low-resolution feature-correlated remote sensing semantic segmentation model based on SAM is trained;

[0057] The feature extraction module 52 is used to obtain a remote sensing RGB image to be segmented, input the remote sensing RGB image into a trained SAM-based high- and low-resolution feature association remote sensing semantic segmentation model, and extract low-resolution features with global receptive field and contour perception and high-resolution features with semantic perception through the low-resolution branch and high-resolution branch of the encoder respectively;

[0058] The semantic segmentation module 53 is used to input low-resolution features with global receptive field and contour perception, and high-resolution features with semantic perception into the decoder, complete feature fusion through layer-by-layer upsampling fusion, and finally output the semantic segmentation result of the original image size.

[0059] Further, if Figure 5 As shown, based on the above-mentioned remote sensing semantic segmentation method and system based on SAM and multi-scale feature association, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 5 Only some of the components of the terminal are shown, but it should be understood that implementation of all of the shown components is not required, and more or fewer components may be implemented instead.

[0060] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard drive or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. Furthermore, the memory 20 may include both the internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software installed in the terminal and various types of data, such as program code of the terminal. The memory 20 may also be used to temporarily store data that has been output or is about to be output. In one embodiment, the memory 20 stores a remote sensing semantic segmentation program 40 based on the association of SAM and multi-scale features. This remote sensing semantic segmentation program 40 based on the association of SAM and multi-scale features can be executed by the processor 10, thereby implementing the remote sensing semantic segmentation method based on the association of SAM and multi-scale features described in this application.

[0061] In some embodiments, the processor 10 can be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the remote sensing semantic segmentation method based on SAM and multi-scale feature association.

[0062] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The processor 10, memory 20, and display 30 of the terminal communicate with each other via a system bus.

[0063] In one embodiment, when the processor 10 executes the remote sensing semantic segmentation program 40 based on SAM and multi-scale feature association in the memory 20, the following steps are implemented:

[0064] Establishing a SAM-based high- and low-resolution feature-correlated remote sensing semantic segmentation model, wherein the SAM-based high- and low-resolution feature-correlated remote sensing semantic segmentation model includes an encoder and a decoder, and training the SAM-based high- and low-resolution feature-correlated remote sensing semantic segmentation model;

[0065] Obtain a remote sensing RGB image to be segmented, input the remote sensing RGB image into a trained SAM-based high- and low-resolution feature association remote sensing semantic segmentation model, and extract low-resolution features with global receptive field and contour perception and high-resolution features with semantic perception through the low-resolution branch and high-resolution branch of the encoder respectively;

[0066] The low-resolution features with global receptive field and contour perception, as well as the high-resolution features with semantic perception are input into the decoder, and feature fusion is completed through layer-by-layer upsampling fusion, and finally the semantic segmentation result of the original image size is output.

[0067] The encoder is used to extract image features, and the decoder is used to decode the extracted image features into target tasks and output them.

[0068] The training of the SAM-based high- and low-resolution feature-correlated remote sensing semantic segmentation model specifically includes:

[0069] Acquire a remote sensing RGB image dataset, gradually extract image samples from the remote sensing RGB image dataset, and input them into the SAM-based high- and low-resolution feature association remote sensing semantic segmentation model;

[0070] Extract low-resolution features and high-resolution features respectively through the low-resolution branch and the high-resolution branch of the SAM-based high- and low-resolution feature association remote sensing semantic segmentation model;

[0071] Inputting the low-resolution features and the high-resolution features into the decoder, completing feature fusion through layer-by-layer upsampling fusion, and outputting a semantic segmentation result of the original image size;

[0072] The loss calculation from the semantic segmentation result to the true label is completed through the true label and cross entropy loss function. The model training is completed through back propagation and parameter update to obtain a trained SAM-based high- and low-resolution feature association remote sensing semantic segmentation model.

[0073] Among them, the low-resolution branch is based on SAM ViT-B. It achieves adaptation of downstream tasks by adding adaptive features generated by the adapter before each Transformer layer, and inserts the adapter module in each Transformer layer to enhance the adaptation of features to tasks, and finally outputs low-resolution features with semantics.

[0074] Among them, the high-resolution branch uses a lightweight detail edge enhancement module to complete the extraction of high-resolution features; the detail edge enhancement module includes a detail feature extraction module and an edge enhancement module; the detail feature extraction module consists of convolution operation, layer normalization, and ReLU activation, and extracts shallow detail features based on input features; the edge enhancement module performs average pooling on the input features to weaken the detail performance, subtracts the features with weakened details from the input features, and then adds the details to the input feature map to enhance the details of the original features; finally, the output of the detail feature extraction module is spliced ​​with the output of the edge enhancement module, and the two are fused using convolution, and the detail edge enhancement features are output as high-resolution features.

[0075] The step of extracting low-resolution features with global receptive field and contour perception and high-resolution features with semantic perception through the low-resolution branch and high-resolution branch of the encoder respectively includes:

[0076] Designing a high-resolution and low-resolution feature association and update module, receiving the high-resolution features and the low-resolution features through the high-resolution and low-resolution feature association and update module, using the high-resolution features and the low-resolution features as query vectors, and using the corresponding high-resolution neighborhood and low-resolution neighborhood as key and value vectors, respectively, to perform cross-attention calculations to obtain contour-aware low-resolution features and semantic-aware high-resolution features;

[0077] The low-resolution branch outputs low-resolution features with global receptive field and contour perception;

[0078] The high-resolution branch outputs semantically-aware high-resolution features at three scales.

[0079] The low-resolution features with global receptive field and contour perception, and the high-resolution features with semantic perception are input into the decoder, and feature fusion is completed by upsampling and fusion layer by layer, and finally the semantic segmentation result of the original image size is output, which specifically includes:

[0080] The low-resolution features with global receptive field and contour perception, as well as the high-resolution features with semantic perception at three scales are input into the decoder;

[0081] The decoder performs layer-by-layer upsampling fusion to complete low-resolution features with global receptive field and contour perception, as well as high-resolution feature fusion with semantic perception at three scales, and finally outputs a semantic segmentation result with the same size as the remote sensing RGB image.

[0082] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a remote sensing semantic segmentation program based on the association of SAM and multi-scale features. When the remote sensing semantic segmentation program based on the association of SAM and multi-scale features is executed by a processor, the steps of the remote sensing semantic segmentation method based on the association of SAM and multi-scale features as described above are implemented.

[0083] In summary, the present invention provides a remote sensing semantic segmentation method, system, terminal, and storage medium based on SAM and multi-scale feature association. The method includes: establishing a SAM-based high- and low-resolution feature-associated remote sensing semantic segmentation model, the SAM-based high- and low-resolution feature-associated remote sensing semantic segmentation model including an encoder and a decoder, and training the SAM-based high- and low-resolution feature-associated remote sensing semantic segmentation model; obtaining a remote sensing RGB image to be segmented, inputting the remote sensing RGB image into the trained SAM-based high- and low-resolution feature-associated remote sensing semantic segmentation model, extracting low-resolution features with global receptive field and contour perception and high-resolution features with semantic perception through the low-resolution branch and high-resolution branch of the encoder, respectively; inputting the low-resolution features with global receptive field and contour perception and the high-resolution features with semantic perception into the decoder, completing feature fusion through layer-by-layer upsampling and fusion, and ultimately outputting a semantic segmentation result of the original image size. The present invention significantly reduces the model computational complexity, achieves better segmentation performance, and improves the accuracy of the semantic segmentation results.

[0084] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or terminal comprising the element.

[0085] Of course, those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium that can be read by a computer. When executed, the program can include the processes in the above-described method embodiments. The computer-readable storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0086] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A remote sensing semantic segmentation method based on SAM and multi-scale feature association, characterized in that: The remote sensing semantic segmentation method based on SAM and multi-scale feature association includes: Establishing a SAM-based high- and low-resolution feature-correlated remote sensing semantic segmentation model, wherein the SAM-based high- and low-resolution feature-correlated remote sensing semantic segmentation model includes an encoder and a decoder, and training the SAM-based high- and low-resolution feature-correlated remote sensing semantic segmentation model; Obtain a remote sensing RGB image to be segmented, input the remote sensing RGB image into a trained SAM-based high- and low-resolution feature association remote sensing semantic segmentation model, and extract low-resolution features with global receptive field and contour perception and high-resolution features with semantic perception through the low-resolution branch and high-resolution branch of the encoder respectively; Inputting low-resolution features with global receptive field and contour perception, and high-resolution features with semantic perception into the decoder, completing feature fusion through layer-by-layer upsampling fusion, and finally outputting a semantic segmentation result of the original image size; The low-resolution branch is based on SAM ViT-B. It achieves adaptation of downstream tasks by adding adaptive features generated by adapters before each Transformer layer, and inserts adapter modules in each Transformer layer to enhance the adaptation of features to tasks. Finally, it outputs low-resolution features with semantic meaning. The high-resolution branch uses a lightweight detail edge enhancement module to complete the extraction of high-resolution features; the detail edge enhancement module includes a detail feature extraction module and an edge enhancement module; the detail feature extraction module consists of convolution operation, layer normalization, and ReLU activation, and extracts shallow detail features based on input features; the edge enhancement module performs average pooling on the input features to weaken the detail performance, subtracts the features with weakened details from the input features, and then adds the details to the input feature map to enhance the details of the original features; finally, the output of the detail feature extraction module is spliced ​​with the output of the edge enhancement module, and the two are fused using convolution, and the detail edge enhancement features are output as high-resolution features.

2. The remote sensing semantic segmentation method based on SAM and multi-scale feature association according to claim 1, characterized in that: The encoder is used to extract image features, and the decoder is used to decode the extracted image features into target tasks and output them.

3. The remote sensing semantic segmentation method based on SAM and multi-scale feature association according to claim 1, characterized in that: The training of the SAM-based high- and low-resolution feature-associated remote sensing semantic segmentation model specifically includes: Acquire a remote sensing RGB image dataset, gradually extract image samples from the remote sensing RGB image dataset, and input them into the SAM-based high- and low-resolution feature association remote sensing semantic segmentation model; Extract low-resolution features and high-resolution features respectively through the low-resolution branch and the high-resolution branch of the SAM-based high- and low-resolution feature association remote sensing semantic segmentation model; Inputting the low-resolution features and the high-resolution features into the decoder, completing feature fusion through layer-by-layer upsampling fusion, and outputting a semantic segmentation result of the original image size; The loss calculation from the semantic segmentation result to the true label is completed through the true label and cross entropy loss function. The model training is completed through back propagation and parameter update to obtain a trained SAM-based high- and low-resolution feature association remote sensing semantic segmentation model.

4. The remote sensing semantic segmentation method based on SAM and multi-scale feature association according to claim 1, characterized in that: The extracting of low-resolution features with global receptive field and contour perception and high-resolution features with semantic perception by the low-resolution branch and the high-resolution branch of the encoder respectively includes: Designing a high-resolution and low-resolution feature association and update module, receiving the high-resolution features and the low-resolution features through the high-resolution and low-resolution feature association and update module, using the high-resolution features and the low-resolution features as query vectors, and using the corresponding high-resolution neighborhood and low-resolution neighborhood as key and value vectors, respectively, to perform cross-attention calculations to obtain contour-aware low-resolution features and semantic-aware high-resolution features; The low-resolution branch outputs low-resolution features with global receptive field and contour perception; The high-resolution branch outputs semantically-aware high-resolution features at three scales.

5. The remote sensing semantic segmentation method based on SAM and multi-scale feature association according to claim 4 is characterized in that: The low-resolution features with global receptive field and contour perception, and the high-resolution features with semantic perception are input into the decoder, and feature fusion is completed by upsampling and fusion layer by layer, and finally the semantic segmentation result of the original image size is output, which specifically includes: The low-resolution features with global receptive field and contour perception, as well as the high-resolution features with semantic perception at three scales are input into the decoder; The decoder performs layer-by-layer upsampling fusion to complete low-resolution features with global receptive field and contour perception, as well as high-resolution feature fusion with semantic perception at three scales, and finally outputs a semantic segmentation result with the same size as the remote sensing RGB image.

6. A remote sensing semantic segmentation system based on SAM and multi-scale feature association, characterized in that: The remote sensing semantic segmentation system based on the association of SAM and multi-scale features is applied to the remote sensing semantic segmentation method based on the association of SAM and multi-scale features according to any one of claims 1 to 5, and the remote sensing semantic segmentation system based on the association of SAM and multi-scale features includes: A model building and training module is used to build a high- and low-resolution feature-correlated remote sensing semantic segmentation model based on SAM, wherein the high- and low-resolution feature-correlated remote sensing semantic segmentation model based on SAM includes an encoder and a decoder, and the high- and low-resolution feature-correlated remote sensing semantic segmentation model based on SAM is trained; A feature extraction module is used to obtain a remote sensing RGB image to be segmented, input the remote sensing RGB image into a trained SAM-based high- and low-resolution feature association remote sensing semantic segmentation model, and extract low-resolution features with global receptive field and contour perception and high-resolution features with semantic perception through the low-resolution branch and high-resolution branch of the encoder respectively; The semantic segmentation module is used to input low-resolution features with global receptive field and contour perception, as well as high-resolution features with semantic perception, into the decoder, complete feature fusion through layer-by-layer upsampling fusion, and finally output the semantic segmentation result of the original image size.

7. A terminal, characterized in that: The terminal includes: a memory, a processor, and a remote sensing semantic segmentation program based on the association of SAM and multi-scale features stored in the memory and runnable on the processor. When the remote sensing semantic segmentation program based on the association of SAM and multi-scale features is executed by the processor, the steps of the remote sensing semantic segmentation method based on the association of SAM and multi-scale features as described in any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a remote sensing semantic segmentation program based on the association of SAM and multi-scale features. When the remote sensing semantic segmentation program based on the association of SAM and multi-scale features is executed by the processor, the steps of the remote sensing semantic segmentation method based on the association of SAM and multi-scale features as described in any one of claims 1 to 5 are implemented.

Images