High-precision remote sensing image semantic segmentation method based on pyramid decoder

Abstract

The invention specifically discloses a high-precision remote sensing image semantic segmentation method based on a pyramid decoder and a multi-scale feature interactive attention module. The method comprises the following steps: firstly, extracting three complementary level features of low-level details, high-level semantics and an original image through a lightweight backbone network; and then inputting the multi-scale features into a network taking an encoder-decoder structure as a core, introducing a pyramid residual context module in a decoding stage, and explicitly enhancing high-level semantics by using pyramid pooling and residual nonlinear transformation to suppress redundant information. The multi-scale feature interactive attention module adaptively calculates space-channel weights of low-level details, high-level semantics and original features, so that differential fusion is realized, and feature conflicts are reduced. And after feature fusion is completed, compensating coding compression loss by optimizing jump connection, and finally outputting a 1024 * 1024 pixel-level semantic segmentation result. The method can be widely applied to high-resolution remote sensing scenes such as urban planning, disaster assessment and environment monitoring.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and remote sensing image processing, and particularly relates to a high-precision remote sensing image semantic segmentation method based on a pyramid decoder BACKGROUND

[0002] With the acceleration of urbanization, urban remote sensing images play an increasingly important role in city management, environmental monitoring, and disaster assessment. Semantic segmentation, as a key technology in deep learning, provides strong support for the automatic analysis of remote sensing images. By classifying each pixel in the image, it effectively extracts information such as buildings, roads, and vegetation in the city from remote sensing images, providing data support for urban planning and management.

[0003] With the wide application of deep learning technology in image processing, semantic segmentation based on convolutional neural networks (CNN) has significantly improved the accuracy and efficiency of remote sensing image processing. However, urban remote sensing images often contain complex landform shapes, scale differences, and rich detail information, which poses many challenges to traditional CNN models when processing such images.

[0004] To solve these problems, researchers have introduced attention mechanisms to enhance the modeling of global context information. By calculating the correlation between input features, the attention mechanism can dynamically focus on the most critical areas of the image, thereby improving the ability to extract details and semantic information. However, the uneven distribution of features in urban remote sensing images and the challenges of different levels of feature fusion limit the effectiveness of existing attention mechanisms (such as Transformer) in specific tasks. Although Transformer can effectively improve the global context modeling capability, its global correlation calculation requires correlation between all input features, which may lead to conflicts between detail information and semantic information, thereby affecting the detail recovery accuracy and segmentation effect.

[0005] In addition, the pyramid structure, as an effective means of expanding the receptive field, can capture information at different scales through multi-scale feature fusion. However, the pyramid structure often leads to the repetition of homogeneous semantic information during feature fusion, causing feature conflicts and semantic ambiguity problems. Especially in the interaction between low-level details and high-level semantic information, the excessive existence of redundant information may affect the learning efficiency and segmentation accuracy of the model. Therefore, how to effectively handle the fusion problem between low-level and high-level features has become a key challenge to improve the segmentation accuracy of remote sensing images. SUMMARY

[0006] 1. A high-precision pyramid decoding network for remote sensing image semantic segmentation, characterized by the following steps:

[0007] Step S1: Use an encoder to extract features from the input remote sensing image to obtain a multi-scale feature set containing low-level detail features, high-level semantic features, and original image features;

[0008] Step S2: The high-level semantic features are fed into the pyramid residual context module PRCBlock, which sequentially performs multi-scale pooling, bilinear interpolation to restore resolution, channel expansion convolution, GELU activation, channel restoration convolution and residual connection to output enhanced high-level features.

[0009] Step S3: Input the enhanced high-level features, low-level detail features and original image features into the multi-scale feature interaction attention module TFSim, and sequentially perform upsampling alignment, convolutional batch normalization ReLU processing, spatial mean and variance calculation and standardization, Sigmoid-constrained dynamic weight map generation and weighted fusion to obtain the final fused feature map;

[0010] Step S4: Input the final fused feature map into the segmentation head, map it to the number of semantic categories through 1×1 convolution, and then upsample it to the original spatial resolution of the input image through bilinear interpolation to output the semantic segmentation result.

[0011] Preferably, the PRCBlock processing procedure in step S2 specifically includes:

[0012] S21: Multi-scale pooling is performed on high-level semantic features through the pyramid pooling module. The pooling scales include 1×1, 2×2, 3×3 and 6×6.

[0013] S22: Perform bilinear interpolation on the pooling results at each scale to restore the spatial resolution of the input feature map;

[0014] S23: The recovered multi-scale features are concatenated by channels, and the channel dimensions are transformed and nonlinearly processed sequentially through extended convolutional layers, GELU activation layers and recovery convolutional layers.

[0015] S24: Add the processed features to the original high-level semantic features through residual connections to output enhanced high-level features.

[0016] Preferably, the dynamic weight generation process of the TFSim module in step S3 includes:

[0017] S31: Upsample the high-level semantic features to make their spatial resolution consistent with the low-level detail features;

[0018] S32: Perform convolution, batch normalization, and ReLU activation on the low-level detailed features in sequence;

[0019] S33: The processed high-level features, low-level features and original image features are added element by element to obtain the fused features;

[0020] S34: Calculate the spatial mean μ and variance σ of the fusion features. 2 And standardize the process;

[0021] S35: Generate a spatial attention weight map using the Sigmoid function to perform weighted fusion of input features.

[0022] Preferably, the encoder uses ResNet18 or lsknet_b1 as the backbone network, wherein the fourth stage output of lsknet_b1 is used as high-level semantic features, the second stage output is used as low-level detail features, and the first stage output is used as original image features.

[0023] Preferably, the processing of the segmentation head in step S4 includes:

[0024] S41: Map the number of channels to the number of semantic categories using a 1×1 convolution;

[0025] S42: Restore the image resolution to the input image resolution by bilinear interpolation upsampling;

[0026] S43: For each pixel, take the category corresponding to the maximum Softmax probability as the segmentation result.

[0027] Preferably, the encoder module is used to extract multi-scale features; the PRCBlock module is configured with a pyramid pooling submodule and a residual context refinement submodule; the TFSim module is configured with a feature alignment unit and a dynamic weight generation unit; and the segmentation head module contains a 1×1 convolutional layer and an upsampling layer.

[0028] Preferably, the residual context refinement submodule in the PRCBlock module includes, in sequence: a channel expansion convolutional layer that expands the number of input channels to four times the original number of channels; a GELU activation layer; a channel restoration convolutional layer that restores the number of channels to the original dimension; and a residual summation unit.

[0029] Preferably, the dynamic weight generation unit of the TFSim module includes: a spatial statistical calculation unit for calculating the mean and variance of the feature map; a standardization processing unit configured with an adjustable smoothing coefficient λ; and a Sigmoid activation unit for generating spatial weights in the interval [0, 1].

[0030] Preferably, the method is applicable to multi-class semantic segmentation tasks of high-resolution remote sensing images and improves segmentation accuracy and detail preservation capabilities.

[0031] Compared with the prior art, the present invention has the following beneficial effects.

[0032] This invention relates to the fields of computer vision and remote sensing image processing. First, a lightweight backbone network extracts features from three complementary levels: low-level details, high-level semantics, and the original image. Then, multi-scale features are input into a network with an encoder-decoder structure at its core. During the decoding stage, a Pyramid Residual Contextual Block (PRCBlock) is introduced to explicitly enhance high-level semantics and suppress redundant information using pyramid pooling and residual nonlinear transformation. Further, a Transformative Feature-Scale Interaction Module (TFSim) adaptively calculates the spatial-channel weights of low-level details, high-level semantics, and the original features, achieving differentiated fusion and reducing feature conflicts. After feature fusion is complete, skip connections are optimized to compensate for encoding compression loss, ultimately outputting a 1024×1024 pixel-level semantic segmentation result. This invention can be widely applied to high-resolution remote sensing scenarios such as urban planning, disaster assessment, and environmental monitoring. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating the high-precision remote sensing image semantic segmentation method based on a pyramid decoder as described in this invention.

[0034] Figure 2 This is a schematic diagram of the overall architecture of the algorithm of the present invention, showing the entire process from image input to segmentation output, including the Pyramid Residual Contextual Block (PRCBlock) and the Transformative Feature-Scale Interaction Module (TFSim).

[0035] Figure 3 The image segmentation results are obtained using trained weights in this invention. Detailed Implementation

[0036] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0037] The flowchart of the high-precision remote sensing image semantic segmentation method based on pyramid decoder described in this invention is as follows: Figure 1 As shown, the specific steps include the following:

[0038] Step 1: Based on the encoder-decoder structure, the network's ability to extract contextual information from the input image is enhanced by introducing the Pyramid Residual Contextual Block (PRCBlock) and the Transformative Feature-Scale Interaction Module (TFSim).

[0039] The network backbone offers two options: ResNet18 and lsknet_b1. ResNet18 is a lighter network with faster computation speed, but slightly lower segmentation accuracy; while lsknet_b1 is an efficient feature extraction model that, although slower in computation, provides higher segmentation accuracy. We initialize both backbone networks using pre-trained weights to fully leverage prior learning experience.

[0040] In the backbone network, the output of the feature maps resides in three different layers, representing low-level features, high-level features, and the original features of the image. lsknet_b1 is responsible for extracting multi-scale features from the input image and generating a series of feature maps, which are then passed to the decoder for further refinement and segmentation. During this process, the output from the fourth stage of lsknet_b1 is processed as high-level features to accurately perform the semantic segmentation task.

[0041] The decoder further processes multi-scale features from the backbone network using the PRCBlock and TFSim modules. PRCBlock focuses on high-level features from the fourth stage, refining and enhancing them to capture more semantic information; TFSim processes low-level, high-level, and original features simultaneously, improving feature stability and discriminative power through feature transformation. By fusing these features from different levels, the decoder ultimately generates the semantic segmentation output map.

[0042] Step 2: The PRCBlock module is mainly used to process high-level semantic features, aiming to improve the expression and refinement of semantic information. By combining pyramid pooling (PSPModule) and convolutional feature refinement (ResidualContextual Block, RCBlock), PRCBlock can effectively enhance the expression of high-level features, thereby improving the model's segmentation accuracy for complex scenes.

[0043] First, PRCBlock utilizes the PSPModule for multi-scale pooling operations. The PSPModule generates feature maps of different sizes through multiple pooling operations at different scales (e.g., 1×1, 2×2, 3×3, 6×6, etc.). Each scale of pooling extracts different levels of semantic information, from local details to global contextual information. The pooled feature maps are then restored to the same spatial resolution as the input feature maps through bilinear interpolation, ensuring effective fusion between features of different scales. This process can be expressed by the following formula:

[0044] Pyramid i =Interp(Pool i (X),size(X)),i=1,2,3,4

[0045] Among them, Pool i The `Interp` parameter represents pooling operations at different scales, where `Interp` represents interpolating the pooled feature map back to its original size. `X` is the input feature map, and `size(X)` is the spatial size of the input feature map. Next, the feature map processed by the `PSPModule` is fed into the `RCBlock` module for further nonlinear transformation and refinement. `RCBlock` performs linear transformations of the features through a series of convolutional layers and nonlinear processing of the features through activation functions, thereby improving the expressive power of the features. The core computational steps of `RCBlock` can be represented as follows:

[0046]

[0047] Conv1 expands the number of channels to a higher dimension and then performs a non-linear transformation using the GELU activation function. Conv2 restores the number of channels in the feature map to the original number of channels and further enhances the features using an activation function. This is the refined feature map, where X is the input feature. The residual connections in RCBlock ensure effective fusion of the refined features with the original input features, thereby further preserving important detail information. Finally, the output feature map is calculated using the following formula:

[0048]

[0049] Through this residual connection, RCBlock can improve the feature representation capability while avoiding information loss, ensuring that the refined features can effectively supplement the original input features.

[0050] Step 3: TFSim (Transformative Feature-Scale Interaction Module) is introduced as an attention mechanism, which aims to dynamically adjust the fusion method of low-level details, high-level semantics and original image features.

[0051] We categorize the feature maps we process into three types: low-level detail features X low It can capture fine-grained information of images, but lacks high-level semantics; high-level semantic features X high It is rich in semantic information, but has low spatial resolution and loses some details; and the original image features X raw Features are typically features of the input image that have undergone preliminary convolution processing. They provide basic contextual information about the image, but may contain noise or redundancy.

[0052] First, the high-level semantic feature map X is upsampled. high The spatial resolution is restored to obtain the processed high-level feature map:

[0053]

[0054] Low-level detail feature map X low The low-level feature map is processed by convolution, batch normalization and ReLU activation. The purpose of this process is to further extract detailed information from the low-level features and remove redundant or noisy components, thereby retaining more meaningful details.

[0055]

[0056] Following this, the low-level detail feature maps, the upsampled high-level feature maps, and the original image features are fused together. This fusion process helps the model better capture fine-grained details in the image while preserving global semantic information.

[0057]

[0058] For the input feature map X fused We first calculate its spatial mean and variance:

[0059]

[0060] Where H and W are the height and width of the feature map X, respectively, and μ(X) and σ are... 2 (X) represent the mean and variance of the feature map, respectively.

[0061] Next, we standardize each feature map to remove scale effects and ensure that the model can more effectively focus on the differences between features:

[0062]

[0063] Here, λ is a small constant used to prevent numerical instability when the variance is zero. Standardized feature map. This will ensure that the feature maps have zero mean and unit variance, thus guaranteeing a consistent scale in space. The standardized feature maps... We can use a sigmoid function to compute the attention weights at each location. These weights reflect the importance of each feature in the final fusion process:

[0064]

[0065] A is a weighted graph representing the importance of each position in the feature map. The Sigmoid function restricts the weight of each feature to the range [0, 1], thus strengthening the important parts of the feature map while suppressing the unimportant parts.

[0066] Finally, the calculated dynamic weights A are used to weight the input feature map X, resulting in the final fused feature map:

[0067]

[0068] This process ensures that more important parts of the feature map receive greater weight, while less important parts are suppressed, thereby optimizing the feature fusion effect.

[0069] Step 4: The network training uniformly uses the AdamW optimizer, with a base learning rate of 4e-4 and a cosine learning rate adjustment strategy; the epoch is 40, and the batch size is fixed at 4. Output a high-resolution semantic segmentation result of 1024×1024.

[0070] Step 5: Using the network weights obtained in Step 4, input the remote sensing image of the city to be detected, and obtain the prediction result, such as... Figure 3 As shown,

[0071] The above embodiments are only for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A high-precision pyramid decoding network for semantic segmentation of remote sensing images, characterized in that, Includes the following steps: Step S1: Use an encoder to extract features from the input remote sensing image to obtain a multi-scale feature set containing low-level detail features, high-level semantic features, and original image features; Step S2: The high-level semantic features are fed into the pyramid residual context module PRCBlock, which sequentially performs multi-scale pooling, bilinear interpolation to restore resolution, channel expansion convolution, GELU activation, channel restoration convolution and residual connection to output enhanced high-level features. Step S3: Input the enhanced high-level features, low-level detail features and original image features into the multi-scale feature interaction attention module TFSim, and sequentially perform upsampling alignment, convolutional batch normalization ReLU processing, spatial mean and variance calculation and standardization, Sigmoid-constrained dynamic weight map generation and weighted fusion to obtain the final fused feature map; Step S4: Input the final fused feature map into the segmentation head, map it to the number of semantic categories through 1×1 convolution, and then upsample it to the original spatial resolution of the input image through bilinear interpolation to output the semantic segmentation result.

2. The method according to claim 1, characterized in that, The PRCBlock processing procedure in step S2 specifically includes: S21: Multi-scale pooling is performed on high-level semantic features through the pyramid pooling module. The pooling scales include 1×1, 2×2, 3×3 and 6×6. S22: Perform bilinear interpolation on the pooling results at each scale to restore the spatial resolution of the input feature map; S23: The recovered multi-scale features are concatenated by channels, and the channel dimensions are transformed and nonlinearly processed by the extended convolutional layer, the GELU activation layer and the recovery convolutional layer in sequence. S24: Add the processed features to the original high-level semantic features through residual connections to output enhanced high-level features.

3. The method according to claim 1, characterized in that, The dynamic weight generation process of the TFSim module in step S3 includes: S31: Upsample the high-level semantic features to make their spatial resolution consistent with the low-level detail features; S32: Perform convolution, batch normalization, and ReLU activation on the low-level detailed features in sequence; S33: The processed high-level features, low-level features, and original image features are added element by element to obtain the fused features; S34: Calculate the spatial mean μ and variance σ of the fusion features. 2 And standardize the process; S35: Generate a spatial attention weight map using the Sigmoid function to perform weighted fusion of input features.

4. The method according to claim 1, characterized in that, The encoder uses ResNet18 or lsknet_b1 as the backbone network, where the fourth stage output of lsknet_b1 is used as high-level semantic features, the second stage output is used as low-level detail features, and the first stage output is used as the original image features.

5. The method according to claim 1, characterized in that, The processing of the segmentation head in step S4 includes: S41: Map the number of channels to the number of semantic categories using a 1×1 convolution; S42: Restore the image resolution to the input image resolution by bilinear interpolation upsampling; S43: For each pixel, take the category corresponding to the maximum Softmax probability as the segmentation result.

6. A semantic segmentation system for implementing the method of any one of claims 1-5, characterized in that, include: The encoder module is used to extract multi-scale features; The PRCBlock module is configured with a pyramid pooling submodule and a residual context refinement submodule; The TFSim module is configured with a feature alignment unit and a dynamic weight generation unit; The segmentation head module contains a 1×1 convolutional layer and an upsampling layer.

7. The system according to claim 6, characterized in that, The residual context refinement submodule in the PRCBlock module includes, in sequence: a channel expansion convolutional layer, which expands the number of input channels to four times the original number of channels; a GELU activation layer; a channel restoration convolutional layer, which restores the number of channels to the original dimension; and a residual summation unit.

8. The system according to claim 6, characterized in that, The dynamic weight generation unit of the TFSim module includes: a spatial statistical calculation unit for calculating the mean and variance of the feature map; a standardization processing unit configured with an adjustable smoothing coefficient λ; and a Sigmoid activation unit for generating spatial weights in the interval [0, 1].

9. The high-precision remote sensing image semantic segmentation method based on a pyramid decoder according to claim 1, characterized in that, The method is applicable to multi-class semantic segmentation tasks of high-resolution remote sensing images and improves segmentation accuracy and detail preservation.

Images