A post-fusion feature enhancement decoding method for remote sensing image semantic segmentation
By introducing a post-fusion feature enhancement module into the decoder, the problems of blurred boundaries and insufficient segmentation of small targets in the semantic segmentation of remote sensing images are solved, improving segmentation accuracy and small target recovery, while keeping computational overhead controllable.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV
- Filing Date
- 2026-04-08
- Publication Date
- 2026-07-03
AI Technical Summary
Existing semantic segmentation methods for remote sensing images are prone to boundary blurring, broken slender structures, and local missegmentation in areas such as road edges, building edges, and water body boundaries. Furthermore, they struggle to fully coordinate high-level semantic information with low-level spatial detail information, resulting in insufficient feature representation.
A post-fusion feature enhancement module is introduced into the decoder. Through local modeling, channel recalibration, and residual fusion, the segmentation quality of boundaries and small targets is improved, and the ability to express local spatial details of the features after multi-scale fusion is enhanced.
It improves the overall accuracy and semantic consistency of semantic segmentation of remote sensing images, the recoverability of small targets, while keeping the computational cost controllable and is suitable for different backbone network configurations.
Smart Images
Figure CN122023830B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of intelligent interpretation of remote sensing images and computer vision semantic segmentation technology, specifically relating to a post-fusion feature enhancement decoding method for semantic segmentation of remote sensing images. Background Technology
[0002] Semantic segmentation aims to classify pixels in an image into categories, and in remote sensing scenarios, it can be used for land use / cover mapping, urban feature extraction, and change monitoring. However, due to the large differences in target scale, dense distribution of small targets, complex feature boundaries, and strong background texture interference often present in high-resolution remote sensing images, existing segmentation methods are prone to boundary blurring, fragmented elongated structures, and local misclassification in areas such as roads, building edges, and water boundaries.
[0003] In existing remote sensing image semantic segmentation methods, the technical approach based on hierarchical encoders and lightweight decoders has been applied. Taking SegFormer-like methods as an example, lightweight decoders such as SegFormer typically employ All-MLP-style linear projection and upsampling stitching fusion: features at each scale are projected 1×1 to unify channels, upsampled to the same scale, stitched together, and fused through 1×1 convolution, followed by direct pixel-level classification. In its decoding stage, features at different scales are typically first unified in channels, then upsampled to the same spatial resolution before stitching and fusion, and the segmentation result is directly output based on the fusion result. This type of method can achieve multi-scale feature aggregation, but the fused features usually go directly into the classification layer, lacking further enhancement processing for the fusion result. Therefore, the utilization of local spatial neighborhood relationships, boundary detail information, and cross-channel semantic interaction remains limited. Furthermore, since channel unification and stitching fusion mainly complete scale alignment and information convergence, it is difficult to simultaneously ensure sufficient coordination of high-level semantic information and low-level spatial detail information. Therefore, in areas with complex textures, small target areas, and boundary areas, insufficient feature representation is prone to occur.
[0004] To address the aforementioned issues, directly adding complex convolutional stacking or enhancement modules to the decoding end could increase the number of parameters and computational load, thus affecting the application requirements of lightweight decoding structures. Therefore, how to further enhance the features after multi-scale fusion without excessively increasing computational complexity to improve local detail representation and semantic interaction capabilities is a technical problem that needs to be solved in the existing technology. Summary of the Invention
[0005] In response to the shortcomings of existing methods pointed out in the background art, this invention proposes a post-fusion feature enhancement decoding method for semantic segmentation of remote sensing images. A post-fusion enhancement module is introduced between the fusion convolution and classification layers, enabling the decoder to have the ability to refine local details after fusion, thereby improving the segmentation quality of boundaries and small targets and enhancing the ability of multi-scale fused features to express local spatial details (especially boundaries and small targets).
[0006] Technical Solution: To solve the above-mentioned technical problems, the present invention adopts the following technical solution:
[0007] A post-fusion feature enhancement decoding method for semantic segmentation of remote sensing images includes the following steps:
[0008] S1. Input remote sensing imagery;
[0009] S2. Multi-scale features are extracted using the Transformer encoder. ;
[0010] S3. Perform channel-unified mapping on features at each scale through the decoder to unify the channel dimension;
[0011] S4. Perform upsampling on the features after unified mapping through the S3 channel, and align the upsampling to the same spatial scale.
[0012] S5. The outputs of S4 are concatenated along the channel dimension and fused features are obtained through fusion convolution.
[0013] S6. Construct a post-fusion feature enhancement module to obtain enhanced features through local modeling, channel recalibration, and residual fusion.
[0014] S7. Input the enhanced features into the classification layer for classification prediction and output the segmentation logits.
[0015] As a preferred option, the specific implementation details in S1 are as follows:
[0016] Remote sensing images are satellite remote sensing images or aerial remote sensing images. Remote sensing images are stored in raster format and include multiple spectral bands, including visible light bands and near-infrared bands. Remote sensing images are derived from publicly available remote sensing data, commercial remote sensing data, and self-acquired remote sensing data.
[0017] As a preferred option, the specific implementation details in S3 are as follows:
[0018] Perform channel-unified mapping on features at all scales to unify the number of channels. Let the first The feature map at each scale is Then the first The feature map obtained after channel unification mapping of the feature maps at each scale is: The calculation process is as follows:
[0019] ;
[0020] in, An index representing the feature scale / level; express Convolution operation.
[0021] As a preferred option, the specific implementation details in S4 are as follows:
[0022] The first The feature map obtained after channel unification mapping of feature maps at each scale. Upsampling to target spatial size Alignment features are obtained. The calculation process is as follows:
[0023] ;
[0024] in, Indicates an upsampling operation; Indicates the height of the target space; Indicates the width of the target space.
[0025] As a preferred option, the specific implementation details in S5 are as follows:
[0026] Alignment features after S4 processing By stitching along the channel dimension, the stitching feature is obtained. The calculation process is as follows:
[0027] ;
[0028] in, This indicates a splicing operation along the channel dimension;
[0029] Subsequently, the splicing features were analyzed. Apply Fusion convolutions are used to perform channel fusion to obtain fused features. The calculation process is as follows:
[0030] ;
[0031] in, express Convolution operation.
[0032] As a preferred option, the specific implementation details in S6 are as follows:
[0033] S601, Channel Expansion: via Convolution transforms the channels from Expand to The intermediate feature map after channel expansion is obtained. ;
[0034] S602, Local Spatial Blending: Intermediate Feature Map After Channel Expansion Apply Depth convolution yields intermediate feature maps after local spatial modeling. , used to capture local boundaries and texture context;
[0035] S603, Channel Blending: Intermediate Feature Map After Local Spatial Modeling Apply Pointwise convolution is performed to obtain intermediate feature maps after channel blending. To achieve cross-channel semantic interaction;
[0036] S604, Channel Restoration: via Convolution blends the intermediate feature maps of the channels descend The channel yields the intermediate feature map after channel reconstruction. ;
[0037] S605, Channel Recalibration: Re-calibrating the intermediate feature map after channel restoration. Apply channel attention to obtain the weighted feature map after channel recalibration. This is used to adaptively emphasize channels with higher information content.
[0038] S606, Residual Fusion: Output ;
[0039] in, This represents the enhanced feature map output by the post-fusion feature enhancement module; Indicates fusion characteristics; This represents the weighted feature map after channel recalibration.
[0040] As a preferred embodiment, in S605, the channel attention module is an efficient channel attention ECA module, which specifically includes global pooling, one-dimensional convolution, sigmoid mapping, and channel weighting.
[0041] Intermediate feature map after channel restoration Global average pooling is performed to obtain channel description vectors; one-dimensional convolution is performed on the channel description vectors and then passed through the Sigmoid function to obtain channel weights; intermediate feature maps are then constructed based on the channel weights. Perform channel-by-channel weighting to obtain the weighted feature map after channel recalibration. .
[0042] As a preferred option, the intermediate feature map after channel restoration The specific calculation process for obtaining the channel description vector by performing global average pooling is as follows:
[0043] ;
[0044] in, This represents the intermediate feature map after channel reconstruction. The The channel description value is obtained by performing global average pooling on all spatial locations for each channel; and These represent the intermediate feature maps after channel reconstruction. Height and width in spatial dimensions; This represents the intermediate feature map after channel restoration; This represents the intermediate feature map after channel reconstruction. Channel index, This represents the intermediate feature map after channel reconstruction. Position index in the height direction, This represents the intermediate feature map after channel reconstruction. Position index in the width direction.
[0045] Beneficial effects: Compared with the prior art, the present invention has the following beneficial effects:
[0046] (1) Improved overall segmentation accuracy of the present invention: The present invention introduces a post-fusion feature enhancement module after multi-scale feature fusion and before classification prediction to refine the fused features in a learnable manner. On the LoveDA dataset, when using the MiT-B0 backbone, the mIoU of the present invention is 51.60±0.40, which is higher than the SegFormer baseline of 50.50±0.83; when using the MiT-B2 backbone, the mIoU of the present invention is 53.82±0.31, which is higher than the SegFormer baseline of 52.52±0.08. On the ISPRS Vaihingen dataset, when using the MiT-B0 backbone, the mIoU of the present invention is improved from 72.06±0.10 to 72.82±0.23; when using the MiT-B2 backbone, the mIoU is improved from 74.01±0.12 to 74.84±0.41. The above results show that the present invention achieves consistent improvement under different datasets and different backbone capacities.
[0047] (2) Improved semantic consistency of boundary neighborhood and recoverability of small targets: Under the unified settings (boundary_tol_ratio=0.002, trimap_band_px=2, size_bins=(1024,9216), ignore_index=255) and repeated experiments with three random seeds (0, 7, 42), the proposed solution shows improvement in Trimap mIoU and small target related indicators: on LoveDA (MiT-B2), Trimap mIoU increased from 26.30±0.12 to 27.89±0.79; on Vaihingen (MiT-B2), Trimap mIoU increased from 26.30±0.12 to 27.89±0.79. The mIoU improved from 59.27±0.41 to 60.23±0.16; simultaneously, the mIoU of the Vaihingen (MiT-B2) small target improved from 48.11±0.83 to 49.41±0.97, and the mRecall of the small target improved from 61.85±0.92 to 63.01±0.72. These results support the present invention's claims to improve the semantic consistency of the boundary neighborhood and the recoverability of small targets.
[0048] (3) Controllable Incremental Overhead and Accuracy-Efficiency Trade-off: The post-fusion enhancement module of this invention introduces a 0.53M parameter increment and increases 8.7G FLOPs under both MiT-B0 and MiT-B2 backbones. In terms of throughput, MiT-B2 decreased from 24.1 FPS to 21.1 FPS on the LoveDA dataset and from 67.7 FPS to 64.9 FPS on the Vaihingen dataset; MiT-B0 decreased from 56.5 FPS to 41.7 FPS on the LoveDA dataset and from 105.2 FPS to 100.6 FPS on the Vaihingen dataset. The above results show that this invention maintains controllable computational overhead while achieving improved accuracy.
[0049] (4) Pluggable portability and rational composition: The enhancement module of this invention is located between the fusion convolution and classification layers, with input and output channel dimensions consistent with the fusion features, making it easy to apply in a pluggable manner to different backbone configurations. In the LoveDA (MiT-B2) ablation experiment, removing the entire post-fusion enhancement module reduced the optimal mIoU from 53.82±0.31 to 52.52±0.08; removing the residual connection reduced the optimal mIoU to 52.58±0.21; removing the ECA attention reduced the optimal mIoU to 53.27±0.21; adjusting the depth convolution kernel size or reducing the expansion ratio also led to a performance decrease. This result supports the structural rationality and reproducibility of the present invention from the perspective of compositional contribution.
[0050] In summary, this invention achieves consistent accuracy improvement on both LoveDA and ISPRS Vaihingen datasets and on both MiT-B0 and MiT-B2 capacity backbones through a post-fusion feature enhancement decoding structure. It also demonstrates reproducible improvements in semantic consistency of boundary neighborhoods and recoverability of small targets, while maintaining a controllable increase in parameter quantity and computational complexity. Attached Figure Description
[0051] Figure 1 This is a flowchart of the method of the present invention;
[0052] Figure 2 This is a schematic diagram of the overall structure of the semantic segmentation network of this invention;
[0053] Figure 3 This is a schematic diagram of the existing SegFormer decoding structure;
[0054] Figure 4 This is a schematic diagram of the decoding structure of the present invention;
[0055] Figure 5 This is a schematic diagram of the Post-Fusing Feature Enhancement Module (PFEB) structure of the present invention;
[0056] Figure 6 This is a schematic diagram of the channel attention module of the present invention;
[0057] Figure 7 This is a visual comparison diagram of the segmentation results of the method of this invention and existing methods on the LoveDA dataset;
[0058] Figure 8 This is a visual comparison diagram of the segmentation results of the method of this invention and existing methods on the ISPRS Vaihingen dataset. Detailed Implementation
[0059] The present invention will be further illustrated below with reference to specific embodiments. These embodiments are implemented based on the technical solutions of the present invention, and it should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.
[0060] like Figure 1 , Figure 2 As shown, this embodiment provides a post-fusion feature enhancement decoding method for semantic segmentation of remote sensing images, which can be applied to practical scenarios of remote sensing image interpretation, such as land use / cover mapping, urban element extraction and change monitoring, etc. In the above scenarios, the object to be processed is remote sensing image data, which is stored in the form of a two-dimensional raster and contains one or more spectral bands; the output of this embodiment is a pixel-level semantic segmentation result consistent with the input image space, used to indicate the land cover category or target category to which each pixel belongs.
[0061] The input remote sensing image can be preprocessed and then input into the semantic segmentation network for inference. The preprocessing includes at least cropping or dividing the remote sensing image into blocks according to a preset size and normalizing or standardizing the pixel values. Then, a Transformer encoder extracts multi-scale features, and a decoder completes channel unification, upsampling alignment, and feature fusion. A post-fusion feature enhancement module is inserted between the fusion convolution and classification layers to perform local spatial mixing and channel recalibration on the fused features, and outputs the enhanced features through residuals. Finally, the enhanced features are input into the classification layer to obtain segmentation logits, and pixel-level class probabilities are obtained through Softmax, thereby generating a semantic segmentation mask or vectorized ground feature boundary results aligned with the input image.
[0062] Through the above process, this embodiment can segment slender structures and complex boundary areas (such as roads, water boundaries, building outlines, etc.) in remote sensing images at the pixel level, providing basic data for subsequent mapping, statistics and change detection.
[0063] In this embodiment, a lightweight feature enhancement module is introduced after multi-scale feature fusion to improve the segmentation quality of boundaries and small targets. Post-fusion feature enhancement decoding: Keeping the original multi-scale feature fusion process unchanged, after the multi-scale features are spliced and fused and before the classifier, a lightweight post-fusion feature enhancement module is inserted to perform local spatial mixing and channel recalibration on the fused features, and to achieve incremental refinement output through residual method.
[0064] Taking the four-scale features of SegFormer (an image segmentation model based on Transformer) as an example, the method of the present invention can be implemented according to the following steps:
[0065] S1. Input remote sensing imagery;
[0066] The remote sensing image is at least one of satellite remote sensing image or aerial remote sensing image. The remote sensing image is stored in a raster format and may contain multiple spectral bands. The spectral bands include at least one band in the visible light band and / or at least one band in the near-infrared band. The remote sensing image may be derived from publicly available remote sensing data, commercial remote sensing data, or self-acquired remote sensing data.
[0067] S2. Multi-scale features are extracted using the Transformer encoder. ;
[0068] The Transformer encoder outputs feature maps at four levels: Feature maps at each scale .
[0069] in, An index representing the feature scale / level. ; and The spatial dimensions of the corresponding layer feature map. Indicates the first The height of each level feature map in the spatial dimension Indicates the first The width of each level of feature map in the spatial dimension, and satisfying The input image spatial size is 1 / 4 of the scale. It is 1 / 8 scale. It is 1 / 16 scale. It is 1 / 32 scale; For the first The number of channels in the multi-scale feature map. The multi-scale features are used for the subsequent channel unification mapping of S3 and scale alignment of S4.
[0070] In S2, such as Figure 4 As shown, the input image is first subjected to overlapping image patch embedding. Specifically, the input image is segmented and feature extracted using convolutional projection, where overlapping regions are preserved between adjacent image patches to obtain initial embedding features containing local continuous spatial information. Then, the initial embedding features are normalized and used as input to Transformer module 1.
[0071] The Transformer module 1 is operated on, specifically including efficient self-attention, hybrid feedforward network, and overlapping image patch merging.
[0072] The implementation of efficient self-attention involves: linearly mapping the input features to obtain query features, key features, and value features; compressing the spatial dimensions of the key features and value features to reduce the computational overhead, and then calculating their relevance with the query features to obtain attention weights; weighting and aggregating the value features according to the attention weights to obtain the self-attention output features; and then performing a residual connection between the self-attention output features and the corresponding input features.
[0073] The implementation of the hybrid feedforward network is as follows: channel mapping is performed on the efficient self-attention output features to obtain expanded intermediate features; then, a local spatial mixing operation is applied to the intermediate features to introduce spatial neighborhood information; subsequently, nonlinear transformation and channel restoration are performed on the features after local spatial mixing to obtain the output features of the hybrid feedforward network; and the output features are residually connected with the input features of the hybrid feedforward network.
[0074] The implementation of overlapping image patch merging is as follows: the output features of the current stage are subjected to downsampling mapping operation with overlapping receptive fields to reduce the spatial size of the feature map and increase the channel dimension, so as to obtain the input features of the Transformer module in the next stage; overlapping image patch merging retains the local correlation information between adjacent regions while completing the scale reduction, which is used to construct a hierarchical multi-scale feature representation.
[0075] The Transformer encoder comprises Transformer module 1, Transformer module 2, Transformer module 3, and Transformer module 4 connected in sequence. Specifically, the initial features from the overlapping image patch embedding output are input to Transformer module 1, processed by an efficient self-attention and hybrid feedforward network to obtain a first-level feature map, which is then input to Transformer module 2 after overlapping image patch merging. Transformer module 2 performs the same processing flow as Transformer module 1 on the input features to obtain a second-level feature map, which is then input to Transformer module 3 after overlapping image patch merging. Transformer module 3 performs the same processing flow as described above on the input features to obtain a third-level feature map, which is then input to Transformer module 4 after overlapping image patch merging. Transformer module 4 performs efficient self-attention and hybrid feedforward network processing on the input features and outputs a fourth-level feature map. The first-level to fourth-level feature maps correspond to multi-scale features with different spatial resolutions and are used for subsequent channel unification mapping in S3 and scale alignment in S4.
[0076] S3. Perform channel-unified mapping on features at each scale through the decoder to unify the channel dimension;
[0077] The decoder performs a 1×1 projection of features at each scale to unify the channel dimensions.
[0078] Perform channel-unified mapping on features at all scales to unify the number of channels. Let the first... The feature map at each scale is Then the first The feature map obtained after channel unification mapping of the feature maps at each scale is: The calculation process is as follows:
[0079]
[0080] in, An index representing the feature scale / level. Indicates the first The feature map obtained after channel-unified mapping of feature maps at each scale; Indicates the first The number of channels in a scale feature map; and The spatial dimensions of the corresponding layer feature map. Indicates the first The height of each level feature map in the spatial dimension Indicates the first The width of each level feature map in the spatial dimension; Indicates the number of channels; express Convolution operation is used to convert the channel dimension from Transform into and maintain space size constant.
[0081] exist Figure 4 In this process, the unified channel mapping is implemented by a multilayer perceptron. The multilayer perceptron acts on the channel vectors corresponding to each spatial location, mapping the channel dimensions while maintaining the spatial size. Functionally, it corresponds to... Convolution projection. Let the first... The feature map at each scale is Then the first The feature maps obtained after the feature maps at each scale are subjected to channel-unified mapping (specifically implemented by a multilayer perceptron mapping) are as follows: The calculation process is as follows:
[0082]
[0083] in, Indicates the action on the first A multilayer perceptron mapping module for feature maps at various scales is used to convert the channel dimensions from... Transformed to a uniform number of channels and maintain space size constant; An index representing the feature scale / level; Indicates the first The feature map obtained after channel-unified mapping of feature maps at each scale; Indicates the first The number of channels in a scale feature map; and The spatial dimensions of the corresponding layer feature map. Indicates the first The height of each level feature map in the spatial dimension Indicates the first The width of each level feature map in the spatial dimension; Indicates the number of channels.
[0084] S4. Perform upsampling on the features after unified mapping through the S3 channel, and align the upsampling to the same spatial scale.
[0085] Specifically, the first The feature map obtained after channel unification mapping of feature maps at each scale. Upsampling to target spatial size Alignment features are obtained. The calculation process is as follows:
[0086]
[0087] in, Indicates alignment features; Indicates an upsampling operation; This represents the height of the target spatial dimension, which corresponds to the spatial height of the first-level feature map. This represents the width of the target space, which corresponds to the spatial width of the first-level feature map. Indicates the first The feature map is obtained by uniformly mapping the feature maps of each scale through channels. Indicates the number of channels.
[0088] In this embodiment, upsampling is achieved using interpolation methods, such as bilinear interpolation; the target spatial size can be taken from the highest resolution scale features. The spatial dimensions, i.e.:
[0089]
[0090] in, This represents the operation of obtaining the spatial dimensions of the input feature map. This represents the height of the target spatial dimension, which corresponds to the spatial height of the first-level feature map. This represents the width of the target space, which corresponds to the spatial width of the first-level feature map. This represents the feature map after channel-unified mapping of the feature map at the first scale.
[0091] In this step, the target spatial size is taken as the channel-unified mapping feature map of the first scale. The space dimensions, therefore and Feature maps corresponding to the first scale after channel-unified mapping The spatial height and width.
[0092] S5, splice and connect in the channel dimension Convolution is used for feature fusion;
[0093] In S5, the alignment features processed by S4 will be... By stitching along the channel dimension, the stitching feature is obtained. The calculation process is as follows:
[0094]
[0095] in, This indicates a splicing operation along the channel dimension; This indicates the number of scale features involved in the stitching (in this embodiment). ). For the number of channels, This represents the height of the target spatial dimension, which corresponds to the spatial height of the first-level feature map. This represents the width of the target space, which corresponds to the spatial width of the first-level feature map.
[0096] In this step, and These correspond to the height and width of the target space after S4 alignment, respectively.
[0097] Subsequently, the splicing features were analyzed. Apply Fusion convolutions are used to perform channel fusion to obtain fused features. The calculation process is as follows:
[0098]
[0099] in, express Convolution operation is applied to concatenated features. This allows for the fusion of spliced multi-scale features along the channel dimension and the output of a preset number of channels. The fusion characteristics. For the number of channels, This represents the height of the target spatial dimension, which corresponds to the spatial height of the first-level feature map. This represents the width of the target space, which corresponds to the spatial width of the first-level feature map.
[0100] In this step, and These correspond to the height and width of the target space after S4 alignment, respectively.
[0101] S6. Construct the Post-Fusion Feature Enhancement Module (PFEB) to obtain enhanced features through local modeling, channel recalibration, and residual fusion.
[0102] like Figure 4 As shown, the fusion features Input the post-fusion feature enhancement module to obtain the enhanced feature map output by the post-fusion feature enhancement module. Their relationship can be expressed as: .
[0103] in, , The fused features are obtained through S5 fusion convolution. For the number of channels, This represents the height of the target spatial dimension, which corresponds to the spatial height of the first-level feature map. This represents the width of the target space, which corresponds to the spatial width of the first-level feature map. This represents the mapping process of the post-fusion feature enhancement module; This represents the enhanced feature map output by the post-fusion feature enhancement module.
[0104] In this step, and These correspond to the height and width of the target space after S4 alignment, respectively.
[0105] like Figure 5 As shown, the post-fusion feature enhancement module uses fused features For input, a lightweight refinement structure is adopted, consisting of channel expansion—depthmography—pointwise convolution—channel restoration—channel attention—residual fusion; specifically:
[0106] S601, Channel Expansion: via Convolution transforms the channels from Expand to The intermediate feature map after channel expansion is obtained. ;
[0107] Fusion features Apply Convolution is used to perform channel expansion, resulting in intermediate feature maps with expanded channels. The calculation process is as follows:
[0108]
[0109] in, This represents the intermediate feature map after channel expansion, where the number of channels is increased from... Expand to ; Indicates the channel expansion ratio; express Convolution operation, applied to fused features To change its channel number from Expand to The intermediate feature map after channel expansion is obtained. .
[0110] S602, Local Spatial Blending: Intermediate Feature Map After Channel Expansion Apply Depthwise Convolution (For odd numbers, such as 3), to obtain the intermediate feature map after local spatial modeling. It is used to capture local boundaries and texture context.
[0111] The specific calculation process is as follows:
[0112]
[0113] in, express Depthwise convolution is a channel-wise convolution, meaning that the convolution operation is performed independently on each input channel. When the number is odd, symmetrical padding can maintain the same output space size. Indicates the channel expansion ratio; For the number of channels, This represents the height of the target spatial dimension, which corresponds to the spatial height of the first-level feature map. This represents the width of the target space, which corresponds to the spatial width of the first-level feature map. This represents an intermediate feature map after local spatial modeling.
[0114] In this step, and These correspond to the height and width of the target space after S4 alignment, respectively. Therefore, the feature maps before and after local spatial blending are both in terms of spatial size. .
[0115] S603, Channel Blending: Intermediate Feature Map After Local Spatial Modeling Apply Pointwise convolution (CVC) yields intermediate feature maps after channel blending. This enables cross-channel semantic interaction.
[0116] Intermediate feature map after local spatial modeling Apply Pointwise convolution yields intermediate feature maps after channel blending. The calculation process is as follows:
[0117]
[0118] in, This is an intermediate feature map after channel mixing; for Pointwise convolution is used to recombine and fuse responses from different channels to achieve cross-channel semantic interaction. Intermediate feature map after modeling the local space; Indicates the channel expansion ratio; For the number of channels, This represents the height of the target spatial dimension, which corresponds to the spatial height of the first-level feature map. This represents the width of the target space, which corresponds to the spatial width of the first-level feature map.
[0119] In this embodiment, the intermediate feature map after channel mixing The number of channels and the intermediate feature map after local spatial modeling Consistent, still Intermediate feature map after channel mixing The number of channels can be set to a preset value as needed to adapt to the subsequent channel restoration operation of S604.
[0120] S604, Channel Restoration: via Convolution blends the intermediate feature maps of the channels descend The channel yields the intermediate feature map after channel reconstruction. ;
[0121] Intermediate feature map after channel mixing Apply Convolution yields intermediate feature maps after channel restoration. The calculation process is as follows:
[0122]
[0123] in, express Convolution operation; This represents the intermediate feature map after channel restoration; Indicates the channel expansion ratio; Indicates the number of channels; This represents the height of the target spatial dimension, which corresponds to the spatial height of the first-level feature map. This represents the width of the target space, which corresponds to the spatial width of the first-level feature map. and These represent the intermediate feature maps after channel reconstruction. Height and width in spatial dimensions.
[0124] In this step, and These correspond to the height and width of the target space dimensions after S4 alignment, respectively. Channel restoration is used to map the extended channel representations within the enhancement module back to channel dimensions consistent with the fusion features, enabling subsequent channel attention and residual fusion.
[0125] S605, Channel Recalibration: Re-calibrating the intermediate feature map after channel restoration. Apply channel attention (e.g., efficient channel attention ECA) to obtain weighted feature maps after channel recalibration. It is used to adaptively emphasize channels with higher information content.
[0126] like Figure 6 The diagram shown is a schematic of the channel attention module of the present invention, which specifically includes global pooling, one-dimensional convolution, sigmoid mapping, and channel weighting.
[0127] (1) Global average pooling: Let the intermediate feature map after channel restoration be... The channel description vector is obtained by global average pooling. ,in, The Each component is denoted as The calculation process is as follows:
[0128]
[0129] in, This represents the intermediate feature map after channel reconstruction. The The channel description value is obtained by performing global average pooling on all spatial locations for each channel; Indicates the number of channels. and These represent the intermediate feature maps after channel reconstruction. Height and width in spatial dimensions; This represents the intermediate feature map obtained after channel restoration processing; This represents the intermediate feature map after channel reconstruction. Channel index, This represents the intermediate feature map after channel reconstruction. Position index in the height direction, This represents the intermediate feature map after channel reconstruction. Position index in the width direction.
[0130] (2) One-dimensional convolution: describing the channel vector One-dimensional convolution is obtained ;
[0131]
[0132] in, Represents the channel description vector The channel response vector obtained after applying a one-dimensional convolution. This represents a one-dimensional convolution operation performed along the channel dimension, used to model the local interaction relationships between adjacent channels.
[0133] (3) Sigmoid mapping: to obtain channel weights ,in, For the Sigmoid function; Represents the channel description vector The channel response vector obtained after applying a one-dimensional convolution.
[0134] (4) Channel weighting: based on channel weights Intermediate feature map after channel restoration The weighted feature map after channel recalibration is obtained by weighting each channel sequentially. ;
[0135]
[0136] in, This is the weighted feature map after channel recalibration; For the first The weight of each channel, This represents the intermediate feature map after channel reconstruction. Channel index, This represents the intermediate feature map after channel reconstruction. Position index in the height direction, This represents the intermediate feature map after channel reconstruction. Position index in the width direction.
[0137] like Figure 5 As shown, after the efficient channel attention ECA (Efficient Channel Attention) performs weighted processing on the channel-reconstructed features, a random deactivation process needs to be applied to the weighted features. Specifically, let the weighted feature map after channel recalibration be... The feature map after random deactivation is denoted as The calculation process can be expressed as follows:
[0138]
[0139] in, This indicates a random deactivation operation, used to determine the deactivation probability. Randomly set some feature responses in the input features to zero; Indicates the probability of random inactivation; This represents the weighted feature map after channel recalibration; This represents the feature map after random deactivation.
[0140] Random deactivation can be implemented using a random mask, the calculation process of which can be expressed as follows:
[0141]
[0142] in, This represents the feature map after random deactivation. This represents element-wise multiplication; This represents the weighted feature map after channel recalibration. A random mask corresponding to the shape, where elements are used to characterize whether the feature response at the corresponding location is preserved. This represents the weighted feature map after channel recalibration.
[0143] The random deactivation step is set after efficient channel attention to reduce excessive dependencies between features during training.
[0144] S606, Residual Fusion: Output .
[0145] in, This represents the weighted feature map after channel recalibration; This represents the enhanced feature map output by the post-fusion feature enhancement module; This indicates the fusion feature.
[0146] Optional parameter: Channel expansion ratio (For example ); depthwise convolution kernel size (in, Indicates based on the number of channels Determine the size of the one-dimensional convolution kernel The mapping function is used to adaptively determine the kernel size as the number of channels changes, for example... or The attention module can employ lightweight channel attention structures such as efficient channel attention (ECA). Figure 6 middle, This represents the input feature map of the channel attention module; This represents the output feature map after channel attention weighting.
[0147] S7. Input the enhanced features into the classification layer for classification prediction and output segmentation logits;
[0148] The enhanced feature map output by the post-fusion feature enhancement module Input classification layer (e.g.) The convolutional process outputs segmentation logits, which yield the semantic segmentation result.
[0149] exist Figure 4 In this model, the classification layer is implemented using a multilayer perceptron. The multilayer perceptron acts on the channel vectors corresponding to each spatial location to perform category mapping on the enhanced feature map, functionally corresponding to... Convolutional classification layer. Specifically, the enhanced feature map output by the post-fusion feature enhancement module is... Then the logits are denoted as The calculation process is as follows:
[0150]
[0151] in, This represents the classification mapping module, used to process the enhanced feature map output by the post-fusion feature enhancement module. The channel characteristics at each spatial location are determined by Mapped to the number of semantic categories ; This indicates the segmentation of logits; This represents the enhanced feature map output by the post-fusion feature enhancement module; Indicates the number of channels; This represents the height of the target spatial dimension, which corresponds to the spatial height of the first-level feature map. This represents the width of the target space, which corresponds to the spatial width of the first-level feature map. This represents the number of semantic categories. The corresponding semantic segmentation result can be obtained based on the segmentation logits.
[0152] In this step, and These correspond to the height and width of the target space after S4 alignment, respectively.
[0153] like Figure 3 The diagram shows the existing SegFormer decoding structure, which includes projection, upsampling, splicing, fusion, and classification. This differs from the traditional SegFormer decoder which directly classifies components after fusion. Figure 4 As shown, this invention introduces a post-fusion enhancement module between the fusion convolutional and classification layers, enabling the decoder to achieve local refinement after fusion. This difference represents an improvement at the network structure level and can be used to enhance semantic consistency in boundary neighborhoods and the recoverability of small objects.
[0154] In this embodiment, the specific implementation process is described in pseudocode as follows:
[0155] Input: Remote sensing image encoder ; Fusion Convolution Mapping process of the post-fusion feature enhancement module Classifier .
[0156] Output: logits .
[0157] 1) ;
[0158] 2) For each : / / Convolutional unified channel is ;
[0159] 3) To : / / Bilinear interpolation aligned to a 1 / 4 scale;
[0160] 4) / / Channel dimension splicing;
[0161] 5) / / Convolutional fusion;
[0162] 6) / / Post-fusion enhancement;
[0163] 7)
[0164] 8) Return ;
[0165] in, This indicates an upsampling operation, which is implemented using interpolation methods, such as bilinear interpolation. Features of fusion; Features of splicing; This represents the enhanced feature map output by the post-fusion feature enhancement module; Indicates the first Each scale feature map is processed Channel-unified features obtained after channel-unified mapping of convolution; for , In subsequent steps, an upsampling operation is performed to... Align to the same spatial scale. express Convolution operation; This indicates a splicing operation along the channel dimension; For the first Feature maps at various scales.
[0166] In this embodiment, the pseudocode for the mapping process Enhance (PFEB) of the post-fusion feature enhancement module is as follows:
[0167] Input: Fusion features .
[0168] Parameter: Channel expansion ratio Depth convolution kernel size ( (for odd numbers); attention module (e.g., ECA); residual switch .
[0169] 1) Channel expansion: ,in, Indicates the use of channel expansion Convolution operation has the following output channels: . This represents the intermediate feature map after channel expansion.
[0170] 2) Local spatial mixing: ,in, express Depthwise convolution is a channel-wise convolution operation. This represents an intermediate feature map after local spatial modeling.
[0171] 3) Channel mixing: ,in, express Pointwise convolution is used to achieve cross-channel information interaction. This is an intermediate feature map after channel mixing.
[0172] 4) Channel restoration: ,in, Indicates the channel restoration Convolution operation has the following output channels: . This represents the intermediate feature map obtained after channel restoration processing.
[0173] 5) Channel attention: ,in, This is the weighted feature map after channel recalibration.
[0174] 6) Residual fusion: If If true, then ;otherwise .
[0175] 7) Output: Enhanced feature map output by the fusion feature enhancement module after return. .
[0176] The above convolution operations can be paired with normalization (such as BatchNorm) and activation functions (such as ReLU function, GELU function), but are not limited to these.
[0177] like Figure 7 As shown, from left to right, the images are: input image, ground truth annotation, prediction results from the contrast method (UPerNet+Swin-T), prediction results from the baseline method (SegFormer+MiT-B2), and prediction results from the method of this invention (post-fusion enhancement). The image also shows the background, bare land, buildings, woodland, roads, farmland, and water bodies.
[0178] like Figure 8 As shown, from left to right, the images are: input image, ground truth annotation, prediction results from the contrast method (PSPNet+ResNet-101), prediction results from the baseline method (SegFormer+MiT-B2), and prediction results from the method of this invention (post-fusion enhancement). The image also shows impermeable surfaces, trees, buildings, cars, low vegetation, and debris.
[0179] In this embodiment, the present invention can be implemented based on the PyTorch and MMSegmentation frameworks, using SegFormer MiT-B0 or MiT-B2 as the encoder backbone. During training, settings such as uniform cropping size (e.g., 512×512), AdamW optimizer, learning rate warm-up + polynomial decay can be adopted. PyTorch is an open-source deep learning framework; MMSegmentation is an image segmentation framework in OpenMMLab; SegFormer is a Transformer architecture for semantic segmentation.
[0180] This invention may also include, but is not limited to, the following extended embodiments:
[0181] (1) Attention modules are replaceable: Channel attention is not limited to ECA, but can be replaced by channel attention mechanisms such as SE (Squeeze-and-Excitation), CBAM (Convolutional Block Attention Module), coordinate attention, or other channel / spatial attention structures, as long as adaptive recalibration of the fused features is achieved.
[0182] (2) Local modeling operators can be replaced: depthwise convolution can be replaced by local modeling structures such as grouped convolution, dilated convolution or large kernel convolution to enhance the local expression of boundary neighborhood and slender structure.
[0183] (3) The insertion position is expandable: the enhancement module can be placed not only after fusion and before classification, but also in series / parallel at multiple positions before multi-scale splicing or in the decoding stage to achieve phased refinement.
[0184] To verify the effectiveness of the technical solution of the present invention, this embodiment conducts a comparative experiment on a publicly available remote sensing semantic segmentation dataset and uses three random seeds (0, 7, 42) for repeated training. The results are given as mean ± standard deviation.
[0185] On the LoveDA dataset, when using the MiT-B0 backbone, the mIoU (mean crossover ratio) of the proposed solution is 51.60±0.40, which is higher than the 50.50±0.83 of the SegFormer baseline; when using the MiT-B2 backbone, the mIoU of the proposed solution is 53.82±0.31, which is higher than the 52.52±0.08 of the SegFormer baseline.
[0186] On the ISPRS Vaihingen dataset, when using the MiT-B0 backbone, the mIoU of the proposed solution improved from 72.06±0.10 to 72.82±0.23; when using the MiT-B2 backbone, the mIoU of the proposed solution improved from 74.01±0.12 to 74.84±0.41.
[0187] The above comparison shows that the present invention achieves consistent performance improvements under different datasets and different backbone capacity settings.
[0188] Under the unified boundary and scale-sensitive evaluation settings (boundary_tol_ratio = 0.002, trimap_band_px = 2, size_bins = (1024, 9216), ignore_index = 255, where 255 represents the ignored label value that is not included in the evaluation statistics):
[0189] This invention achieves improvements in Trimap mIoU (mean intersection-union ratio) and small target metrics: LoveDA (MiT-B2) Trimap mIoU increased from 26.30±0.12 to 27.89±0.79; Vaihingen (MiT-B2) Trimap mIoU increased from 59.27±0.41 to 60.23±0.16, and small target mIoU increased from 48.11±0.83 to 49.41±0.97, and small target mRecall increased from 61.85±0.92 to 63.01±0.72. These results support the invention's claims to improve the semantic consistency of boundary neighborhoods and the recoverability of small targets.
[0190] Furthermore, efficiency evaluation results show that the proposed solution introduces a 0.53M parameter increment and increases 8.7G FLOPs (floating-point operations) under both MiT-B0 and MiT-B2 backbones. In terms of throughput, MiT-B2 reduces from 24.1 FPS (frames per second) to 21.1 FPS on LoveDA and from 67.7 FPS to 64.9 FPS on Vaihingen, indicating that while achieving improved accuracy, the computational overhead remains within a controllable range.
[0191] This invention enhances the expressive power of multi-scale fused features for local spatial details (especially boundaries and small objects) while maintaining the lightweight nature of the Transformer semantic segmentation model decoder. It also enables learnable refinement of fused features during the decoding stage without significantly increasing the number of parameters or computational complexity, and avoids excessive perturbation of fused features leading to training instability. Furthermore, this invention allows the enhancement module to be pluggable, facilitating its use across different backbone network capacities (such as MiT-B0 and MiT-B2).
[0192] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A post-fusion feature enhancement decoding method for semantic segmentation of remote sensing images, characterized in that, Includes the following steps: S1. Input remote sensing imagery; S2. Multi-scale features are extracted via the Transformer encoder. ; S3. Perform channel-unified mapping on features at each scale through the decoder to unify the channel dimension; S4. Perform upsampling on the features after unified mapping through the S3 channel, and align the upsampling to the same spatial scale. S5. The outputs of S4 are concatenated along the channel dimension and fused features are obtained through fusion convolution. S6. Construct a post-fusion feature enhancement module to obtain enhanced features through local modeling, channel recalibration, and residual fusion. S601, Channel Expansion: via Convolution transforms the channels from Expand to The intermediate feature map after channel expansion is obtained. ; S602, Local Spatial Blending: Intermediate Feature Map After Channel Expansion Apply Depth-wise convolution yields intermediate feature maps after local spatial modeling. , used to capture local boundaries and texture context; S603, Channel Blending: Intermediate Feature Map After Local Spatial Modeling Apply Pointwise convolution is performed to obtain intermediate feature maps after channel blending. To achieve cross-channel semantic interaction; S604, Channel Restoration: via Convolution blends the intermediate feature maps of the channels descend The channel yields the intermediate feature map after channel reconstruction. ; S605, Channel Recalibration: Re-calibrating the intermediate feature map after channel restoration. Apply channel attention to obtain the weighted feature map after channel recalibration. This is used to adaptively emphasize channels with higher information content. In S605, the channel attention module is an efficient channel attention ECA module, which specifically includes global pooling, one-dimensional convolution, sigmoid mapping, and channel weighting. Intermediate feature map after channel restoration Global average pooling is performed to obtain the channel description vector; one-dimensional convolution is performed on the channel description vector and the channel weights are obtained by passing the Sigmoid function; The intermediate feature map after channel reconstruction based on channel weights Perform channel-by-channel weighting to obtain the weighted feature map after channel recalibration. ; S606, Residual Fusion: Output ; in, This represents the enhanced feature map output by the post-fusion feature enhancement module; Indicates fusion characteristics; This represents the weighted feature map after channel recalibration; S7. Input the enhanced features into the classification layer for classification prediction and output the segmentation logits.
2. The post-fusion feature enhancement decoding method for semantic segmentation of remote sensing images according to claim 1, characterized in that, In S1, the specific implementation details are as follows: Remote sensing images are satellite remote sensing images or aerial remote sensing images. Remote sensing images are stored in raster format and include multiple spectral bands, including visible light bands and near-infrared bands. Remote sensing images are derived from publicly available remote sensing data, commercial remote sensing data, and self-acquired remote sensing data.
3. The post-fusion feature enhancement decoding method for semantic segmentation of remote sensing images according to claim 1, characterized in that, In S3, the specific implementation details are as follows: Perform channel-unified mapping on features at all scales to unify the number of channels. Let the first The feature map at each scale is Then the first The feature map obtained after channel unification mapping of the feature maps at each scale is: The calculation process is as follows: ； in, An index representing the feature scale / level; express Convolution operation.
4. The post-fusion feature enhancement decoding method for semantic segmentation of remote sensing images according to claim 1, characterized in that, In S4, the specific implementation details are as follows: The first The feature map obtained after channel unification mapping of feature maps at each scale. Upsampling to target spatial size Alignment features are obtained. The calculation process is as follows: ； in, Indicates an upsampling operation; Indicates the height of the target space; Indicates the width of the target space.
5. The post-fusion feature enhancement decoding method for semantic segmentation of remote sensing images according to claim 1, characterized in that, In S5, the specific implementation details are as follows: Alignment features after S4 processing By stitching along the channel dimension, the stitching feature is obtained. The calculation process is as follows: ； in, This indicates a splicing operation along the channel dimension; Subsequently, the splicing features were analyzed. Apply Fusion convolutions are used to perform channel fusion to obtain fused features. The calculation process is as follows: ； in, express Convolution operation.
6. The post-fusion feature enhancement decoding method for semantic segmentation of remote sensing images according to claim 1, characterized in that, Intermediate feature map after channel restoration The specific calculation process for obtaining the channel description vector by performing global average pooling is as follows: ； in, This represents the intermediate feature map after channel reconstruction. The The channel description value is obtained by performing global average pooling on all spatial locations for each channel; and These represent the intermediate feature maps after channel reconstruction. Height and width in spatial dimensions; This represents the intermediate feature map after channel restoration; This represents the intermediate feature map after channel reconstruction. Channel index, This represents the intermediate feature map after channel reconstruction. Position index in the height direction, This represents the intermediate feature map after channel reconstruction. Position index in the width direction.