A semantic segmentation model for satellite remote sensing images coupled with a frequency domain decoder

Abstract

This invention relates to a semantic segmentation model for satellite remote sensing images coupled with a frequency domain information decoder. Based on the Mask2Former framework, this invention proposes a coupled frequency domain information decoder (CFDD). Unlike existing methods that limit frequency domain processing to the encoder, the coupled frequency domain decoder directly integrates frequency domain analysis into the decoder stage through two complementary paths: (1) the original pixel decoder (OPD) already present in Mask2Former, used to robustly extract spatial features; (2) the novel multi-scale frequency enhancement chain (MSFEC) decoder proposed in this invention, used to systematically process frequency domain features. The multi-scale frequency enhancement chain decoder realizes frequency domain feature extraction through three core modules: frequency band decomposition and processing module (FBDP), frequency band attention mechanism module (FBAM), and gated residual fusion module (GRF).

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing methods, specifically to a method for semantic segmentation models of high-resolution satellite remote sensing images that combines frequency domain information to develop a decoder structure. Background Technology

[0002] Traditional methods cannot utilize the rich spatial semantic information inherent within images, relying only on local spectral, texture, and edge features, leading to numerous misclassifications. With the development of deep learning, numerous models and methods have emerged; however, most existing methods focus on utilizing spatial domain information, emphasizing spatial feature extraction and fusion, while neglecting frequency domain features. Therefore, the accuracy of deep learning semantic segmentation is limited in cases involving highly heterogeneous land cover within categories or requiring accurate boundary localization of land classes. Existing frequency-domain-based improvements have the following shortcomings: FcaNet, with its simple scalar compression, easily loses key features; the MFFNet model, with its fixed-position decomposition, lacks adaptability. The SFFNet (Spatial and Frequency Domain Fusion Network) framework proposes a sequential feature enhancement method using a two-stage network structure. First, a spatial domain network extracts features and performs preliminary segmentation of the input image; then, frequency domain information is introduced in the second stage. This method obtains multi-scale frequency information through frequency domain transformation and utilizes a cross-scale alignment mechanism to fuse frequency and spatial features, thereby improving segmentation accuracy. However, in this technical solution, frequency domain features, as a subsequent supplement to spatial features, exhibit a clear sequential processing flow. If there is coding loss in the spatial features of the first stage, the fusion of frequency domain information in the second stage cannot compensate for it. Overall, the participation of frequency domain information in the decoding stage is relatively low, and a decoding structure parallel to the spatial decoding path is not formed.

[0003] In summary, in existing methods for fusing the frequency domain, frequency domain processing is mostly used in the encoder structure of deep learning models and is not fully coupled to the decoder stage of deep learning models; moreover, there is a lack of attention and gating mechanisms for the frequency domain, which affects the network's utilization of frequency features. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a semantic segmentation model for high-resolution satellite remote sensing images with a coupled frequency domain information decoder structure.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: The model proposed in this invention employs a decoder approach that couples frequency information during the design of the decoder module. Within the decoder, one path retains the original pixel decoder from the Mask2former model, while the other path utilizes the multi-scale frequency enhancement chain (MSFEC) decoder proposed in the patent.

[0006] In the semantic segmentation model of this invention, the encoder still generates multi-scale spatial features step by step through multiple Swin-tiny modules to obtain information representations containing different spatial scales and semantic levels, and outputs multi-scale encoded features as input to the subsequent coupled frequency information decoder.

[0007] This patent proposes and employs a coupled frequency domain information decoder during the decoding process. The first branch is the original pixel decoder, which processes the spatial domain features output by the feature encoding module.

[0008] The second branch is the multi-scale frequency enhancement chain decoder, which performs multi-scale frequency conversion and processing on the input features through the frequency domain decomposition and processing module, adaptively assigns weights to the enhanced frequency components and performs weighted fusion using the frequency band attention mechanism module, and adaptively controls the enhancement intensity of the frequency domain enhanced features based on the difference between the original features and the frequency domain features using the gated residual fusion module, so as to achieve the effect of suppressing invalid or noise enhancement while preserving the original key semantic information.

[0009] Finally, the proposed decoder with coupled frequency domain information will weighted couple the data output from the original pixel decoder and the multi-scale frequency enhancement chain decoder.

[0010] Specifically, the processing procedure of the multi-scale frequency enhancement chain decoder is as follows: Step 1.1: The frequency band decomposition and enhancement module transforms the input features from the spatial domain to the frequency domain to obtain different frequency components of the multi-scale coded features. Then, based on the frequency domain intensity, the frequency components are equally divided into frequency bands (generally recommended to be 3). Different frequency bands correspond to frequency domain information at different scales. After each frequency band is restored to the spatial domain through inverse frequency domain transformation, it is processed by the frequency band processing unit to highlight structural or boundary information that is more discriminative for the current frequency band, resulting in enhanced frequency band features. Step 1.2: Use the frequency band attention mechanism module to perform weighted fusion of the enhanced frequency band features; Step 1.3: The gated residual fusion module adaptively controls the introduction intensity of frequency domain enhancement features based on the difference between the original features and the fused frequency domain features. While preserving the original key semantic information, it suppresses invalid or noise enhancement to obtain the output features.

[0011] Furthermore, in step 1.3, the frequency band weighted fusion method is to multiply each frequency band by the learnable weight of the corresponding frequency band, and then sum the results of multiplication of all frequency bands to obtain the result.

[0012] Furthermore, in steps 1.1-1.3, a complete multi-scale frequency enhancement chain (MSFEC) is formed through multi-scale frequency band decomposition and processing, frequency band attention mechanism, and gated residual fusion.

[0013] Furthermore, the coupling formula of the dual-path coupling module is as follows: F_out = α · F_spatial + β · F_freq in, F_spatial This represents the output characteristics of the spatial domain decoding path. F_freq This represents the output characteristics of the frequency domain decoding path. α and β These are the learnable weight parameters for the spatial domain decoding path and the frequency domain decoding path, respectively, used to balance the contributions of the two decoding paths to the final segmentation result.

[0014] Furthermore, the learnable weights are obtained through multiple training sessions, with the training set consisting of labeled satellite remote sensing images.

[0015] The beneficial effects of this invention are as follows: Unlike existing methods that limit frequency domain processing to the encoder and spatial / frequency domain information distribution processing, this invention directly integrates frequency domain analysis into the decoder stage, enabling frequency domain features to directly participate in the detail recovery process. The decoder stage is closer to the final output, and frequency domain enhancement directly affects the reconstruction of boundary details.

[0016] This invention designs a systematic processing flow for frequency domain features. By integrating frequency band decomposition and processing, attention weighting and gating mechanisms, it achieves refined modeling of frequency components at different scales, rather than simply utilizing and splicing sequential features.

[0017] This invention preserves the spatial domain semantic understanding capability of the original pixel decoder path in the Mask2former model, and enhances the boundary detail capability in the frequency domain through the multi-scale frequency enhancement chain (MSFEC) path. Furthermore, the two paths are adaptively fused through learnable parameters to reconstruct the segmentation result. Therefore, this invention proposes a coupled frequency decoder that achieves an optimal balance between semantic understanding and boundary accuracy.

[0018] By introducing frequency domain processing into the decoder, the coupled frequency decoder significantly improves the boundary segmentation accuracy, especially the ability to identify small targets and complex boundaries, and the boundary F1 score is expected to be improved.

[0019] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the model framework of the present invention. Detailed Implementation

[0021] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0022] like Figure 1 As shown, this invention proposes a coupled frequency domain decoder based on the Mask2Former framework. Unlike existing methods that limit frequency domain processing to the encoder, the coupled frequency domain decoder integrates frequency domain analysis directly into the decoder stage through two complementary paths: (1) the original pixel decoder (OPD) already present in Mask2Former, used for robust extraction of spatial features; (2) the novel multi-scale frequency enhancement chain (MSFEC) decoder proposed in this invention, used for systematic frequency domain feature processing. The multi-scale frequency enhancement chain decoder realizes frequency domain feature extraction through three core modules: frequency band decomposition and enhancement module (FBDE), frequency band attention mechanism module (FBAM), and gated residual fusion module (GRF).

[0023] The algorithm flow of this invention is as follows: Feature encoding stage Multi-scale spatial features are generated step by step through four Swin-tiny modules to obtain information representations containing different spatial scales and semantic levels. The multi-scale features output by the feature extraction module are used as inputs for subsequent decoding stages to support further analysis of spatial structure and semantic information.

[0024] (II) Dual-path decoding stage Path 1 is the Original Pixel Decoder (OPD), which is responsible for spatial domain feature processing.

[0025] Path 2 is a multi-scale frequency enhancement chain (MSFEC) decoder, responsible for frequency domain feature processing. The specific process of Path 2 is as follows: ① Frequency domain decomposition and frequency band construction The decoding module first transforms the two input low-scale encoded features from the spatial domain to the frequency domain to obtain the frequency domain features of the low-scale spatial encoded features. This frequency domain transformation can employ the discrete Hartley transform, whose one-dimensional form can be expressed as: H( k ) = Σ x ( n ) · cas(2 πnk / N ) in, x ( n ) represents the input feature sequence. H ( k ) represents the corresponding frequency domain coefficients. kFor frequency index, N The sequence length is given. This transformation is only one implementation method; this invention uses the Discrete Hartley Transform as the frequency domain transformation form.

[0026] ② Frequency band division and processing Based on the intensity in the frequency domain, each scale is divided into three equal frequency bands, corresponding to low, medium, and high frequency bands respectively. Frequency band division can be achieved by normalizing the frequency domain intensity and constructing a frequency band mask. Different frequency bands correspond to structural information at different scales. After each frequency band is restored to the spatial domain through inverse frequency domain transformation, it is processed by a 3×3 convolution module with normalization and ReLU activation function to achieve feature alignment and facilitate subsequent processing, enabling subsequent attention fusion and threshold residual fusion.

[0027] ③ Frequency band attention weighted fusion Since the features processed by different frequency bands at each scale contribute differently to the image semantic segmentation task, a frequency band attention mechanism is further introduced to perform weighted fusion of features. The frequency band weighted fusion process can be represented as follows: F_freq = Σ w_b · F_b in, F_b Indicates the first b Features corresponding to frequency bands w_b These are the learnable weight parameters corresponding to this frequency band.

[0028] ④ Gated residual fusion mechanism To balance the relationship between the original spatial domain features and the frequency domain enhanced features, a gated residual fusion mechanism is introduced. This gated mechanism adaptively controls the intensity of the frequency domain enhanced features based on the differences between the original features and the frequency domain features, thus preserving the original key semantic information while suppressing ineffective or noisy enhancements.

[0029] Dual-path decoding stage After the pixel decoder and frequency decoder complete decoding simultaneously, the outputs of the two proposed coupled paths are combined using a weighted method to obtain the final segmentation result. This coupling process can be represented as follows: F_out = α · F_spatial + β · F_freq in, F_spatial This represents the output characteristics of the spatial domain decoding path. F_freq This represents the output characteristics of the frequency domain decoding path. α and β These are learnable weight parameters used to balance the contributions of the two decoding paths to the final segmentation result.

[0030] Based on the aforementioned technical solutions and their implementation mechanisms, this invention achieves significant technical results in high-resolution remote sensing image semantic segmentation tasks by introducing a frequency domain processing mechanism during the decoding stage and coordinating its integration with the spatial domain decoding path. Its beneficial effects can be reasonably explained through the following reasoning process.

[0031] 1. Improve the clarity and continuity of ground feature boundaries. Ground feature boundaries in images typically correspond to high-frequency variation regions. However, traditional segmentation methods relying solely on spatial domain decoding are susceptible to local noise and texture disturbances during pixel-by-pixel reconstruction, leading to blurred or broken boundaries. This invention introduces a frequency domain branch during the decoding stage, explicitly modeling frequency components at different scales, allowing high-frequency boundary information to directly participate in the decoding and reconstruction process. Since frequency domain analysis is insensitive to fluctuations in individual pixels, this design can provide stable constraints on boundary positions, theoretically effectively mitigating boundary blurring and breakage, and improving the clarity and continuity of ground feature boundaries.

[0032] 2. Enhance the segmentation stability of land features with high intraclass heterogeneity. For land cover types with high intra-class heterogeneity, such as arable land, there are often significant differences in crop species, growth stages, and textural structures. Traditional spatial domain segmentation methods easily misjudge these local differences as class differences, leading to fragmentation within the same class. The frequency domain branch of this invention provides structural consistency constraints for the decoding process through the structural statistical characteristics reflected by frequency components. Based on semantic decoding to determine class affiliation, frequency domain information further maintains the spatial connectivity and integrity of the same class, thereby reducing the risk of missegmentation caused by local heterogeneity.

[0033] 3. Achieving a synergistic balance between semantic consistency and boundary precision In semantic segmentation tasks, there is often a trade-off between semantic consistency and boundary accuracy. This invention maintains stable semantic understanding by preserving the spatial domain decoding path, while introducing a frequency domain decoding path to enhance boundary and detail representation capabilities. Furthermore, it adaptively fuses these two types of features using learnable parameters. This design allows for the introduction of effective frequency domain enhancements while suppressing ineffective or noisy enhancements, thereby improving the boundary accuracy of the segmentation results without compromising the original semantic consistency.

[0034] 4. Improve the robustness and generalization ability of the model. Remote sensing images from different regions and under different imaging conditions often exhibit significant differences in texture and appearance. This invention introduces frequency domain modeling during the decoding stage, utilizing the scale statistics of frequency components to reduce the dependence on specific textures and appearances to some extent. Simultaneously, the fusion process of spatial and frequency domain features is adaptive, enabling the model to dynamically adjust the contribution ratio of the two types of features based on data characteristics. Therefore, this invention demonstrates better robustness and generalization potential in cross-regional and cross-scenario applications. Example

[0035] 1. Urban scene segmentation in the ISPRS Potsdam dataset Scene description: High-resolution aerial image of Potsdam, Germany, with a spatial resolution of 5cm, RGB three-band, and 6 land cover types.

[0036] Key Achievements: The proposed method achieves state-of-the-art performance on the Potsdam dataset, with an mIoU of 78.47% and an mBF of 72.14%, where the mBF is 5.27% higher than Mask2Former. Specifically, the segmentation performance for the "car" class is significantly improved—the boundary F1 score reaches 92.94%, which is better than Mask2Former's 88.25%, fully demonstrating the advantages of the proposed method in segmenting small object boundaries.

[0037] 2. Splitting the ISPRS Vaihingen dataset The method proposed in this invention achieves an mIoU of 76.33% on the ISPRS Vaihingen dataset, which is 1% higher than the baseline model Mask2Former.

[0038] In the boundary F1 score evaluation, the method proposed in the patent outperformed other models. Its mBF (mBF) reached 71.35%, demonstrating outstanding performance in key categories of urban scenarios. For impermeable surfaces, the BF reached 82.69% (Mask2Former: 81.52%), improving the accuracy of urban road networks; for buildings, the BF score reached 83.89% (Mask2Former: 82.31%), ensuring the integrity of building edges.

[0039] 3. Segmentation effect of Gaofen-2 (GF-2) satellite imagery Using Gaofen-2 satellite Class 2A data with a spatial resolution of 0.8m, land cover was classified into seven categories (cultivated land, artificial land, bare land, forest land, grassland, roads, and water surfaces), with 3046 samples. The overall accuracy across all categories was mIoU = 80.78%, an improvement of 1.52% compared to the baseline model Mask2former. Specifically, the IoU for cultivated land was improved by 2.40% compared to the baseline model Mask2former, and for bare land by 2.27%. This demonstrates the effectiveness of the method.

[0040] 3. Application results of the independent test study area Three independent test study areas were selected in Hunan Province, each containing different land cover types. Details are as follows: Study Area 1 (mainly woodland, 23.76 km²): mIoU 67.07%, surpassing all comparison methods and improving by 1.20% compared to the baseline model Mask2former.

[0041] Study Area 2 (mainly urban area, 16.81 km²): mIoU 71.03%, surpassing all comparison methods and improving by 1.10% compared to the baseline model Mask2former.

[0042] Study Area 3 (farmland irrigation network area, 12.74 km²): mIoU 71.03%, surpassing all comparison methods and improving by 1.50% compared to the baseline model Mask2former.

[0043] Model evaluation metrics In this embodiment, IoU, mIoU, boundary F1 score (BF), and mean boundary F1 score (mBF) are used.

[0044] IoUi: Measures the degree of overlap between the predicted result and the actual result for the i-th class. mIoU: The average IoU across all categories Boundary Precision (BF): This evaluates boundary detection performance by comprehensively considering boundary precision and boundary recall, while mBF (mBF) achieves an overall evaluation by averaging the BF across all classes. The mathematical expressions for each metric are as follows: ; in N Indicates the number of categories. For the first... i kind, TP i Indicates a true positive result. FP i This indicates a false positive. FN i Indicates a false negative. TN i Indicates the first iTrue negative of the class. In boundary assessment, P b Indicates the boundary accuracy. R b Indicates the edge recall rate; TP b This represents the number of true positives for the boundary pixels. FP b This indicates the number of false positives among the boundary pixels. FN b Indicates the number of false negatives among boundary pixels; BF i This represents the boundary F1 score.

[0045] The above description provides examples of the preferred embodiments of the present invention. Parts not detailed herein are common knowledge to those skilled in the art. The scope of protection of the present invention is determined by the claims. Any equivalent modifications based on the technical teachings of the present invention are also within the scope of protection of the present invention.

Claims

1. A semantic segmentation model for satellite remote sensing images coupled with a frequency domain information decoder, characterized in that, It includes a feature encoding module, a raw pixel decoder, a multi-scale frequency enhancement chain decoder, and a dual-path coupling module. The feature encoding module is used to generate multi-scale spatial features step by step through multiple Swin-tiny modules to obtain information representations containing different spatial scales and semantic levels, and the output multi-scale features are used as inputs for subsequent decoders. The raw pixel decoder is used to process the spatial domain features output by the feature encoder; The multi-scale frequency enhancement chain decoder performs multi-scale frequency conversion and processing on the input features through the "frequency band decomposition and processing module," and adaptively assigns weights to the enhanced frequency components and performs weighted fusion using the "frequency band attention mechanism module." The "gated residual fusion module" adaptively adjusts the intensity of frequency domain information enhancement based on the difference between the original features and the frequency domain features, suppressing noise while preserving the original key semantic information. The dual-path coupling module performs weighted coupling of the data output from the original pixel decoder and the multi-scale frequency enhancement chain decoder.

2. The semantic segmentation model for satellite remote sensing images using a decoder coupled with frequency domain information according to claim 1, characterized in that, The processing procedure of the multi-scale frequency enhancement chain decoder is as follows: Step 1.1: The frequency band decomposition and processing module transforms the input features from the spatial domain to the frequency domain to obtain frequency domain representations of different frequency components of different features. After obtaining the frequency domain representations of different frequency components, the frequency components are divided into multiple frequency bands according to the frequency domain intensity. Different frequency bands correspond to frequency domain information of different scales. After being restored to the spatial domain by inverse frequency domain transformation, they are enhanced by frequency band processing to obtain the enhanced features of each frequency band. Step 1.2: Use the frequency band attention mechanism module to perform weighted fusion of the enhanced frequency band features; Step 1.3: The gated residual fusion module adaptively controls the introduction intensity of frequency domain enhancement features based on the difference between the original features and the fused frequency domain features. While preserving the original key semantic information, it suppresses invalid or noise enhancement to obtain the output features.

3. The semantic segmentation model for satellite remote sensing images with coupled frequency domain information decoder according to claim 2, characterized in that, In steps 1.1-1.3, a complete multi-scale frequency enhancement chain is formed through multi-scale frequency band decomposition and processing, frequency band attention mechanism, and gated residual fusion.

4. The semantic segmentation model for satellite remote sensing images with a coupled frequency domain decoder according to claim 1, characterized in that, The coupling formula for the dual-path coupling module is: F_out = α · F_spatial + β · F_freq in, F_spatial This represents the output characteristics of the spatial domain decoding path. F_freq This represents the output characteristics of the frequency domain decoding path. α and β These are the learnable weight parameters for the spatial domain decoding path and the frequency domain decoding path, respectively, used to balance the contributions of the two decoding paths to the final segmentation result.

5. A semantic segmentation model for satellite remote sensing images with a coupled frequency domain decoder according to claim 3 or 4, characterized in that, The learnable weights are obtained through multiple training sessions, with the training set consisting of labeled satellite remote sensing images.

Images