Remote sensing image segmentation method and system based on lightweight UMFormer

Abstract

The present application relates to the technical field of remote sensing image processing, and specifically discloses a remote sensing image segmentation method, system and product based on a light-weight UMFormer. The present application is based on the light-weight and moderately accurate UMFormer, adopts a two-stage optimization strategy of first improving segmentation accuracy and then balancing light-weight efficiency, and realizes high-precision and light-weight segmentation of remote sensing images. The LEUMFormer model constructed adopts a DecoupleNet light-weight backbone network, suppresses channel redundancy, and realizes model light-weight on the basis of precision improvement. Through the MSAF module, linear complexity is realized to enhance and extract farmland multi-scale features and direction-sensitive structures. In the decoding stage, the SCIA module is introduced, the spatial structure and spectral channel features are dynamically fused through the double-branch parallel structure and adaptive gating. In addition, the BEH module is constructed to explicitly strengthen the subtle boundary at a very low computational cost, solving the problems of segmentation blur and fracture.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and in particular to a remote sensing image segmentation method and system based on the lightweight UMFormer. Background Technology

[0002] In recent years, high-resolution remote sensing image semantic segmentation has become a core supporting technology for smart agriculture, land monitoring, and ecological protection. Farmland scenes are characterized by clear plot boundaries, a high proportion of large homogeneous areas, sparse distribution of small-scale agricultural facilities, and susceptibility to light and weather interference, requiring models to simultaneously possess global context modeling capabilities, local detail capture capabilities, and efficient and lightweight inference capabilities.

[0003] Farmland segmentation places greater emphasis on small model size, high speed, accurate boundaries, and edge deployment capability. Current remote sensing image segmentation is shifting from "pursuing high accuracy" to "balancing high accuracy and lightweight design," primarily achieved through three types of models. The first type is segmentation models based on CNNs (such as UNet and DeepLab series); the second type is global modeling models represented by Transformers (such as Swin Transformer, SegFormer, and UNetFormer); and the third type is visual state-space models centered around Mamba (such as Vision Mamba and UMFormer), as illustrated in the Chinese patent application with publication number CN120388300A. UMFormer, as a lightweight baseline model, integrates CNN, Transformer, and Mamba structures, achieving a good balance between accuracy and efficiency in segmenting multiple datasets. It is currently the most suitable lightweight infrastructure for edge deployment, solving the common problems of weak global capabilities in CNNs, excessive weight in Transformers, and poor detail in early versions of Mamba, while also possessing engineering value through ease of improvement, deployment, and high efficiency.

[0004] However, while the UMFormer baseline model boasts lightweight advantages and moderate accuracy, it still suffers from adaptation limitations in farmland scenarios: the multi-scale attention module exhibits redundant computation, resulting in insufficient modeling of farmland strip structures and orientation-sensitive features; it lacks an adaptive balancing mechanism for spatial and channel attention, making it prone to misclassification of spectrally similar regions; and deep downsampling leads to the loss of boundary details and discontinuous plot edges. Furthermore, the model employs ResNet18 as its backbone network, which suffers from feature channel redundancy, resulting in incomplete lightweighting and further limiting inference speed improvements, making it difficult to meet the requirements of real-time deployment on edge devices.

[0005] Therefore, there is an urgent need to propose a remote sensing image segmentation method based on the lightweight UMFormer. This method should be based on the lightweight and moderately accurate UMFormer, prioritizing the improvement of segmentation accuracy without compromising its lightweight advantages, and then further reducing the computational load and parameter count of the model. Summary of the Invention

[0006] To address the problems existing in the prior art, this application provides a remote sensing image segmentation method and system based on lightweight UMFormer, which achieves high-precision and lightweight segmentation of remote sensing images.

[0007] This application provides a remote sensing image segmentation method based on lightweight UMFormer, including: Construct the LEUMFormer model, including a lightweight coding network, a perceptual fusion network, and a global-local decoding network; The backbone network of the lightweight coding network is DecoupleNet, which replaces the original ResNet18, and is used to input farmland scene image datasets into multi-level feature extraction units to extract features at different scales. The perceptual fusion network employs the MSAF module to fuse features at different scales and extract directional features; The global-local decoding network includes multiple levels of decoding units. The first-level decoding unit includes a decoder GL-VSSBlock, which decodes the output features of the MSAF module. The remaining multiple decoding units include a sequentially connected Spatial and Channel Adaptive Integration (SCIA) module, a decoder GL-VSS Block, and a Boundary Enhancement (BEH) module. The SCIA module is used to extract spatial attention features and channel attention features through parallel spatial attention branches and channel attention branches, respectively, and then adaptively fuse them. The BEH module is used to enhance and repair the boundary features of the fused features. The trained LEUMFormer model was used to segment images of farmland scenes; This implementation scheme constructs a farmland remote sensing segmentation model with higher accuracy, lighter weight, and clearer boundaries. A dedicated feature enhancement module is designed specifically for the characteristics of farmland scenes to strengthen plot boundary features, thereby improving segmentation accuracy and boundary continuity. Simultaneously, by replacing redundant backbone networks to eliminate channel redundancy, it further reduces computational load and parameter count, and increases inference speed while improving accuracy.

[0008] As a further optimization of the above scheme, in the LEUMFormer model, during the network forward pass, the last deep feature output by the lightweight encoder is fused with the three shallow features in the MSAF module, and the output of the MSAF module is used as the input of the first-level decoding unit; the first three-level features output by the multi-level feature extraction unit are passed to the parallel decoding unit of the same level through skip connections, and are concatenated with the output features of the previous level decoding unit, and used as the input of the current level decoding unit SCIA module for feature enhancement; the fused and enhanced features are input to the GL-VSS Block of the same level decoding unit for decoding and reconstruction, and its output is used as the input of the BEH module of the same level decoding unit.

[0009] This implementation scheme performs channel pruning on the output features of each stage of the encoder and optimizes the feature transfer method of skip connections, thereby reducing channel dimension and computational load while retaining key information.

[0010] As a further optimization of the above scheme, the MSAF module adopts a mechanism of multi-scale feature fusion first and orientation-aware enhancement later. Through the synergistic optimization of depthwise separable convolution and attention mechanism, it achieves an efficient balance between multi-scale feature fusion and orientation-sensitive feature extraction while maintaining linear computational complexity.

[0011] As a further optimization of the above scheme, the processing procedure of the MSAF module includes: It receives four feature maps at different scales from the input of a lightweight encoder, including three shallow feature maps and one deep feature map; After channel compression of the three shallow features using 1×1 convolution, preprocessing is performed, and the features are downsampled to the same spatial resolution as the deep features to obtain a four-layer single-scale feature map. The preprocessing includes ReLU function activation and batch normalization performed sequentially. After concatenating the features of the four single-scale feature maps, a multi-scale fused feature map is obtained by channel fusion through 1×1 convolution, which integrates shallow detail features and deep semantic features, providing a foundation for subsequent attention weighting. Multi-scale enhanced feature maps are obtained by enhancing multi-scale fused feature maps through depthwise separable convolution and attention mechanisms. By processing the multi-scale enhanced feature map with asymmetric convolution kernels of different scales, a directional enhanced feature map is obtained; After feature extraction from the orientation enhancement feature map and the multi-scale enhancement feature map, they are fused using a 1×1 convolution to obtain the initial fused feature map.

[0012] In this implementation scheme, during the acquisition of the four-layer single-scale feature maps, activation functions are used to introduce nonlinear feature transformations to enhance the model's expressive power, batch normalization is used to accelerate network training convergence, and 1×1 convolution is used to compress the number of feature channels to reduce computational overhead while retaining key feature information. Then, feature concatenation is performed to initially integrate multi-scale information, thereby promoting full interaction between low-level detailed information and high-level semantic information. Finally, 1×1 convolution is used to perform channel fusion and dimensionality adjustment on the concatenated high-dimensional features to ensure uniform feature map dimensions.

[0013] As a further optimization of the above scheme, the depthwise separable convolution and attention mechanism is processed as follows: By weighting the multi-scale fusion feature map in the channel dimension using CAM, a channel fusion feature map is obtained to highlight the discriminative channel response; Subsequently, three sets of deep separable convolutions with different receptive fields are used to extract multi-scale contextual features of the channel fusion feature map in parallel. The features are then fused by adding them element by element. The fused features are input into the SAM module and weighted in the spatial dimension to obtain the multi-scale enhanced feature map.

[0014] In this implementation, the MSAF module further introduces a collaborative design of multi-scale convolutional groups and attention mechanisms to adaptively enhance key region features. The channel attention weights generated by CAM reflect the importance of each channel to the farmland segmentation task, and key channel features are strengthened through weighted enhancement. Separable convolutions with different receptive fields are used in parallel to extract spatial features at different scales, which are then fused to achieve multi-scale spatial information complementarity. SAM enhances the spatial dimension of the fused feature map, highlighting key spatial regions (such as field ridges and ditch locations).

[0015] As a further optimization of the above scheme, the asymmetric convolution kernel includes three sets of paired horizontal convolution kernels and vertical convolution kernels, each set of horizontal convolution kernels and vertical convolution kernels having the same scale and being transposes of each other. The asymmetric convolution kernel is processed as follows: The multi-scale enhanced feature map is subjected to layer normalization. Then, orientation-related features are extracted by three sets of horizontal and vertical convolution kernels. After element-wise addition, channel fusion is performed using 1×1 convolution. The fused result is then residually connected to the multi-scale enhanced feature map to obtain the orientation-enhanced feature map, so as to maintain the stability of gradient propagation.

[0016] In this implementation, the MSAF module further introduces asymmetric convolution kernels in the horizontal and vertical directions to model orientation-sensitive features. Layer normalization is used to stabilize the input distribution of the directional convolutions. Asymmetric horizontal and vertical convolution kernels are used for horizontal and vertical features, respectively, such as horizontal field ridges and vertical ditches. Finally, channel fusion enhances the robustness of the directional features.

[0017] As a further optimization of the above solution, the SCIA module's processing procedure includes: Receive the spliced ​​feature map from the SCIA module; The concatenated feature maps are input into the parallel spatial attention branch and channel attention branch respectively to obtain spatial attention-enhanced feature maps and channel attention-enhanced feature maps; The two feature maps are dynamically fused using an adaptive fusion gating system: the two feature maps are concatenated and then passed through a 1×1 convolutional layer and a sigmoid activation function to generate fusion weights. The fusion weights are used to perform a weighted summation of the two feature maps to obtain an initial attention-enhanced feature map. The initial attention-enhanced feature map is then fused with the concatenated feature map through a residual connection to obtain the final attention-enhanced feature map.

[0018] In this implementation, the SCIA module adaptively balances attention weights in spatial and channel dimensions, thereby more effectively modeling direction-sensitive feature relationships in farmland scenes, embedding local information into global information, and supplementing the information that may be lost in the feature map after layers of processing.

[0019] As a further optimization of the above solution, the BEH module processes the following: Receive the input feature map from the BEH module; Edge responses are extracted from the input feature map through a boundary detection layer to obtain a boundary-enhanced feature map: In the boundary detection layer, a 3×3 convolution kernel is used to convolve the input feature map to capture feature gradient changes. The convolutional feature map is normalized to a boundary probability map between 0 and 1 by the Sigmoid activation function, which represents the confidence that each position belongs to the boundary. This allows the model to focus on boundary regions with drastic feature changes. The input feature map and the boundary probability map are fused along the channel dimension to obtain the boundary enhancement feature map. The boundary enhancement feature map is nonlinearly transformed by the boundary feature refinement layer to obtain the boundary refinement feature map. The boundary feature refinement layer adopts a depthwise separable convolution structure, which includes a 3×3 depthwise convolution, batch normalization, point convolution and SiLU activation function, ensuring the lightweight nature of the module, while effectively fusing boundary information and original features to generate the refined boundary enhancement features. Finally, the boundary refinement feature map is added to the input feature map through residual connection to serve as the final output feature map of the BEH module, thus realizing progressive feature enhancement.

[0020] In this implementation, the BEH module enhances the representation of boundary features during the decoding process through explicit edge detection and feature fusion mechanisms. In farmland scenes, boundary structures such as field ridges and ditches possess important semantic information, but these subtle boundary features are easily smoothed or lost during deep feature extraction. The BEH module enhances the response of boundary regions at the feature level with minimal computational cost, thereby improving the continuity and accuracy of boundary regions in the final segmentation result. This fusion method combines spatial boundary prior information with channel features to form a boundary-enhanced feature representation. The boundary probability map, as an additional guiding channel, provides clear geometric cues for subsequent feature refinement.

[0021] This application provides a remote sensing image segmentation system based on the lightweight UMFormer, including: The model building module is used to construct the LEUMFormer model, including a lightweight encoder, a multi-scale orientation-aware fusion module, and an enhanced global-local decoder. The backbone network of the lightweight encoder is DecoupleNet. The end of the lightweight encoder introduces a multi-scale attention fusion (MSAF) module, which explicitly models orientation features at different scales through multi-scale convolution and orientation-aware branches. The enhanced global-local decoder includes a spatial and channel integration and adaptive balancing (SCIA) module, which models spatial location dependency and channel feature dependency respectively through parallel spatial attention branches and channel attention branches, and introduces learnable weights to achieve adaptive fusion of the two. The enhanced global-local decoder also includes a boundary enhancement (BEH) module, which strengthens and repairs boundary features through feature interaction between the edge detection branch and the main segmentation branch. The segmentation module is used to segment farmland scene images using the trained model.

[0022] This application provides a computer program product including a computer program that, when run on a computer device, causes the computer device to execute a remote sensing image segmentation method based on lightweight UMFormer.

[0023] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: 1. This invention fully leverages the characteristics of farmland scenarios, such as clearly defined boundaries, significant spectral features, and the coexistence of large homogeneous areas and small-scale agricultural facilities. Through multi-scale feature fusion, orientation-sensitive structure modeling, adaptive weighting of spatial and channel features, and explicit boundary reinforcement, it significantly enhances the model's ability to express and discriminate key farmland features. Experimental data shows that on the LoveDA multi-category land cover dataset, the model's average intersection-over-union (mIoU) increased from 67.1% to 70.3%, a relative improvement of 3.2%. On the FGFD dedicated farmland segmentation dataset, mIoU increased from 85.6% to 86.2%, a relative improvement of 0.6%. The segmentation accuracy of key land cover categories such as agricultural land, roads, water bodies, field ridges, and ditches is comprehensively improved, effectively solving problems such as blurred boundaries, fragmentation, misclassification of spectrally similar areas, and missed detection of small-scale facilities. The segmentation results are more complete, continuous, and closely reflect the actual farmland distribution.

[0024] 2. This invention adopts an innovative technical approach that first enhances feature representation capabilities to ensure segmentation accuracy, and then performs lightweight network structure reconstruction to improve operational efficiency. This fundamentally avoids the sharp drop in accuracy caused by traditional lightweight methods that directly compress channels and reduce network depth. While significantly improving segmentation accuracy, the model's computational load is reduced by 18.61%, and the number of parameters is reduced by 38.52%, achieving optimal overall performance with higher accuracy, smaller model size, and faster speed. This solves the long-standing core contradiction in the field of farmland remote sensing image segmentation: the difficulty of balancing accuracy and computational efficiency.

[0025] 3. This invention replaces the traditional ResNet18 with a lightweight DecoupleNet backbone, utilizing feature integration downsampling and multi-branch feature decoupling mechanisms to efficiently suppress feature channel redundancy while preserving small targets, details, and boundary features. The final model has only 7.58M parameters and a computational cost reduced to 38.97G, far lighter than mainstream segmentation models such as RS3mamba, DBBANet, and UnetFormer. It has low memory usage, low inference latency, and low hardware dependency, and can be deployed on resource-constrained edge platforms such as drones, embedded terminals, portable monitoring equipment, and mobile terminals, meeting practical engineering needs such as real-time field monitoring in smart agriculture, real-time segmentation of drone aerial photography, and rapid field interpretation.

[0026] 4. This invention specifically optimizes for the typical characteristics of farmland, such as susceptibility to light interference, strip-shaped plot distribution, narrow field ridges and ditches, and sparse and uneven agricultural facilities. The MSAF module enhances the modeling ability for orientation-sensitive structures, the SCIA module dynamically balances spatial structure and spectral channel features to enhance anti-interference capabilities, and the BEH module explicitly strengthens plot boundaries with extremely low computational cost. The model maintains stable segmentation results on remote sensing images of farmland under different terrains, planting structures, lighting conditions, and regions. Its generalization ability and scene adaptability are significantly better than general remote sensing segmentation models, giving it greater engineering practical value. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0028] Figure 1 This is an overall workflow diagram of the remote sensing image segmentation method based on lightweight UMFormer provided in the embodiments of the present invention; Figure 2 This is an overall architecture diagram of the LEUMFormer model provided in this embodiment of the invention; Figure 3 This is a structural diagram of the MSAF module provided in an embodiment of the present invention; Figure 4 This is a structural diagram of the SCIA module provided in an embodiment of the present invention; Figure 5 This is a structural diagram of the BEH module provided in an embodiment of the present invention; Figure 6 This is a visualization of the LEUMFormer model provided in this embodiment of the invention on the LoveDA dataset. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0030] See Figure 1 and Figure 2 This application provides a remote sensing image segmentation method based on lightweight UMFormer, including the following steps: S1. Acquire a dataset of farmland scene images, including the FGFD and Loveda datasets, and perform data preprocessing and data augmentation. Data preprocessing includes, but is not limited to, filtering and histogram equalization to improve the visual quality of the images and enhance the segmentation accuracy of the model. Data augmentation includes, but is not limited to, flipping, rotating, and scaling to increase the model's generalization ability.

[0031] S2, constructing the LEUMFormer model: including a lightweight encoder, a multi-scale orientation-aware fusion module, and an enhanced global-local decoder; the backbone network of the lightweight encoder is DecoupleNet, which has efficient feature decoupling capabilities, replacing the original ResNet18; the end of the lightweight encoder introduces a multi-scale attention fusion (MSAF) module, which explicitly models orientation features at different scales through multi-scale convolution and orientation-aware branches, and achieves adaptive fusion of cross-scale features; the enhanced global-local decoder includes a spatial and channel integration and adaptive balancing (SCIA) module, which models spatial location dependence and channel feature dependence respectively through parallel spatial attention branches and channel attention branches, and introduces learnable weights to achieve adaptive fusion of the two; the enhanced global-local decoder includes a boundary enhancement head (BEH) module, which strengthens and repairs boundary features through feature interaction between the edge detection branch and the main segmentation branch.

[0032] S3, Training and Validating the LEUMFormer Model: Implement and train the LEUMFormer model code on the platform. After data preprocessing and data augmentation, the collected dataset is divided into training and test sets according to the proportions. Train the model on the training set. The loss function is defined as a combination of soft cross-entropy (SCE) loss and Dice loss. Validate the model on the test set.

[0033] S4, Image Segmentation: After the LEUMFormer model is trained, it is used to segment the image and output the semantic segmentation results of the farmland image.

[0034] In this invention, a LEUMFormer model is constructed based on the lightweight UMFormer: the backbone network is replaced with DecoupleNet, and SCIA, MSAF, BEH, and the existing GL_VSS Block module are introduced sequentially. During the forward pass of the LEUMFormer model, the first three shallow features output by the lightweight encoder are passed to the decoding stage through skip connections and concatenated with the features output by the previous layer decoder, and then input into the SCIA module for feature enhancement; the last deep feature output by the lightweight encoder is input into the MSAF module along with the last three shallow features for feature fusion; the fused and enhanced features are then input into the GL-VSS Block of the enhanced global-local decoder for decoding and reconstruction.

[0035] In a specific embodiment of the present invention, a lightweight encoder is first constructed, with a lightweight DecoupleNet backbone network consisting of alternating stacks of a Feature Integration Down-sampling (FID) module and a Multi-Branch Feature Decoupling (MBFD) module. During processing, the farmland remote sensing image is input into the Stem module to complete the initial convolutional feature embedding, generating an initial feature map. Subsequently, the initial feature map is downsampled and feature decoupling is extracted level by level through alternating stacks of multi-level FID and MBFD modules. In this application, there are five levels, outputting five feature maps at different scales, denoted as the first, second, third, fourth, and fifth feature maps, with output scales of C×H / 4×W / 4, C×H / 4×W / 4, C×H / 8×W / 8, C×H / 16×W / 16, and C×H / 32×W / 32, respectively. The first feature map is a high-resolution shallow feature map output by the first-level encoder, the second to fourth feature maps are shallow feature maps output by the second to fourth-level encoders respectively, the fifth feature map is a deep feature map, the second to fifth feature maps are the multi-scale features of the input MSAF module, and the features concatenated with the decoder features are the skip connection features of the fourth to second feature maps.

[0036] In remote sensing farmland image segmentation tasks, farmland areas exhibit significant multi-scale feature distribution characteristics, while agricultural infrastructure such as field ridges and ditches possess prominent linear structural features. Although existing multi-scale attention modules can model multi-scale contextual dependencies, the Transformer-based self-attention mechanism has quadratic computational complexity, failing to fully meet the modeling needs of orientation-sensitive features in farmland scenarios.

[0037] In a specific embodiment of the present invention, an MSAF module is introduced at the end of the lightweight encoder. See also Figure 3 The MSAF module employs a mechanism of multi-scale feature fusion followed by orientation-aware enhancement. Through the synergistic optimization of depthwise separable convolution and attention mechanisms, it achieves a balance between multi-scale feature fusion and orientation-sensitive feature extraction. Four feature maps at different scales extracted by the lightweight encoder are input into the MSAF module for processing. The processing procedure of this module is as follows: (1) After channel compression of the shallow feature map using 1×1 convolution, preprocessing is performed. The preprocessing includes ReLU activation and batch normalization performed sequentially, and then downsampled to the level of the deep feature map. Figure 1 With the desired spatial resolution, four single-scale feature maps are obtained, namely the compressed second to fifth feature maps, represented as follows: ;in, For the compressed first The feature map is aligned with the dimensions of the corresponding deep feature map and is used for subsequent multi-scale feature fusion. The first extracted for the lightweight encoder Feature map, i.e. , , , These correspond to the outputs of encoder levels 2-5, generated through the feature integration downsampling and multi-branch feature decoupling mechanism of DecoupleNet. They contain basic features of farmland scenes at different scales, such as plot outlines and preliminary features of agricultural facilities. The ReLU activation function is used, BN represents batch normalization, and Conv1×1 represents a 1×1 convolution operation.

[0038] (2) After concatenating the features of the four single-scale feature maps, channel fusion is performed through 1×1 convolution to obtain a multi-scale fused feature map, which is represented as: ;in, This indicates a feature concatenation operation. This is a multi-scale fused feature map.

[0039] The 1×1 convolution operation facilitates full interaction between low-level detail information and high-level semantic information. The resulting multi-scale fusion feature map integrates shallow detail features and deep semantic features, providing a foundation for subsequent attention weighting.

[0040] (3) To address the significant differences in the scale of features in farmland scenarios, MSAF further introduces a collaborative design of multi-scale convolutional groups and attention mechanisms to adaptively enhance key region features. That is, multi-scale enhanced feature maps are obtained by enhancing the multi-scale fused feature maps through depthwise separable convolution and attention mechanisms.

[0041] (31) Specifically, the channel fusion feature map is obtained by weighting the fused features, i.e., the multi-scale fused feature map, in the channel dimension through the Channel Attention Module (CAM) to highlight the discriminative channel response, as shown in the following: ; This indicates the channel attention module. This represents the channel fusion feature map.

[0042] (32) Subsequently, three sets of depthwise separable convolutions (3×3, 5×5, and 7×7) with different receptive fields are used in parallel to extract multi-scale contextual features, and these features are fused by element-wise addition to obtain a multi-scale aggregated feature map. The fused features are then input into the Spatial Attention Module (SAM) to enhance salient regions in the spatial dimension, resulting in a multi-scale enhanced feature map, expressed as: ; This is a multi-scale enhanced feature map that simultaneously captures multi-scale contextual information and spatially significant regional features of farmland (such as contiguous farmland). express The results generated by depthwise separable convolution with three different receptive fields of 3×3, 5×5 and 7×7 are multi-scale aggregated feature maps. 3×3 captures local details and 7×7 captures large-scale contiguous farmland features. This indicates an element-wise addition operation; SAM stands for Spatial Attention Module, used to obtain spatial attention weights.

[0043] In the above operations, three sets of differentiated depthwise separable convolutions are used to extract contextual information from different receptive fields, solving the problem of semantic ambiguity of large-scale ground features caused by insufficient receptive fields at a single scale. The channel attention module adaptively enhances the semantic information of key channels, while spatial attention adaptively focuses on key regions where ground features are located, suppressing background noise and improving the semantic discriminativeness and spatial localization capability of the features. The overall parallel processing mechanism improves feature quality while controlling overall processing time, meeting the real-time processing requirements of large-scale farmland remote sensing images.

[0044] (4) To effectively capture the linear boundary structures commonly found in farmland scenes, the module further introduces asymmetric convolution kernels in the horizontal and vertical directions to model orientation-sensitive features. That is, the multi-scale enhanced feature map is processed by asymmetric convolution kernels of different scales to obtain the orientation-enhanced feature map.

[0045] Specifically, the multi-scale enhanced feature maps are first normalized layer by layer. Then, orientation-related features are extracted using three sets of horizontal and vertical convolutional kernels, resulting in six orientation feature maps (three each in the horizontal and vertical directions). These capture linear structural features (such as horizontal field ridges and vertical ditches) in the farmland scene. A 1×1 convolution is used to integrate the channels, and then the features are added element-wise, fusing orientation feature maps with different kernel sizes to enhance the robustness of the orientation features and obtain the orientation-enhanced feature map. Residual connections are introduced to maintain the stability of gradient propagation; that is, the orientation-enhanced feature map is the result of residual connections between the fused and multi-scale enhanced feature maps. (Horizontal orientation feature map) The expression is: ; Vertical feature map The expression is: ; in, Presentation layer normalization operation, for Normalization is performed to stabilize the input distribution of the directional convolution; For horizontal convolution (k = 7, 11, 21), specifically extract horizontal features; This is a vertical convolution specifically designed to extract vertical features. It involves element-wise addition.

[0046] The horizontal and vertical convolution kernels mentioned above are structurally symmetrical and satisfy a strict matrix transpose relationship. Functionally, they are complementary in direction. The former slides only in the horizontal direction to extract horizontal textures and long horizontal features (such as field ridges and horizontal roads), while the latter slides only in the vertical direction to extract vertical textures and long vertical features (such as ditches and longitudinal field boundaries).

[0047] The above operation employs a multi-scale paired asymmetric convolution design, configuring a set of complementary convolution kernels for each scale. Through three sets of asymmetric convolutions at different scales, horizontal and vertical features of short, medium, and long distances are extracted respectively, covering slender features of different lengths in farmland scenes. This solves the problems of insufficient extraction and fragmented segmentation of slender structural features by traditional convolution, significantly improving the segmentation continuity of targets such as roads and field ridges. This asymmetric convolution kernel complements the aforementioned channel / spatial attention module in terms of "global semantics, local localization, and directional details." Subsequently, residual connections between the directional enhancement feature map and the multi-scale enhancement feature map are used to enhance directional information while preserving the semantic information of the original features, avoiding information loss during feature enhancement.

[0048] (5) The output of the multi-scale attention branch and the output of the multi-directional attention branch are concatenated, that is, the multi-scale enhanced feature map and the directional enhanced feature map are concatenated, and the final fusion is completed by 1×1 convolution to obtain the output feature. This is the initial fused feature map. This feature map simultaneously integrates multi-scale contextual information and orientation-sensitive features, providing a more discriminative feature representation for subsequent decoding and fine-grained segmentation.

[0049] It is important to understand that the MSAF module introduced in this application first performs multi-scale alignment and channel fusion of the input to effectively avoid information fragmentation between layers. Then, it improves the semantic discriminativeness and spatial localization capability of features through collaborative optimization of depthwise separable convolution and attention mechanism. Finally, based on asymmetric convolution for direction-aware enhancement, it specifically extracts horizontal and vertical features of slender features such as roads and field ridges in farmland scenes, solving the problems of fragmented segmentation and blurred edges of slender structures in traditional methods. This module achieves an efficient balance between multi-scale feature fusion and direction-sensitive feature extraction, and can adapt to the real-time processing requirements of large-scale farmland remote sensing images.

[0050] In a specific embodiment of the present invention, an enhanced global-local decoder is constructed. During the decoding stage, the first three shallow feature maps output by the lightweight encoder are... , , The data is passed to the decoder via skip connections and concatenated with the features output from the previous layer after processing by the GL_VSS module. The enhanced global-local decoder consists of four decoding units and an upsampled output unit. The first-layer decoding unit contains only a GL-VSS module (Global-Local Visual State Space Block), used to complete the initial fused feature map. Global-local context modeling, output ; The second to fourth layer decoding units each include a SCIA module, a GL-VSS module, and a BEH module connected in sequence. The processing procedure is as follows: ... and The jump-connection feature is concatenated, input to the second-layer decoding unit, and output after processing. ; will and The jump-connection feature is concatenated, input to the third-layer decoding unit, and output after processing. ; will and The jump-connection feature is concatenated, input to the fourth-layer decoding unit, and output after processing. ; Upsampling output unit receives and The splicing result of the jump connection feature is processed by bilinear interpolation upsampling to the input image size, and then the semantic segmentation result of the farmland image is output.

[0051] It's important to understand that traditional decoding structures often rely on local convolution or fixed receptive field modeling, which is insufficient for modeling large-scale continuous features and long-distance texture relationships in farmland. The GL-VSS module in this solution, as the core component of the decoding unit, achieves adaptive modeling of global dependencies and local details. This represents a structural improvement over the original decoding structure, rather than a conventional parameter replacement. Traditional output layers directly output results through a single upsampling, easily leading to blurred boundaries and loss of detail. This solution introduces high-resolution shallow feature skip connections before output, combined with multi-level BEH boundary enhancement, significantly improving the segmentation accuracy of farmland boundaries and slender features, resulting in a substantial structural difference from the original output layer structure. This decoder is not a simple application of existing decoding structures, but rather a new combined structure formed by GL-VSS global-local modeling, SCIA directional awareness attention, BEH boundary enhancement, and multi-level refined skip connection fusion. This overcomes the technical shortcomings of traditional decoders in remote sensing farmland scenes, such as low accuracy, weak long-range modeling, and discontinuous boundaries.

[0052] In a specific embodiment of the present invention, for the second to fourth layer decoding units, the concatenated features input to the SCIA module are sequentially denoted as follows: The features input to the GL-VSS module are denoted as follows: The features input to the BEH module are denoted as follows: , .

[0053] See Figure 4 For the spliced ​​feature map (Right now The SCIA module was designed, and its processing steps include: S1. Input the spliced ​​feature maps into the parallel spatial attention branch and channel attention branch respectively to obtain the spatial attention enhanced feature map and the channel attention enhanced feature map.

[0054] In the spatial attention branch, to capture the regular directional structure in the farmland scene, the module extracts spatial attention from the height and width directions respectively. The concatenated feature maps are transformed to obtain feature maps in two directions, namely the height direction feature map and the width direction feature map. The feature maps in each direction are fused with max pooling and average pooling features through 1×1 convolution to obtain the initial attention weights. After dimensionality reshaping, depthwise separable convolution is used to refine the features, and a learnable dynamic scaling factor is introduced to adaptively adjust the attention intensity to obtain the spatial attention weights. The spatial attention weights are multiplied with the concatenated feature maps to obtain the spatial attention enhanced feature map.

[0055] Specifically, for the features in each direction, the calculation method is as follows: ; Spatial attention weights in a single direction (height or width) are used to enhance the directional structural features of farmland in that direction (such as the extension direction of long strips of farmland). This is the feature map after dimensional transformation, which can be either a feature map in the height direction or a feature map in the width direction. , , representing max pooling and average pooling operations, respectively for Spatial dimension pooling is performed to extract key feature statistics in that direction. This indicates a depthwise separable convolution operation, which refines the features after a 1×1 convolution and enhances orientation-sensitive features. It is a learnable dynamic scaling factor.

[0056] Spatial attention enhancement feature map , represented as: Finally, the original shape is restored through dimensional transformation. , Spatial attention weights for height and width, respectively. , These are feature maps in the height and width directions, respectively.

[0057] In the channel attention branch, to enhance the ability to distinguish the spectral features of different land features in farmland scenes, the module adopts an improved channel attention mechanism. This branch simultaneously utilizes the spatial information of the spliced ​​feature map aggregated by average pooling and max pooling, and then learns the nonlinear relationship between channels through two fully connected layers and the ReLU activation function between layers to calculate the channel attention weights: multiplying the channel attention weights with the spliced ​​feature map yields the channel attention enhanced feature map. Specifically, the expression for the channel attention weight is: ; The channel attention weights reflect the importance of each channel for spectral discrimination of farmland features. (Multilayer Perceptron) performs a non-linear transformation on the pooled feature vectors to learn the dependencies between channels. This represents the feature vector addition operation, which fuses the output features of the MLP corresponding to average pooling and max pooling. This represents the Sigmoid activation function, which is used to normalize the fused feature vector to the [0,1] interval.

[0058] Channel attention enhancement feature map , represented as: .

[0059] S2, dynamically fusing the two features using an adaptive fusion gating system: The two feature maps are concatenated and then passed through a 1×1 convolutional layer and a sigmoid activation function to generate fusion weights. These fusion weights are then used to weight and sum the two feature maps to obtain an initial attention-enhanced feature map. Finally, a residual connection is used to fuse this attention-enhanced feature map with the concatenated feature map, resulting in the final attention-enhanced feature map, which is the feature map input to the GL-VSS module. The fusion weights are represented as follows: ; For adaptive fusion weights, dimensions and , Consistency is used to dynamically balance the contributions of spatial attention features and channel attention features.

[0060] The initial attention-enhanced feature map is represented as follows: ; The initial attention-enhanced feature map takes into account both the spatial structure features and spectral channel discrimination features of the farmland scene, reducing misclassification of spectrally similar regions.

[0061] The aforementioned SCIA module enhances key features while effectively controlling the enhancement magnitude, ensuring the stability of network training. Its output serves as the input to the current decoding unit, the GL-VSS module, thereby fully utilizing multi-scale feature information and significantly improving the segmentation accuracy of complex structures and subtle boundaries in farmland scenes.

[0062] In a specific embodiment of the present invention, a BEH module was constructed, see [link to relevant documentation]. Figure 5 The specific processing procedure is as follows: Receive the input feature map from the BEH module. (Right now The module first extracts edge responses from the features through a lightweight boundary detection layer to obtain a boundary enhancement feature map. Then, a boundary feature refinement layer performs a nonlinear transformation on the boundary enhancement feature map to obtain a boundary refinement feature map. Finally, the boundary refinement feature map is added to the input feature map through a residual connection to obtain the final output feature map of the BEH module, thus achieving progressive feature enhancement.

[0063] Specifically, the boundary detection layer uses a 3×3 convolutional kernel as an edge detector to perform convolution operations on the input features, capture feature gradient changes, and obtain the edge response feature map, expressed as: ; The edge response feature map captures areas in farmland scenes where feature gradients change drastically (i.e., potential boundary areas, such as field ridges and plot edges). It is a 3×3 convolution operation, and the convolution kernel uses an edge detection operator (such as the Sobel operator) to specifically capture gradient changes in the feature map.

[0064] Then, the edge response feature map is normalized to a boundary probability map between 0 and 1 using the Sigmoid activation function, representing the confidence that each position belongs to the boundary. This allows the model to focus on boundary regions with drastic feature changes. The expression is: ; For the boundary probability graph, the dimension is... Consistent, each pixel value represents the confidence level that the location belongs to the farmland boundary.

[0065] Finally, the input feature map and the boundary probability map are fused along the channel dimension to obtain the boundary enhancement feature map, expressed as: ; This represents the boundary enhancement feature map.

[0066] The boundary feature refinement layer employs a depthwise separable convolutional structure, comprising a 3×3 depthwise convolution, batch normalization, pointwise convolution, and the SiLU activation function. This ensures the module's lightweight nature while effectively fusing boundary information with the original features to generate a refined boundary feature map, expressed as: ; The feature map is refined for the boundary, which enhances the feature representation of the boundary region while maintaining lightweight characteristics; This represents a combination of depthwise separable convolution, batch normalization, and SiLU activation function. Depthwise separable convolution performs lightweight feature extraction of fused features, batch normalization stabilizes the feature distribution, and SiLU activation function introduces nonlinear transformation to refine boundary features.

[0067] The final output feature map is expressed as: ; The final output feature map of the BEH module is used to progressively enhance the boundary features through residual connections, ensuring that the boundaries of farmland plots are clear and continuous.

[0068] In this embodiment, the lightweight and moderately accurate UMFormer is used as the sole base model. A two-stage optimization strategy is proposed to first improve accuracy and then balance lightweighting to construct the LEUMFormer segmentation model. Compared with ResNet18, the encoder of this application can significantly reduce the number of model parameters and computational complexity, reduce redundant feature extraction, and improve feature information density. It achieves model lightweighting while ensuring multi-scale feature representation capabilities, making it more suitable for real-time segmentation and edge deployment of farmland remote sensing images. At the same time, it provides high-quality and highly balanced multi-granular features for subsequent multi-scale fusion and boundary enhancement. Through the "two-stage optimization strategy," the segmentation accuracy of farmland scene features (such as fields, roads, and buildings) is greatly improved without significantly increasing the number of parameters, achieving a balance between "lightweight deployment and high-precision segmentation," which is especially suitable for the large-scale batch processing needs of UAV / satellite remote sensing. In addition, the overall architecture of the LEUMFormer segmentation model is designed for farmland scenes, which can effectively distinguish different types of farmland features, reduce missed detections and false detections, and the segmentation results are more in line with the actual application needs of agricultural production.

[0069] This application provides a remote sensing image segmentation system based on the lightweight UMFormer, including: The dataset acquisition module is used to acquire datasets of farmland scene images; The model building module is used to construct the LEUMFormer model, including a lightweight encoder, a multi-scale orientation-aware fusion module, and an enhanced global-local decoder. The backbone network of the lightweight encoder is DecoupleNet. The end of the lightweight encoder introduces a multi-scale attention fusion (MSAF) module, which explicitly models orientation features at different scales through multi-scale convolution and orientation-aware branches. The enhanced global-local decoder includes a spatial and channel integration and adaptive balancing (SCIA) module, which models spatial location dependency and channel feature dependency respectively through parallel spatial attention branches and channel attention branches, and introduces learnable weights to achieve adaptive fusion of the two. The enhanced global-local decoder also includes a boundary enhancement (BEH) module, which strengthens and repairs boundary features through feature interaction between the edge detection branch and the main segmentation branch. The segmentation module is used to segment images using the trained model.

[0070] In this embodiment, high-precision and lightweight segmentation of remote sensing images is achieved through the cooperation of various modules.

[0071] This application provides a computer program product including a computer program that, when run on a computer device, causes the computer device to execute a remote sensing image segmentation method based on the lightweight UMFormer. To verify the effectiveness of UMFormer, experimental and visualization analyses were performed on publicly available data; see [link to relevant documentation]. Figure 6 And Tables 1 to 3. (For example...) Figure 6 As shown, the segmentation visualization results of the remote sensing image segmentation method based on the lightweight UMFormer are presented on the LoveDA dataset.

[0072] Tables 1 and 2 show the results of the method of this application compared with other models on the LoveDA dataset and the FGFD dataset.

[0073] Table 1 Quantitative comparison results on the LoveDA dataset As shown in Table 1, the LEUMFormer model outperforms other models in IoU on the LoveDA dataset for bg (background), bld (buildings), rd (roads), wt (water), br (wasteland), and agr (farmland), only falling behind CMTFNet in frt (forest). It also outperforms other models in mIoU (mean intersection-over-union ratio), OA (overall accuracy), and mF1 (mean F1 score), while having lower Flops (floating-point computation cost) at only 38.97G and lower Param (number of model parameters) at only 7.58M.

[0074] Table 2 Quantitative comparison results on the FGFD dataset As shown in Table 2, the LEUMFormer model outperforms other models in IoU on the FGFD dataset in both background and farmland, as well as in mIoU, OA, and mF1. It also has a lower computational cost (only 38.97 G) and fewer model parameters (only 7.58 M).

[0075] Table 3 shows the results of ablation experiments with the addition of a single component or a combination of components to the baseline model, to further validate the effectiveness of the LEUMFormer model.

[0076] Table 3. Ablation study results for each component of the LoveDA dataset LEUMFormer. As shown in Table 3, the LEUMFormer model constructed in this application achieves the best mIoU, OA, and mF1 on the LoveDA dataset, outperforming models that include only one or more of these structures, while also having the lowest computational cost and number of model parameters. Data from the ablation experiments indicate that the combination of the newly added MSAF module, SCIA module, BEH module, and backbone network DecoupleNet with the baseline model is the optimal combination.

[0077] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0078] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A remote sensing image segmentation method based on lightweight UMFormer, characterized in that, Includes the following steps: Construct the LEUMFormer model, including a lightweight coding network, a perceptual fusion network, and a global-local decoding network; The backbone network of the lightweight coding network is DecoupleNet, which is used to input farmland scene image datasets into multi-level feature extraction units to extract features at different scales; The perceptual fusion network employs the MSAF module to fuse features at different scales and extract directional features; The global-local decoding network includes multiple levels of decoding units. The first-level decoding unit includes a decoder GL-VSSBlock, which decodes the output features of the MSAF module. The remaining multiple decoding units include a sequentially connected Spatial and Channel Adaptive Integration (SCIA) module, a decoder GL-VSS Block, and a Boundary Enhancement (BEH) module. The SCIA module is used to extract spatial attention features and channel attention features through parallel spatial attention branches and channel attention branches, respectively, and then adaptively fuse them. The BEH module is used to enhance and repair the boundary features of the fused features. The trained LEUMFormer model was used to segment images of farmland scenes.

2. The remote sensing image segmentation method based on lightweight UMFormer as described in claim 1, characterized in that, In the LEUMFormer model, during the network forward pass, the last-level deep features output by the lightweight encoder are fused with the first three levels of shallow features in the MSAF module. The output of the MSAF module serves as the input to the first-level decoding unit. The first three levels of features output by the multi-level feature extraction unit are passed to the parallel decoding unit through skip connections and concatenated with the output features of the previous level decoding unit. This concatenation is then used as the input to the SCIA module of the current level decoding unit for feature enhancement. The fused and enhanced features are then input to the GL-VSS Block of the same-level decoding unit for decoding and reconstruction. Its output serves as the input to the BEH module of the same-level decoding unit.

3. The remote sensing image segmentation method based on lightweight UMFormer as described in claim 1, characterized in that, The MSAF module employs a mechanism of first fusing multi-scale features and then enhancing orientation awareness. Through the synergistic optimization of depthwise separable convolution and attention mechanisms, it achieves a balance between multi-scale feature fusion and orientation-sensitive feature extraction.

4. The remote sensing image segmentation method based on lightweight UMFormer as described in claim 1, characterized in that, The processing procedure of the MSAF module includes: After channel compression of the three shallow features using 1×1 convolution, preprocessing is performed, and the features are downsampled to the same spatial resolution as the deep features to obtain a four-layer single-scale feature map. The preprocessing includes ReLU function activation and batch normalization performed sequentially. After concatenating the features of the four single-scale feature maps, channel fusion is performed through 1×1 convolution to obtain a multi-scale fused feature map. Multi-scale enhanced feature maps are obtained by enhancing multi-scale fused feature maps through depthwise separable convolution and attention mechanisms. By processing the multi-scale enhanced feature map with asymmetric convolution kernels of different scales, a directional enhanced feature map is obtained; After concatenating the directional enhancement feature map and the multi-scale enhancement feature map, they are fused using a 1×1 convolution to obtain the initial fused feature map.

5. The remote sensing image segmentation method based on lightweight UMFormer as described in claim 4, characterized in that, The depth-separable convolution and attention mechanisms are processed as follows: The multi-scale fusion feature map is weighted along the channel dimension by CAM to obtain the channel fusion feature map; Three sets of deep separable convolutions with different receptive fields are used to extract multi-scale contextual features of the channel fusion feature map in parallel. The features are then fused by adding them element by element. The fused features are input into the SAM and weighted in the spatial dimension to obtain the multi-scale enhanced feature map.

6. The remote sensing image segmentation method based on lightweight UMFormer as described in claim 4, characterized in that, The asymmetric convolution kernel includes three pairs of horizontal and vertical convolution kernels, each pair of horizontal and vertical convolution kernels having the same scale and being transposes of each other; The asymmetric convolution kernel is processed as follows: The multi-scale enhanced feature map is subjected to layer normalization. Then, orientation-related features are extracted by three sets of horizontal and vertical convolution kernels. After element-wise addition, channel fusion is performed using 1×1 convolution. The fusion result is then residually connected to the multi-scale enhanced feature map to obtain the orientation enhanced feature map.

7. The remote sensing image segmentation method based on lightweight UMFormer as described in claim 1, characterized in that, The SCIA module's processing procedure includes: The concatenated feature maps are input into the parallel spatial attention branch and channel attention branch respectively to obtain spatial attention-enhanced feature maps and channel attention-enhanced feature maps; The two feature maps are dynamically fused using an adaptive fusion gating system: the two feature maps are concatenated and then fed through a 1×1 convolutional layer and a sigmoid activation function to generate fusion weights. The fusion weights are used to perform a weighted sum of the two feature maps, and then fused with the concatenated feature map through a residual connection to obtain the final attention-enhanced feature map.

8. The remote sensing image segmentation method based on lightweight UMFormer as described in claim 1, characterized in that, The BEH module processes the following steps: Receive the input feature map from the BEH module; Edge responses are extracted from the input feature map through a boundary detection layer to obtain a boundary-enhanced feature map: In the boundary detection layer, the input feature map is convolved by a 3×3 convolution kernel. The convolved feature map is normalized to a boundary probability map by the Sigmoid activation function. The input feature map and the boundary probability map are fused in the channel dimension to obtain the boundary enhancement feature map. The boundary enhancement feature map is nonlinearly transformed by the boundary feature refinement layer to obtain the boundary refinement feature map. The boundary feature refinement layer adopts a depthwise separable convolution structure, which includes a 3×3 depthwise convolution, batch normalization, point convolution and SiLU activation function. The boundary refinement feature map is added to the input feature map through residual connections, which is used as the final output feature map of the BEH module.

9. A remote sensing image segmentation system based on lightweight UMFormer, characterized in that, include: The model building module is used to build the LEUMFormer model, which includes a lightweight encoder, a multi-scale orientation-aware fusion module, and an enhanced global-local decoder. The backbone network of the lightweight encoder is DecoupleNet. The end of the lightweight encoder introduces a multi-scale attention fusion MSAF module, which explicitly models the orientation features at different scales through multi-scale convolution and orientation-aware branches. The enhanced global-local decoder includes a spatial and channel integrated and adaptively balanced SCIA module. It models spatial location dependency and channel feature dependency respectively through parallel spatial attention branches and channel attention branches, and introduces learnable weights to achieve adaptive fusion of the two. The enhanced global-local decoder also includes a boundary enhancement (BEH) module, which strengthens and repairs boundary features through feature interaction between the edge detection branch and the main segmentation branch; The segmentation module is used to segment farmland scene images using the trained model.

10. A computer program product comprising a computer program, characterized in that, When it is run on a computer device, it causes the computer device to perform the method of any one of claims 1-8.

Images