UAV remote sensing image segmentation method and system based on U-shaped adaptive EST

Abstract

A U-shaped adaptive EST (Extended Mean Stability) unmanned aerial vehicle (UAV) remote sensing image segmentation method and system, comprising a feature extraction unit, a feature encoder, a skip connection module, a convolutional decoding module, and a projection module. The feature encoder comprises a multi-order EST and a corresponding adaptive image block merging layer. Each order of EST comprises several EST units consisting of a first EST module and a second EST module, each EST module comprising a normalization layer (LN), an enhanced multi-head self-attention module, a residual connection layer, and a multi-layer perceptron (MLP). Based on CNN and Transformer modules, the present invention achieves global self-attention through a hierarchical structure. In the decoder, upsampling is achieved by combining convolutional and deconvolution blocks. The upsampling restores the encoder information and performs corresponding pixel-level segmentation, thereby reducing the loss of edge and position information caused by directly performing image block processing on the original image.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular to a U-shaped adaptive Enhanced Swin Transformer unmanned aerial vehicle (UAV) remote sensing image segmentation method. Background Art

[0002] Thanks to the development of deep learning, computer vision technology has been widely used in remote sensing image segmentation. With the development of deep learning, CNN has achieved good results in the field of image segmentation. However, due to the inherent localization of convolution operations, CNN-based networks have difficulty in taking into account global semantic information.

[0003] With the remarkable success of Transformer in natural language processing in recent years, this method has achieved relatively satisfactory results in capturing global and long-range information and can be well transplanted to downstream tasks. However, at the same time, global characteristics often ignore local feature details, thereby reducing the distinction between foreground and background. In addition, the nonlinearity of multi-head self-attention increases memory and computation, resulting in heavy training and inference overhead. Summary of the Invention

[0004] The present invention aims to address the problem of poor foreground-background distinction in image segmentation by proposing a U-shaped adaptive EST (UAV remote sensing image segmentation method) and a U-shaped adaptive Enhanced Swin Transformer (AESwin-UNet) UAV remote sensing image segmentation method. Taking into account the different yet complementary characteristics of CNNs and Transformers, a hybrid CNN-Transformer network is integrated: the U-shaped adaptive Enhanced Swin Transformer (Adaptive Enhanced Swin Transformer with U-net, AESwin-UNet). This combination combines the advantages of the Transformer's self-attention strategy and convolution, and is suitable for remote sensing image segmentation. It can effectively avoid the loss of some shallow image features, such as corners or edges, and improve segmentation accuracy.

[0005] The technical solution of the present invention is:

[0006] The present invention provides a U-shaped adaptive Enhanced Swin Transformer UAV remote sensing image segmentation method, comprising the following steps:

[0007] Step 1: Extract features from the input image to obtain a high-dimensional feature encoding sequence;

[0008] Step 2: Input the coding sequence into the feature encoder for feature extraction to generate multi-scale hierarchical features;

[0009] Step 3: The multi-scale hierarchical features obtained by the feature encoder are input into the convolutional decoding module through the skip connection module for fusion to complete the restoration of the feature map and obtain the restored feature map;

[0010] Step 4: Project based on the restored feature map to obtain pixel-level segmentation results.

[0011] Furthermore, step 1 specifically includes:

[0012] Step 1-1: Extract features of the input image through the residual neural network ResNet50 to generate a high-dimensional feature map of the input image;

[0013] Step 1-2: Split the high-dimensional feature map into a series of non-overlapping image blocks;

[0014] Step 1-3: Convert the image block segmentation results into an embedding sequence;

[0015] Step 1-4: The embedded sequence is passed through a linear mapping layer to generate a one-dimensional hierarchical feature representation, namely the encoding sequence.

[0016] Furthermore, the feature encoder includes a multi-stage EST (Enhanced Swin Transformer) and a corresponding Deformable Adaptive Patch Merging (DeforAPM) layer, wherein each stage EST includes a plurality of EST units; step 2 specifically includes:

[0017] Step 2-1: Input the coding sequence into the first-order EST for processing;

[0018] Step 2-2: Output the result of the first-order EST to the skip connection module of the corresponding order; at the same time, the result of the first-order EST is downsampled using the Deformable Adaptive Patch Merging (DeforAPM) layer to generate first-order multi-scale features and output them to the second-order EST;

[0019] Step 2-3: The second-order EST processes the output of the first-order EST and outputs the result of the second-order EST to the skip connection module of the corresponding order according to step 2-2; at the same time, the adaptive image block merging layer is used to downsample the result of the second-order EST to generate second-order multi-scale features and output them to the third-order EST;

[0020] Traverse each order of EST, complete the encoding, and generate multi-scale hierarchical features.

[0021] Furthermore, the EST unit in step 2 includes two consecutive first EST modules and a second EST module, each EST module includes a normalization layer LN, an enhanced multi-head self-attention module Multi-Head Self-Attention, a residual connection layer Residual Connection, and a multi-layer perceptron MLP (a two-layer structure based on a Gaussian Error Linear Units (GELU) activation function); wherein, the enhanced multi-head self-attention modules of the first EST module and the second EST module respectively adopt a window-based enhanced multi-head self-attention module W-EMHSA and a shifted window-based enhanced multi-head self-attention module SW-EMHSA;

[0022] The EST unit processes the input coding sequence using the following formula:

[0023]

[0024]

[0025]

[0026]

[0027] Where: x represents the characteristic variable, l represents the number of EST modules in the EST unit; x l-1 represents the input coding sequence; represents the W-EMHSA output of the lth block; x l represents the output of the MLP of the lth block; represents the SW-EMHSA output of the lth block; x l+1 represents the MLP output of the lth block; x l+1 Represents the output encoding sequence.

[0028] Furthermore, the enhanced multi-head self-attention module uses the following formula for calculation:

[0029]

[0030] Where: Q, K, V are the corresponding input code sequence LN(x l )、LN(x l-1 ) After three linear transformations and mappings, the resulting query, key, and value are obtained; M 2 represents the number of image blocks in the window, d represents the embedding dimension of query Q and key value K; B represents the relative position of the current image block within the window range, R represents the receptive field, which is a natural number;

[0031] att represents the multi-scale attention used after the QK correlation matrix, which is calculated using the following formula:

[0032]

[0033] L(X)=LN(FC2(δ(FC1(X)))), (7)

[0034] G(X)=LN(FC2(δ(FC1(GAP(X)))), (8)

[0035] Where: X MS-Attention ∈R k×m×m Represents a feature with multi-scale attention, k represents the number of channels of the feature, and m represents the width and height of the feature; W(X)∈R k×m×m Represents the weight coefficient of multi-scale attention features, L(x)∈R k×m×m Represents local channel context information, G(X)∈R k×1×1 Represents global feature context information, k represents feature dimension; FC1 and FC2 are both fully connected layers for reducing and expanding size; GAP represents average pooling, δ represents the combination of layer normalization and activation function ReLU, σ is Sigmoid function, LN represents layer normalization, and They represent element-wise multiplication and addition, respectively.

[0036] Furthermore, in steps 2-2 and 2-3, the adaptive image block merging layer performs the following steps:

[0037] Convolutional branch processing is used to learn the offset from the feature map output by the corresponding order EST and output it to the trunk merging layer;

[0038] The backbone merging layer is used to downsample the feature map of the corresponding order EST and output it to the next order EST;

[0039] The calculation formula of the adaptive image block merging layer is:

[0040]

[0041] Where: p0 represents the two-dimensional sampling point on the input feature map, x l represents the input feature variable calculated by EST, Represents the feature variable of the expansion dimension, n represents the number of the image block position in the receptive field R, w and p n Represents the sum of the sampling weight value and position in the receptive field R, where p offset is the sampling point offset learned from the convolution branch, G is the bilinear interpolation; q represents the feature area All spatial locations within.

[0042] Furthermore, step 3 specifically includes: using a skip connection module to fuse the multi-scale hierarchical features from the feature encoder with the decoded features of the convolutional decoding module; through linear transformation, the feature dimension extracted by the encoder is made consistent with the up-sampled feature dimension of the convolutional decoding module, thereby realizing the concatenation of shallow and deep features.

[0043] Furthermore, in step 4, projecting the restored feature map specifically includes mapping each channel component feature vector to a required number of categories based on the restored feature map.

[0044] A system for a UAV remote sensing image segmentation method based on a U-shaped adaptive Enhanced Swin Transformer is disclosed. The system comprises a feature extraction unit, a feature encoder, a skip connection module, a convolutional decoding module, and a projection module. The feature encoder comprises a multi-stage Enhanced Swin Transformer (EST) and a corresponding Deformable Adaptive Patch Merging (DeforAPM) layer.

[0045] Among them, each order of EST includes several EST units, and the EST unit includes two consecutive first EST modules and a second EST module. Each EST module includes a normalization layer LN, an enhanced multi-head self-attention module Multi-HeadSelf-Attention, a residual connection layer Residual Connection and a multi-layer perceptron MLP (a two-layer structure based on the Gaussian Error Linear Units (GELU) activation function); among them, the enhanced multi-head self-attention modules of the first EST module and the second EST module respectively adopt the window-based enhanced multi-head self-attention module W-EMHSA and the shift window-based enhanced multi-head self-attention module SW-EMHSA.

[0046] Furthermore, the multi-layer perceptron MLP adopts a two-layer structure based on a Gaussian error linear unit GELU activation function.

[0047] Beneficial effects of the present invention:

[0048] Based on CNN and Transformer modules, this paper proposes a U-shaped encoder-decoder model for remote sensing image segmentation. The encoder consists of an Enhanced Swin Transformer, which implements global self-attention through a hierarchical structure. In the decoder, upsampling is achieved by combining convolutional and deconvolution blocks. Upsampling restores the encoder information and performs corresponding pixel-level segmentation.

[0049] The present invention obtains image blocks on the input high-dimensional feature map instead of on the original image. This method can reduce the loss of edge and position information caused by directly performing image blocks on the original image, and can also solve the complexity of subsequent Transformer encoding.

[0050] This paper proposes an Enhanced Swin Transformer with Enhanced Multi-Head Self-Attention (EMHSA). This EMHSA improves the algorithm by strengthening the calculation of feature correlations to address the obvious redundancy and noise caused by the capture of global context in the Transformer.

[0051] The present invention implements an adaptive hierarchical implementation strategy by adding a deformable adaptive patch merging layer (DeforAPM) after each stage of the Enhanced Swin Transformer and applies it to the Transformer framework to reduce the inevitable pooling information pollution problem under ordinary fixed scale.

[0052] Other features and advantages of the present invention will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The above and other objects, features and advantages of the present invention will become more apparent through a more detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present invention.

[0054] Figure 1 A flow chart of the present invention is shown.

[0055] Figure 2 Shown is the overall model framework diagram of the present invention.

[0056] Figure 3 The figure shows the structure diagram of the enhanced multi-head self-attention EMHSA framework in the present invention.

[0057] Figure 4 The figure shows the framework structure diagram of the multi-scale attention module in the present invention.

[0058] Figure 5 The figure shows the structural diagram of the feature encoder framework in the present invention.

[0059] Figure 6The figure shows the framework structure diagram of the adaptive image block merging layer DAPM in the present invention.

[0060] Figure 7 A schematic diagram showing the comparison of segmentation results of the six models in the embodiment on the WHDLD dataset is shown.

[0061] (a) Manual labeling, (b) U-shaped convolutional model, (c) pyramid pooling model, (d) deep high-resolution recognition model, (e) simple and effective Transformer segmentation model, (f) U-shaped pure Transformer model, and (g) the model of the present invention.

[0062] Figure 8 A schematic diagram comparing the segmentation results of five decoders in the embodiment is shown.

[0063] (a) Manual labeling, (b) asymmetric adaptive pooling upsampling, (c) direct upsampling based on multi-layer perceptron (MLP), (d) upsampling based on image block expansion, and (e) cascaded symmetric upsampling (the method adopted by this invention). DETAILED DESCRIPTION

[0064] The preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0065] A UAV remote sensing image segmentation method based on a U-shaped adaptive Enhanced Swin Transformer includes the following steps:

[0066] Step 1: Extract features from the input image and obtain the encoding sequence of high-dimensional features, specifically:

[0067] Step 1-1: Extract features of the input image through the residual neural network ResNet50 to generate a high-dimensional feature map of the input image;

[0068] Step 1-2: Split the high-dimensional feature map into a series of non-overlapping image blocks;

[0069] Step 1-3: Convert the image block segmentation results into an embedding sequence;

[0070] Step 1-4: The embedded sequence is passed through a linear mapping layer to generate a one-dimensional hierarchical feature representation, namely the encoding sequence.

[0071] Step 2: Input the coding sequence into a feature encoder for feature extraction to generate multi-scale hierarchical features; the feature encoder includes a multi-order EST (Enhanced Swin Transformer) and a corresponding Deformable Adaptive Patch Merging (DeforAPM) layer, wherein each order of EST includes a number of EST units; Step 2 specifically includes:

[0072] Step 2-1: Input the coding sequence into the first-order EST for processing;

[0073] Step 2-2: Output the result of the first-order EST to the skip connection module of the corresponding order; at the same time, the result of the first-order EST is downsampled using the Deformable Adaptive Patch Merging (DeforAPM) layer to generate first-order multi-scale features and output them to the second-order EST;

[0074] Step 2-3: The second-order EST processes the output of the first-order EST and outputs the result of the second-order EST to the skip connection module of the corresponding order according to step 2-2; at the same time, the adaptive image block merging layer is used to downsample the result of the second-order EST to generate second-order multi-scale features and output them to the third-order EST;

[0075] Traverse each order of EST, complete the encoding, and generate multi-scale hierarchical features. In steps 2-2 and 2-3, the adaptive image block merging layer performs the following steps:

[0076] Convolutional branch processing is used to learn the offset from the feature map output by the corresponding order EST and output it to the trunk merging layer;

[0077] The backbone merging layer is used to downsample the feature map of the corresponding order EST and output it to the next order EST;

[0078] The calculation formula of the adaptive image block merging layer is:

[0079]

[0080] Where: p0 represents the two-dimensional sampling point on the input feature map, x l represents the input feature variable calculated by EST, Represents the feature variable of the expansion dimension, n represents the number of the image block position in the receptive field R, w and p n Represents the sum of the sampling weight value and position in the receptive field R, where p offset is the sampling point offset learned from the convolution branch, G is the bilinear interpolation; q represents the feature area All spatial locations within.

[0081] Step 3: The multi-scale hierarchical features obtained by the feature encoder are input into the convolution decoding module through the skip connection module for fusion, and the feature map is restored to obtain the restored feature map; specifically, the skip connection module is used to fuse the multi-scale hierarchical features from the feature encoder with the decoded features of the convolution decoding module; through linear transformation, the feature dimension extracted by the encoder is made consistent with the up-sampled feature dimension of the convolution decoding module, thereby realizing the concatenation of shallow and deep features.

[0082] Step 4: Projection is performed based on the restored feature map to obtain pixel-level segmentation results; the projecting of the restored feature map is specifically as follows: mapping each channel component feature vector to the required number of categories based on the restored feature map.

[0083] In step 2, the EST unit includes two consecutive first EST modules and a second EST module, each of which includes a normalization layer LN, an enhanced multi-head self-attention module Multi-Head Self-Attention, a residual connection layer Residual Connection, and a multi-layer perceptron MLP (a two-layer structure based on the Gaussian Error Linear Units (GELU) activation function); wherein the enhanced multi-head self-attention modules of the first EST module and the second EST module respectively adopt the window-based enhanced multi-head self-attention module W-EMHSA and the shifted window-based enhanced multi-head self-attention module SW-EMHSA;

[0084] The EST unit processes the input coding sequence using the following formula:

[0085]

[0086]

[0087]

[0088]

[0089] Where: x represents the characteristic variable, l represents the number of EST modules in the EST unit; x l-1 represents the input coding sequence; represents the W-EMHSA output of the lth block; x l represents the output of the MLP of the lth block; represents the SW-EMHSA output of the lth block; x l+1 represents the MLP output of the lth block; x l+1 Represents the output coding sequence;

[0090] The enhanced multi-head self-attention module uses the following formula for calculation:

[0091]

[0092] Where: Q, K, V are the corresponding input code sequence LN(x l )、LN(x l-1 ) After three linear transformations and mappings, the resulting query, key, and value are obtained; M 2 represents the number of image blocks in the window, d represents the embedding dimension of query Q and key value K; B represents the relative position of the current image block within the window range, R represents the receptive field, which is a natural number;

[0093] att represents the multi-scale attention used after the QK correlation matrix, which is calculated using the following formula:

[0094]

[0095] L(X)=LN(FC2(δ(FC1(X)))), (7)

[0096] G(X)=LN(FC2(δ(FC1(GAP(X)))), (8)

[0097] Where: X MS-Attention ∈R k×m×m Represents a feature with multi-scale attention, k represents the number of channels of the feature, and m represents the width and height of the feature; W(X)∈R k×m×m Represents the weight coefficient of multi-scale attention features, L(x)∈R k×m×m Represents local channel context information, G(X)∈R k×1×1 Represents global feature context information, k represents feature dimension; FC1 and FC2 are both fully connected layers for reducing and expanding size; GAP represents average pooling, δ represents the combination of layer normalization and activation function ReLU, σ is Sigmoid function, LN represents layer normalization, and They represent element-wise multiplication and addition, respectively.

[0098] A system for a UAV remote sensing image segmentation method based on a U-shaped adaptive Enhanced Swin Transformer is provided, the system comprising a feature extraction unit, a feature encoder, a skip connection module, a convolutional decoding module, and a projection module; wherein,

[0099] The feature encoder includes a multi-stage EST (Enhanced Swin Transformer) and a corresponding adaptive image block merging layer (Deformable Adaptive Patch Merging, DeforAPM);

[0100] Among them, each order of EST includes several EST units, and the EST unit includes two consecutive first EST modules and a second EST module. Each EST module includes a normalization layer LN, an enhanced multi-head self-attention module Multi-HeadSelf-Attention, a residual connection layer Residual Connection and a multi-layer perceptron MLP (a two-layer structure based on the Gaussian Error Linear Units (GELU) activation function); among them, the enhanced multi-head self-attention modules of the first EST module and the second EST module respectively adopt the window-based enhanced multi-head self-attention module W-EMHSA and the shift window-based enhanced multi-head self-attention module SW-EMHSA.

[0101] When implementing:

[0102] A UAV remote sensing image segmentation method based on a U-shaped adaptive Enhanced Swin Transformer includes the following steps:

[0103] Step 1: Extract features from the input image and encode the obtained high-dimensional features through linear mapping;

[0104] Step 2: The encoded sequence obtained in step 1 is used as input and sequentially passes through four feature extraction stages, including multiple Enhanced SwinTransformer blocks and deformable adaptive downsampling layers, to generate a hierarchical feature representation;

[0105] Step 3: The multi-scale contextual features obtained by the Transformer encoder are fused with the features recovered by the convolutional layer-based decoder through skip connections;

[0106] Step 4: After four cascaded upsampling layers, the feature map resolution is restored to the input resolution size W×H, and the resulting upsampled feature map is projected to obtain the pixel-level segmentation prediction result.

[0107] The model was trained on a CPU with 40GB of memory and an Nvidia A100 GPU. The algorithm was implemented using Python 3.8 and the PyTorch 1.7 deep learning framework.

[0108] The specific model training and test parameters are as follows: the input image size and image block size are set to 256×256 (512×512) and 1, respectively.

[0109] We trained all models from scratch for 160k iterations, with an initial learning rate of 0.01 and 15k linear iterations, respectively. All Transformer models used a random depth of 0.2. We used the official ImageNet-based pre-trained weights as initialization parameters. During training, we used a batch size of 4 and a stochastic gradient descent (SGD) optimizer with a momentum of 0.9 and a weight decay of 2e-5 to optimize our backpropagation model.

[0110] Experiments were conducted on two different types of UAV remote sensing image datasets:

[0111] Wuhan Land Surface Detection Dataset (WHDLD): This dataset, captured by a drone at an altitude of 2 meters, consists of 4,940 256×256 images across six categories. This data was cropped from a large remote sensing image of the Wuhan urban area. This paper uses mean intersection over union (mIoU), mean pixel accuracy (Acc), and mean Dice similarity coefficient (DSC) as evaluation metrics to assess six categories (buildings, roads, sidewalks, farmland, bare land, and wasteland).

[0112] LoveDA (Land Cover Dataset for Adaptive Segmentation): This dataset, captured from a 3-meter altitude, contains 9,840 drone-generated images of 1024×1024 pixels across seven categories (buildings, roads, wasteland, water, farmland, forest, and bare land) covering various rural and urban areas. This dataset is more challenging due to the complex background samples and the large number of multi-scale objects. The mean intersection over union (mIoU), mean pixel accuracy (Acc), and mean Dice similarity coefficient (DSC) are still used as evaluation metrics.

[0113] For these two drone remote public datasets, the present invention divides them into training set and test set with a ratio of 7:3. In addition, the present invention also applies data enhancement (e.g., random rotation, flipping and padding) to avoid overfitting.

[0114] This experimental section comprehensively evaluates the U-shaped adaptive EST model on the WHDLD and LoveDA datasets, focusing on evaluation metrics and visualization. To better validate the model's performance, we compared the U-shaped adaptive EST model with nine other segmentation methods, including CNN-based methods: U-shaped convolutional network (U-Net), pyramid pooling model (PSPNet), dilated separable convolutional encoder-decoder (DeepLabV3+), and deep high-resolution recognition network (HRNet); and Transformer-based methods: visual transformer (ViT), segmentation transformer (SETR), simple and effective transformer segmentation model (Segformer), Transformer-based U-shaped strong encoder-decoder (TransUnet), and U-shaped pure Transformer model (Swin-Unet). To ensure fair comparison, all pre-trained models use officially provided data. The ImageNet pre-training model is used for the Transformer model, the VGG11 pre-training model is used for the U-Net, the W32 pre-training model is used for HRNet, and the ResNet-50 pre-training model is used for DeepLabV3+ and PSPNet.

[0115] Table 1 Comparison of segmentation accuracy of each model on the WHDLD dataset

[0116] unit:%

[0117]

[0118] Table 2 Comparison of segmentation accuracy of each model on the LoveDA dataset.

[0119] unit:%

[0120]

[0121] Table 1 shows the comparative results of the U-shaped adaptive EST model on the WHDLD dataset. The experimental results show that the U-shaped adaptive EST model achieves the best segmentation performance, achieving an Intersection-over-Union (IoU) of 64.31%, an accuracy of 76.14%, and a Dice similarity of 76.81%. The U-shaped adaptive EST model achieves higher accuracy improvements for small regions (such as roads and sidewalks) than for large regions (such as buildings and farmland). The accuracy and DSC evaluation metric improve by approximately 4.08% and 3.4%, respectively, and the IoU improves by approximately 3.84%, demonstrating the effectiveness of our network. Compared to the classic Vit and SETR, our approach achieves significant improvements in accuracy. Vit and SETR, however, generally underperform the remaining CNN models, demonstrating the inadequacy of directly porting transformers to semantic segmentation on small and medium-sized datasets. TransUnet, also a hybrid Transformer architecture, achieves good results on small categories, but at the expense of accuracy on other categories. In comparison, Segformer and Swin-Unet achieve better results, with the latter exhibiting improved model stability. These improvements demonstrate that our Transformer achieves state-of-the-art results. Similarly, we also compared several multi-scale CNN-based remote sensing segmentation models, with PSPNet, DeepLabV3+, and HRNet outperforming U-Net, achieving 61.46% IoU, 63.32% accuracy, and 63.37% Dice similarity. The above analysis demonstrates that directly applying the Transformer to small and medium-sized remote sensing datasets underperforms CNN results. This is because the high-level semantic features captured by the Transformer are beneficial for classification, but lack low-level cues for fine-grained shape segmentation. However, CNNs also suffer from underfitting and overfitting issues, which are caused by the local nature of convolutional operations. For example, U-Net achieves excellent segmentation results for large area categories (such as water and agriculture), but suffers from lower accuracy for small categories (such as bare land and roads). In this work, we demonstrate that by integrating a hybrid Transformer encoder with U-shaped cascade upsampling with skip connections, we can better integrate local and global semantic information, resulting in superior segmentation results.

[0122] We also performed a qualitative comparison on the WHDLD dataset. Figure 7As shown in the figure, this method achieves more accurate segmentation results. CNN-based methods, such as U-Net and PSPNet, often suffer from overfitting or underfitting in different regions (for example, in the first column, water is over-segmented by U-Net and under-segmented by PSPNet), which is caused by the local nature of the convolution operation. Furthermore, more complex multi-scale models, such as DeepLabV3+ and HRNet, can achieve better results. In contrast, transformer-based models are better at encoding global context and distinguishing semantics, and they also produce fewer false negatives. However, due to scale limitations, the model cannot capture sufficient feature information, preventing it from achieving excellent results on small and medium-sized datasets. In this work, we demonstrate that the U-shaped adaptive EST model achieves better prediction results on multiple categories, especially for categories with large variations in boundaries and shapes (for example, roads in the second column and buildings on roads in the fourth column). Furthermore, the results in the second and third rows show that our method predicts more accurately for categories with a small proportion, suggesting that the U-shaped adaptive EST model can correct for class imbalance by providing sufficient contextual information for each category. These results further validate the correctness of our method.

[0123] Similarly, we conducted the same experiments on the LoveDA dataset. The experimental results are shown in Table 2. The results demonstrate that the U-shaped adaptive EST model maintains good performance on a more complex and larger dataset, achieving an IoU of 54.39%, an accuracy of 66.81%, and a DSC similarity of 69.5%. Although ViT and TransUnet are slightly less efficient, they achieve segmentation accuracy comparable to most multi-scale CNN models. Furthermore, HRNet achieves comparable results to the Transformer (Segformer). Other transformer methods also show significant improvements across all evaluation metrics. SETR surpasses Segformer and Swin-Unet, achieving accuracy second only to our method. Swin-Unet achieves better results on small-category bare land regions, while SETR achieves better results on large-category farmland regions, but falls short of our method in other aspects. These experiments further demonstrate the advantages of our hybrid approach, which better addresses class imbalance, handles both inter- and intra-class variations, and adapts to diverse datasets.

[0124] To evaluate the proposed U-shaped adaptive EST model framework and verify its performance under different factors, we also conducted various ablation studies, including:

[0125] 1) Different upsampling methods;

[0126] 2) Enhanced Multi-Head Self-Attention (EMHSA) in Enhanced Swin Transformer;

[0127] 3) Deformable adaptive image block merging layer;

[0128] 4) Input image size, pre-trained model size, and model specifications.

[0129] To address the scale change achieved by the block merging layer in the encoder, the present invention designs corresponding upsampling and dimensionality reduction modules in the decoder. To explore the effectiveness of the decoder used in the present invention, the present invention conducted experiments on the network U-shaped adaptive EST model using three methods: U-shaped symmetric cascade upsampling, asymmetric pooling upsampling, and direct upsampling on the WHDLD remote sensing dataset.

[0130] In order to explore the effectiveness of the decoder used in the present invention, four different methods were experimented on the WHDLD dataset, including U-shaped cascade upsampling, asymmetric adaptive pooling upsampling (Adaptive Pyramid Context Network, APC), direct upsampling based on multi-layer perceptron MLP, and the image block expansion upsampling layer corresponding to the image block merging layer proposed in Swin-Unet. The experimental results in Table 3 show that the U-shaped adaptive EST model proposed in the present invention combined with the U-shaped symmetric decoder can obtain better segmentation accuracy. The cascaded U-shaped upsampling structure can better retain the information lost by the encoder and make full use of local and global advantages to obtain better segmentation performance. It is worth mentioning that the performance improvement for small-scale area targets (such as paths) is obvious. However, for large-scale area targets (such as water areas and farmlands), the improvement is not obvious, and some methods are even not as good as other methods. As we can see, compared with the asymmetric upsampling method, the accuracy of road and bare land categories has increased by an average of 14.1% and 6.1%. Among them, due to the loss of information from multiple pooling at different scales, the segmentation effect of APC is not as good as the direct upsampling structure aggregated by multiple multi-layer perceptron MLPs. Figure 8 In

[15] , MLP still loses details because it directly concatenates features from different layers, ignoring information recovery achieved through skip connections and cascaded upsampling. The method based on the image patch expansion layer achieves significantly better results than the previous two methods. It is particularly advantageous in the segmentation of large-scale categories (such as water areas). The model also considers small-scale categories (such as bare land) to some extent, but its performance is still inferior to our method. In summary, the method proposed in this paper can more flexibly handle multiple scales and category imbalances, thereby obtaining more accurate segmentation results.

[0131] In order to verify the enhanced multi-head attention EMHSA proposed in the Enhanced Swin Transformer of the present invention, the present invention is compared with the original MHSA in the experiment, and the results are summarized in Table 4. It is observed that EMHSA with multi-scale attention has higher segmentation performance. For the U-shaped adaptive EST model, the use of EMHSA compared with MHSA increases the average intersection-over-union ratio by 1.21% and the average DSC by 1.22%. Among them, the improvement in segmentation accuracy for building and road categories is more obvious, followed by bare land and paths. The characteristics of these categories are that they account for a small proportion, but the differences are also large. This also proves that our model is effective for misjudgment of results caused by large category differences and category imbalance, because the EMHSA model with attention can better filter out the information we need from the global information.

[0132] The present invention discusses the impact of an adaptive patch merging layer (DAPM) based on model performance. As can be seen in Table 5. The average intersection-over-union ratio and DSC accuracy reached 64.7% and 77.05%, respectively, an improvement of 0.69% and 0.93%, respectively. As expected, DAPM reduces the resolution and takes into account the differences in the receptive fields required for objects of different sizes. There are significant improvements for buildings, roads, and paths. Among them, buildings are more dispersed and have large morphological differences, often appearing as dense small quadrilaterals. On the other hand, pavements and roads, as the smallest proportion in the dataset, are usually sparsely distributed in slender and curved shapes. The improvements in the above three categories further illustrate that DAPM has the ability to improve segmentation effects. It can avoid the loss of information of some small-sized and small-category objects caused by fixed-ratio downsampling, which helps the network improve segmentation effects.

[0133] Table 3 Decoder ablation experiment

[0134] unit:%

[0135]

[0136] Table 4 Transformer ablation experiment

[0137] unit:%

[0138]

[0139] Table 5 Ablation experiments of adaptive image patch merging layer

[0140] unit:%

[0141]

[0142] Table 6 Ablation experiments on input image size

[0143] unit:%

[0144]

[0145]

[0146] Table 7 Ablation experiments on pre-trained model size

[0147] unit:%

[0148]

[0149] Table 8 Ablation experiments on model scale

[0150] unit:%

[0151]

[0152] This paper investigates the impact of input image size and the image size used for training the U-shaped adaptive EST model. Unlike the default Transformer input resolution, training is performed on a higher-resolution 256×256 (512×512) image. The segmentation performance of the average intersection over union (IoU) on the WHDLD dataset is shown in Tables 6 and 7. When the input image size increases from 224×224 to 256×256 (512×512) and the window size remains constant, it is readily apparent that the larger the input size, the better the segmentation performance. This is due to the establishment of more complex dependencies between each element in the input. For pre-training, the size that best matches the dataset is selected to achieve higher segmentation performance. For example, the 224×224 pre-training size is more suitable for the 256×256 input size of the WHDLD dataset than the 384×384 pre-training size. Conversely, on the LoveDA dataset, 384×384 performs better than 224×224 with an input size of 512×512. To ensure running efficiency and realism, we adopt 256×256 and 512×512 as input sizes, and 224×224 and 384×384 pre-training sizes for WHDLD and LoveDA.

[0153] The present invention also discusses the impact of the U-shaped adaptive EST model on different model sizes, as shown in Table 8, mainly including two sizes: the micro "Tiny" and the basic "Base". Among them, the number of modules in the "Tiny" model stage3 is 6, and the number of heads in the multi-head attention is 24; while the "Base" model has a more complex model structure: the number of modules in stage3 is 18, and the number of heads in the multi-head attention is 32. From Tables 4-10, we conclude that larger models have better performance. Taking into account the computational cost and accuracy-speed of the model, we will use the "Tiny" model in the parameter comparison experiment, while the comparison of other modules and the overall model will still use the "Base" model to ensure the segmentation effect of the overall network.

[0154] This paper uses the pre-trained weights provided by the Swin Transformer to address the problem of poor model performance due to insufficient pre-training data. Furthermore, the present invention incorporates convolutions into the pre-encoding and decoding stages, fully integrating the two with skip connections in a U-shaped structure to achieve the desired advantages.

[0155] While various embodiments of the present invention have been described above, the above description is intended to be illustrative, not exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

Claims

1. A U-shaped adaptive EST UAV remote sensing image segmentation method, characterized in that: The following steps are involved: Step 1: Extract features from the input image to obtain a high-dimensional feature encoding sequence; Step 2: Input the coding sequence into the feature encoder for feature extraction to generate multi-scale hierarchical features; Step 3: The multi-scale hierarchical features obtained by the feature encoder are input into the convolutional decoding module through the skip connection module for fusion to complete the restoration of the feature map and obtain the restored feature map; Step 4: Project the restored feature map to obtain pixel-level segmentation results; Step 1 specifically includes: Step 1-1: Extract features of the input image through the residual neural network ResNet50 to generate a high-dimensional feature map of the input image; Step 1-2: Split the high-dimensional feature map into a series of non-overlapping image blocks; Step 1-3: Convert the image block segmentation results into an embedding sequence; Step 1-4: The embedded sequence is passed through a linear mapping layer to generate a one-dimensional hierarchical feature representation, i.e., the encoding sequence; The feature encoder includes a multi-stage EST, namely Enhanced Swin Transformer, and a corresponding adaptive image block merging layer DAPM, wherein each stage EST includes several EST units; step 2 specifically includes: Step 2-1: Input the coding sequence into the first-order EST for processing; Step 2-2: Output the result of the first-order EST to the skip connection module of the corresponding order; at the same time, use the adaptive image block merging layer DAPM to downsample the result of the first-order EST to generate first-order multi-scale features and output them to the second-order EST; Step 2-3: The second-order EST processes the first-order multi-scale features and outputs the results of the second-order EST to the skip connection module of the corresponding order; at the same time, the adaptive image block merging layer is used to downsample the results of the second-order EST to generate second-order multi-scale features and output them to the third-order EST; Output the multi-scale features of the current order EST to the next order EST, and traverse each order EST in turn to generate multi-scale hierarchical features; Step 3 specifically includes: using a skip connection module to fuse the multi-scale hierarchical features from the feature encoder with the decoded features of the convolutional decoding module; through linear transformation, the multi-scale hierarchical feature dimensions extracted by the feature encoder are made consistent with the upsampled feature dimensions of the convolutional decoding module, thereby realizing the concatenation of shallow and deep features; In step 4, projection based on the restored feature map is specifically as follows: mapping each channel component feature vector to the required number of categories based on the restored feature map; In steps 2-2 and 2-3, the adaptive image block merging layer performs the following steps: Convolutional branch processing is used to learn the offset from the feature map output by the corresponding order EST and output it to the trunk merging layer; The backbone merging layer is used to downsample the feature map of the corresponding order EST and output it to the next order EST; The calculation formula of the adaptive image block merging layer is: Where: p0 represents the two-dimensional sampling point on the input feature map, x l represents the input feature variable calculated by EST, Represents the feature variable of the expansion dimension, n represents the number of the image block position in the receptive field R, w and p n Represents the sum of the sampling weight value and position in the receptive field R, where p offset is the sampling point offset learned from the convolution branch, G is the bilinear interpolation; q represents the feature area All spatial locations within.

2. The U-shaped adaptive EST UAV remote sensing image segmentation method according to claim 1 is characterized in that: In step 2, the EST unit includes two consecutive first EST modules and a second EST module. Each EST module includes a normalization layer LN, an enhanced multi-head self-attention module EMHSA, a residual connection layer RC, and a multi-layer perceptron MLP. The enhanced multi-head self-attention modules of the first EST module and the second EST module respectively adopt the window-based enhanced multi-head self-attention module W-EMHSA and the shifted window-based enhanced multi-head self-attention module SW-EMHSA. The EST unit processes the input coding sequence using the following formula: Where: x represents the characteristic variable, l represents the number of EST modules in the EST unit; x l-1 represents the input coding sequence; represents the W-EMHSA output of the lth EST block; x l represents the output of the MLP of the lth EST block; represents the SW-EMHSA output of the lth EST block; x l+1 represents the MLP output of the l+1th EST block.

3. The U-shaped adaptive EST UAV remote sensing image segmentation method according to claim 2 is characterized in that: The enhanced multi-head self-attention module uses the following formula for calculation: Where: Q, K, V are the corresponding input code sequence LN(x l ) After three linear transformations and mappings, the resulting query, key, and value are obtained; M 2 represents the number of image blocks in the window, d represents the embedding dimension of query Q and key value K; B represents the relative position of the current image block within the window range, R represents the receptive field, which is a natural number; att represents the multi-scale attention used after the QK correlation matrix, which is calculated using the following formula: L(X)=LN(FC2(δ(FC1(X)))), (7) G(X)=LN(FC2(δ(FC1(GAP(X)))), (8) Where: X MS-Attention ∈R c×m×m Represents a feature with multi-scale attention, c represents the number of channels of the feature, and m represents the width and height of the feature; W(X)∈R k×m×m Represents the weight coefficient of multi-scale attention features, L(X)∈R k×m×m Represents local channel context information, G(X)∈R k×1×1 Represents global feature context information, k represents feature dimension; FC1 and FC2 are both fully connected layers for reducing and expanding size; GAP represents average pooling, δ represents the combination of normalization layer and activation function ReLU, σ is Sigmoid function, LN represents normalization layer, and ⊕ represent element-wise multiplication and addition respectively.

4. A system used in the U-shaped adaptive EST UAV remote sensing image segmentation method according to any one of claims 1 to 3, characterized in that: The system includes a feature extraction unit, a feature encoder, a skip connection module, a convolutional decoding module and a projection module. The feature encoder includes a multi-order EST and a corresponding adaptive image block merging layer DAPM; Among them, each order of EST includes several EST units, and the EST unit includes two consecutive first EST modules and second EST modules. Each EST module includes a normalization layer LN, an enhanced multi-head self-attention module EMHSA, a residual connection layer ResidualConnection and a multi-layer perceptron MLP; among them, the enhanced multi-head self-attention modules of the first EST module and the second EST module respectively adopt the window-based enhanced multi-head self-attention module W-EMHSA and the shift window-based enhanced multi-head self-attention module SW-EMHSA.

5. The system used in the U-shaped adaptive EST UAV remote sensing image segmentation method according to claim 4 is characterized in that: The multi-layer perceptron MLP adopts a two-layer structure based on the Gaussian error linear unit GELU activation function.

Images