A remote sensing image feature extraction method of a deformable alignment multi-scale fusion network

Abstract

The application discloses a remote sensing image ground object extraction method of a deformable alignment multi-scale fusion network, and relates to the field of remote sensing image intelligent interpretation and ground object extraction. The method comprises the following steps: acquiring a remote sensing semantic segmentation dataset and dividing the dataset into a training dataset and a test dataset; secondly, a deformable alignment multi-scale fusion network DDMGF-Segformer is constructed, which comprises a data preprocessing module, a MiT encoder module, a deformable alignment multi-scale fusion feature enhancement module, a decoder module and an output module; the training dataset is input into the DDMGF-Segformer network for iterative training, a loss value is calculated through a multi-class cross-entropy loss function, and model parameters are iteratively optimized through back propagation; finally, the test dataset is input into the trained DDMGF-Segformer network, and a ground object extraction result is output. The scheme can improve the segmentation precision and boundary quality of ground object categories in a scene with complex texture and significant scale change, and provides a reliable technical scheme for high-resolution remote sensing image ground object extraction.

Description

Technical Field

[0001] This invention relates to the field of remote sensing images, and in particular to a method for extracting ground features from remote sensing images based on a deformable aligned multi-scale fusion enhancement network. Background Technology

[0002] High-resolution remote sensing image semantic segmentation technology, with its advantages of large-area coverage and high timeliness, has become a core tool for land cover classification, urban expansion monitoring, and land resource surveys. The core objective of remote sensing image feature extraction is to obtain pixel-level classification information reflecting the attributes and spatial distribution of features from complex remote sensing images. However, in practical applications, the drastic changes in the scale of remote sensing features, their highly irregular geometric shapes, severe background interference, and information loss during downsampling consistently limit the extraction accuracy and robustness of boundary localization.

[0003] Current remote sensing image ground feature extraction techniques are mainly divided into two categories: traditional methods and deep learning methods. Traditional remote sensing image ground feature extraction methods, such as classifiers based on handcrafted features or conditional random fields, rely heavily on manually designed mathematical rules and lack adaptive learning capabilities. When dealing with complex scenes, they are easily affected by environmental noise, leading to the accumulation of unwrapping and classification errors. When facing regions with strong geometric and topological features, it is difficult to balance classification accuracy and edge detail preservation, often resulting in problems such as ground feature fragmentation, over-smoothing, or local false alarms propagating globally. Furthermore, it is difficult to balance computational efficiency with scene adaptability.

[0004] While existing deep learning methods for remote sensing image feature extraction have made progress in deep feature extraction, they still suffer from the following problems: First, they lack the ability to perceive geometric distortions of remote sensing image features. The sampling and feature extraction modes of mainstream models are fixed, lacking an explicit modeling mechanism for geometric distortions and irregular shapes of features. They cannot dynamically adjust the receptive field to accurately capture the geometric contours and spatial morphological differences of features. When processing distorted features such as tilted buildings and winding roads, it is easy to cause a mismatch between deep features and actual geometric structures, affecting the accuracy of category determination and boundary positioning. Second, the spatial alignment of feature features during upsampling lacks specificity. The decoder does not effectively utilize geometric offset information to match the distribution of features at abrupt changes in terrain, making the segmentation results prone to problems such as blurred feature edges, broken stripes, and discontinuous connections, reducing the completeness and accuracy of feature extraction.

[0005] To address the aforementioned problems in remote sensing image feature extraction, this invention proposes a method for remote sensing image feature extraction based on a deformable aligned multi-scale fusion enhancement network. This method, building upon the existing segformer network, introduces a deformable aligned multi-scale fusion feature enhancement module in the encoder feature extraction stage and a geometrically aligned multi-scale fusion module in the decoder upsampling stage. Through the dynamic offset learning mechanism of deformable convolutional kernels, this invention adaptively adjusts the sampling position, range, and angle of the receptive field based on the geometric steepness and morphological curvature of remote sensing feature edges, explicitly modeling the geometric distortion and irregular morphology of features, correcting geometric offsets at the feature level, and ensuring accurate matching of deep features to the actual geometric structure of features. Simultaneously, during decoder upsampling, deep semantic features at different scales are precisely spatially aligned with geometric offset information to construct multi-scale feature fusion weights, matching the distribution patterns of features at abrupt terrain changes. This method couples dynamic geometric perception with precise spatial alignment, achieving accurate capture of the geometric contours of remote sensing features and continuous, complete extraction of boundaries, improving the extraction accuracy and completeness of the model in scenes with complex geometric features. Summary of the Invention

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is a method for extracting ground features from remote sensing images using a deformable aligned multi-scale fusion network, the method comprising the following steps:

[0007] S1: Obtain the remote sensing image semantic segmentation dataset, and divide the remote sensing image semantic segmentation dataset into a training dataset and a test dataset;

[0008] S2: Construct a deformable aligned multi-scale fusion network DDMGF-Segformer, which includes a data preprocessing module, a MiT encoder module, a deformable aligned multi-scale fusion feature enhancement module, a decoder module, and an output module.

[0009] S3: Input the training dataset into the DDMGF-Segformer network for iterative training, calculate the loss value through the multi-class cross-entropy loss function, and iteratively optimize the model parameters through backpropagation;

[0010] S4: Input the test dataset into the trained DDMGF-Segformer network and output the predicted ground feature extraction results;

[0011] The deformable alignment multi-scale fusion feature enhancement module in S2 includes a feature calibration branch, a local deformable perception branch and a global multi-scale semantic branch set in parallel, a channel splicing module, a fusion enhancement module and a residual connection module.

[0012] The feature calibration branch includes a 1×1 convolutional layer, a batch normalization layer, and a ReLU activation function layer, all connected in series. The 1×1 convolutional layer compresses the dimensionality of the input features. The batch normalization layer stabilizes the feature distribution. The ReLU activation function layer introduces a nonlinear transformation to unify the channel dimensions, normalize the feature distribution, and regularize the basic spatial structure of the remote sensing image features, outputting the basic calibration features of the remote sensing image features.

[0013] The local deformable sensing branch includes a 1×1 convolutional layer, a batch normalization layer, a ReLU activation function layer, a 3×3 convolutional layer, a 3×3 deformable convolutional layer, a batch normalization layer, and a ReLU activation layer, connected in series. The 3×3 convolutional layer has an inflation rate of 2 and a padding rate of 2, used to obtain the offset of ground feature. The 3×3 deformable convolutional layer has an inflation rate of 2 and a padding rate of 2, used to adaptively sample the input features, extract the geometric morphological features of individual ground features, and output local semantic features.

[0014] The global multi-scale semantic branch includes a 1×1 convolutional layer, a batch normalization layer, a ReLU activation function layer, a 3×3 convolutional layer, a 3×3 deformable convolutional layer, a batch normalization layer, and a ReLU activation layer, connected in series. The 3×3 convolutional layer has an inflation rate of 4 and a padding rate of 4, used to obtain the offset of ground feature. The 3×3 deformable convolutional layer has an inflation rate of 4 and a padding rate of 4, used to extract the spatial distribution pattern of ground features in remote sensing images and the global semantic association information, outputting global semantic features.

[0015] The fusion enhancement module includes a 1×1 convolutional layer and a batch normalization layer, which are connected in series. The 1×1 convolutional layer is used to fuse multi-scale features of land features in remote sensing images and map the features back to the target output channel number of the module. The batch normalization layer is used for normalization processing and outputs denoised features.

[0016] The residual connection module includes a channel projection convolutional layer and an element-wise addition operator, which are connected in series. The channel projection convolutional layer includes a 1×1 convolutional kernel to keep the number of input feature channels consistent with the number of output feature channels. The element-wise addition operator adds the features processed by the channel projection convolutional layer to the features output by the fusion enhancement module element-wise, which is used to superimpose the enhanced features after the three-branch fusion, and output features with matching dimensions and enhanced information.

[0017] Furthermore, the encoder module includes a first MiT encoder layer, a second MiT encoder layer, a third MiT encoder layer, and a fourth MiT encoder layer;

[0018] The first MiT encoder layer includes a 7×7 convolutional layer, a layer normalization layer, a multi-head self-attention module, a feedforward network layer, and a residual connection layer. The 7×7 convolutional layer has a stride of 4 and padding of 3, used to map the remote sensing image from RGB three channels to a 64-dimensional feature space and achieve 4x downsampling. The multi-head self-attention module includes one attention head with a spatial reduction ratio of 8, which performs spatial downsampling on key-value features through convolution, while maintaining the original resolution of the query matrix. The feedforward network layer includes a first linear layer, a depthwise separable convolutional layer, a GELU activation function, and a second linear layer. The residual connection layer includes two branches, which connect the self-attention output and the feedforward network output to the input features, respectively.

[0019] The second MiT encoder layer includes a 3×3 convolutional layer, a layer normalization layer, a multi-head self-attention module, a feedforward network layer, and a residual connection layer. The 3×3 convolutional layer has a stride of 2 and padding of 1, used to map features from 64 channels to a 128-dimensional feature space and achieve 2x downsampling. The multi-head self-attention module includes two attention heads with a spatial reduction ratio of 4, which performs spatial downsampling on key-value features through convolution. The feedforward network layer includes a first linear layer, a depthwise separable convolutional layer, a GELU activation function, and a second linear layer. The residual connection layer includes two branches, connecting the self-attention output and the feedforward network output to the input features, respectively.

[0020] The third MiT encoder layer includes a 3×3 convolutional layer, a layer normalization layer, a multi-head self-attention module, a feedforward network layer, and a residual connection layer. The 3×3 convolutional layer has a stride of 2 and padding of 1, used to map features from 128 channels to a 320-dimensional feature space and achieve 2x downsampling. The multi-head self-attention module includes 5 attention heads with a spatial reduction ratio of 2, which performs spatial downsampling on key-value features through convolution. The feedforward network layer includes a first linear layer, a depthwise separable convolutional layer, a GELU activation function, and a second linear layer. The residual connection layer includes two branches, connecting the self-attention output and the feedforward network output to the input features, respectively.

[0021] The fourth MiT encoder layer includes a 3×3 convolutional layer, a layer normalization layer, a multi-head self-attention module, a feedforward network layer, and a residual connection layer. The 3×3 convolutional layer has a stride of 2 and padding of 1, used to map features from 320 channels to a 512-dimensional feature space and achieve 2x downsampling. The multi-head self-attention module includes 8 attention heads with a spatial reduction ratio of 1, employing a full-resolution attention mechanism without key-value feature downsampling. The feedforward network layer includes a first linear layer, a depthwise separable convolutional layer, a GELU activation function, and a second linear layer. The residual connection layer includes two branches, connecting the self-attention output and the feedforward network output to the input features, respectively.

[0022] Furthermore, the channel stitching module of the deformable aligned multi-scale fusion feature enhancement module in S2 adopts tensor stitching with a stitching dimension of 1, which is used to stitch together the remote sensing image ground feature output from three parallel branches.

[0023] Furthermore, the residual connection module of the deformable aligned multi-scale fusion feature enhancement module in S2 includes a channel projection convolutional layer; the channel projection convolutional layer is enabled only when the number of input channels of the residual connection module is inconsistent with the number of output channels of the fusion enhancement module.

[0024] Furthermore, the element-wise addition operator of the deformable aligned multi-scale fusion feature enhancement module in S2 is a residual superposition structure, which is used to preserve the deep semantic information in the original remote sensing image land cover features.

[0025] Furthermore, the decoder module includes a multi-scale feature embedding layer, a feature upsampling alignment layer, a feature splicing and fusion layer, and a classification prediction layer;

[0026] The multi-scale feature embedding layer includes four parallel 1×1 convolutional layers, which respectively transform the channel dimension of the four features output by the encoder, so that features with different numbers of channels can be uniformly mapped to the same embedding dimension.

[0027] The feature upsampling alignment layer includes a first bilinear interpolation operator, a second bilinear interpolation operator, and a third bilinear interpolation operator; the first bilinear interpolation operator is used to perform an upsampling operation on the input features; the second bilinear interpolation operator is used to sequentially upsample features with different spatial resolutions to the same spatial size; and the third bilinear interpolation operator is used to align the spatial resolutions of all features.

[0028] The feature splicing and fusion layer uses tensor splicing to splice the input features along the channel dimension, with a splicing dimension of 1.

[0029] The classification prediction layer includes a 1×1 convolutional layer, a softmax activation layer, and a bilinear interpolation operator. The 1×1 convolutional layer is used to map the input features to a channel dimension that matches the number of land cover categories. The softmax activation layer normalizes the output features of the 1×1 convolutional layer, outputting remote sensing image land cover features with a dimension that is the product of the number of categories and the aligned spatial size. The bilinear interpolation operator is used to upsample the remote sensing image land cover features to the original remote sensing image size, outputting category probability features.

[0030] Furthermore, the output module includes a remote sensing image land cover category conversion layer, a size upsampling layer, and a large-format mosaic layer;

[0031] The remote sensing image land cover category conversion layer includes an argmax operation module, which performs pixel-by-pixel index mapping on the category probability features to generate a land cover category label map;

[0032] The size upsampling layer includes a bilinear interpolation operator for upsampling the category label map from the network output size to the original image size;

[0033] The large-format mosaic layer includes a sliding window segmentation module, a sub-block inference module, and a weighted fusion module. The sliding window segmentation module divides the large-format image into overlapping sub-blocks. The sub-block inference module is used to predict each sub-block independently. The weighted fusion module uses a weighted average mechanism to complete the mosaicking of sub-block results and outputs the remote sensing image ground feature extraction results.

[0034] The beneficial effects of adopting the above technical solution are as follows:

[0035] 1. To address the insufficient perception of geometric distortion in remote sensing image feature extraction, a deformable alignment multi-scale fusion feature enhancement module is proposed. This module generates feature sampling offsets through a deformable convolutional network in a three-branch parallel architecture, adaptively resampling and spatially calibrating the deep features output by the encoder. This dynamically adapts to irregular feature morphologies in remote sensing images, such as tilted buildings, curved roads, and scattered tree distribution. The feature calibration branch in the module performs lightweight feature alignment, while the local / global feature branches utilize deformable convolutions with different dilation rates to achieve multi-scale geometric correction. This effectively corrects spatial position shifts and feature misalignments caused by multiple downsampling, ensuring precise alignment between strong semantic features and the true geometric boundaries of the features, thus improving the model's ability to extract features with complex geometric structures from remote sensing images.

[0036] 2. To address the issue of inaccurate spatial alignment in the upsampling of remote sensing image feature extraction decoders, a deformable alignment multi-scale fusion feature enhancement module is proposed. This module outputs enhanced deep features, providing the decoder with basic features that combine accurate geometric alignment with multi-scale semantic information. When the decoder performs upsampling based on these enhanced features, it can achieve differentiated upsampling by relying on the geometric information pre-calibrated by the module. During resolution restoration, it enhances details of remote sensing image feature edges and steep high-gradient areas such as building corners and road edges, effectively suppressing common defects in remote sensing image feature extraction such as blurred feature edges and road breaks. At the same time, the module's fusion enhancement unit and residual connection layer achieve noise suppression and feature enhancement, balancing noise suppression and geometric edge fidelity under complex backgrounds of remote sensing images, such as cloud cover, shadow interference, and low signal-to-noise ratio conditions. This ensures the topological continuity of the remote sensing image feature extraction results and improves the accuracy and robustness of remote sensing image feature edge extraction. Attached Figure Description

[0037] Figure 1This is a flowchart illustrating the method described in an embodiment of the present invention.

[0038] Figure 2 This is a schematic diagram of the deformable aligned multi-scale fusion network of the method described in the embodiments of the present invention.

[0039] Figure 3 This is a schematic diagram of the deformable aligned multi-scale fusion feature enhancement module according to an embodiment of the present invention.

[0040] Figure 4 Test data for embodiments of the present invention

[0041] Figure 5 The truth label for the test data in this embodiment of the invention.

[0042] Figure 6 The ground feature extraction results of the segformer network in this embodiment of the invention.

[0043] Figure 7 The ground feature extraction results of the DDMGF-Segformer network in this embodiment of the invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0045] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] A schematic diagram of a remote sensing image feature extraction method based on deformable aligned multi-scale fusion enhancement network is shown below. Figure 1 As shown, it includes the following steps:

[0048] S1: Obtain the Potsdam remote sensing image semantic segmentation dataset, and divide the remote sensing image semantic segmentation dataset into a training dataset and a test dataset;

[0049] S2: Construct a deformable aligned multi-scale fusion network DDMGF-Segformer. The DDMGF-Segformer network includes a data preprocessing module, a MiT encoder module, a deformable aligned multi-scale fusion feature enhancement module, a decoder module, and an output module. The parameter information of the DDMGF-Segformer network is shown in Table 1.

[0050] In this diagram, Mix Transformer Stages 1 through 4 represent the first to fourth encoding layers of the MiT encoder module, respectively. The DDMGF feature calibration branch, DDMGF local deformable perception branch, and DDMGF global multi-scale semantic branch represent the three parallel processing branches of the deformable alignment multi-scale fusion module. The DDMGF channel stitching layer performs channel dimension stitching of the three-branch features. The DDMGF fusion unit performs channel dimension restoration and noise suppression of the stitched features. The DDMGF residual connection layer performs residual fusion of the fused features and the original input features. SegFormerHead MLP embedding represents the multi-scale feature embedding layer, used for multi-layer perceptron linear embedding and upsampling alignment of the encoder's fourth-level output features. The SegFormerHead channel stitching layer represents the feature upsampling alignment layer of the decoder module, used for multi-scale feature channel stitching. SegFormerHead linear fusion represents the feature stitching fusion layer of the decoder module, used for linear feature fusion. The SegFormerHead prediction head represents the classification prediction layer of the decoder module, used for pixel-level category prediction. The Detect classification layer represents the final pixel-level land cover category determination stage.

[0051] Table 1 - DDMGF-Segformer Network Parameter Information

[0052]

[0053] A schematic diagram of the deformable aligned multi-scale fusion network DDMGF-Segformer is shown below. Figure 2 As shown. The specific process is as follows:

[0054] The data preprocessing module is used to convert raw RGB remote sensing images with inconsistent formats and scales into a standard tensor format that can be directly trained or inferred by the network. Simultaneously, it enhances the model's generalization ability through data augmentation during the training phase. This module takes raw RGB remote sensing images as input. First, it reads .tif format remote sensing image files using a custom data loader, converting the original height c × width h × channel w color remote sensing images into a format standard for deep learning frameworks. It then generates batch tensors through batch reading and performs normalization according to the ImageNet pre-training standard. The calculation formula is as follows:

[0055]

[0056] Eliminating differences in illumination and radiation scales between different remote sensing data sources, For the first The mean of the channel, For the first Standard deviation of the channel For the normalized data, In the batch input of raw remote sensing images, the first Passage, height ,width The pixel values ​​of the location are used; bilinear interpolation is used to uniformly adjust the standardized image to a fixed size of 512×512 to ensure dimensionality consistency during batch training; during the inference phase, the standardized tensor is directly output as the network input, and during the training phase, it is subjected to multi-dimensional random data augmentation operations such as random scaling, random cropping, horizontal flipping, random rotation, and RGB channel photometric distortion, and finally outputs a standardized training tensor with dimensions [B,3,512,512] as the input of the encoder module.

[0057] The MiT encoder module employs four encoding layers of the original Segformer B1 network architecture: the first MiT encoder layer, the second MiT encoder layer, the third MiT encoder layer, and the fourth MiT encoder layer, denoted as Stage1, Stage2, Stage3, and Stage4. Using the standard tensor output from the data preprocessing module as input, it extracts multi-scale hierarchical features from shallow textures to high-level semantics through progressive downsampling operations including block embedding, Transformer block feature extraction, and block merging, forming a feature pyramid. The core Transformer block of this module consists of a multi-head self-attention network and a feedforward network, combined with residual connections and layer normalization. Stage 1 performs 7×7 convolutional block embedding, sine and cosine positional encoding, and one Transformer block feature extraction on the input tensor, outputting shallow texture features c1 with 4x downsampling and 64 channels. Stage 2 performs 3×3 convolutional block merging and three Transformer block deep feature extraction on c1, outputting intermediate preliminary semantic features c2 with 8x downsampling and 128 channels. Stage 3 performs 3×3 convolutional block merging and 18 Transformer block deep feature learning on c2, outputting intermediate semantic features c3 with 16x downsampling and 320 channels. Stage 4 performs 3×3 convolutional block merging and three Transformer block high-level semantic abstraction on c3, outputting high-level semantic features c4 with 32x downsampling and 512 channels. Finally, c1, c2, c3, and c4 constitute a multi-scale feature pyramid, realizing a feature hierarchy expression from local texture to global semantics and from fine-grained to coarse-grained, laying the foundation for multi-scale fusion of the decoder.

[0058] The deformable alignment multi-scale fusion feature enhancement module enhances the feature c4 with dimensions [B, 512, 16, 16] output from Stage4. Using feature c4 as input, a three-branch parallel architecture is employed to achieve multi-dimensional feature enhancement, ultimately outputting an enhanced deep semantic feature c4e with the same number of channels and dimensions as the input c4. The first branch is the feature calibration branch, consisting of a 1×1 convolutional layer, a BatchNorm2d layer, and a ReLU activation layer connected in series. This branch compresses the input c4 to 128 dimensions and performs non-linear correction, achieving lightweight feature space alignment and outputting a feature with dimensions [B, 128, 16, 16]. The basic alignment features; the second branch is the local deformable perception branch, which consists of a 1×1 convolutional layer, a BatchNorm2d layer, a ReLU activation layer, a 3×3 convolutional layer, a 3×3 deformable convolutional layer, a BatchNorm2d layer, and a ReLU activation layer connected in series. The 3×3 convolutional layer has a dilation rate of 2 and a padding rate of 2, and the 3×3 deformable convolutional layer has a dilation rate of 2 and a padding rate of 2. The 3×3 convolutional layer generates a sampling offset map, which guides the deformable convolutional sampling points to adaptively offset according to the deformation of the ground objects, thereby realizing feature space alignment correction and outputting semantic features in the mid-receptive field with dimensions [B, 128, 16, 16]. The third branch is the global multi-receptive feature. The scale semantic branch consists of a 1×1 convolutional layer, a BatchNorm2d layer, a ReLU activation layer, a 3×3 convolutional layer, a 3×3 deformable convolutional layer, a BatchNorm2d layer, and a ReLU activation layer connected in series. The 3×3 convolutional layer has a dilation rate of 4 and padding of 4, and the 3×3 deformable convolutional layer also has a dilation rate of 4 and padding of 4. The deformable convolution with a larger dilation rate expands the receptive field, capturing global contextual information of a large area of ​​ground features, and outputs a large receptive field global feature with dimensions [B, 128, 16, 16]. The output features from the three branches are then fused by a channel concatenation operator into a concatenated feature of [B, 384, 16, 16]. The input is fed into a fusion unit consisting of a 1×1 convolutional layer and a BatchNorm2d layer. The 1×1 convolutional layer maps the concatenated features back to 512-dimensional channels, and the BatchNorm2d layer performs feature noise suppression, suppressing spurious activations caused by noise such as illumination changes and shadows. Finally, the output features of the fusion unit are fused with the original input c4 through a residual connection layer. When the channel dimensions do not match, c4 is first adapted to the channels through a 1×1 convolutional projection layer, and then the features are added by an element-wise addition operator to ensure the continuity of the feature flow and avoid the loss of deep feature information. The final output is an enhanced feature c4e with dimensions [B, 512, 16, 16].

[0059] The decoder module takes features c1, c2, and c3 output from the MiT encoder module and the enhanced feature c4e as input. Through four steps—multi-scale feature embedding, spatial upsampling alignment, channel stitching and fusion, and classification prediction—it achieves deep fusion of multi-scale features and gradual restoration of spatial resolution, ultimately generating pixel-level land cover category probability distributions. First, the module uses a multi-scale feature embedding layer to perform linear embedding of the four levels of features using a single fully connected layer in a multilayer perceptron, mapping the different channel numbers (64, 128, 320, and 512) to a 256-dimensional common embedding space. Then, a feature upsampling alignment layer uses bilinear interpolation to upsample the embedded c2, c3, and c4e by 2x, 4x, and 8x respectively, aligning them to the same 128×128 size as c1. Finally, a feature stitching and fusion layer stitches the four-level aligned features together in the channel dimension to form 1024 channels. The fused features are then refined using a combination of 1×1 and 3×3 convolutions, and the channels are compressed to 256. At the same time, random deactivation regularization is performed to prevent the decoder from overfitting. Finally, a classification prediction layer is used. First, a 1×1 convolution maps the refined features of 256 channels to a 6-channel class score map. Then, a Softmax activation function is performed on the channel dimension to convert the class scores into probability values ​​between 0 and 1, while ensuring that the sum of the probabilities of all classes for a single pixel is 1. The final output is a land cover class probability map with dimensions [B,6,128,128].

[0060] The output processing module takes the category probability map output by the decoder as input, and transforms the probability distribution predicted by the network into the final pixel-level land feature segmentation map through a three-layer network of pixel-by-pixel category determination layer, size restoration layer and large-format image sliding window inference layer. It also solves the problem of inference memory limitation of large-format remote sensing images, and finally outputs a pixel-level land feature segmentation map with the same size as the original remote sensing image, realizing pixel-level accurate land feature extraction. The pixel-by-pixel category determination layer first performs a maximum value indexing operation on the probability map in the channel dimension to generate a numerical label map. Then, it establishes a one-to-one correspondence between numerical labels from 0 to 5 and six types of land features: impervious surfaces, buildings, low vegetation, trees, cars, and debris, realizing the conversion from pixel-level category probability distribution to discrete land feature category labels. The size restoration layer uses bilinear interpolation to restore the numerical label map from 128×128 to the size of the original remote sensing image. After interpolation, the pixel values ​​are rounded to maintain the discreteness of the numerical labels. The large-format image sliding window inference layer is only used in the inference stage. For large-format remote sensing images, it is first divided into multiple overlapping sub-blocks with a sub-block size of 1024×1024, an inter-block step size of 768×768, and an overlapping area of ​​256×256. Then, each sub-block is sequentially input into the preceding module to obtain the corresponding sub-block label map. Subsequently, a distance-weighted average strategy is used to achieve seamless fusion of the overlapping areas of multiple sub-blocks. Finally, the fused sub-block label maps are stitched together and the overlapping areas are removed to generate a complete segmentation map of the large-format remote sensing image.

[0061] S3: Input the training dataset into the DDMGF-Segformer network for iterative training, calculate the loss value through the multi-class cross-entropy loss function, and iteratively optimize the model parameters through backpropagation;

[0062] (1) Set the training hyperparameter configuration and network initialization parameters; the initial learning rate is 0.001, the momentum coefficient is 0.9, the weight decay is 0.0005, the number of training batches is 8 and the number of training rounds is 105; the stochastic gradient descent optimizer is used to update the parameters, and the learning rate strategy adopts fixed learning rate scheduling to ensure training stability;

[0063] (2) During training, overfitting of the network is prevented by batch normalization, Dropout regularization and model checkpoint saving strategy; regularization with a dropout rate of 0.1 is applied before the classification prediction layer of the decoder;

[0064] (3) The stochastic gradient descent optimization algorithm was used to optimize the deformable aligned multi-scale fusion enhancement network DDMGF-Segformer. During the optimization process, a multi-class cross-entropy loss function was used in combination with a class weight balancing strategy as the training objective. Among them, the weighted multi-class cross-entropy loss function, which considers the imbalance of land cover classes, guides the model to optimize both segmentation accuracy and class balance. Its calculation formula is as follows:

[0065]

[0066] in, Total number of pixels For the number of land cover categories, For the first Class weight coefficient, For pixels The true category label, Pixels predicted by the network Category The probability, This is a multi-class cross-entropy loss. The loss function is calculated by crossing the probability distribution after Softmax activation with the true label, ensuring the accuracy of the network's output probability distribution and the reliability of class discrimination.

[0067] (4) Repeat steps (2) and (3) for iterative training until the deformable aligned multi-scale fusion enhanced semantic segmentation network DDMGF-Segformer converges, and obtain the final network model and weights for semantic segmentation of land cover in remote sensing images. Save the optimal model weights. The final saved model weight file contains complete network parameters and can be directly used for inference prediction on test datasets and land cover segmentation tasks in actual remote sensing images.

[0068] S4: Input the test dataset into the trained DDMGF-Segformer network and output the predicted ground feature extraction results.

[0069] This embodiment selects the classic Segformer model as a control and conducts comparative tests under the same dataset and experimental environment. The results of the comparative experiments are shown in Table 2. The definitions of each classification index are as follows: Intersection over Union (IoU) is the ratio of the intersection to the union of the predicted result and the ground truth label of a single land cover class, used to characterize the segmentation accuracy and boundary matching degree of a single land cover class; Mean Intersection over Union (mIoU) is the average IoU of all land cover classes, which is the core index for measuring the overall comprehensive segmentation performance of the model; Overall Accuracy (OA) is the proportion of correctly classified pixels in the entire image to the total number of pixels, reflecting the overall accuracy of the model's global classification.

[0070] Table 2 - Comparative Experiment Results

[0071]

[0072] The visualization results include: test data from embodiments of the present invention, such as... Figure 4 As shown, the truth labels of the test data in this embodiment of the invention are as follows: Figure 5 As shown, the ground feature extraction results of the segformer network in this embodiment of the invention are as follows: Figure 6 As shown, the ground feature extraction results of the DDMGF-Segformer network in this embodiment of the invention are as follows: Figure 7 As shown.

Claims

1. A method for extracting ground features from remote sensing images using a deformable aligned multi-scale fusion network, characterized in that, The method includes the following steps: S1: Obtain the remote sensing image semantic segmentation dataset, and divide the remote sensing image semantic segmentation dataset into a training dataset and a test dataset; S2: Construct a deformable aligned multi-scale fusion network DDMGF-Segformer, which includes a data preprocessing module, a MiT encoder module, a deformable aligned multi-scale fusion feature enhancement module, a decoder module, and an output module. S3: Input the training dataset into the DDMGF-Segformer network for iterative training, calculate the loss value through the multi-class cross-entropy loss function, and iteratively optimize the model parameters through backpropagation; S4: Input the test dataset into the trained DDMGF-Segformer network and output the predicted ground feature extraction results; The deformable alignment multi-scale fusion feature enhancement module in S2 includes a feature calibration branch, a local deformable perception branch and a global multi-scale semantic branch set in parallel, a channel splicing module, a fusion enhancement module and a residual connection module. The feature calibration branch includes a 1×1 convolutional layer, a batch normalization layer, and a ReLU activation function layer, all connected in series. The 1×1 convolutional layer compresses the dimensionality of the input features. The batch normalization layer stabilizes the feature distribution. The ReLU activation function layer introduces a nonlinear transformation to unify the channel dimensions, normalize the feature distribution, and regularize the basic spatial structure of the remote sensing image features, outputting the basic calibration features of the remote sensing image features. The local deformable sensing branch includes a 1×1 convolutional layer, a batch normalization layer, a ReLU activation function layer, a 3×3 convolutional layer, a 3×3 deformable convolutional layer, a batch normalization layer, and a ReLU activation layer, connected in series. The 3×3 convolutional layer has an inflation rate of 2 and a padding rate of 2, used to obtain the offset of ground feature. The 3×3 deformable convolutional layer has an inflation rate of 2 and a padding rate of 2, used to adaptively sample the input features, extract the geometric morphological features of individual ground features, and output local semantic features. The global multi-scale semantic branch includes a 1×1 convolutional layer, a batch normalization layer, a ReLU activation function layer, a 3×3 convolutional layer, a 3×3 deformable convolutional layer, a batch normalization layer, and a ReLU activation layer, connected in series. The 3×3 convolutional layer has an inflation rate of 4 and a padding rate of 4, used to obtain the offset of ground feature. The 3×3 deformable convolutional layer has an inflation rate of 4 and a padding rate of 4, used to extract the spatial distribution pattern of ground features in remote sensing images and the global semantic association information, outputting global semantic features. The fusion enhancement module includes a 1×1 convolutional layer and a batch normalization layer, which are connected in series. The 1×1 convolutional layer is used to fuse multi-scale features of land features in remote sensing images and map the features back to the target output channel number of the module. The batch normalization layer is used for normalization processing and outputs denoised features. The residual connection module includes a channel projection convolutional layer and an element-wise addition operator, which are connected in series. The channel projection convolutional layer includes a 1×1 convolutional kernel to keep the number of input feature channels consistent with the number of output feature channels. The element-wise addition operator adds the features processed by the channel projection convolutional layer to the features output by the fusion enhancement module element-wise, which is used to superimpose the enhanced features after the three-branch fusion, and output features with matching dimensions and enhanced information.

2. The method for extracting ground features from remote sensing images using a deformable aligned multi-scale fusion enhancement network according to claim 1, characterized in that, The encoder module includes a first MiT encoder layer, a second MiT encoder layer, a third MiT encoder layer, and a fourth MiT encoder layer; The first MiT encoder layer includes a 7×7 convolutional layer, a layer normalization layer, a multi-head self-attention module, a feedforward network layer, and a residual connection layer. The 7×7 convolutional layer has a stride of 4 and padding of 3, used to map the remote sensing image from RGB three channels to a 64-dimensional feature space and achieve 4x downsampling. The multi-head self-attention module includes one attention head with a spatial reduction ratio of 8, which performs spatial downsampling on key-value features through convolution, while maintaining the original resolution of the query matrix. The feedforward network layer includes a first linear layer, a depthwise separable convolutional layer, a GELU activation function, and a second linear layer. The residual connection layer includes two branches, which connect the self-attention output and the feedforward network output to the input features, respectively. The second MiT encoder layer includes a 3×3 convolutional layer, a layer normalization layer, a multi-head self-attention module, a feedforward network layer, and a residual connection layer. The 3×3 convolutional layer has a stride of 2 and padding of 1, used to map features from 64 channels to a 128-dimensional feature space and achieve 2x downsampling. The multi-head self-attention module includes two attention heads with a spatial reduction ratio of 4, which performs spatial downsampling on key-value features through convolution. The feedforward network layer includes a first linear layer, a depthwise separable convolutional layer, a GELU activation function, and a second linear layer. The residual connection layer includes two branches, connecting the self-attention output and the feedforward network output to the input features, respectively. The third MiT encoder layer includes a 3×3 convolutional layer, a layer normalization layer, a multi-head self-attention module, a feedforward network layer, and a residual connection layer. The 3×3 convolutional layer has a stride of 2 and padding of 1, used to map features from 128 channels to a 320-dimensional feature space and achieve 2x downsampling. The multi-head self-attention module includes 5 attention heads with a spatial reduction ratio of 2, which performs spatial downsampling on key-value features through convolution. The feedforward network layer includes a first linear layer, a depthwise separable convolutional layer, a GELU activation function, and a second linear layer. The residual connection layer includes two branches, connecting the self-attention output and the feedforward network output to the input features, respectively. The fourth MiT encoder layer includes a 3×3 convolutional layer, a layer normalization layer, a multi-head self-attention module, a feedforward network layer, and a residual connection layer. The 3×3 convolutional layer has a stride of 2 and padding of 1, used to map features from 320 channels to a 512-dimensional feature space and achieve 2x downsampling. The multi-head self-attention module includes 8 attention heads with a spatial reduction ratio of 1, employing a full-resolution attention mechanism without key-value feature downsampling. The feedforward network layer includes a first linear layer, a depthwise separable convolutional layer, a GELU activation function, and a second linear layer. The residual connection layer includes two branches, connecting the self-attention output and the feedforward network output to the input features, respectively.

3. The method for extracting ground features from remote sensing images using a deformable aligned multi-scale fusion enhancement network according to claim 1, characterized in that, The channel stitching module of the deformable aligned multi-scale fusion feature enhancement module in S2 adopts tensor stitching with a stitching dimension of 1, and is used to stitch together the remote sensing image features output by three parallel branches.

4. The method for extracting ground features from remote sensing images using a deformable aligned multi-scale fusion enhancement network according to claim 1, characterized in that, The residual connection module of the deformable aligned multi-scale fusion feature enhancement module in S2 includes a channel projection convolutional layer; the channel projection convolutional layer is enabled only when the number of input channels of the residual connection module is inconsistent with the number of output channels of the fusion enhancement module.

5. The method for extracting ground features from remote sensing images using a deformable aligned multi-scale fusion enhancement network according to claim 1, characterized in that, The element-wise addition operator of the deformable aligned multi-scale fusion feature enhancement module in S2 is a residual stacking structure, which is used to preserve the deep semantic information in the ground feature features of the original remote sensing image.

6. The method for extracting ground features from remote sensing images using a deformable aligned multi-scale fusion enhancement network according to claim 1, characterized in that, The decoder module includes a multi-scale feature embedding layer, a feature upsampling alignment layer, a feature splicing and fusion layer, and a classification prediction layer. The multi-scale feature embedding layer includes four parallel 1×1 convolutional layers, which respectively transform the channel dimension of the four features output by the encoder, so that features with different numbers of channels can be uniformly mapped to the same embedding dimension. The feature upsampling alignment layer includes a first bilinear interpolation operator, a second bilinear interpolation operator, and a third bilinear interpolation operator; the first bilinear interpolation operator is used to perform an upsampling operation on the input features; the second bilinear interpolation operator is used to sequentially upsample features with different spatial resolutions to the same spatial size; and the third bilinear interpolation operator is used to align the spatial resolutions of all features. The feature splicing and fusion layer uses tensor splicing to splice the input features along the channel dimension, with a splicing dimension of 1. The classification prediction layer includes a 1×1 convolutional layer, a softmax activation layer, and a bilinear interpolation operator. The 1×1 convolutional layer is used to map the input features to a channel dimension that matches the number of land cover categories. The softmax activation layer normalizes the output features of the 1×1 convolutional layer, outputting remote sensing image land cover features with a dimension that is the product of the number of categories and the aligned spatial size. The bilinear interpolation operator is used to upsample the remote sensing image land cover features to the original remote sensing image size, outputting category probability features.

7. The method for extracting ground features from remote sensing images using a deformable aligned multi-scale fusion enhancement network according to claim 1, characterized in that, The output module includes a remote sensing image land cover category conversion layer, a size upsampling layer, and a large-format mosaic layer; The remote sensing image land cover category conversion layer includes an argmax operation module, which performs pixel-by-pixel index mapping on the category probability features to generate a land cover category label map; The size upsampling layer includes a bilinear interpolation operator for upsampling the category label map from the network output size to the original image size; The large-format mosaic layer includes a sliding window segmentation module, a sub-block inference module, and a weighted fusion module. The sliding window segmentation module divides the large-format image into overlapping sub-blocks. The sub-block inference module is used to predict each sub-block independently. The weighted fusion module uses a weighted average mechanism to complete the mosaicking of sub-block results and outputs the remote sensing image ground feature extraction results.

Images