Building extraction method for high-resolution remote sensing image based on multi-scale hybrid transformer model with boundary refinement

Abstract

The application relates to a high-resolution remote sensing image building extraction method based on a multi-scale hybrid Transformer model of boundary refinement, and belongs to the technical field of remote sensing image recognition. The method comprises the following steps: performing data preprocessing and data enhancement on an acquired remote sensing optical image; inputting the processed training set data into a hybrid CNN and a Transformer encoder to extract four feature maps of different scales according to four stages; sequentially inputting the encoded features of the fourth stage to the first stage into a decoder for decoding, wherein the low-level features extracted from the first two stages are input into an edge enhancement module for edge refinement, the decoder combines the low-level features of the previous stage encoder step by step and is fused through a group feature alignment fusion module, and finally a prediction map is obtained; the above model is trained by using a back propagation method, and after the training is completed, the model is used for remote sensing image building extraction. The application can effectively improve the building extraction precision of remote sensing images.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image recognition technology, and relates to a method for extracting buildings from high-resolution remote sensing images based on a multi-scale hybrid Transformer model with boundary refinement. Background Technology

[0002] With the continuous development of remote sensing technology and the widespread application of high-resolution remote sensing imagery, building extraction from high-resolution remote sensing images has attracted widespread attention in the remote sensing community. Accurate building extraction is crucial for population estimation, disaster assessment, and environmentally friendly urban planning. However, due to the diversity of building appearance and size, the complexity of the surrounding environment, and occlusion and shadows, accurate building extraction from high-resolution remote sensing imagery remains a significant challenge. In addressing multi-scale object segmentation problems, many deep learning-based methods have been developed for building extraction. For example, PSPNet utilizes a spatial pyramid pooling module to inherit multi-scale context, but the extraction of large-scale context comes at the cost of spatial resolution. The Deeplab family integrates a hollow spatial pyramid pooling module into a deep convolutional neural network, capturing multi-scale contextual information without losing resolution.

[0003] In recent years, with the rapid development of Convolutional Neural Networks (CNNs), the task of building extraction has seen significant breakthroughs. The introduction of Fully Convolutional Networks (FCNs) was the first to apply a fully convolutional network structure to this field, sparking the interest of many researchers who focused on designing encoder-decoder architectures to achieve more accurate building extraction. They proposed numerous CNN-based building extraction frameworks. These methods have improved the accuracy of building extraction to some extent. However, these methods are limited by the convolutional operations on buildings, making it difficult to effectively model global contextual information. Due to the vast differences in building shape and size, and the potentially complex surrounding environment (e.g., the presence of trees or shadows), it is difficult to accurately infer the building target from local contextual information.

[0004] In recent years, with the rapid development of Convolutional Neural Networks (CNNs), building extraction tasks have seen significant breakthroughs. The introduction of Fully Convolutional Networks (FCNs) was the first to apply a fully convolutional network structure to this field. Compared to CNN-based methods, Transformers possess strong global modeling capabilities. While Transformers demonstrate advantages in extracting global information, local details (such as building edges) are also crucial in building extraction tasks. These details provide key information about the building's shape, structure, and boundaries. In this context, introducing CNN methods into Transformers to enhance local information extraction capabilities becomes key to balancing local and global contextual information. By fusing the global modeling capabilities of Transformers with the local feature extraction capabilities of CNNs, hybrid Transformers can better understand the overall structure of buildings in images, thereby improving the accuracy and robustness of building extraction. Although sequential, parallel, or cross-processing of CNN and Transformer models can improve the extraction of both local and global features, this undoubtedly increases computational costs significantly.

[0005] In the encoder, low-level features carry local details such as building edges; however, they also contain a significant amount of background noise from non-building areas. Directly fusing these low-level features with high-level features in the decoder can lead to semantic ambiguity and weakened edge representations. This makes it difficult for the model to distinguish building edge pixels when recovering spatial details, resulting in blurred building edge extraction. Furthermore, during feature aggregation, high-level feature maps need to be upsampled to the same size as low-level feature maps. However, due to misalignment between different levels of feature maps, the model may fail to accurately capture building boundary information, affecting the accuracy of building extraction. Additionally, small buildings may be obscured by trees, other buildings, or shading structures, or cast shadows due to poor lighting conditions, causing the model to fail to detect them correctly, thus impacting the accuracy and robustness of the entire building extraction process. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide a high-resolution remote sensing image building extraction method based on a multi-scale hybrid Transformer model with boundary refinement, so as to solve the problems of missing or incorrect extraction of small buildings and irregular or blurry building boundaries in current building extraction methods.

[0007] To achieve the above objectives, the present invention provides the following technical solution: A method for extracting buildings from high-resolution remote sensing images based on a multi-scale hybrid Transformer model with boundary refinement, the method comprising: S1. Perform data preprocessing and data augmentation on the acquired remote sensing optical images; S2. Input the processed training set data into the hybrid CNN and Transformer encoder to extract feature maps of four different scales in four stages. S3. The encoded features from the fourth stage to the first stage are sequentially input into the decoder for decoding. The low-level features extracted in the first two stages are input into the edge enhancement module for edge refinement. The decoder combines the low-level features of the encoder in the previous stage step by step and fuses them through the group feature alignment and fusion module to finally obtain the prediction map. S4. The above model is trained using the backpropagation method. After training, it is used for building extraction from remote sensing images.

[0008] Furthermore, in step S1, in terms of data preprocessing, the acquired remote sensing building dataset images are cropped using an overlapping method, and the label pixels of the labeled images are converted to 0 or 1. Then, the cropped dataset is divided into training set, validation set and test set according to the proportion. In terms of data augmentation, data augmentation is applied to the training set, including random scaling, random horizontal and vertical flipping, random Gaussian noise, contrast-limited adaptive histogram equalization, random rotation, and random cropping.

[0009] Furthermore, in step S2, the given RGB image is divided into several image blocks of the same size, and then these image blocks are input into the multi-scale hybrid Transformer encoder. After passing through each stage, four feature maps with a resolution of {1 / 4, 1 / 8, 1 / 16, 1 / 32} of the original image are obtained, where the number of feature map channels in each stage is denoted as C1~C4. The multi-scale hybrid Transformer encoder has four stages, and the feature maps encoded by the four stages are denoted as follows: , , , Correspondingly, four decoders are set up for the four stages, wherein the feature map in the decoder of the first stage is... The feature map in the second-stage decoder is The feature map in the third stage decoder is The feature map in the fourth stage decoder is Among them, the features obtained in the first and second stages , These are low-level features; during decoding, the features in the fourth stage... Decode first, then proceed in sequence. , , ; Each stage of the multi-scale hybrid Transformer encoder includes an overlapping patch merging layer and multiple Hybridformer Blocks; Each overlapping patch merging layer consists of a convolutional layer and a LayerNorm regularization layer, and the feature map vector after passing through the overlapping patch merging layer is fed into the Hybridformer Block; each Hybridformer Block includes: a multi-scale local perception module and a global perception module. The Multi-Scale Local Perception Module (MSLP) specifically works as follows: First, the feature map is input into a depthwise convolution IDConv that can adapt to different inputs; then, the feature map after IDConv is divided into three groups of feature maps according to the channel dimension, where i=1,2,3, and they are input into 1×1, 3×3, and 5×5 depthwise convolutions, respectively. The results are concatenated and fused according to the channel dimension to obtain a feature map with multi-scale information; Two fully connected layers and Gaussian error linear units (GELUs) are used together to enhance the nonlinear representation of local features, ultimately generating a feature map. The entire MSLP module is represented as follows:

[0010]

[0011]

[0012] in, IDConv This represents a depthwise convolution that depends on the input. ChannelSplit This indicates that the feature map is divided according to the channel dimension, and GELU represents Gaussian error linear unit.

[0013] The global awareness module specifically includes an efficient self-attention layer and a hybrid feedforward position encoding Mix-FFN layer. The efficient self-attention layer downsamples K and V from the original Q, K, V. The Mix-FFN layer includes a 3×3 convolutional layer. There is a regularization layer before entering the efficient self-attention and Mix-FFN modules, and residual connections are used after each module.

[0014] Furthermore, step S3 includes the following steps: S31. First, the characteristics of the fourth stage... The input is fed into the decoder to obtain the decoded high-level features. ; High-level features decoded in the fourth stage Advanced features of the third stage The input group feature alignment and fusion module obtains the third-stage alignment features. ; S32, the low-level features of the second stage Advanced features after decoding in the fourth stage Third-stage alignment features The input is fed into the edge enhancement module to obtain the low-level features enhanced in the second stage. The low-level features enhanced in the second stage are then aligned with the features from the third stage. The input group alignment module obtains the second-stage alignment features. ; S33, the low-level features of the first stage Advanced features after decoding in the fourth stage Third-stage alignment features and second-stage alignment features The input edge enhancement module obtains the low-level features enhanced in the first stage, and then aligns these low-level features with the features from the second stage. The input group feature alignment and fusion module performs alignment and fusion to obtain the final prediction map, which is then used as the final building segmentation result.

[0015] Furthermore, in step S31, the process of feature alignment and fusion through the group feature alignment and fusion module is as follows: Given high-level features of the decoder and the low-level features of the encoder in the previous stage ,Will Performing 1×1 convolution and bilinear interpolation upsampling makes Dimensions and Consistently obtained features Then and Simultaneously, a 1×1 convolution is used to compress the number of channels to obtain the feature map. and :

[0016]

[0017]

[0018] The two feature maps are concatenated, and the concatenated feature map is input into a 3×3 convolutional layer to obtain the predicted semantic flow field. The semantic flow field is represented as:

[0019] Then Divided into G groups according to channel dimension, i.e. ,in Each sub-feature is associated with a point on the semantic flow field spatial grid. Mapped to using simple addition operations New points Then, a differentiable bilinear sampling mechanism is used to... Linear interpolation is performed on the four nearest neighbor pixel values ​​to obtain ; Then align each set of features Connect along the channel dimension to generate the final result Specifically, it is expressed as:

[0020]

[0021]

[0022] in express Mid-twist point The adjacent points involved, This represents the bilinear kernel weights estimated locally by the distorted mesh; Finally, the data is fused using a concatenation operation and a 3×3 convolution. and The specific process is as follows:

[0023] in, This represents the high-level features after alignment. This represents the low-level features of the encoder in the previous stage.

[0024] Further, in step S32, the feature map of the second stage from the four stages of feature maps generated by the multi-scale hybrid Transformer encoder is input into the boundary enhancement module, and the fused feature map output from the last two stages of the decoder is also input into the boundary enhancement module. The second-stage edge enhancement module includes: Features in the second-stage encoder Features in the third and fourth stages of the decoder As input; Use 3 1×1 convolutions to Unify the dimensions to the same Consistent, uniformly upsampling to the same Same size; After unification Concatenate along the channel dimension, then use 3×3 convolution to generate and Weight matrices W of the same dimension; Using Fast Fourier Transform (FFT) layers With weight matrix Simultaneously convert to the frequency domain, and then... and Perform element-wise multiplication and use Inverse Fast Fourier Transform (IFFT) to... Generating feature maps by converting from the frequency domain back to the spatial domain. Next Inputting LN and MLP generates features that ultimately possess strong edge information and architectural semantic information. ; The process of the second-stage edge enhancement module is represented as follows:

[0025]

[0026]

[0027] in, Representative has A 1×1 convolution with 1 convolution kernel, MLP stands for Multilayer Perceptron, FFT stands for Fast Fourier Transform, IFFT stands for Inverse Fast Fourier Transform, Up stands for Upsampling, and LN stands for LayerNorm Regularization.

[0028] Further, in step S33, the feature map of the first stage of the four-stage feature maps generated by the multi-scale hybrid Transformer encoder is input into the boundary enhancement module, and the fused feature map output from the last three stages of the decoder is input into the boundary enhancement module; wherein, the process of obtaining enhanced features through the first-stage edge enhancement module includes: Features from the first stage encoder Features in the second, third, and fourth stages of the decoder As input; Use 3 1×1 convolutions to Unify the dimensions to the same Consistent, uniformly upsampling to the same Same size; After unification Concatenate along the channel dimension, then use 3×3 convolution to generate and Weight matrices W of the same dimension; Using Fast Fourier Transform (FFT) layers With weight matrix Simultaneously convert to the frequency domain, and then and Perform element-wise multiplication and use Inverse Fast Fourier Transform (IFFT) to... Return from the frequency domain to the spatial domain and Adding them together yields the feature map. Next Inputting LN and MLP generates features that ultimately possess strong edge information and architectural semantic information. ; The process of the first-stage edge enhancement module is represented as follows:

[0029]

[0030]

[0031] in, Represents the weight matrix. Representative has A 1×1 convolution with 1 convolution kernel, MLP stands for Multilayer Perceptron, FFT stands for Fast Fourier Transform, IFFT stands for Inverse Fast Fourier Transform, Up stands for Upsampling, and LN stands for LayerNorm Regularization.

[0032] Furthermore, in step S4, the AdamW optimizer and poly learning rate adjustment strategy are used to calculate the loss between the predicted map and the real map using the cross-entropy loss function, and the loss is reduced through backpropagation. The loss calculation method is as follows: calculate the cross-entropy loss between the final predicted image obtained in S4 and the ground truth image. The cross-entropy loss calculation formula is as follows:

[0033] Where N represents the total number of pixels in the image, This represents the true label of pixel t. This represents the probability that pixel t belongs to category 1.

[0034] The beneficial effects of this invention are as follows: First, this invention proposes a high-resolution remote sensing image building extraction framework based on a multi-scale hybrid Transformer model with boundary refinement, which can fully combine the advantages of CNN and Transformer to extract local and global information at multiple scales.

[0035] Second, this invention proposes an edge enhancement module that controls the information flow from the encoder to the decoder, utilizes high-level features with rich semantic information in the decoder to enhance the edge representation of low-level features in the encoder, and suppresses irrelevant background noise in non-building areas, thereby improving the segmentation results of building boundaries.

[0036] Third, this invention proposes a group alignment feature fusion module, which aligns feature maps of different levels at adjacent levels, enabling the effective fusion of high-level features containing rich contextual information and low-level features containing rich spatial detail information.

[0037] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0038] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a flowchart of the high-resolution remote sensing image building extraction method based on a multi-scale hybrid Transformer model with boundary refinement according to the present invention. Figure 2 This is a model architecture diagram of the high-resolution remote sensing image building extraction method based on the multi-scale hybrid Transformer model with boundary refinement in this invention. Figure 3 This is a schematic diagram of the Transformer Block module structure in the encoder of the present invention; Figure 4 This is a schematic diagram illustrating an example of the architectural extraction effect of the present invention. Detailed Implementation

[0039] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0040] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0041] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0042] Please see Figures 1-4 This is a method for extracting buildings from high-resolution remote sensing images based on a multi-scale hybrid Transformer model with boundary refinement.

[0043] like Figure 1 The flowchart shown is for a high-resolution remote sensing image building extraction method based on a multi-scale hybrid Transformer model with boundary refinement, which includes: S1. Perform data preprocessing and data augmentation on the acquired remote sensing optical images; S2. Input the processed training set data into the hybrid CNN and Transformer encoder to extract feature maps of four different scales in four stages. S3. The encoded features from the fourth stage to the first stage are sequentially input into the decoder for decoding. The low-level features extracted in the first two stages are input into the edge enhancement module for edge refinement. The decoder combines the low-level features of the encoder in the previous stage step by step and fuses them through the group feature alignment and fusion module to finally obtain the prediction map. S4. The above model is trained using the backpropagation method. After training, it is used for building extraction from remote sensing images.

[0044] Example This embodiment uses the Massachusetts architectural dataset as a data sample to provide a detailed description of the extraction method of the present invention.

[0045] Furthermore, in step S1, the acquired remote sensing building dataset images are cropped using an overlapping method, and the label pixels of the labeled images are converted to 0 or 1. Then, the cropped dataset is divided into training set, validation set and test set in a ratio of 8:1:1, and data augmentation is used during the training process. Specifically, the original 5000×5000 images of the Massachusetts architecture dataset were cropped to 256×256 using a sliding window with a stride of 128. The dataset was then divided into three subsets: training set, validation set, and test set. The pixel values ​​of the building areas in the labeled images of the dataset were set to 1, and the values ​​of other pixels were set to 0.

[0046] During model training, data augmentation was applied to the training dataset, including random scaling, random horizontal and vertical flipping, random Gaussian noise, contrast-limited adaptive histogram equalization, random rotation, and random cropping. The random scaling ratio ranged from 0.5 to 2.0. Random horizontal and vertical flipping, random Gaussian noise, contrast-limited adaptive histogram equalization, and random rotation were performed with a 50% probability. The random rotation angle was within the range of (-180°, 180°). Finally, the processed images were randomly cropped to a size of 256×256. All augmentation methods were applied to the training set.

[0047] Furthermore, in step S2, a multi-scale hybrid Transformer encoder is used to extract architectural features from the input remote sensing image to obtain feature maps at four different scales. Specifically, firstly, the given 256×256×3 RGB image is divided into 4×4 image blocks. Then, these image blocks are input into a multi-scale hybrid Transformer encoder. After passing through each stage, four feature maps with resolutions of {1 / 4, 1 / 8, 1 / 16, 1 / 32} of the original image are obtained, where the number of feature map channels C1~C4 in each stage are {64, 128, 256, 512}, respectively.

[0048] The multi-scale hybrid Transformer encoder has four stages, and the feature maps output by the four stages are denoted as follows: , , , Correspondingly, four decoders are set up for the four stages, wherein the feature map in the decoder of the first stage is... The feature map in the second-stage decoder is The feature map in the third stage decoder is The feature map in the fourth stage decoder is Among them, the features obtained in the first and second stages , These are low-level features. During decoding, the features in the fourth stage... Decode first, then proceed in sequence. , , .like Figure 2 As shown, each stage includes an overlapping patch merging layer and multiple Hybridformer Blocks. Each Hybridformer is a feature extraction module that combines CNN and Transformer. In this invention, the number of Hybridformer Blocks in each stage is set to {3, 4, 18, 3} respectively. Each overlapping patch merging layer consists of a convolutional layer with a kernel size of 7, a stride of 4, and padding of 3, and a LayerNorm regularization layer. The feature map vector after passing through the overlapping patch merging layer is then fed into HybridformerBlock, as follows: Figure 3 As shown, each Hybridformer Block includes: a multi-scale local perception module and a global perception module; Specifically, the multi-scale local perception module first inputs the feature map into an input-dependent depthwise convolution (IDConv), enabling it to dynamically adapt to changes in the input.

[0049] To capture multi-scale information, the feature map after IDConv is divided into three groups of feature maps according to the channel dimension, where i=1,2,3, and will be input into 1×1, 3×3 and 5×5 depth convolutions respectively; The results are then cascaded and fused along the channel dimension to obtain a feature map with rich multi-scale information.

[0050] Finally, two fully connected layers and Gaussian error linear unit (GELU) activation are used together to enhance the nonlinear representation of local features, ultimately generating a feature map. The entire MSLP module can be expressed by the following formula:

[0051]

[0052]

[0053] Here, IDConv represents a depthwise convolution that depends on the input, ChannelSplit represents splitting the feature map along the channel dimension, and GELU represents a Gaussian error linear unit.

[0054] The global awareness module specifically consists of an efficient self-attention layer and a Mix-FFN layer (mix-feedforward position encoding). The efficient self-attention layer downsamples K and V from the original Q, K, and V to reduce computational complexity. The Mix-FFN layer consists of a 3×3 convolutional layer. A regularization layer precedes the efficient self-attention and Mix-FFN modules, and residual connections are used after each module.

[0055] Furthermore, step S3 includes the following steps: S31. First, the characteristics of the fourth stage... The input is fed into the decoder to obtain the decoded high-level features. ; High-level features decoded in the fourth stage Advanced features of the third stage The input group feature alignment and fusion module obtains the third-stage alignment features. ; S32, the low-level features of the second stage Advanced features after decoding in the fourth stage Third-stage alignment features The input is fed into the edge enhancement module to obtain the low-level features enhanced in the second stage. The low-level features enhanced in the second stage are then aligned with the features from the third stage. The input group alignment module obtains the second-stage alignment features. ; S33, the low-level features of the first stage Advanced features after decoding in the fourth stage Third-stage alignment features and second-stage alignment features The input edge enhancement module obtains the low-level features enhanced in the first stage, and then aligns these low-level features with the features from the second stage. The input group feature alignment and fusion module performs alignment and fusion to obtain the final prediction map, which is then used as the final building segmentation result.

[0056] More specifically, in step S31, the 8×8×512-dimensional high-level features output from the fourth stage of the encoder are first input into the decoder, and these high-level features are simultaneously input into the group feature alignment and fusion module along with the encoder features from the previous stage for fusion. The group feature alignment and fusion module includes: First, the high-level features of the decoder are given. and the low-level features of the encoder in the previous stage ,Will Performing 1×1 convolution and bilinear interpolation upsampling makes Dimensions and Consistently obtained features Then, and Simultaneously, 1×1 convolution is used to compress the number of channels to reduce computational cost, ultimately yielding the feature map. and :

[0057]

[0058]

[0059] Then, the two feature maps are concatenated, and the concatenated feature map is input into a 3×3 convolutional layer to obtain the predicted semantic flow field. The semantic flow field is represented as:

[0060] Then, Divided into G groups according to channel dimension, i.e. ,in Each sub-feature is associated with a point on the semantic flow field spatial grid. Mapped to using simple addition operations New points Then, a differentiable bilinear sampling mechanism is used to... Linear interpolation is performed on the four nearest neighbor pixel values ​​to obtain ; Then align each set of features Connect along the channel dimension to generate the final result Specifically, it is expressed as:

[0061]

[0062]

[0063] in express Mid-twist point The adjacent points involved, This represents the bilinear kernel weights estimated locally by the distorted mesh. Finally, this embodiment fuses the weights using a connection operation and a 3×3 convolution. and The specific process is as follows:

[0064] in This represents the high-level features after alignment. This represents the low-level features of the encoder in the previous stage.

[0065] Then, the features obtained through the group feature alignment and fusion module are used as high-level features in the decoder and subjected to the same operation as the low-level feature maps of the second and first stages of the decoder, ultimately resulting in a feature map with dimensions of 64×64×64. A 1×1 convolution is then used to classify the feature map. Finally, bilinear interpolation is used to upsample the image to the original size, yielding the final building segmentation result.

[0066] Furthermore, in step S32, the feature map of the second stage from the four stages of feature maps generated by the multi-scale hybrid Transformer encoder is input into the boundary enhancement module, and the fused feature map output from the last two stages of the decoder is also input into the boundary enhancement module. The second-stage edge enhancement module includes: First, the features in the second-stage encoder... Features in the second and third stage decoders As input; for feature maps Because their dimensions are Unlike other implementations, this example uses three 1×1 convolutions to unify their dimensions to the same level. Consistent, then uniformly upsampled to the same level. They are of the same size, then concatenated along the channel dimension, and then a 3×3 convolution is used to generate a sequence of equal sizes. Weight matrices of the same dimension Then, we use a Fast Fourier Transform (FFT) layer to... With weight matrix Simultaneously convert to the frequency domain, and then and Perform element-wise multiplication and use Inverse Fast Fourier Transform (IFFT) to... Generating feature maps by converting from the frequency domain back to the spatial domain. Next Inputting LN and MLP generates features that ultimately possess strong edge information and architectural semantic information. The second-stage edge enhancement module can be represented as:

[0067]

[0068]

[0069] in, Representative has A 1×1 convolution with 1 convolution kernel, MLP stands for Multilayer Perceptron, FFT stands for Fast Fourier Transform, IFFT stands for Inverse Fast Fourier Transform, Up stands for Upsampling, and LN stands for LayerNorm Regularization.

[0070] Furthermore, in step S33, the feature map of the first stage of the four-stage feature maps generated by the multi-scale hybrid Transformer encoder is input into the boundary enhancement module, while the fused feature map output from the last three stages of the decoder is input into the boundary enhancement module.

[0071] The first-stage edge enhancement module includes: First, the features in the first-stage encoder... Features in the second, third, and fourth stages of the decoder As input; for feature maps Because their dimensions are Unlike other implementations, this example uses three 1×1 convolutions to unify their dimensions to the same level. Consistent, then uniformly upsampled to the same level. They are of the same size, then concatenated along the channel dimension, and then a 3×3 convolution is used to generate a sequence of equal sizes. Weight matrices of the same dimension Then, a Fast Fourier Transform (FFT) layer is used to... With weight matrix Simultaneously convert to the frequency domain, and then and Perform element-wise multiplication and use Inverse Fast Fourier Transform (IFFT) to... Generating feature maps by converting from the frequency domain back to the spatial domain. Next Inputting LN and MLP generates features that ultimately possess strong edge information and architectural semantic information. The first-stage edge enhancement module can be represented as:

[0072]

[0073]

[0074] in, Representative has A 1×1 convolution with 1 convolution kernel, MLP stands for Multilayer Perceptron, FFT stands for Fast Fourier Transform, IFFT stands for Inverse Fast Fourier Transform, Up stands for Upsampling, and LN stands for LayerNorm Regularization.

[0075] Additionally, in step S4, during the model training process of the present invention, the AdamW optimizer and poly learning rate adjustment strategy are used, and the cross-entropy loss function is used to calculate the loss between the predicted map and the real map, and the loss is reduced through backpropagation. Specifically, batch training is employed with a batch size of 16. This invention uses the AdamW optimizer and a Poly learning rate adjustment strategy, with the learning rate set to 0.0006 and the weight decay set to 0.00025.

[0076] The loss calculation method is as follows: calculate the cross-entropy loss between the final predicted image obtained in S4 and the ground truth image. The cross-entropy loss calculation formula is as follows:

[0077] Where N represents the total number of pixels in the image, This represents the true label of pixel t. This represents the probability that pixel t belongs to category 1.

[0078] In this embodiment, OA, Precision, Recall, F1 Score, and Iou are selected as evaluation metrics. Test data is input into the trained model, and the confusion matrix is ​​calculated between the predicted results and the true values. Specifically, the evaluation metric Precision represents the accuracy rate, and the equation is:

[0079] The evaluation metric, Recall, is calculated using the following equation:

[0080] The evaluation metric F1_Score is the harmonic mean of precision and recall, and its equation is:

[0081] The evaluation index IoU is the intersection-union ratio, and the equation is:

[0082] In the equation, TP, TN, FP, and FN represent the following: TP is the number of samples correctly predicted as positive; TN is the number of samples correctly predicted as negative; FP is the number of samples incorrectly predicted as positive; and FN is the number of samples incorrectly predicted as negative.

[0083] Four test images were selected using the test set data to perform the test, and the final building extraction results were obtained, such as... Figure 4 As shown, even with shadows and other terrain features around the building, the model can still accurately segment the building's boundary, proving the effectiveness of this invention in extracting building boundary details.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A high-resolution remote sensing image building extraction method based on boundary refinement and multi-scale hybrid Transformer model, characterized by: The method includes: S1. Perform data preprocessing and data augmentation on the acquired remote sensing optical images; S2. Input the processed training set data into the hybrid CNN and Transformer encoder to extract feature maps of four different scales in four stages. In step S2, the given RGB image is divided into several image blocks of the same size, and then these image blocks are input into the multi-scale hybrid Transformer encoder. After passing through each stage, four feature maps with a resolution of {1 / 4, 1 / 8, 1 / 16, 1 / 32} of the original image are obtained. The number of feature map channels in each stage is denoted as C1~C4. The multi-scale hybrid Transformer encoder has four stages, and the feature maps encoded by the four stages are denoted as follows: , , , Correspondingly, four decoders are set up for four stages, where the feature map in the decoder of the first stage is... The feature map in the second-stage decoder is The feature map in the third stage decoder is The feature map in the fourth stage decoder is Among them, the features obtained in the first and second stages , These are low-level features; during decoding, the features in the fourth stage... Decode first, then proceed in sequence. , , ; Each stage of the multi-scale hybrid Transformer encoder includes an overlapping patch merging layer and multiple Hybridformer Blocks; Each overlapping patch merging layer consists of a convolutional layer and a LayerNorm regularization layer, and the feature map vector after passing through the overlapping patch merging layer is fed into the Hybridformer Block; each Hybridformer Block includes: a multi-scale local perception module and a global perception module. The Multi-Scale Local Perception Module (MSLP) specifically works as follows: First, the feature map is input into the depthwise convolution IDConv, which can adapt to different inputs; then, the feature map after IDConv is divided into three groups of feature maps according to the channel dimension, and input into the depthwise convolutions 1×1, 3×3 and 5×5 respectively. The results are concatenated and fused according to the channel dimension to obtain a feature map with multi-scale information; Two fully connected layers and Gaussian error linear units (GELUs) are used together to enhance the nonlinear representation of local features, ultimately generating a feature map. The entire MSLP module is represented as follows: Where IDConv represents a depthwise convolution that depends on the input, ChannelSplit represents splitting the feature map along the channel dimension, and GELU represents a Gaussian error linear unit; The global awareness module specifically consists of an efficient self-attention layer and a hybrid feedforward position encoding Mix-FFN layer. The efficient self-attention layer downsamples K and V from the original Q, K, and V. The Mix-FFN layer consists of a 3×3 convolutional layer. There is a regularization layer before entering the efficient self-attention and Mix-FFN modules, and residual connections are used after each module. S3. The encoded features from the fourth stage to the first stage are sequentially input into the decoder for decoding. The low-level features extracted from the first two stages are input into the edge enhancement module for edge refinement. The decoder combines the low-level features of the encoder from the previous stage step by step and fuses them through the group feature alignment fusion module to finally obtain the prediction map. The fusion process of the group feature alignment fusion module is as follows: Given the high-level features of the decoder and the low-level features of the encoder from the previous stage, the dimensions are first unified by upsampling through 1×1 convolution and bilinear interpolation, and then the number of channels is compressed by 1×1 convolution. The two feature maps are concatenated and then predicted by 3×3 convolution. The semantic flow field is divided into G groups according to the channel dimension. Spatial transformation is performed by differentiable bilinear sampling to realize the spatial alignment of low-level features to high-level features. Finally, the fusion is completed by concatenation operation and 3×3 convolution. S4. The above model is trained using the backpropagation method. After training, it is used for building extraction from remote sensing images.

2. The method for extracting buildings from high-resolution remote sensing images based on a multi-scale hybrid Transformer model with boundary refinement according to claim 1, characterized in that: In step S1, in terms of data preprocessing, the acquired remote sensing building dataset images are cropped using an overlapping method, and the label pixels of the labeled images are converted to 0 or 1. Then, the cropped dataset is divided into training set, validation set and test set according to the proportion. In terms of data augmentation, data augmentation is applied to the training set, including random scaling, random horizontal and vertical flipping, random Gaussian noise, contrast-limited adaptive histogram equalization, random rotation, and random cropping.

3. The method for extracting buildings from high-resolution remote sensing images based on a multi-scale hybrid Transformer model with boundary refinement according to claim 1, characterized in that: Step S3 includes the following steps: S31. First, the characteristics of the fourth stage... The input is fed into the decoder to obtain the decoded high-level features. ; High-level features decoded in the fourth stage Advanced features of the third stage The input group feature alignment and fusion module obtains the third-stage alignment features. ; S32, the low-level features of the second stage Advanced features after decoding in the fourth stage Third-stage alignment features The input is fed into the edge enhancement module to obtain the low-level features enhanced in the second stage. The low-level features enhanced in the second stage are then aligned with the features from the third stage. The input group alignment module obtains the second-stage alignment features. ; S33, the low-level features of the first stage Advanced features after decoding in the fourth stage Third-stage alignment features and second-stage alignment features The input edge enhancement module obtains the low-level features enhanced in the first stage, and then aligns these low-level features with the features from the second stage. The input group feature alignment and fusion module performs alignment and fusion to obtain the final prediction map, which is then used as the final building segmentation result.

4. The method for extracting buildings from high-resolution remote sensing images based on a multi-scale hybrid Transformer model with boundary refinement according to claim 3, characterized in that: In step S31, the process of feature alignment and fusion through the group feature alignment and fusion module is as follows: Given high-level features of the decoder and the low-level features of the encoder in the previous stage ,Will Performing 1×1 convolution and bilinear interpolation upsampling makes Dimensions and Consistently obtained features Then and Simultaneously, a 1×1 convolution is used to compress the number of channels to obtain the feature map. and : The two feature maps are concatenated, and the concatenated feature map is input into a 3×3 convolutional layer to obtain the predicted semantic flow field. The semantic flow field is represented as: Then Divide into G groups according to channel dimension, that is ,in Each sub-feature is associated with a point on the semantic flow field spatial grid. Mapped to using simple addition operations New points Then, a differentiable bilinear sampling mechanism is used to... Linear interpolation is performed on the four nearest neighbor pixel values ​​to obtain ; Then align each set of features Connect along the channel dimension to generate the final result Specifically, it is expressed as: in express Mid-twist point The adjacent points involved, This represents the bilinear kernel weights estimated locally by the distorted mesh; Finally, the data is fused using a concatenation operation and a 3×3 convolution. and , The final fused features output by the group feature alignment and fusion module are as follows: in This represents the high-level features after alignment. This represents the low-level features of the encoder in the previous stage.

5. The method for extracting buildings from high-resolution remote sensing images based on a multi-scale hybrid Transformer model with boundary refinement according to claim 4, characterized in that: In step S32, the feature map of the second stage from the four stages of the multi-scale hybrid Transformer encoder is input into the boundary enhancement module, and the fused feature map from the last two stages of the decoder is also input into the boundary enhancement module. The second-stage edge enhancement module includes: Features in the second-stage encoder Features in the third and fourth stages of the decoder As input; Use 3 1×1 convolutions to Unify the dimensions to the same Consistent, uniformly upsampling to the same Same size; After unification Concatenate along the channel dimension, then use 3×3 convolution to generate and Weight matrices of the same dimension ; Using Fast Fourier Transform (FFT) layers With weight matrix Simultaneously convert to the frequency domain, and then and Perform element-wise multiplication and use Inverse Fast Fourier Transform (IFFT) to... Generating feature maps by converting from the frequency domain back to the spatial domain. Next Inputting LN and MLP generates features that ultimately possess strong edge information and architectural semantic information. ; The process of the second-stage edge enhancement module is represented as follows: in, Representative has A 1×1 convolution with 1 convolution kernel, MLP stands for Multilayer Perceptron, FFT stands for Fast Fourier Transform, IFFT stands for Inverse Fast Fourier Transform, Up stands for Upsampling, and LN stands for LayerNorm Regularization.

6. The method for extracting buildings from high-resolution remote sensing images based on a multi-scale hybrid Transformer model with boundary refinement according to claim 3, characterized in that: In step S33, the feature map of the first stage of the four-stage feature maps generated by the multi-scale hybrid Transformer encoder is input into the boundary enhancement module, and the fused feature map output from the last three stages of the decoder is also input into the boundary enhancement module; wherein, the process of obtaining enhanced features through the first-stage edge enhancement module includes: Features from the first stage encoder Features in the second, third, and fourth stages of the decoder As input; Use 3 1×1 convolutions to Unify the dimensions to the same Consistent, uniformly upsampling to the same Same size; After unification Concatenate along the channel dimension, then use 3×3 convolution to generate and Weight matrices of the same dimension ; Using Fast Fourier Transform (FFT) layers With weight matrix Simultaneously convert to the frequency domain, and then and Perform element-wise multiplication and use Inverse Fast Fourier Transform (IFFT) to... Return from the frequency domain to the spatial domain and Adding them together yields the feature map. Next Inputting LN and MLP generates features that ultimately possess strong edge information and architectural semantic information. ; The process of the first-stage edge enhancement module is represented as follows: in, Representative has A 1×1 convolution with 1 convolution kernel, MLP stands for Multilayer Perceptron, FFT stands for Fast Fourier Transform, IFFT stands for Inverse Fast Fourier Transform, Up stands for Upsampling, and LN stands for LayerNorm Regularization.

7. The method for extracting buildings from high-resolution remote sensing images based on a multi-scale hybrid Transformer model with boundary refinement according to claim 3, characterized in that: In step S4, the AdamW optimizer and poly learning rate adjustment strategy are used to calculate the loss between the predicted map and the real map using the cross-entropy loss function, and the loss is reduced through backpropagation. The loss calculation method is as follows: calculate the cross-entropy loss between the final predicted image obtained in S4 and the ground truth image. The cross-entropy loss calculation formula is as follows: Where N represents the total number of pixels in the image, This represents the true label of pixel t. This represents the probability that pixel t belongs to category 1.

Images