A Parallel Building Extraction Method from Remote Sensing Imagery Based on Multi-Scale Attention

Abstract

This invention discloses a parallel building extraction method for remote sensing images based on multi-scale attention. The invention designs and implements a building extraction network, MGAR, that incorporates an attention mechanism. This invention utilizes multi-head attention to perform hierarchical weighted processing on high-level semantic information to extract the optimal feature combination with a more complete representation effect. Then, a gating structure is used to fuse each dimension's feature map with the corresponding low-level semantic information from the encoding end to compensate for the loss of local building detail information.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image technology, and specifically to a method for parallel extraction of buildings from remote sensing images based on multi-scale attention. Background Technology

[0002] In the era of remote sensing big data, the continuous emergence of high-resolution, hyperspectral, and other high-quality remote sensing images has provided rich data support for Earth science research and effectively promoted the development of spatial information technology. Building extraction from remote sensing images is one of the most important research areas in remote sensing data mining, with significant practical application value in urban planning, ecological environment construction, land use analysis, military surveying, and many other fields. However, significant differences in building shape, structure, size, and texture, coupled with complex background information and numerous ground features in remote sensing images, lead to problems such as confusion, missed detections, and dense noise points in the extraction results. Therefore, how to quickly and accurately extract clearly outlined and structurally complete building targets from remote sensing images remains a challenge that needs to be overcome.

[0003] With the rapid development of deep learning technology, semantic segmentation models based on deep convolutional neural networks (DCNNs) have been widely applied to building extraction tasks in remote sensing images. Among them, building segmentation methods based on fully convolutional networks (FCNs) are the most popular. Compared with traditional convolutional neural networks (CNNs), FCNs use deconvolution instead of fully connected layers, effectively solving the problem of high computational complexity. Researchers have applied FCNs and their improved versions, such as UNet, SegNet, and PSPNet, to building extraction tasks and have achieved numerous research results. Zhang Cuijun et al. proposed asymmetric convolutional blocks to replace conventional convolution operations based on the UNet network, enhancing the model's ability to learn changing features. Addressing the problem of missing building edge details, He Zhimeng et al. proposed the E-UNet algorithm, which combines dilated convolutions with UNet, achieving better edge perception than the traditional UNet structure. Zhang Chunsen et al. improved upon the SegNet network by combining the Lorentz function sparsity constraint factor.

[0004] The LSPNet semantic segmentation model significantly improves the extraction performance of large building targets. While the aforementioned studies have achieved good building extraction results, most of their multi-scale feature extraction and fusion methods rely on aggregating multi-scale contextual information with the same weights. This fails to distinguish the importance of building targets of different scales in the image, as well as the dependence of building target pixels on different spatial locations, leading to problems such as false detections and significant internal noise.

[0005] Therefore, providing a parallel extraction method for remote sensing images based on multi-scale attention, which utilizes positional encoding and scaling dot product operations to perform hierarchical weighting operations on feature maps to obtain the optimal feature combination with the highest matching degree to the image, and designs a gating process to fuse the corresponding low-level semantic information at the encoding end to refine the image restoration process, has become a problem worthy of research. Summary of the Invention

[0006] The purpose of this invention is to provide a multi-scale attention-based parallel building extraction method for remote sensing images. This method utilizes positional encoding and scaled dot product operations to perform hierarchical weighting operations on feature maps to obtain the optimal feature combination with the highest image matching performance. It also designs a gating process to fuse low-level semantic information from the corresponding encoding end, refining the image restoration process. Based on the adaptive weighting concept, this invention proposes a building extraction network that incorporates an attention mechanism.

[0007] MGAR (Multi-scale Gating Attention with ResNet50) is a building extraction network that first combines dilated convolution and pyramid pooling into the original ResNet50 structure to obtain rich multi-scale semantic information. Based on this, it uses positional encoding and scaled dot product operations to perform hierarchical weighting operations on the feature maps to obtain the optimal feature combination with the highest matching accuracy to the image. Simultaneously, to compensate for the loss of local details in buildings, a gating process is further designed to fuse low-level semantic information from the corresponding encoding end, thus refining the image restoration process.

[0008] The objective of this invention is achieved as follows:

[0009] A method for parallel building extraction from remote sensing images based on multi-scale attention includes the following steps:

[0010] Step S1: Establish the MGAR architecture for building extraction network to improve the target omission and noise density problems in the deep learning-based remote sensing image building extraction method: use multi-head attention to perform hierarchical weighted processing on high-level semantic information to extract the optimal feature combination with a more complete representation effect; then use a gating structure to fuse each dimension feature map with the low-level semantic information of the corresponding encoding end to make up for the loss of local building detail information.

[0011] Step S2: Conduct experiments on the Massachusetts Building and WHU Building public datasets to verify the effectiveness of the MGAR architecture for building extraction proposed in Step 1.

[0012] The specific steps of step S1 are as follows:

[0013] Step S1.1 Multi-scale feature interpretation

[0014] A multi-scale feature interpretation module (MGA) is proposed at the decoding end, and further refined into an attention module and a gating unit to extract the optimal feature combination with a more complete representation effect.

[0015] Step S1.2 Feature interpretation based on multi-head attention mechanism. The multi-head attention mechanism selects important information from a large number of interference factors by assigning different weights to each component in the feature sequence and uses multiple independent attention heads for parallel computation, which can effectively prevent model overfitting while improving computational efficiency.

[0016] First, the feature map [H,W,C] is transformed into a feature sequence [H×W,C] using the Flatten operation. Then, position encoding is performed on each component of the feature sequence.

[0017]

[0018] Where pos is the position of the pixel in the sequence, i is the i-th channel of the input feature, and d model For sequence dimensions; positional encoding uses sine and cosine functions to assign a unique identifier to each pixel to record the positional information of positive sample pixels, and to enable any pixel in the feature sequence to be linearly represented by other positional information;

[0019] Finally, the attention score is normalized using the Softmax function, and the weight vector is multiplied by V to complete the weighted calculation of the single-dimensional feature map.

[0020]

[0021] In equation (3), W i Q , The transition matrices for the initial parameters Q, K, and V under the i-th (i∈{1,...,h}) linear transformation are respectively, and head i The outputs of i attention heads correspond to four scales of the feature maps. A final linear transformation is then used to aggregate the multi-head feature maps to obtain the final result.

[0022] MultiHead(Q,K,V)=Concat(head1,head2,head3,head4)W o (5)

[0023] In equation (5), MultiHead(MH) is the multi-scale hybrid feature matrix after weighting by the multi-head attention mechanism, Concat is the connection operation, and W o Here is the transition matrix;

[0024] Step S1.3 Feature fusion based on gating mechanism: In the process of image restoration at the decoding end, in order to retain more detailed information, the multi-scale feature interpretation module (MGA) adopts a gating method to adjust the feature fusion process. Based on step S1.2, in order to make up for the problem of loss of local details of buildings, a gating process is further designed to fuse the corresponding low-level semantic information at the encoding end to refine the image restoration process.

[0025] Using feature map C of the same level encoder n , Upper-level multi-scale feature map M n Compared with the feature map MH after multi-head attention processing n Feature fusion is performed as input.

[0026] H n =sigmoid(conv(C n ))·M n +C n +MH n (6)

[0027] G n =sigmoid(conv(H n (7)

[0028] P n =G n ·C n +(1-G n )·MH n (8)

[0029] Where n represents the current feature level (n∈{1,...,l}), l represents the model level, and C n For encoder feature map, H n For mixed feature output, G n ∈(0,1) represents a gated unit, Conv is a regular convolution operation with a 1×1 kernel, and P n This is the result of the fusion.

[0030] In step 2, three public datasets are used for experimental verification. The three public datasets include the Massachusetts Buildings Dataset, the Wuhan Aerial Photography Dataset, and the Wuhan Satellite Dataset II.

[0031] The beneficial effects of this invention are as follows: This invention designs and implements a building extraction network MGAR that incorporates an attention mechanism. This invention utilizes multi-head attention to perform hierarchical weighted processing on high-level semantic information to extract the optimal feature combination with a more complete representation effect; then, a gating structure is used to fuse each dimension of the feature map with the corresponding low-level semantic information at the encoding end to compensate for the loss of local building detail information. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the MGAR (Building Extraction Network) structure proposed in this invention.

[0033] Figure 2 This is a diagram of the pyramid pooling module used in this invention;

[0034] Figure 3 This is a schematic diagram of the multi-scale feature interpretation module proposed in this invention;

[0035] Figure 4 Examples of training samples from the Massachusetts building dataset, Wuhan aerial photography dataset, and Wuhan satellite dataset II used in this invention;

[0036] Figure 5 This is an example image of extraction from a Massachusetts building dataset according to the present invention;

[0037] Figure 6 These are examples of extraction of different structures from the Wuhan aerial photography dataset and the Wuhan satellite dataset, as presented in this invention.

[0038] Figure 7 This is an example image showing the extraction of different structures from an aerial photography dataset in Wuhan during an ablation experiment, based on the present invention. Detailed Implementation

[0039] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0040] like Figure 1As shown, the building extraction network MGAR proposed in this invention consists of an encoding and decoding path. At the encoding end, convolution and pooling operations are used to convert low-level building details in remote sensing images into high-level semantic features. At the decoding end, a binarized image is restored to obtain a prediction result of the same size as the original image. Multi-head attention is introduced to perform hierarchical weighted processing on the high-level semantic information to extract the optimal feature combination with a more complete representation effect. Then, a gating structure is introduced to fuse each dimension of feature map with the corresponding low-level semantic information at the encoding end to compensate for the loss of local building details.

[0041] like Figure 2 The pyramid pooling structure module shown, as the output of the backbone, contains shallow features C1, C2, and C3, which contain more fine-grained information such as building edges and textures. After multiple pooling operations, each pixel of the top-level feature C4 corresponds to a larger overlapping receptive field, fusing rich semantic information from that point and its surrounding area. Therefore, this paper uses C4 as the global feature map for image restoration. However, buildings in remote sensing images vary greatly in shape and size. C4 is only the result of a single pooling of C3. This means that features of buildings with sizes similar to or the same as the pooling size receive more weight, while features with significantly different pooling sizes are often ignored. To ensure that both large and small buildings can be well identified and preserved, the building extraction network MGAR uses a PPM (Pyramid Pooling Module) at the connection between the encoder and decoder to process C4 and obtain a multi-scale feature map.

[0042] like Figure 3 The diagram shows the overall structure of the multi-scale feature interpretation module, which can be further refined into an attention module and a gating unit. The multi-head attention mechanism can filter out important information from a large number of interference factors by assigning different weights to each component in the feature sequence. It also uses multiple independent attention heads for parallel computation, which can improve computational efficiency and effectively prevent model overfitting.

[0043] like Figure 4The three public datasets are shown. The Massachusetts Buildings Dataset

[20] consists of 151 aerial images of the Boston area with a spatial resolution of 1m, covering an area of ​​about 340km^2, with a resolution of 1500×1500 pixels per image. To improve the robustness of the model to unknown data, three methods of data augmentation were used: rotation by 180°, horizontal flipping, and vertical flipping. The training set was expanded to 548 images, and the original test set and validation set were combined into 14 images for model accuracy testing. The Wuhan Aerial Dataset

[21] comes from the New Zealand Land Information Service website and contains 0.075m aerial images of Christchurch and New Zealand areas covering 450km^2. The original dataset contains 8189 images of size 512×512 and corresponding labels. During the experiment, the dataset was divided into 4736 images (130,500 buildings in total) as the training set and 3452 images (56,500 buildings in total) as the test set. The Wuhan Satellite Dataset II (WHU Satellite Dataset II)

[21] consists of 6 adjacent satellite images, covering 550 km^2 of East Asia, with a ground resolution of 0.45 m, and contains 34085 buildings. The original images were seamlessly cropped into 17388 data samples of size 512×512, of which 13662 images (25749 buildings in total) were used for model training and the remaining 3726 images (8358 buildings in total) were used for model testing. In order to adapt to the memory space size, the images of the above three datasets were uniformly cut into size 128×128 to feed into the model during actual training.

[0044] like Figure 5 The results are shown below, based on five images from the Massachusetts test set with densely packed buildings: In region a, the buildings are relatively evenly distributed overall. While the buildings within the circular frame have irregular shapes, MGAR can still extract complete building targets with clear outlines and relatively independent features. In regions b and d, where buildings are densely packed, the MGAR building extraction network effectively avoids the problem of small targets clustering by recognizing the importance of small-scale features and detects targets missed by the labels. In region b, although the identified large building targets have some salt-and-pepper noise, the model can accurately preserve the edges and outlines of large targets, demonstrating good robustness to noise interference.

[0045] like Figure 6The diagram shows a comparison of extraction performance using different model structures. It is evident that the method of this invention is more accurate in extracting large buildings and less susceptible to interference from internal target factors. The salt-and-pepper noise problem is also effectively mitigated compared to Unet (a variant of a fully convolutional neural network), RAPNet (a modified pyramid network), and GAM (gated and attention network). Due to the sparse distribution of buildings and complex ground feature interference in Wuhan satellite imagery, the extraction difficulty of this dataset is increased. As shown in region c, due to terrain and vegetation interference, Unet (a variant of a fully convolutional neural network) and RAPNet (a modified pyramid network) exhibit varying degrees of adjacent target adhesion and target misjudgment problems. GAM (gated and attention network) and GSM (gated spatial storage network) fail to extract small building targets well in region d, while the building extraction network MGAR maintains the independence between targets well and extracts small targets more clearly and accurately.

[0046] like Figure 7 The images show examples of different structures extracted from the Wuhan aerial photography dataset during the ablation experiment. In region a, it's clear that the image labels have missing targets at the marked locations; Model 1 failed to identify the building target. Model 4, however, utilizes a multi-scale method to enhance its feature perception capabilities, not only extracting the target but also achieving a more complete extraction compared to Models 2 and 3 through multi-head attention and gating optimization. In region b, it can be seen that Model 4 uses multi-head attention to interpret features at different scales, resulting in clearer and more complete outlines for small targets. Furthermore, in the extraction of large building targets in region c, although Models 2, 3, and 4 achieve similar results, Model 4 further refines the selection of shallow features through gating, obtaining clearer building edges.

Claims

1. A method for parallel building extraction from remote sensing images based on multi-scale attention, characterized in that: Includes the following steps: Step S1: Establish the MGAR architecture for building extraction network to improve the target omission and noise density problems in the deep learning-based remote sensing image building extraction method; use multi-head attention to perform hierarchical weighted processing on high-level semantic information to extract the optimal feature combination with a more complete representation effect; and then use a gating structure to fuse each feature map with the low-level semantic information of the corresponding encoding end to make up for the loss of local building detail information. The specific operation steps of step S1 are as follows: Step S1.1 Multi-scale feature interpretation: a multi-scale feature interpretation module MGA is proposed at the decoding end, and further refined into attention module and gating unit to extract the optimal feature combination with a more complete representation effect. Step S1.2 Feature interpretation based on multi-head attention mechanism. The multi-head attention mechanism selects important information from a large number of interference factors by assigning different weights to each component in the feature sequence and uses multiple independent attention heads for parallel computation, which can effectively prevent model overfitting while improving computational efficiency. First, the feature map [H,W,C] is transformed into a feature sequence [H×W,C] using the Flatten operation. Then, position encoding is performed on each component of the feature sequence. Where pos is the position of the pixel in the sequence, i is the i-th channel of the input feature, and d model For sequence dimensions; positional encoding uses sine and cosine functions to assign a unique identifier to each pixel to record the positional information of positive sample pixels, and to enable any pixel in the feature sequence to be linearly represented by other positional information; Finally, the attention score is normalized using the Softmax function, and the weight vector is multiplied by V to complete the weighted calculation of the single-dimensional feature map. In equation (3), W i Q , The transition matrices for the initial parameters Q, K, and V under the i-th (i∈{1,...,h}) linear transformation are respectively, and head i The outputs of i attention heads correspond to four scales of the feature map; then, a linear transformation is used to aggregate the multi-head feature maps to obtain the final result. MultiHead(Q,K,V)=Concat(head1,head2,head3,head4)W O (5) In equation (5), MultiHead(MH) is the multi-scale hybrid feature matrix after weighting by the multi-head attention mechanism, Concat is the connection operation, and W O Here is the transition matrix; Step S1.3 Feature fusion based on gating mechanism: In the process of image restoration at the decoding end, in order to retain more detailed information, the multi-scale feature interpretation module (MGA) adopts a gating method to adjust the feature fusion process. Based on step S1.2, in order to make up for the problem of loss of local details of buildings, a gating process is further designed to fuse the corresponding low-level semantic information at the encoding end to refine the image restoration process. Using feature map C of the same level encoder n , Upper-level multi-scale feature map M n Compared with the feature map MH after multi-head attention processing n Feature fusion is performed as input. H n =sigmoid(conv(C n ))·M n +C n +MH n (6) G n =sigmoid(conv(H n )) (7) P n =G n ·C n +(1-G n )·MH n (8) Where n represents the current feature level (n∈{1,...,l}), l represents the model level, and C n For encoder feature map, H n For mixed feature output, G n ∈(0,1) represents a gated unit, Conv is a regular convolution operation with a 1×1 kernel, and P n The result of the fusion; Step S2: Conduct experiments on the Massachusetts Building and WHU Building public datasets to verify the effectiveness of the MGAR architecture for building extraction proposed in Step 1.

2. The method for parallel building extraction from remote sensing images based on multi-scale attention according to claim 1, characterized in that: In step S2, three public datasets are used for experimental verification. The three public datasets include the Massachusetts Buildings Dataset, the Wuhan Aerial Photography Dataset, and the Wuhan Satellite Dataset II.

Images