Segformer-based monomer planar building detection method

Abstract

The application provides a kind of monomer surface building detection method based on SegFormer.The method includes step 1: constructing building extraction network based on improved SegFormer, including encoder and decoder;The encoder includes four Transform blocks connected in turn from shallow to deep;The decoder includes first 1x1 convolution layer, ASPP module and semantic fusion module connected in turn from shallow to deep;Wherein, the remote sensing image is input into the first Transform block, the output of the first Transform block is used as one of the inputs of the semantic fusion module, the output of the fourth Transform block is used as the input of the first 1x1 convolution layer, and the output of the ASPP module is used as another input of the semantic fusion module;The output of the semantic fusion module is the detection result;Step 2: training the building extraction network based on improved SegFormer to obtain a building extraction network model;Step 3: using the trained building extraction network model to extract buildings from the input remote sensing image to obtain the detection result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of building detection, and in particular to a monomer planar building detection method based on SegFormer. BACKGROUND

[0002] As an important artificial ground object, buildings are the center of human production and living activities. They can provide important references for city planning, disaster monitoring, traffic management, and scientific planning of ecological environment. Unlike traditional natural images, the building area on remote sensing images is a mixed settlement composed of house, vegetation, road, and soil land and other ground object types. Due to the differences in material structure and other aspects of ground objects, buildings on remote sensing images will produce different differences. When the ground object distribution is complex, a large amount of non-house information is mixed into the detection network, and the classification detection effect is poor. Traditional image processing selects features by hand, which is time-consuming and laborious, and cannot meet the requirements of real-time updating and large-scale scene application. At present, there is a big gap between the research and actual application of intelligent mapping technology.

[0003] In recent years, high-resolution remote sensing classification methods based on convolutional neural networks have also been studied in the field of house information extraction. Related research summarizes the semantic representation ability of neural networks in road network extraction, house detection, and crop classification. The FCN network proposes a feature extraction module, which is often used for building semantic segmentation. The UNet network proposes the idea of skip connection based on encoder-decoder, which fuses deep and shallow features to improve the accuracy of semantic segmentation. Compared with the FCN network, it extracts more complete features. The Hrnet network effectively reduces feature information loss by cascading multi-scale resolution features, making the extracted multi-scale features more rich, and achieving pixel-level extraction of house information. The PSPNet network aggregates multi-scale features in different regions through a pyramid pooling module and a pyramid scene parsing module, and the integrity of the building edge is improved to a certain extent. The Deeplab series network aggregates multi-scale information features by introducing deep separable convolution, and obtains better edge perception effect. The SegFormer network improves the invalid extraction of non-house features to a certain extent by pruning the position encoding and using the MLP (Multilayer Perception) multilayer perception for feature extraction. Although the above feature extraction networks have high detection accuracy, the information loss problem has not been effectively improved, and the detection network does not focus on extracting house features and suppressing the extraction of non-house information, so the house extraction effect in complex environments needs to be further improved.

[0004] Existing research indicates that high-resolution remote sensing building extraction methods based on deep learning have significant advantages when supported by a large number of samples, but some problems still exist: On the one hand, most existing studies directly apply convolutional neural network models to the field of remote sensing, and there is a relative lack of network models specifically built for building detection or house classification in remote sensing images; on the other hand, buildings in remote sensing images are affected by factors such as sensor data acquisition methods, different scales, imaging conditions, complex terrain, building shadows, and building tilt, requiring the exploration of more efficient ways to obtain semantic information and detailed features of buildings. Summary of the Invention

[0005] To address the limitations of existing convolutional neural network models in building extraction from remote sensing images and their poor building feature extraction capabilities, this invention provides a method for detecting single-unit planar buildings based on SegFormer.

[0006] The present invention provides a method for detecting single-unit planar buildings based on SegFormer, comprising:

[0007] Step 1: Construct a building extraction network based on the improved SegFormer, including an encoder and a decoder. The encoder consists of four Transformer blocks connected sequentially from shallow to deep layers. The decoder consists of a first 1×1 convolutional layer, an ASPP module, and a semantic fusion module connected sequentially from shallow to deep layers. Remote sensing imagery is input to the first Transformer block, and the output of the first Transformer block serves as one input to the semantic fusion module. The output of the fourth Transformer block serves as the input to the first 1×1 convolutional layer, and the output of the ASPP module serves as the other input to the semantic fusion module. The output of the semantic fusion module is the detection result.

[0008] Step 2: Train the building extraction network based on the improved SegFormer to obtain the building extraction network model;

[0009] Step 3: Use the trained building extraction network model to extract buildings from the input remote sensing image and obtain the detection results.

[0010] Furthermore, the semantic fusion module includes a second 1×1 convolutional layer, an upsampling layer, a network cascade layer, a transposed convolutional sampling module, and a third 1×1 convolutional layer;

[0011] The output of the first Transformer block is used as the input of the second 1×1 convolutional layer, and the output of the ASPP module is used as the input of the upsampling layer.

[0012] The output of the second 1*1 convolutional layer and the output of the up-sampling layer are processed by a network concatenation layer, and then are sequentially input to a transposed convolutional sampling module and a third 1*1 convolutional layer, and the output of the third 1*1 convolutional layer is a detection result.

[0013] Further, the transposed convolutional sampling module comprises two 3*3 two-dimensional convolutional layers, two 4*4 two-dimensional transposed convolutional layers and a 1*1 two-dimensional convolutional layer connected in sequence from shallow to deep.

[0014] Further, for the first four layers of the transposed convolutional sampling module, a two-dimensional normalization layer and a Relu activation layer are sequentially interpolated after each convolutional operation.

[0015] Further, the data processing process of the ASPP module is as follows:

[0016] The input feature F is augmented into i groups, denoted as ASPP i , i = 1, 2, 3, 4; the grouped features ASPP i are respectively processed by a dilated convolutional layer with an expansion rate of 1, 3, 11 and 17 to obtain processed grouped features ASPP i .

[0017] The input feature F is pooled and then up-sampled and operated in parallel, and is spliced and fused with all the grouped features ASPP i , and the spliced and fused features are sequentially input to a 1*1 convolutional layer and an up-sampling module, and the output of the up-sampling module is the final output of the ASPP module.

[0018] Further, in step 2, a transfer learning strategy is adopted, and the building extraction network based on the improved SegFormer is trained using the pre-trained weights on the ImageNet-1K dataset.

[0019] The present application has the following advantages:

[0020] (1) In view of the problem that the existing SegFormer model has poor applicability in building extraction of remote sensing images, the present application improves the traditional SegFormer network, optimizes the detection network structure, and strengthens the sampling of building semantic and detail features.

[0021] (2) The semantic fusion module fuses the low-level feature information extracted from the input remote sensing image by the encoder with the high-level feature information extracted, and cascades the interlayer receptive field, so as to weaken the influence of spatial heterogeneity and complex ground object background, and strengthen the detection of building features.

[0022] (3) In view of the problems of multiple interference, missing and misjudgment of semantic and detailed feature information of buildings in the extraction of house features in remote sensing images, the decoding output part of the network model is optimized, and an ASPP module is used, which explores multi-scale context information by applying multi-sampling rate atrous convolution and multi-receptive field convolution on the input feature map; rich context information is captured on different resolutions by pooling operation, and spatial information is gradually recovered to capture clear target boundary;

[0023] (4) By combining the ASPP module with the semantic fusion module, the shallow rough information and the deep fine information are aggregated, the feature information interaction of different layers is realized by shuffling, the problems of missing of local building detail information and lack of correlation of long-distance information are overcome, the feature information missing caused by the difference of image resolution due to the redundancy of data source is effectively relieved, and the problems of missing of multi-scale house edge details and misclassification of small houses in building extraction are solved. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 The structure diagram of the building extraction network based on the improved SegFormer provided for the embodiments of the present application is shown;

[0025] Figure 2 The sampling method diagram provided for the embodiments of the present application is shown;

[0026] Figure 3 The structure diagram of the transposed convolution sampling module provided for the embodiments of the present application is shown;

[0027] Figure 4 The interpolation position diagram of the normalization activation layer provided for the embodiments of the present application is shown;

[0028] Figure 5 The Shuffle shuffling diagram provided for the embodiments of the present application is shown;

[0029] Figure 6 The diagram of the semantic fusion module provided for the embodiments of the present application is shown;

[0030] Figure 7 The data processing process diagram of the ASPP module provided for the embodiments of the present application is shown;

[0031] Figure 8 The reasonable dilation rate atrous convolution diagram provided for the prior art is shown;

[0032] Figure 9 The grid effect diagram of the atrous convolution provided for the prior art is shown;

[0033] Figure 10The schematic diagram of the three data sets provided by the embodiment of the present application is shown in the following figure: (a) AISD; (b) WHU; (c) MBD;

[0034] Figure 11 The change of the loss value and the segmentation accuracy of the original SegFormer model and the improved SegFormer network model of the present application on the MBD data set in the training process is shown in the following figure: (a) the original model training accuracy change curve; (b) the improved model training accuracy change curve; (c) the original model training loss change curve; (d) the improved model training loss change curve;

[0035] Figure 12 The experimental prediction results of the existing network and the method of the present application are shown in the following figure: (a) the original image; (h) the data set label; (b)-(f) the detection results of the existing network, which are Hrnet, PSPNet, UNet, Deeplabv3+, and the original SegFormer, respectively; (g) the detection result of the improved network of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be described clearly in combination with the drawings in the embodiment of the present application. Obviously, the described embodiment is a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0037] Embodiment 1

[0038] The embodiment of the present application provides a single building detection method based on SegFormer, which comprises the following steps:

[0039] S101: constructing a building extraction network based on improved SegFormer;

[0040] Specifically, as Figure 1As shown, comprising an encoder and a decoder; the encoder includes four Transform blocks (namely Transform Block1, Transform Block2, Transform Block3 and Transform Block4) connected in turn from shallow to deep; the decoder includes a first 1x1 convolution layer, an ASPP module and a semantic fusion module connected in turn from shallow to deep; wherein the remote sensing image is input to the first Transform block, the output of the first Transform block is used as one of the inputs of the semantic fusion module, the output of the fourth Transform block is used as the input of the first 1x1 convolution layer, and the output of the ASPP module is used as another input of the semantic fusion module; the output of the semantic fusion module is the detection result.

[0041] Specifically, the ASPP (Atrous Spatial Pyramid Pooling) module is composed of a series of different dilated rate hollow convolution and spatial pyramid pooling structure, which extracts multi-scale information of the image by parallel multiple different dilated rate hollow convolution, and obtains global information of the image by introducing global average pooling (GAP). The ASPP module overcomes the shortcomings of local information loss and long-distance information lack of correlation caused by grid effect when using single hollow convolution, can obtain different scale feature information without using pooling layer, and clear high-resolution building edge information.

[0042] S102: training the building extraction network based on the improved SegFormer to obtain a building extraction network model;

[0043] S103: using the trained building extraction network model to perform building extraction on the input remote sensing image to obtain a detection result.

[0044] In view of the problem that the existing SegFormer model has poor applicability in building extraction of remote sensing images, the present application improves the traditional SegFormer network, optimizes the detection network structure, and strengthens the sampling of building semantic and detail features. The semantic fusion network is used to fuse the low-level feature information extracted from the input remote sensing image by the first Transform block in the encoder and the high-level feature information extracted by the fourth Transform block, and the inter-layer receptive field is cascaded, so as to weaken the influence of spatial heterogeneity and complex ground object background, and strengthen the detection of building features.

[0045] In addition, in view of the problems of much interference, missing and misjudgment of semantic and detail feature information of buildings in house feature extraction of remote sensing images, the network model decoding output part is optimized, and the ASPP module is adopted, the module explores multi-scale context information by applying multi-sampling rate hollow convolution and multi-receiving field convolution on the input feature map; rich context information is captured on different resolutions by pooling operation, and spatial information is gradually recovered to capture clear target boundary; further, by combining the ASPP module with the semantic fusion module, shallow rough information and deep fine information are aggregated, different layer feature information interaction is realized through Shuffle, the problems of missing of local building detail information and lack of correlation of long-distance information are overcome, the feature information missing caused by data source redundancy and differentiation of image resolution is effectively relieved, so that the problems of missing of multi-scale house edge details and misjudgment of small houses in building extraction are solved.

[0046] Embodiment 2

[0047] Remote sensing image information is complex and has much interference, the higher the discrimination between detected ground objects and background information, the easier to identify stable and strong anti-interference features, therefore, from the perspective of strengthening building feature expression and inhibiting useless background information, on the basis of the above embodiment, the sampling module is coupled with the transpose convolution network layer in the semantic fusion module in the embodiment of the application.

[0048] As shown in Figure 1 The semantic fusion module includes a second 1x1 convolution layer, an up-sampling layer, a network concatenation layer, a transpose convolution sampling module and a third 1x1 convolution layer; the output of the first Transformer block is taken as the input of the second 1x1 convolution layer, and the output of the ASPP module is taken as the input of the up-sampling layer; the output of the second 1x1 convolution layer and the output of the up-sampling layer are processed through the network concatenation layer, and then are sequentially input to the transpose convolution sampling module and the third 1x1 convolution layer, and the output of the third 1x1 convolution layer is the detection result.

[0049] Specifically, after multiple convolution operations on the image, the size of the feature map will be continuously reduced. The transpose convolution can effectively couple high and low layer feature information, concatenate interlayer receptive field, and solve the problem of overflow of non-building information caused by inconsistent image resolution or different feature map scales.

[0050] In the convolution operation C, the convolution Y output for the n-dimensional vector X can be represented as: CX=Y, and the transpose convolution can be represented as: X=C T Y. The purpose is to restore more semantic information by rearranging the input and kernel, adding holes and padding, and the specific implementation form is as follows:

[0051] Ignoring the number of channels, assume a stride of s and a fill block of p. The input tensor shape is n. h ×n w The convolution kernel shape is k h ×k w The transpose operation produces n h ×n w There are n intermediate results. Each intermediate result is an (n) intermediate result. h +k h -1)×(n w +k w A tensor with parameters initialized to 0 (-1). The method for calculating the intermediate tensor: multiply each element of the input tensor by the convolution kernel to obtain k. h ×k w The intermediate tensor is replaced by a portion of the intermediate tensor. The position of the replaced portion in each intermediate tensor corresponds to the position of an element in the input tensor. Finally, all intermediate results are summed to obtain the final result. The intermediate tensor calculation formula (1) is as follows:

[0052] Y[i:(i+h),j:(j+w)]+=X[i,j]*K (1)

[0053] When n and k are both 2, stride s = 1, padding p = 0, and the input parameters are {0, 1, 2, 3}, the operation form is as follows: Figure 2 As shown.

[0054] Based on the above theory, this embodiment designs a transposed convolution sampling module as follows: Figure 3 Illustration: Following the MLP output of a feature map with 4C channels, two 3×3 2D convolutional layers, two 4×4 2D transposed convolutional layers, and one 1×1 2D convolutional layer are embedded in the downsampling part. Through the transposed convolution operation, while preserving the blurred boundaries, some information is effectively recovered, thereby outputting more targeted detailed and semantic information features.

[0055] Furthermore, since convolution operations can only abstract local features of building images and ignore the relationships between pixels, inconsistencies between object classes are caused, greatly affecting the reliability and completeness of building edge segmentation. Adding an additional normalized activation layer after the convolution operation, through a regularization expression to prevent overfitting, makes the selection of feature parameters more stable, avoids the influence of outliers and extreme values, suppresses noise interference, enhances the network's expressive power, and focuses more on saliency and information-rich feature channels and spatial locations, thus making it easier to obtain the internal correlation of images or features, further solving the problem of accurate extraction of differences between building classes. The normalized activation layer is set based on the introduction of a transposed convolution sampling module; the setting method is described in [link to documentation]. Figure 4For the first four layers of the transposed convolution sampling module, a two-dimensional normalization layer and a ReLU activation layer are interpolated sequentially after each convolution operation. By fusing multiple normalization activation layers, the efficiency of filtering semantic information about buildings is improved by compressing the number of parameters, suppressing gradient vanishing, and accelerating convergence. Effective filtering and classification of intra-class feature information is achieved by relying on the embedded activation layers.

[0056] In this embodiment, a semantic fusion module (SFM) is used. Figure 5 The method fuses the detailed information obtained from shallow, coarse feature extraction with the deep, refined semantic information obtained after ASPP multi-scale hollow pyramid pooling network computation, and then performs channel shuffling on the fused information. Figure 6 This method rearranges the original grouped feature channels by "reshaping-transpose-reshaping," and then uses network autonomous learning to predict the "semantic flow" information between features of different resolutions. This corrects coarse features into fine features with higher resolution, aligning pixels between features of different resolutions. Furthermore, by shuffling information from different channels to harmonize multi-scale information, the method enhances information exchange between feature channels of different groups, resulting in a feature layer rich in multi-scale information. This effectively transmits semantic information from deep layers to shallow layers, achieving effective fusion between features of different resolutions.

[0057] Example 3

[0058] Conventional fusion methods that simply skip and stitch semantically dissimilar features cannot solve the semantic gap problem caused by the lack of multi-scale features in the network design itself, and will severely limit the network's ability to extract large-scale house edges and small house objects themselves. Therefore, based on the above embodiments, the ASPP module in this embodiment of the invention has been further designed.

[0059] like Figure 7 As shown, the data processing procedure of the ASPP module in this embodiment is as follows: the input feature F is augmented into i groups, denoted as ASPP. i i = 1, 2, 3, 4; Grouping feature ASPP i Each feature is processed through dilated convolutional layers with dilation rates of 1, 3, 11, and 17, respectively, to obtain the processed grouped features ASPP. i The input feature F is pooled, then upsampled and processed in parallel, and then combined with all grouped features ASPP. i The concatenated and fused features are then sequentially input into a 1×1 convolutional layer and an upsampling module. The output of the upsampling module is the final output of the ASPP module.

[0060] Specifically, the deep network features are expanded at different inflation rates, the ASPP module is used to capture multi-scale feature context information, and a global average pooling module is introduced to supplement and improve the context information.

[0061] Thus, after the shallow features output by the first Transformer block and the deep features output by the ASPP module in this embodiment are input into the semantic fusion module, the semantic fusion module establishes the corresponding relationship between pixels at different levels of features, integrates the deep semantic information into the shallow image detail information in a bottom-up manner, expands the input feature map and groups it according to the inflation coefficient, extracts features of different scales using dilated kernels of different sizes, and mixes the extracted features to realize information interaction at different levels and enhance the expression of global semantic information.

[0062] Compared with the traditional convolutional layer and pooling layer, the advantages of the dilated convolution lie in the following two aspects: ① After replacing the traditional convolutional layer with the dilated convolution, the increase in the number of parameters is avoided as much as possible, and the new parameters that are reasonably inflated can be trained, so that the performance can be continuously optimized during the training process. ② The size of the receptive field can be changed by setting different dilation rates to obtain feature maps at different resolutions, thereby reducing the loss of position information in the down-sampling process.

[0063] In this embodiment, the conventional convolution is replaced with the dilated convolution, and the size of the receptive field in the pooling process is changed by changing the dilation rate of the dilated convolution. It is found in experiments that different inflation rates also have a certain impact on the performance of the model. The dilated convolution mode with a reasonable inflation rate is as follows Figure 8 Through research, it is found that: ① Using the dilated convolution with the same inflation rate will cause the convolution kernel to be discontinuous, resulting in a "grid effect", as shown in Figure 9 . ② The combination of inflation rates cannot contain a common divisor greater than 1, and the value of the inflation rate R i needs to meet formula (2) as follows. ③ Assuming that the inflation rates of N dilated convolutions with a kernel size of k x k are [r i ,...r n ], then R2≤k needs to be met, where r i represents the inflation rate of the i-th dilated convolution, R i represents the maximum inflation rate of the i-th dilated convolution, and Rn=r n by default. Based on the above conclusion, the dilated convolution combination with inflation rates of 1, 3, 11, and 17 is designed in this embodiment.

[0064] R i = max[R i+1 -2r i ,R i+1 -2(R i+1 -r i ),r i] (2)

[0065] Example 4

[0066] To verify the effectiveness and generalization performance of the proposed method, the following experimental data is also provided in this embodiment.

[0067] I. Experimental data set

[0068] AISD (Aerial Image Segmentation Dataset) (see Abdollahi A, Pradhan B, Alamri A M. An ensemble architecture of deep convolutional Segnet and Unet networks for building semantic segmentation from high-resolution aerial images [J]. Geocarto International, 2022, 37(12): 3355-3370.) and MBD (Massachusetts Buildings Dataset) (see Wang L, Fang S, Meng X, et al. Building extraction with vision transformer [J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-11) and WHU aerial remote sensing image dataset (see http: / / study.rsgis.whu.edu.cn / pages / download / ) are used for testing and comparing the results of multiple methods.

[0069] (1) AISD semantic segmentation dataset, which contains four types of ground object labels: houses, roads, water systems, and vegetation. In this experiment, only the building label is used for analysis. It contains 1672 images with a resolution of 3328*2816 pixels. The images are from Google Earth, and the labels are from Open Street Map pixel-level labels. The coverage area includes Berlin, Chicago, Paris, Potsdam, and Zurich, with a resolution of 0.3 meters.

[0070] (2) MBD Massachusetts Building Dataset, which contains 151 images with a resolution of 1500x1500 pixels. The coverage area includes urban and suburban areas in the Boston area. Each image covers an area of 2.25 square kilometers with a resolution of 1 meter.

[0071] (3) WHU Wuhan University Building Dataset, containing 204 512x512 pixel images in the Satellite Dataset I (Global Cities) column, covering global city satellite images with a resolution of 0.3 meters.

[0072] The above three data sets are shown in the following table: Figure 10

[0073] The data set division is shown in Table 1:

[0074] Table 1: Data sets used in the experiment

[0075]

[0076] II. Experimental environment and parameter setting

[0077] The experimental operating system is Windows 11, equipped with a 12th Gen Intel(R) Core(TM) i9-12900H CPU and a NVIDIA GeForce RTX 3080 TI GPU, and the deep learning framework is torch 1.11.0+cu115. The experiment uses a transfer learning strategy, using pre-trained weights on the ImageNet-1K dataset for training. The network model improves the accuracy of the detection result by relying on its unsaturated continuous learning ability through transfer learning. The training process is divided into a frozen stage and an unfreezing stage, and the training configuration information is shown in Table 2:

[0078] Table 2: Data set training configuration

[0079]

[0080] III. Ablation experiment

[0081] To explore the contribution of the improvements made by the invention to the network model, an ablation experiment was conducted on the AISD and MBD, WHU data sets using SegFormer as the baseline system. The experimental parameters and experimental environment were kept consistent, and the results are shown in Table 1.

[0082] (I) Ablation experiment with different mechanism modules

[0083] ​To verify the effectiveness of the improved mechanism, the present experiment respectively carries out ablation experiment on AISD and MBD, WHU data sets with SegFormer (referred to as segf) as the baseline system, and evaluates the model performance improvement effect through mIoU (mean Intersection over Union), F1 score, Precision, Recall and other evaluation indexes, wherein mIoU refers to the average value of the intersection and union percentage of the predicted house pixel number and the true house pixel; F1 score comprehensively considers the influence of precision and recall; Precision refers to the percentage of pixels that the model can correctly predict the house and the true house; Recall refers to the percentage of the true house that is correctly predicted. The network training process is visualized, and the improved network effect is comprehensively analyzed.

[0084] (1) Ablation experiment of different improved mechanisms

[0085] The method of adding different mechanisms is respectively experimented on three data sets and the original network model, and the evaluation indexes are compared as shown in the following table 3.

[0086] Table 3: Comparison of experimental precision of improved network and original network

[0087]

[0088] From the experimental results in table 3, when the experiment is carried out on AISD and MBD, WHU data sets, the precision evaluation indexes of the improved network of the present application are all better than those of the original network, which verifies the effectiveness of the improved network of the present application in building extraction. Among them, the improved network has the best improvement effect on the precision indexes of MBD data set, the mIoU is improved by 9.76%, the F1 is improved by 10.54%, the Precision is improved by 10.58%, and the Recall is improved by 10.45%. It is analyzed that the original network has weak applicability to the data set, and the mIoU is only 66.09%. After optimizing the network, the improved network has stronger feature extraction and feature fusion capability, can better recover and pay attention to different layer semantic information and feature information, and effectively improves the overall segmentation precision of the model.

[0089] (2) Visualization of training process

[0090] Figure 11 The change of loss value and segmentation precision of the original SegFormer model and the improved SegFormer network model in the training process in the MBD data set experiment is shown in the figure.

[0091] After adding the transposed convolution network module, the loss value of the improved network model is smaller than that of the original network model, and the segmentation precision is higher than that of the original network model. Figure 11From (a) and (b), it can be seen that the improved network has smaller precision curve fluctuations and a more flat overall training curve in the training process, and can reach a higher precision than the original network in a shorter time. By comparing the dynamic change curves of precision and loss in the training process, it can be seen that the improved network has smaller fluctuations and a more flat overall training curve. Figure 11 From (c) and (d), it can be seen that the improved network can dynamically adjust according to the context information, more efficiently supplement the detail features and semantic features, and has good extraction precision, generalization ability and universality.

[0092] (II) Ablation experiment of multiple normalization activation layers

[0093] In the improvement strategy of stacking normalization activation layers, in order to further understand the effectiveness and contribution of the added modules in building the transpose convolutional network, and to explore the effect of cascading multiple layers of normalization activation on network expression ability and precision improvement, the multiple accumulation method is adopted on the MBD dataset to analyze the improvement effect. The addition method is shown in Figure 4 .

[0094] Table 4: Ablation experiment of normalization activation layer

[0095]

[0096] As can be seen from Table 4, with the multiple superposition of normalization activation layers, the selected evaluation indicators mIoU, F1, Precision and Recall are all improved. Among them, mIoU is improved by 3.11%, F1 is improved by 1.27%, Precision is improved by 0.58%, and Recall is improved by 1.81%, and the precision is improved the most when four layers of normalization activation layers are embedded in the convolutional network. The experiment verifies that the introduction of multiple normalization activation layers improves the building detection ability of the improved network.

[0097] (III) Ablation experiment of dilated spatial pyramid pooling decoding module expansion rate

[0098] Table 5: Effect of different expansion rates on ASPP module performance

[0099]

[0100] The experiment also verifies the influence of different expansion rate combinations on the performance of the ASPP module. The comparative experimental results based on the MBD data set are shown in Table 5. As can be seen from Table 5, the non-identical continuous convolution effect is better: when the combination is 1, 3, 6, and 9, the accuracy is improved by 0.33% than the combination 1, 2, 2, and 2; the large expansion kernel coprime combination effect is better, when the combination is 1, 5, 11, and 17, the accuracy is improved by 0.27% than the combination 1, 6, 12, and 18; the second expansion rate is not greater than the maximum size of the convolution kernel: when the combination is 1, 3, 11, and 17, the accuracy is improved by 2.15% than the combination 1, 5, 11, and 17. Based on this, the non-continuous, common divisor of 1, and large expansion rate 1, 3, 11, and 17 are selected for the hollow convolution combination, and in the experiment, the combination is improved by 1.96%-3.93% than other combinations. The experimental results prove that reasonable expansion rate not only does not cause the loss of relevant information, but also can improve the effect of capturing different scale target context information.

[0101] Four, comparative experiment

[0102] The experimental data set is derived from AISD, MBD, and WHU public data sets, and is analyzed based on high-precision pixel-level labels to accurately represent the range, distribution, and geometric contour of the house on the image. In this experiment, the original data set label information is visualized on the image and compared with the experimental results. In order to verify the effectiveness of the improved algorithm, several classical mainstream deep learning semantic segmentation algorithms such as Hrnet, PSPNet, UNet, Deeplabv3+, and the original SegFormer network are selected for comparison with the algorithm of the present application, and the comparison results are as follows:

[0103] Table 6: Accuracy comparison of different methods

[0104]

[0105] As can be seen from Table 6, compared with the mainstream methods in recent years and the original SegFormer model, the improved method is superior to the comparison method in overall experimental accuracy. Among them, on the AISD data set, the mIoU reaches 88.43%, the F1 score is 93.81%, the Precision is 93.85%, and the recall rate is 93.77%; on the MBD data set, the mIoU reaches 75.85%, the F1 score is 88.24%, the Precision is 93.07%, and the recall rate is 83.89%; on the WHU data set, the mIoU reaches 81.69%, the F1 score is 91.09%, the Precision is 92.46%, and the recall rate is 89.76%. On the three data sets, the mIoU value of the improved method is optimally improved by 17.68%, 30.44%, and 15.26% compared with the comparison method, and the Precision accuracy value is optimally improved by 11.17%, 20.02%, and 11.86%.

[0106] From the precision value comparison, it is found that the experimental precision of all algorithms is obviously lower than that of the other two data sets when using different extraction methods in the MBD data set, which is due to the fact that all images in the MBD data set use large-scale scene images, such as Figure 10 The image scale is greatly different from the other two data sets, and the building style, spectral feature and shadow feature are more diverse and complex in the large-scale scene, so the segmentation network does not have very ideal experimental effect on the MBD data set. However, after the network is optimized, the improved network effectively improves the detection precision of the MBD data set, wherein mIoU is improved by 9.76%, F1 is improved by 10.54%, Precision is improved by 10.58%, and Recall is improved by 10.45%. The experimental detection precision is improved by an order of magnitude, which indicates that after the network is improved, the applicability of the large-scale scene image is enhanced, and the network performs better on large-scale and large-scale data sets, and has stronger robustness to complex buildings and variable environment, which verifies the effectiveness of the algorithm improvement.

[0107] Figure 12 The experimental prediction results of Hrnet, PSPNet, UNet, Deeplabv3+, original SegFormer algorithm and the algorithm of the application are shown in Table 1. Based on the detection result visualization diagram, the performance of different algorithms is evaluated, wherein column (a) is the original image, column (h) is the data set label, columns (b)-(f) are the detection results of the comparison network, and column (g) is the detection result of the improved network. As can be seen from the figure, when testing on the same image, the improved network has better detection effect than other networks, mainly in the following aspects:

[0108] (1) The edge integrity is better, and the boundary transition is more regular and smooth: the building form of the house is more regular, and the original image and the label of the data set are shown in the first and second rows of Table 1. Figure 12 As can be seen from the first and second rows of Table 1, in the detection of a single independent house, Hrnet, PSPNet and SegFormer have poor ability to depict the right-angle edge of the house, UNet and the improved network can accurately locate the edge line of the house; but in the edge detection, the improved network is more complete and reasonable than the UNet network in the edge detection detail transition, and can better distinguish the background and the house boundary; and compared with the original SegFormer network, the improved network better corrects the regular edge of the house, and the detection effect is obviously improved based on the original network.

[0109] (2) Strengthen the construction of building feature information, effectively suppress the missed detection and misdetected caused by the similarity between ground objects, and have strong multi-scale generalization ability and improved detection rate of small buildings: in Figure 12In the 3rd and 4th rows of the middle, when the small houses in the original drawing are similar in color to the surrounding background objects and have a large difference in size from the surrounding houses, the buildings are missing in the PSPNet and Deeplabv3+ experiments, and although the Hrnet, UNet and SegFormer experiments can realize the positioning of part of the houses, the buildings are not effectively detected, the improved network accurately positions the small buildings and detects the distribution range and edges thereof.

[0110] (3) The detail feature information is relatively rich and complete, and has strong distinguishing ability for the houses and the surrounding background. Figure 12 As can be seen from the 5th and 6th rows in the middle, when the polygonal houses are detected, the edges of several comparative algorithms are obviously fitted, the edge details at the corners and bends are seriously lost, and the distribution of the houses is misjudged in a large range, while the improved network restores more complete details than other networks, and the difference between the distribution range and the contour is not large.

[0111] In order to realize high-precision extraction of buildings in remote sensing images, the improved sampling mechanism and decoding structure remote sensing image building detection method based on SegFormer is proposed, which is used for improving the problems of building missing and misjudgment, poor edge integrity, poor intelligent interpretation effect of remote sensing images caused by building inter-class similarity and intra-class inconsistency, and the main improvement content is to introduce a transpose convolution network to optimize the sampling mode, and a spatial pyramid pooling decoding module is used to fuse the dilated convolution, so that the algorithm can effectively filter the positive information of the building, suppress and filter the negative information, and alleviate the problems of semantic information and detail feature loss. Ablation experiments and comparative experiments are carried out on AISD, MBD and WHU remote sensing image datasets, and the experimental results show that the improved method has more accurate boundary segmentation, reduces building missing and misjudgment, and can be applied to large-scale scene detection and intelligent interpretation, and provides strong support for surveying and mapping production.

[0112] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for detecting single-unit planar buildings based on SegFormer, characterized in that, include: Step 1: Construct a building extraction network based on the improved SegFormer, including an encoder and a decoder. The encoder consists of four Transformer blocks connected sequentially from shallow to deep layers. The decoder consists of a first 1×1 convolutional layer, an ASPP module, and a semantic fusion module connected sequentially from shallow to deep layers. Remote sensing imagery is input to the first Transformer block, and the output of the first Transformer block serves as one input to the semantic fusion module. The output of the fourth Transformer block serves as the input to the first 1×1 convolutional layer, and the output of the ASPP module serves as the other input to the semantic fusion module. The output of the semantic fusion module is the detection result. The semantic fusion module includes a second 1×1 convolutional layer, an upsampling layer, a network cascade layer, a transposed convolutional sampling module, and a third 1×1 convolutional layer. The output of the first Transformer block is used as the input of the second 1×1 convolutional layer, and the output of the ASPP module is used as the input of the upsampling layer. The outputs of the second 1×1 convolutional layer and the upsampling layer are processed by the network cascade layer and then sequentially input to the transposed convolutional sampling module and the third 1×1 convolutional layer. The output of the third 1×1 convolutional layer is the detection result. The data processing procedure of the ASPP module is as follows: The input features F are augmented into i groups, denoted as ASPP. i i=1,2,3,4; Grouping feature ASPP i Each feature is processed through dilated convolutional layers with dilation rates of 1, 3, 11, and 17, respectively, to obtain the processed grouped features ASPP. i '; The input feature F is pooled, upsampled, and processed in parallel, then combined with all grouped features ASPP. i The concatenated and fused features are then sequentially input into a 1×1 convolutional layer and an upsampling module. The output of the upsampling module is the final output of the ASPP module. Step 2: Train the building extraction network based on the improved SegFormer to obtain the building extraction network model; Step 3: Use the trained building extraction network model to extract buildings from the input remote sensing image and obtain the detection results.

2. The method for detecting single-unit planar buildings based on SegFormer according to claim 1, characterized in that, The transposed convolutional sampling module consists of two 3×3 two-dimensional convolutional layers, two 4×4 two-dimensional transposed convolutional layers, and one 1×1 two-dimensional convolutional layer connected sequentially from shallow to deep layers.

3. The method for detecting single-unit planar buildings based on SegFormer according to claim 2, characterized in that, For the first four layers of the transposed convolution sampling module, a two-dimensional normalization layer and a ReLU activation layer are interpolated sequentially after each convolution operation.

4. The method for detecting single-unit planar buildings based on SegFormer according to claim 1, characterized in that, In step 2, a transfer learning strategy is adopted, and the building extraction network based on the improved SegFormer is trained using pre-trained weights on the ImageNet-1K dataset.

Images