High-resolution remote sensing image building roof extraction and height estimation method based on deep learning

Abstract

The invention discloses a high-resolution remote sensing image building roof extraction and height estimation method based on deep learning, and relates to the technical field of remote sensing data processing and application. Comprising the following steps: acquiring a self-made city building data set, a WHU global city data set, an Inria data set, a Vaihinge data set and a Potsdam data set, preprocessing the data sets, constructing a building roof extraction model MA-UNet + + and an evaluation index, verifying and training the MA-UNet + + model by using the self-made city building data set, the WHU global city data set and the Inria data set to obtain an optimal MA-UNet + + model, and establishing a building roof extraction model based on the optimal MA-UNet + + model. A building height estimation model SDLHNet and an evaluation index are constructed, verification and training are performed by using a Vaihingen data set and a Potsdam data set, an optimal SDLHNet model is obtained, fusion analysis is performed on the optimal MA-UNet + + model and the optimal SDLHNet model based on the Potsdam data set, and comprehensive performance and benefits are determined. According to the method, the accuracy of extracting the multi-dimensional information of the building in the high-resolution remote sensing image can be improved.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing data processing and application technology, and in particular to a method for extracting and estimating the height of a building roof from a high-resolution remote sensing image based on deep learning. Background Art

[0002] With the continuous advancement of my country's urbanization process, smart cities, digital twins, and real-life 3D China construction have received widespread attention, which has put forward new requirements and challenges for urban planning and management. Buildings are important man-made features that have gradually evolved in the process of human development and are important areas for production and various life activities. The multi-dimensional information of buildings has a wide range of application value not only in urban planning, land grading, and map navigation, but also in regional population estimation, economic development assessment, topographic map production and updating, and basic geographic data expansion. Therefore, how to quickly and accurately obtain the location, outline, and height information of buildings has become particularly important. The traditional method of manual data collection is labor-intensive, inefficient, and has poor timeliness. It is difficult to meet the rapid update of building information at a large scale. Remote sensing technology provides the necessary data foundation for the automatic extraction of buildings in a large range, and provides a technical guarantee for the rapid extraction of buildings in a large spatial scale.

[0003] For the task of extracting building roofs, traditional methods mainly rely on image processing techniques, such as edge detection, morphological operations, texture analysis, and threshold segmentation. These methods extract buildings by identifying features such as geometric shapes, color textures, etc. in images. They generally have low accuracy and poor generalization ability when processing high-resolution images. Accurately understanding the height information of buildings is of great significance for urban three-dimensional modeling, disaster risk assessment, etc. However, there are currently few studies on analyzing building height information using high-resolution remote sensing images. This is due to the two-dimensional nature of remote sensing images. It is difficult to estimate three-dimensional information directly from a single image, especially in the absence of auxiliary data. Existing methods based on stereo pairs, radar remote sensing, shadow analysis, etc. have a series of shortcomings such as high image quality requirements, long processing time, susceptibility to terrain restrictions, and low automation.

[0004] Therefore, it is an urgent problem for technical personnel in this field to propose a method for extracting and estimating building roofs from high-resolution remote sensing images based on deep learning to solve the problems existing in the prior art. Summary of the invention

[0005] In view of this, the present invention provides a method for extracting and estimating building roofs from high-resolution remote sensing images based on deep learning, which helps to improve the accuracy of extracting multi-dimensional information of buildings in high-resolution remote sensing images.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A method for extracting and estimating building roofs from high-resolution remote sensing images based on deep learning, comprising the following steps:

[0008] S1. Obtain self-made urban building dataset, WHU global city dataset, Inria dataset, Vaihingen dataset and Potsdam dataset, and preprocess the dataset;

[0009] S2. Construct the building roof extraction model MA-UNet++ and evaluation indicators, and use the self-made urban building dataset, WHU global city dataset and Inria dataset to verify and train the MA-UNet++ model, and obtain the optimal MA-UNet++ model;

[0010] S3, construct the building height estimation model SDLHNet and evaluation indicators, use the Vaihingen dataset and Potsdam dataset for verification and training, and obtain the optimal SDLHNet model;

[0011] S4. The optimal MA-UNet++ model and the optimal SDLHNet model are fused and analyzed based on the Potsdam dataset to determine the comprehensive performance and benefits.

[0012] In the above method, optionally, the self-made urban building dataset in S1 is any urban building dataset.

[0013] In the above method, optionally, the MA-UNet++ model in S2 includes:

[0014] The multi-scale feature grouping convolution module is used to replace the original encoder module of UNet++, Cross EntropyLoss is used as the loss function of the UNet++ model, and spatial and channel attention mechanism modules are added to the model.

[0015] The above method is optional. The evaluation indicators in S2 include: intersection over union, recall rate, precision and F1 score. The model training hyperparameters are set. 42 rounds of training are performed on the WHU global city dataset and 32 rounds of training are performed on the Inria dataset.

[0016] The above method is optional. Construction of the SDLHNet model in S3 includes: using the SwinTransformer architecture as the feature extraction module of SDLHNet, adding a shift window mechanism to integrate information, using a dense dilated convolutional pyramid module to organize each dilated convolutional layer, and adding a local plane guidance module to merge and output features generated by different decoders at different stages.

[0017] In the above method, optionally, the evaluation indicators in S3 include: Rel, RMSE, RMSE(log), δ1, δ2 and δ3.

[0018] In the above method, optionally, the fusion analysis in S4 includes adding a height attribute to each extracted building, combining the original image with the stretched height value, and forming a three-dimensional visual effect.

[0019] It can be seen from the above technical solution that compared with the prior art, the present invention provides a method for extracting and estimating building roofs from high-resolution remote sensing images based on deep learning, and the beneficial effects include: being able to improve the accuracy of extracting multi-dimensional information of buildings in high-resolution remote sensing images, being helpful in constructing building models, having great application reference value for the development of smart cities and the construction of real three-dimensional China, and being able to solve more practical problems in the production field. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0021] Figure 1 A flowchart of a method for extracting and estimating building roofs from high-resolution remote sensing images based on deep learning disclosed in the present invention;

[0022] Figure 2 This is a schematic diagram of the MA-UNet++ model disclosed in the present invention;

[0023] Figure 3 The schematic diagram of the SDLHNet model disclosed in the present invention

[0024] Figure 4 It is a rendering of the building roof extraction of the self-made urban building data set disclosed in the embodiment of the present invention;

[0025] Figure 5 This is a rendering of the building height estimation effect of the Vaihingen dataset disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0028] Reference Figure 1 As shown, the present invention discloses a method for extracting and estimating the building roof from high-resolution remote sensing images based on deep learning, comprising the following steps:

[0029] S1. Obtain self-made urban building dataset, WHU global city dataset, Inria dataset, Vaihingen dataset and Potsdam dataset, and preprocess the dataset;

[0030] S2. Construct the building roof extraction model MA-UNet++ and evaluation indicators, and use the self-made urban building dataset, WHU global city dataset and Inria dataset to verify and train the MA-UNet++ model, and obtain the optimal MA-UNet++ model;

[0031] S3, construct the building height estimation model SDLHNet and evaluation indicators, use the Vaihingen dataset and Potsdam dataset for verification and training, and obtain the optimal SDLHNet model;

[0032] S4. The optimal MA-UNet++ model and the optimal SDLHNet model are fused and analyzed based on the Potsdam dataset to determine the comprehensive performance and benefits.

[0033] Furthermore, the self-made urban building dataset in S1 is any urban building dataset.

[0034] Furthermore, in S1, in order to reduce the impact of various errors generated during the acquisition process of remote sensing images on the building roof extraction task, it is necessary to perform preprocessing operations such as geometric correction, radiation correction, filter smoothing, and band fusion on the remote sensing images before cropping and segmentation. For the self-made urban building dataset, the processed image data needs to be imported into the QGIS software to make labels, and manually combined with Google historical image data, Baidu 3D street view data, etc. to annotate and verify them one by one. At the same time, data enhancement can effectively increase the diversity of data and improve the robustness and reliability of the model in practical applications without actually increasing the cost of data collection by applying operations such as rotation and flipping to the data.

[0035] Further, see Figure 2 As shown, the MA-UNet++ model in S2 includes:

[0036] The multi-scale feature grouping convolution module is used to replace the original encoder module of UNet++, Cross EntropyLoss is used as the loss function of the UNet++ model, and spatial and channel attention mechanism modules are added to the model.

[0037] Specifically, the image first extracts features at the multi-scale feature extraction module, and then passes the features to the decoder using jump connections; the convolution block in the decoder uses the output of the previous layer and the jump connection from the encoder to generate a high-resolution feature map. In order to improve the model's focus on image features, a spatial and channel squeezing excitation module is added after each layer of the decoder; a deep supervision mechanism is also connected to the middle layer of the network to monitor and improve the overall performance of the model; finally, after multiple convolution blocks and upsampling processing, the feature map image is converted into the required number of channels for output.

[0038] Furthermore, the multi-scale feature grouping convolution module is based on the fusion of the "grouping convolution" idea in ResNeXt and the "layered connection" strategy in Res2Net, where the base width is set to 4 and the base is set to 8;

[0039] Cross Entropy Loss has the best comprehensive performance and is used to replace the original loss function to further improve the serious imbalance between background and building categories in the building roof extraction task, thereby accelerating the convergence of the model and improving the accuracy and efficiency of building roof extraction.

[0040] The spatial and channel attention mechanism modules adopt a parallel structure, allowing the model to simultaneously learn dependencies in both spatial and channel dimensions, enhancing the model's adaptability and generalization capabilities to different types of image features, especially when processing complex or diverse image data.

[0041] Furthermore, the evaluation indicators in S2 include: intersection over union, recall, precision and F1 score. The model training hyperparameters are set, and 42 rounds of training are performed on the WHU global city dataset and 32 rounds of training are performed on the Inria dataset.

[0042] Furthermore, when training based on the WHU global city dataset, the optimal model generated in the 32nd round was saved. The model performed well on this dataset in terms of accuracy indicators, with an IoU of 80.34%, indicating that the model has high accuracy in identifying buildings; the F1-Score is 88.84%, which is very close to its precision (88.72%) and recall (89.12%), further reflecting the accuracy and balance of the model;

[0043] When training the Inria dataset, the optimal model produced in the 22nd round was saved, in which the three indicators of Recall, Precision, and F1-Score all exceeded 90%, namely 93.49%, 94.10%, and 93.80% respectively, and the IoU was as high as 88.48%; compared with the WHU global city dataset, these indicators have been improved. By comparing the extraction results of the model on these two datasets with significant features, the overall performance of the model can be effectively tested.

[0044] Further, see Figure 3 As shown in Figure 1, the SDLHNet model constructed in S3 includes: using the Swin Transformer architecture as the feature extraction module of SDLHNet, adding a shift window mechanism to integrate information, using a dense atrous convolution pyramid module to organize each atrous convolution layer, and adding a local plane guidance module to merge and output the features generated by different decoders at different stages.

[0045] Specifically, the encoder part uses SwinTransformer as the feature extraction module of the model, and reduces the spatial resolution of the feature map through repeated strided convolution and spatial pooling layers, thereby compressing the image information and extracting important features; after the encoder, the DenseASPP layer is used to capture the contextual information of the image by adopting convolution kernels with different expansion rates; in the decoding stage, the network gradually increases the spatial resolution of the feature map through the decoder to restore the original size of the image and predict the height value of each pixel. In this process, multiple LPG layers are inserted to estimate the local height based on the local plane parameters, providing detailed multi-scale information for the final height estimation; after the last upsampling layer, fusion is performed through the final convolution layer to output a height map with the same spatial resolution as the input image.

[0046] Furthermore, the Swin Transformer architecture increases the receptive field of the model by gradually reducing the resolution of the feature map, thereby improving the model's ability to obtain image information. At the same time, it simulates the multi-scale feature extraction method in the convolutional network, and can better handle visual tasks that require fine-grained information, such as image height estimation.

[0047] The shift window mechanism is a mechanism introduced based on the self-attention mechanism, the core concept of the original Transformer model, to address the high computational cost of processing large-size images. By alternating the position of the window between consecutive layers, it effectively integrates local and global information and avoids the problem of quadratic growth of computational complexity with the increase of image size.

[0048] The dense atrous convolutional pyramid module is different from the existing spatial pyramid pooling (ASPP) structure. This module organizes the atrous convolutional layers in a densely connected form. The layers with large dilation rates are placed at the top, and the layers with small dilation rates are placed at the bottom. The output of each layer is connected to the input feature map and then transmitted to the subsequent layer. By sharing information between the dilated convolutional layers, the density of feature extraction is improved, with a larger receptive field and better scale diversity.

[0049] The local plane guidance module is used to improve the building height estimation accuracy of the model. This module is added to different stages of the decoder to merge and output the features generated by different decoders at different stages, so that the network can understand and predict height information at different scales, thereby achieving more accurate height estimation.

[0050] Furthermore, the evaluation indicators in S3 include: Rel, RMSE, RMSE(log), δ1, δ2 and δ3.

[0051] Specifically, we selected images containing a variety of land object categories, and further quantitatively compared the height estimation results of the model from both point and surface perspectives. During the comparison process, we randomly selected two points containing the edge of the building, and compared the model's height estimation value at this position with the true value. At the same time, we used the existing classification boundaries to divide the background and various land object elements, and calculated the average height value of all pixels within the building range for evaluation from a holistic perspective. From the analysis of the results, we can see that SDLHNet can estimate the building height in the sample more accurately, with the overall estimation error controlled within 1.5 meters and the error percentage less than 1%. Its excellent performance at the edge of the building further verifies the model's multi-scale feature acquisition and analysis capabilities.

[0052] Furthermore, in order to qualitatively judge the model's overall height estimation effect on the sample, as well as the degree of distinction between buildings and surrounding features, the Vaihingen dataset and the Potsdam dataset were specially selected for detailed comparative analysis with various models; the results show that SDLHNet can accurately estimate the heights of large and small buildings, and at the same time can well distinguish the height differences between the main building, walls, roads and vegetation.

[0053] Furthermore, the fusion analysis in S4 includes adding height attributes to each extracted building and combining it with the original image stretching height value to form a three-dimensional visual effect.

[0054] In a specific embodiment, self-made urban buildings were selected to construct an urban building dataset. The roof extraction task used four common accuracy indicators, and compared nine popular network models, five "U-shaped" network improvement models and five other network models. The height estimation task selected four classic network models and eight existing research methods, and compared and verified them on six accuracy indicators from both quantitative and qualitative perspectives.

[0055] The results obtained by using the self-made urban building dataset and Vaihingen dataset in this example are as follows Figure 4 and Figure 5 As shown in the figure, it can be seen that the building roof extracted by the method of the present invention can more comprehensively and finely express the structural information of the building, and can more accurately and balancedly estimate the height information of the building.

[0056] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0057] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for extracting and estimating building roofs from high-resolution remote sensing images based on deep learning, characterized in that: The following steps are involved: S1. Obtain self-made urban building dataset, WHU global city dataset, Inria dataset, Vaihingen dataset and Potsdam dataset, and preprocess the dataset; S2. Construct the building roof extraction model MA-UNet++ and evaluation indicators, and use the self-made urban building dataset, WHU global city dataset and Inria dataset to verify and train the MA-UNet++ model, and obtain the optimal MA-UNet++ model; S3, construct the building height estimation model SDLHNet and evaluation indicators, use the Vaihingen dataset and Potsdam dataset for verification and training, and obtain the optimal SDLHNet model; S4. The optimal MA-UNet++ model and the optimal SDLHNet model are fused and analyzed based on the Potsdam dataset to determine the comprehensive performance and benefits.

2. According to the method of claim 1, a high-resolution remote sensing image building roof extraction and height estimation method based on deep learning is characterized in that: The self-made urban building dataset in S1 is any urban building dataset.

3. The method for extracting and estimating building roofs from high-resolution remote sensing images based on deep learning according to claim 1, characterized in that: The MA-UNet++ model in S2 includes: The multi-scale feature grouping convolution module is used to replace the original encoder module of UNet++, Cross Entropy Loss is used as the loss function of the UNet++ model, and spatial and channel attention mechanism modules are added to the model.

4. The method for extracting and estimating building roofs from high-resolution remote sensing images based on deep learning according to claim 1, characterized in that: The evaluation indicators in S2 include: intersection over union, recall rate, precision and F1 score. The model training hyperparameters are set, and 42 rounds of training are performed on the WHU global city dataset and 32 rounds of training are performed on the Inria dataset.

5. The method for extracting and estimating building roofs from high-resolution remote sensing images based on deep learning according to claim 1, characterized in that: The SDLHNet model constructed in S3 includes: using the SwinTransformer architecture as the feature extraction module of SDLHNet, adding a shift window mechanism to integrate information, using a dense atrous convolution pyramid module to organize each atrous convolution layer, and adding a local plane guidance module to merge and output the features generated by different decoders at different stages.

6. The method for extracting and estimating building roofs from high-resolution remote sensing images based on deep learning according to claim 1, characterized in that: The evaluation indicators in S3 include: Rel, RMSE, RMSE(log), δ1, δ2 and δ3.

7. The method for extracting and estimating building roofs from high-resolution remote sensing images based on deep learning according to claim 1, characterized in that: The fusion analysis in S4 includes adding height attributes to each extracted building and combining it with the original image stretching height value to form a three-dimensional visual effect.

Images