A method for multi-scale building roof identification using GF-2 remote sensing imagery

Abstract

This invention discloses a method for multi-scale building roof identification using GF-2 remote sensing imagery. Based on the DeepLabv3+ network structure, this method adds a normalized attention mechanism (NAM) to the spatial pyramid pooling structure during the encoding stage to suppress insignificant features and better extract multi-scale features of building roofs. It replaces the traditional dilated convolution in the ASPP structure with depthwise separable convolutions with different dilation rates, combining the advantages of both to improve model efficiency. During feature fusion in the decoding stage, the features output from the bottom and middle layers of the ResNet101 backbone network are processed through a self-attention and convolution ensemble module to obtain more refined identification results. Using this invention to extract building roofs can address the problem of imprecise roof edge recognition and effectively optimize the identification accuracy of building roofs at different scales.

Description

Technical Field

[0001] This invention relates to spatiotemporal remote sensing, building roof image segmentation, multi-scale feature extraction of building roofs, and attention mechanism technology, specifically to an ADSN_DeepLabv3+ network based on multi-scale building roof extraction from GF-2 remote sensing images. Background Technology

[0002] Buildings are the primary physical spaces upon which people depend for survival, and rooftops are a crucial element of architecture. Information such as the number of buildings and the size of their roofs is essential for applications including land use, thematic mapping of buildings, demographics, and rooftop utilization. However, traditional building rooftop statistics rely on manual interpretation, consuming significant human resources. Automatic rooftop extraction based on GF-2 remote sensing imagery offers an efficient and low-cost solution for extracting building rooftop information. Multi-scale rooftop extraction using GF-2 remote sensing imagery greatly reduces human resource requirements, facilitates dynamic building monitoring, and achieves higher accuracy.

[0003] To address these issues, this patent proposes an ADSN_DeepLabv3+ network for multi-scale building roof extraction based on GF-2 remote sensing images, aiming to solve the problems of insufficient utilization of multi-scale information and imprecise edge recognition of building roofs. This method, based on the DeepLabv3+ network structure, adds a normalized attention mechanism (NAM) module to the Atrous Spatial Pyramid Pooling (ASPP) structure during the encoding stage to suppress insignificant features and better extract multi-scale features of building roofs. It replaces the traditional dilated convolution in the ASPP structure with depthwise separable convolutions with different dilation rates, combining the advantages of both depthwise separable convolutions and dilated convolutions to improve the model's efficiency. During feature fusion in the decoding stage, the features output from the bottom and middle layers of the ResNet101 backbone network are processed through a self-attention and convolution ensemble module (ACmix) to enhance the model's representational power, capture more features, and fuse them with the features output from the encoder, enriching the geometric details of the image for more refined recognition results. This invention can address the problem of imprecise identification of building roof edges, and also effectively optimizes the identification accuracy of roofs of buildings of different scales. Summary of the Invention

[0004] The technical problem solved by this invention is: to address the issues of insufficient utilization of multi-scale information of building roofs and imprecise edge recognition based on GF-2 remote sensing images, an ADSN_DeepLabv3+ network based on multi-scale building roof extraction from GF-2 remote sensing images is proposed to achieve multi-scale building roof extraction and ensure accurate recognition.

[0005] The technical solution of this invention is as follows: a method for multi-scale building roof identification using GF-2 remote sensing imagery. First, GF-2 remote sensing data is acquired, stored, preprocessed, and partitioned to form a building roof sample library. Based on this sample library, an encoder-decoder structure of the ADSN_DeepLabv3+ network is constructed. In the encoder, the backbone network is a ResNet101 network, and the output feature map is fed into the ASPP structure. The ASPP module is improved by constructing depthwise separable convolutions (DSDConv) with different dilation rates to replace the traditional dilated convolutions. This obtains multiple larger receptive fields while reducing the number of parameters and computational complexity, thus improving the model's efficiency. Attention mechanisms can help neural networks suppress less significant features in channels or space. Therefore, NAM is introduced into the ASPP structure. After each DSDConv convolution operation, a NAM structure is added to suppress insignificant features, better extracting multi-scale features and fusing them to ultimately obtain a high-level feature map with more detailed semantic information in the encoding region. In the decoding stage, an ACmix structure is embedded. The large-scale features output from the shallow and mid-layers of the ResNet 101 backbone network contain rich small-scale information. The ACmix structure learns from these large-scale output features and captures more features through convolution and self-attention mechanisms within the ACmix network. Finally, the output feature map is fused with the high-level feature map obtained from the encoder to obtain a more refined recognition result. Based on the constructed ADSN_DeepLabv3+ network, training is performed on the training set to optimize model parameters until the ADSN_DeepLabv3+ model converges. Based on the converged ADSN_DeepLabv3+ building roof extraction model, predictions are made on the test set, outputting the building roof extraction results. The specific steps are as follows:

[0006] (1) Acquire GF-2 remote sensing data, store and preprocess the data.

[0007] In step (1), the acquired GF-2 remote sensing data is reduced using a standard data structure.

[0008] Furthermore, efficient storage of GF-2 remote sensing data is achieved. This can be done using either relational databases (structured methods) or NoSQL databases (unstructured methods) for logical data storage and management; and can utilize centralized high-performance storage devices or distributed clusters for physical storage and optimization of building remote sensing image data.

[0009] Furthermore, the remote sensing image data of buildings undergoes preprocessing, specifically including the following steps:

[0010] (a) Writing spatial information. While ensuring accuracy, the spatial information of the study area is written into the high-resolution remote sensing image.

[0011] (b) Georegistration. Use ArcGIS tools to georegister the building roof vector map and high-resolution remote sensing image.

[0012] (c) Image cropping. The registered sample label data and the original remote sensing image are cropped into 500*500 pixel images.

[0013] (d) Image filtering. Compare the original data in (c) with the corresponding sample label data, and filter out images with no buildings or a large number of unlabeled buildings. At this point, the building rooftop photovoltaic sample database is completed.

[0014] (2) Perform roof identification on the building remote sensing image dataset with spatial information obtained in step (1), and construct a multi-scale building roof extraction model ADSN_DeepLabv3+. The specific steps include:

[0015] (a) Constructing the encoding structure of the ADSN_DeepLabv3+ network. The encoder is mainly used to extract multi-scale features of building roofs. ResNet101 is selected as the backbone network. Using the residual structure, through four layers of convolution, the feature map is reduced to 1 / 16 of its original size. The deep feature map passed through the backbone network is input into the improved ASPP structure to extract multi-scale features of building roofs. The improved ASPP structure includes a 1×1 convolution, three 3×3 depthwise separable dilated convolutions with dilation rates of (6, 12, 18) and a global pooling operation. A NAM module is introduced, and the feature map obtained from each operation is input into the NAM to suppress inconspicuous features. The feature map output after passing through the NAM is fused as the output of the improved ASPP structure, which better extracts the multi-scale features of the building. Finally, a 1×1 convolution is used to obtain the high-level feature map output after feature extraction by the encoder.

[0016] (b) Constructing the decoding structure of the ADSN_DeepLabv3+ network. The decoder's main role is to improve the segmentation results. The large-scale features output from the shallow and mid-level layers in the backbone network contain rich small-scale information. Therefore, an ACmix structure is embedded in the decoding stage to learn the large-scale features output from the shallow and mid-level layers, capturing more features through convolution and self-attention mechanisms within ACmix. The mid-level feature map obtained through the ACmix structure is fused with the high-level feature map output from the decoding stage after 4x upsampling. After 3×3 convolution and 2x upsampling, it is fused with the shallow feature map obtained through the ACmix structure, making better use of the deep and shallow features of the building roof. Finally, the recognition result of the building roof is obtained through 3×3 convolution and 2x upsampling. The ACmix structure and upsampling layer can better utilize the multi-scale features of the building roof for feature fusion, recover the spatial information of the building roof, and obtain more refined results.

[0017] (3) Based on the constructed ADSN_DeepLabv3+ building roof extraction network, train on the training set and optimize the model parameters until the ADSN_DeepLabv3+ network converges.

[0018] (a) Training of multi-scale building roof feature model. Initially, the model learning rate was set to 0.001, and 200 training sessions were performed on the training and validation sets. The model training loss value and the results of Acc, Precision, Recall, F1-Score were observed. The Acc and F1-Score values ​​were improved by parameter tuning.

[0019] (b) Multi-scale building roof feature model validation. The best model from the training rounds in process (a) is retained. This model is used to perform the first round of validation on the validation set to generate a prediction set. Images with lower prediction results are viewed. Data of the same category are added to the training set so that the model can learn that type of feature.

[0020] (c) Repeat steps (a) and (b) until the model accuracy reaches a usable level. Retain the optimal model. The building roof extraction model is now complete.

[0021] (4) Based on the converged ADSN_DeepLabv3+ building roof extraction network, the test set is predicted, and the building roof extraction result is obtained by using the output probability. The specific steps include: using the model to extract building roof features from the test set and storing the extraction results.

[0022] The advantages of this invention compared to the prior art are:

[0023] 1. In the encoding stage, depthwise separable convolutions (DSDConv) with different dilation rates are constructed to replace traditional dilated convolutions. This convolution combines the advantages of dilated convolutions and depthwise separable convolutions, obtaining multiple large receptive fields while reducing the number of parameters and computational complexity, thus improving the efficiency of the model.

[0024] 2. Introducing NAM into the ASPP structure, adding a NAM structure after each DSDConv convolution operation, suppresses insignificant features and extracts multi-scale features better.

[0025] 3. An ACmix structure is embedded in the decoding stage. The large-scale features output from the shallow and mid-layers of the ResNet 101 backbone network contain rich small-scale information. The ACmix structure learns from these large-scale features and captures more features through convolution and self-attention mechanisms within the ACmix architecture. Finally, the output feature map is fused with the high-level feature map obtained from the encoder to obtain a more refined recognition result. Attached Figure Description

[0026] Figure 1 This is the overall flowchart of the present invention.

[0027] Figure 2 This is a network structure diagram for ADSN_DeepLabv3+. Detailed Implementation

[0028] To enable those skilled in the art to better understand the embodiments of the present invention, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and implementation methods.

[0029] like Figure 1 , Figure 2 As shown, the present invention includes the following steps:

[0030] 1. Data Storage and Preprocessing: GF-2 remote sensing data can be stored using centralized high-performance computing facilities, utilizing high-performance servers, high-speed networks, and high-speed storage device clusters to physically store high-resolution remote sensing data. Alternatively, a distributed cluster approach can be used, utilizing a distributed computing environment and storage structure for the physical storage of high-resolution remote sensing data of buildings. Data preprocessing steps include: writing spatial information, georegistration, image cropping, and image filtering.

[0031] Spatial information is written into the high-resolution remote sensing imagery, ensuring accuracy.

[0032] Georeferencing. ArcGIS tools were used to georeferencing the building roof vector map with the high-resolution remote sensing imagery.

[0033] Image cropping. The registered sample label data and the original remote sensing image are cropped into 500*500 pixel images.

[0034] Image filtering. The raw data is compared with the corresponding sample label data to filter out images without buildings or with a large number of unlabeled buildings. At this point, the building rooftop photovoltaic sample database is completed.

[0035] 2. Perform roof identification on the building remote sensing image dataset and construct a multi-scale building roof extraction model ADSN_DeepLabv3+ network. The specific steps include: constructing a multi-scale building roof feature learning model, training the multi-scale building roof feature model, testing the multi-scale building roof feature model, optimizing the model, and storing the optimal model.

[0036] Constructing a building roof feature learning model: First, the encoding structure of the ADSN_DeepLabv3+ network is constructed. The encoder is mainly used to extract multi-scale features of the building roof. ResNet101 is chosen as the backbone network. Utilizing a residual structure, four convolutional layers reduce the feature map to 1 / 16 of its original size. The deep feature map from the backbone network is then input into an improved ASPP structure to extract multi-scale features of the building roof. The improved ASPP structure includes a 1×1 convolution, three 3×3 depthwise separable dilated convolutions with dilation rates of (6, 12, 18), and a global pooling operation. A NAM module is introduced, inputting the feature map obtained from each operation into the NAM to suppress inconspicuous features. The feature maps output from the NAM are then fused as the output of the improved ASPP structure, resulting in better extraction of multi-scale features of the building. Finally, a 1×1 convolution is used to obtain the high-level feature map output after feature extraction by the encoder. Constructing the decoding structure of the ADSN_DeepLabv3+ network: The decoder's main role is to improve the segmentation results. The large-scale features output from the shallow and mid-level layers of the backbone network contain rich small-scale information. Therefore, an ACmix structure is embedded in the decoding stage to learn the large-scale features from the shallow and mid-level outputs, capturing more features through convolution and self-attention mechanisms within ACmix. The mid-level feature map from the ACmix structure is fused with the high-level feature map output from the decoding stage after 4x upsampling. After 3×3 convolution and 2x upsampling, it is fused with the shallow feature map from the ACmix structure, making better use of the deep and shallow features of the building roof. Finally, the building roof recognition result is obtained after 3×3 convolution and 2x upsampling. The ACmix structure and upsampling layer can better utilize the multi-scale features of the building roof for feature fusion, recovering the spatial information of the building roof and obtaining more refined results.

[0037] Multi-scale building roof feature model training: Initially, the model learning rate was set to 0.01, and 200 training cycles were performed on the training and validation sets. The training results, such as ACC, precision, recall, and F1, were observed. While ensuring that the precision and recall values ​​were not too different, the precision value was improved by parameter tuning.

[0038] Multi-scale building roof feature model validation: The best model from the training rounds in process (a) is retained. The model is used to perform the first round of validation on the validation set to generate a prediction set. The images with lower prediction results are viewed. Data of the same category are added to the training set so that the model can learn the features of that category.

[0039] Model tuning: Repeat the process of model training and testing until the model accuracy reaches a usable level. Keep the optimal model, and the building extraction model construction is completed.

[0040] Optimal storage model: Using centralized high-performance computing facilities or distributed computing environments and storage structures, retain the model from the last round of optimization.

[0041] 3. Extract the roof features of all buildings from the remote sensing image dataset based on the multi-scale building roof feature model. The specific steps include: extracting roof features of all buildings and storing the prediction results.

[0042] Multi-scale building roof feature extraction: The entire dataset of remote sensing images of building roofs is placed into the test set. The optimal multi-scale building roof feature extraction model is used to extract the building roof features. The precision set of the building roof feature extraction results is the extracted building roof data.

[0043] Storage prediction results: The data extracted from the building roof will be physically stored using centralized high-performance computing facilities or distributed computing environments and storage structures.

[0044] It should be noted that the method of this embodiment is applicable to multi-scale building roof feature extraction.

[0045] The embodiments of the present invention have been described in detail above. Specific implementation methods have been used to illustrate the present invention. The description of the above embodiments is only for the purpose of helping to understand the method of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for multi-scale building roof identification using GF-2 remote sensing imagery, characterized in that, The implementation steps of this method are as follows: (1) Obtain GF-2 remote sensing data, store and preprocess GF-2 remote sensing data, obtain building roof data, mark building roofs, construct a building roof extraction sample database, and randomly split the dataset into training set and test set in an 8:2 ratio. (2) Construct the coding structure of the ADSN_DeepLabv3+ network for multi-scale building roof extraction based on GF-2 remote sensing images, and obtain the high-level feature map aux of multi-scale building roofs. The coding structure of the ADSN_DeepLabv3+ network for multi-scale building roof extraction includes a backbone network module, an improved ASPP module, and a feature output module. The backbone network module is a ResNet101 network, which utilizes a residual structure and includes four convolutional groups and one pooling layer to extract features from building roofs. The improved ASPP module includes a 1×1 convolution, three 3×3 depthwise separable dilated convolutions with dilation rates of (6, 12, 18) and a global pooling operation. A NAM module is introduced, and the feature map obtained from each operation is input into the NAM to suppress inconspicuous features. The feature map output after passing through the NAM is fused as the output of the improved ASPP structure, which can better extract multi-scale features of buildings. The feature output module includes a 1×1 convolution, which performs a 1×1 convolution operation on the feature map output by the improved ASPP module to obtain a high-level feature map output after feature extraction by the encoder. (3) Construct the decoding structure of the multi-scale building roof extraction network ADSN_DeepLabv3+ to obtain the final building roof recognition result; the decoding structure of the multi-scale building roof extraction network ADSN_DeepLabv3+ includes an upsampling module and a feature fusion module; The upsampling module includes performing a 4x upsampling operation on the high-level feature map generated in the decoding stage, and performing two 2x upsampling operations on the feature map after feature fusion, in order to restore the spatial detail information of the recognition result; The feature fusion module includes an embedded self-attention and convolutional ensemble module (ACmix) structure to learn large-scale features from the shallow and mid-layer outputs of the backbone network, and captures more features through convolution and self-attention mechanisms within ACmix. The mid-level feature map obtained through the ACmix structure is fused with the high-level feature map output from the decoding stage after 4x upsampling. After 3×3 convolution and 2x upsampling, it is fused with the shallow feature map obtained through the ACmix structure to better utilize the deep and shallow features of the building roof. Finally, the building roof recognition result is obtained after 3×3 convolution and 2x upsampling. (4) ADSN_DeepLabv3+ is constructed based on the decoding structure and the encoding structure. The encoding structure is mainly used for multi-scale feature extraction, and the decoding structure is used for feature fusion and recovery of spatial detail information to obtain more refined building roof extraction results. (5) Based on the constructed ADSN_DeepLabv3+ multi-scale building roof extraction model, train it on the training set, observe the loss value of the model training and the results such as Acc, Precision, Recall, F1-Score, etc., and adjust the learning rate and number of iterations by parameter tuning until the ADSN_DeepLabv3+ model converges. (6) Based on the converged ADSN_DeepLabv3+ multi-scale building roof extraction model, the test set is predicted, and the building roof extraction result is obtained by using the output probability.

2. The method for multi-scale building roof identification using GF-2 remote sensing imagery according to claim 1, characterized in that, In step (1), the storage of GF-2 remote sensing images can be carried out using either centralized high-performance storage devices or distributed cluster methods for physical storage and optimization of data; the processing of GF-2 remote sensing image data includes writing spatial information, georegistration, image segmentation, and image filtering; the division of GF-2 remote sensing image data adopts the publicly available division method of 8:2 ratio, splitting it into training set and test set.

3. The method for multi-scale building roof identification using GF-2 remote sensing imagery according to claim 1, characterized in that, In step (2), a multi-scale building roof extraction network ADSN_DeepLabv3+ is constructed using an encoding structure, including: (1) In the ASPP structure, construct depthwise separable convolutions (DSDConv) with different dilation rates to replace the dilated convolutions in ASPP. (2) The normalization-based attention mechanism module NAM is introduced into the ASPP structure, and the NAM structure is added after each DSDConv convolution operation.

4. The method for multi-scale building roof identification using GF-2 remote sensing imagery according to claim 1, characterized in that, In step (3), a feature fusion module is added to construct the decoding structure of the multi-scale building roof extraction network ADSN_DeepLabv3+, so as to extract detailed building roof recognition results.

5. The method for multi-scale building roof identification using GF-2 remote sensing imagery according to claim 1, characterized in that, In step (4), ADSN_DeepLabv3+ is constructed based on the decoding and encoding structures, and an improved encoding-decoding structure is used for multi-scale building roof extraction. The ADSN_DeepLabv3+ model is divided into two parts: an encoder and a decoder. The building roof feature extraction is used as the encoder, and the output high-level feature map is used as the input of the decoder. The middle and shallow feature maps of the backbone network are used as the input of feature fusion. The decoder is used to upsample and restore the spatial detail information of the image, and to fuse deep features with shallow features to obtain more refined building roof recognition results.

6. The method for multi-scale building roof identification using GF-2 remote sensing imagery according to claim 1, characterized in that, In step (5), the constructed ADSN_DeepLabv3+ building roof extraction model is trained on the training set, including: (1) Initialize the ADSN_DeepLabv3+ building extraction model parameters, fix the cropping size of the training set, the number of input images per batch, and the initial learning rate; (2) The multi-scale building roof recognition network is trained using the PyTorch framework. The loss value between the real label and the probability map is calculated based on the segmentation loss function, which is composed of binary cross-entropy loss and mean square loss. (3) The network parameters are continuously updated through backfeedback until the network converges.

7. The method for multi-scale building roof identification using GF-2 remote sensing imagery according to claim 1, characterized in that, In step (6), predictions are made on the test set based on the converged ADSN_DeepLabv3+ building roof extraction model, including: (1) Input the test set image and normalize it; (2) Input the normalized image into the trained model to obtain the building roof segmentation map of the test set.

Images