Building extraction method based on contour guidance and structure attention in fully convolutional network

Abstract

The present invention discloses a method for extracting buildings using a fully convolutional network based on contour guidance and structure attention. The method comprises: acquiring a remote sensing image; inputting the remote sensing image into a trained fully convolutional network based on contour guidance and structure attention, outputting a building intensity map through the fully convolutional network based on contour guidance and structure attention; and extracting a binary image of the building intensity map based on a threshold algorithm to obtain a building extraction result. The fully convolutional network based on contour guidance and structure attention in embodiments of the present invention can better focus on building edge information through contour guidance and structure attention, thereby improving the edge accuracy of building extraction and exhibiting good robustness and practicality.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing technology, and in particular to a building extraction method based on a full convolutional network with contour guidance and structure attention. Background Art

[0002] Buildings are one of the primary man-made objects on Earth. Automatically and accurately extracting buildings from remote sensing data is of great importance in cadastral mapping, urban monitoring, disaster management, and many other geospatial applications. Remote sensing technology enables users to efficiently and reproducibly collect data covering large areas. Furthermore, with advances in remote sensing technology, very high-resolution (VHR) remote sensing data has become available, improving the quality of extracted building boundaries. However, in practical applications, automatically extracting buildings from remote sensing data remains challenging. Buildings vary in shape, size, height, location, and material, resulting in large intra-class variability and low inter-class variability. Therefore, developing automatic and robust algorithms for extracting buildings from VHR remote sensing data is an important and meaningful task in the remote sensing community. The key to distinguishing buildings from non-buildings is to extract effective features. Therefore, existing building extraction algorithms can be broadly categorized into methods based on handcrafted features and those based on deep learning (DL). However, building extraction methods based on handcrafted features typically have limited generalization capabilities, while DL-based methods ignore the rich spatial information in building shape priors or encoder layers.

[0003] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0004] The technical problem to be solved by the present invention is that, in response to the above-mentioned defects of the prior art, a fully convolutional network building extraction method based on contour guidance and structure attention is provided, aiming to solve the problem that the building extraction algorithms in the prior art based on artificially designed features usually have limited generalization capabilities, and the methods based on deep learning ignore the building shape prior or the rich spatial information in the encoder layer.

[0005] The technical solutions adopted by the present invention to solve the problem are as follows:

[0006] In a first aspect, an embodiment of the present invention provides a method for extracting buildings using a fully convolutional network based on contour guidance and structure attention, wherein the method comprises:

[0007] Acquisition of remote sensing images;

[0008] Inputting the remote sensing image into a trained contour-guided and structure-attention-based fully convolutional network, and outputting a building intensity map through the contour-guided and structure-attention-based fully convolutional network;

[0009] Based on a threshold algorithm, a binary image of the building intensity map is extracted to obtain a building extraction result.

[0010] In one implementation, the fully convolutional network based on contour guidance and structure attention includes an encoder, a contour guidance module and a decoder, wherein the encoder is obtained by improving the CNN classification model; the contour guidance module includes several residual modules; and the decoder is a symmetric module constructed based on the encoder.

[0011] In one implementation, inputting the remote sensing image into a trained contour-guided and structure-attention-based fully convolutional network, and outputting a building intensity map through the contour-guided and structure-attention-based fully convolutional network includes:

[0012] Inputting the remote sensing image into the encoder, and outputting a multi-scale feature image through the encoder;

[0013] Inputting the multi-scale feature image into the contour guidance module, and outputting the contour feature image through the contour guidance module;

[0014] The contour feature image and the multi-scale feature image are input into the decoder, and the decoder outputs a building intensity map.

[0015] In one implementation, the training process of the contour-guided and structure-attention-based full convolutional network is as follows:

[0016] Acquire a training sample set, wherein the training sample set includes a plurality of training samples, and the training samples include training image data and at least one true value image label corresponding to the training image data;

[0017] Inputting the training image data into a preset initial network model, and outputting a predicted building intensity image through the initial network model;

[0018] Obtaining a total loss function based on the predicted building intensity image and the true image label;

[0019] Based on the total loss function, the initial network model is trained to obtain a fully convolutional network based on contour guidance and structure attention.

[0020] In one implementation, obtaining a total loss function based on the predicted building intensity image and the true image label includes:

[0021] Obtaining a weight matrix through the predicted building intensity image and the true value image label;

[0022] Based on the weight matrix, a weighted binary cross entropy loss function is constructed; wherein the weighted binary cross entropy loss function is used to reflect the loss function of the building boundary;

[0023] Based on the weight matrix, a weighted intersection-over-union loss function is constructed; wherein the weighted intersection-over-union loss function is used to reflect the loss function of the overlap between buildings;

[0024] Obtaining a structural similarity loss function according to the predicted building intensity image and the true value image label, wherein the structural similarity loss function is used to characterize the similarity of building structures;

[0025] The weighted binary cross entropy loss function, the weighted intersection-over-union loss function, and the structural similarity loss function are added to obtain several regional loss functions and edge loss functions, and the several regional loss functions and the edge loss functions are weightedly summed to obtain a total loss function.

[0026] In one implementation, obtaining a weight matrix according to the predicted building intensity image and the true image label includes:

[0027] Acquire a neighborhood of each pixel of the predicted building intensity image based on a neighborhood perception technology;

[0028] A weight matrix is ​​obtained according to the neighborhood of each pixel and the true value of each pixel in the true value image label.

[0029] In one implementation, obtaining a structural similarity loss function based on the predicted building intensity image and the true image label includes:

[0030] Obtaining a mask image corresponding to the predicted building intensity image;

[0031] Cutting out pixel blocks of the same size from corresponding positions in the mask image and the ground truth image label respectively; wherein the pixel blocks are composed of pixel points surrounded by a preset length and a preset width;

[0032] A structural similarity loss function is obtained based on pixel values ​​in the pixel block.

[0033] In one implementation, extracting the binary image of the building intensity map based on a threshold algorithm to obtain a building extraction result includes:

[0034] Based on a threshold algorithm, the binary image of the building intensity map is classified to obtain a building extraction result.

[0035] In a second aspect, an embodiment of the present invention further provides a fully convolutional network building extraction device based on contour guidance and structure attention, wherein the device comprises:

[0036] Remote sensing image acquisition module, used for acquiring remote sensing images;

[0037] a building intensity map acquisition module, configured to input the remote sensing image into a trained contour-guided and structure-attention-based fully convolutional network, and output a building intensity map through the contour-guided and structure-attention-based fully convolutional network;

[0038] The building extraction result obtaining module is used to extract the binary image of the building intensity map based on a threshold algorithm to obtain a building extraction result.

[0039] In a third aspect, an embodiment of the present invention further provides an intelligent terminal comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, wherein the one or more programs include a method for executing a fully convolutional network building extraction method based on contour guidance and structure attention as described in any one of the above.

[0040] In a fourth aspect, an embodiment of the present invention also provides a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the contour-guided and structure-attention-based fully convolutional network building extraction method as described in any one of the above.

[0041] Beneficial effects of the present invention: The embodiment of the present invention first acquires a remote sensing image; then inputs the remote sensing image into a trained full convolutional network based on contour guidance and structure attention, and outputs a building intensity map through the full convolutional network based on contour guidance and structure attention; finally, based on a threshold algorithm, a binary image of the building intensity map is extracted to obtain a building extraction result; it can be seen that the full convolutional network based on contour guidance and structure attention in the embodiment of the present invention can better focus on the edge information of the building through contour guidance and structure attention, thereby improving the edge accuracy of building extraction, and has good robustness and usability. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 A schematic flow chart of the method for extracting buildings using a fully convolutional network based on contour guidance and structure attention provided in an embodiment of the present invention.

[0044] Figure 2 Schematic diagram of the overall framework of the building extraction method based on the contour-guided and local structure-aware encoder-decoder network (CGSABENet) provided in an embodiment of the present invention.

[0045] Figure 3 This is a schematic diagram of the specific structure of the contour guidance module in the remote sensing image building extraction method provided by an embodiment of the present invention.

[0046] Figure 4 Provided for the embodiment of the present invention is a specific structural diagram of a symmetric encoder and deep supervision in the remote sensing image building extraction method provided by the embodiment of the present invention.

[0047] Figure 5 A schematic diagram of remote sensing images in training samples provided by an embodiment of the present invention.

[0048] Figure 6 A schematic diagram of the true value of the building area in the training sample provided by an embodiment of the present invention.

[0049] Figure 7 A schematic diagram of the true value of building outlines in training samples provided by an embodiment of the present invention.

[0050] Figure 8 Schematic diagram of qualitative comparison of the results of the building extraction method provided by an embodiment of the present invention and other comparative methods on dataset A.

[0051] Figure 9 Schematic diagram of qualitative comparison of the results of the building extraction method provided by an embodiment of the present invention and other comparative methods in dataset B.

[0052] Figure 10 This is an experimental flow chart of the building extraction method provided in an embodiment of the present invention.

[0053] Figure 11 This is a block diagram of the principle of a fully convolutional network building extraction device based on contour guidance and structure attention provided by an embodiment of the present invention.

[0054] Figure 12 This is a block diagram of the internal structure of the smart terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] This invention discloses a method for extracting buildings using a fully convolutional network based on contour guidance and structure attention. To clarify the objectives, technical solutions, and effects of the invention, the invention is further described below with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are intended only to illustrate the invention and are not intended to limit the invention.

[0056] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the description of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0057] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0058] In the prior art, existing building extraction algorithms can be roughly divided into methods based on human-designed features and methods based on deep learning (DL). Building extraction methods based on human-designed features mainly rely on low- / mid-level features designed by experts based on implicit or inherent characteristics of buildings, such as geometric features (e.g., corners, edges, lines, and contours), spatial spectral features (e.g., Morphological Building Index (MBI)), contextual features (e.g., shadows), shape features, and structural features or object-level features. These methods can extract buildings to a certain extent in specific tasks, but they are unable to extract high-level semantic information, resulting in poor performance in complex scenes. The background in VHR remote sensing images is particularly complex. Therefore, building extraction methods based on human-designed features usually have limited generalization capabilities. Recent deep learning-based methods have demonstrated strong feature learning and classification performance, and have promoted a new round of research on automatic building extraction.

[0059] Deep learning-based building extraction methods automatically learn discriminative features from training data without relying on manual feature design. This approach benefits from the development of convolutional neural networks (CNNs). The fully convolutional network (FCN) is the most popular CNN architecture for building extraction. FCNs extend the original CNN architecture by replacing fully connected layers with fully convolutional layers to achieve efficient pixel-to-pixel dense prediction. Despite their superior performance, traditional FCNs have two inherent limitations. First, repeated downsampling and coarse upsampling layers lose spatial detail, resulting in low boundary localization accuracy. Second, the receptive field of FCNs grows linearly with increasing network depth. This slow growth still fails to capture global context, leading to misclassification of objects at multiple scales. Numerous studies have addressed these two issues, including the U-Net and the DeepLab series, which utilizes dilated convolutions to increase the receptive field. The U-Net employs an encoder-decoder architecture that concatenates low-level features to high-level features via skip connections, helping to recover spatial information lost due to downsampling. The DeepLab series reduces downsampling by dilated convolution and introduces the Atrous Spatial Pyramid Pooling (ASPP) module to fuse multi-scale contextual information. These methods have alleviated these two problems to a certain extent and have become typical and widely used FCN architectures. However, there are still problems when applying these classic semantic segmentation methods to extract buildings from VHR remote sensing images. On the one hand, compared with objects in natural images, buildings in VHR remote sensing images have diverse appearances, complex peripheries, and greater scale differences. On the other hand, buildings are typical man-made objects with rich morphological features. The loss of spatial detail information leads to inaccurate building edges, which limits their potential for practical applications. Therefore, effectively extracting features to obtain accurate building boundaries while preserving the spatial details of VHR remote sensing data is a research frontier in the remote sensing community.

[0060] Many methods have been proposed to improve the quality of extracted building boundaries. The most straightforward approach is to add post-processing steps, such as probabilistic graphical models and empirical rules. While post-processing can improve extraction results, these methods are typically multi-stage and complex. Some studies have used semantic edge detection networks to improve the quality of extracted boundaries. While these methods achieve high-quality results, buildings have highly structured shapes and boundaries, rather than full object edges. Consequently, these approaches increase the model's computational burden without achieving optimal results. Highly structured building shape priors can be encoded into the model through building silhouette learning. However, silhouette learning suffers from a severe sample imbalance issue, as silhouette pixels only account for a small fraction of all sample pixels. Therefore, silhouette learning must overcome this issue while preserving the structural characteristics of buildings. However, the most commonly used binary cross-entropy loss function focuses solely on pixel-level similarity, resulting in a loss of structural information and sensitivity to sample imbalance. Inspired by these two observations, researchers have proposed a method that combines a multi-task learning framework with a total loss function to simultaneously learn building regions and silhouettes to improve building extraction results. However, existing methods only supervise building edges in the last decoder layer, which still suffers from the loss of detailed spatial information in the encoder. In reality, the encoder layer has finer details about the original image. In summary, these methods improve the quality of extracted building boundaries to some extent, but still ignore the building shape prior or the rich spatial information in the encoder layer.

[0061] To address the challenges of the prior art, this embodiment provides a building extraction method using a contour-guided and structure-attention-based fully convolutional network. This method utilizes contour-guided and structure-attention-based fully convolutional networks to better focus on building edge information, thereby improving the edge accuracy of building extraction and providing robustness and usability. In a specific implementation, a remote sensing image is first acquired; this image is then input into a trained contour-guided and structure-attention-based fully convolutional network, which then outputs a building intensity map. Finally, a thresholding algorithm is used to extract a binary image of the building intensity map, yielding a building extraction result.

[0062] Exemplary Methods

[0063] This embodiment provides a fully convolutional network building extraction method based on contour guidance and structure attention, which can be applied to smart terminals for remote sensing. Figure 1-2 As shown, the method includes:

[0064] Step S100: acquiring a remote sensing image;

[0065] Specifically, remote sensing images can be downloaded from remote sensing satellite data publicly available in domestic and foreign databases, such as NASA, ESA, etc. Remote sensing images are used to prepare for obtaining building intensity maps later.

[0066] After obtaining the remote sensing image data, you can perform the following operations: Figure 1-2 The following steps are shown: S200, inputting the remote sensing image into a trained contour-guided and structure-attention based full convolutional network, and outputting a building intensity map through the contour-guided and structure-attention based full convolutional network;

[0067] In this embodiment, the full convolutional network based on contour guidance and structural attention includes an encoder, a contour guidance module and a decoder, wherein the encoder is obtained by improving the CNN classification model, and the encoder can learn the multi-scale features of different remote sensing images for generating subsequent edge attention features and building extraction processes; in one implementation, the CNN classification model is a ResNet model, as shown in Table 1, and the encoder is obtained by improving the CNN classification model, specifically by replacing the previous downsampling convolution and downsampling pooling in the ResNet model with ordinary convolution blocks, so that the encoder only needs to downsample the input image three times, thereby improving the spatial information retention capability of the model. The contour guidance module includes several residual modules; such as Figure 3 As shown, the contour guidance module is used to optimize the contour or edge expression capability of the encoder feature, and the feature image can generate multiple images of different scales, the same depth, and representing different features through the contour guidance module. Figure 4 As shown, the decoder is a symmetric module constructed based on the encoder.

[0068] Table 1 Encoder (Improved ResNet model)

[0069] name enter Output operate nuclear filling step length Input H*W*3 H*W*64 Conv+BN+ReLU 3 1 1 E0 H*W*64 H*W*64 ResNetBasicBlock*3 3 1 1 E1 H*W*64 H / 2*W / 2*64 ResNetBasicBlock*4 3 1 2 E2 H / 2*W / 2*64 H / 4*W / 4*128 ResNetBasicBlock*6 3 1 2 E3 H / 4*W / 4*128 H / 8*W / 8*256 ResNetBasicBlock*3 3 1 2

[0070] In order to obtain a building strength map, step S200 includes the following steps:

[0071] S201, inputting the remote sensing image into the encoder, and outputting a multi-scale feature image through the encoder;

[0072] S202, inputting the multi-scale feature image into the contour guidance module, and outputting the contour feature image through the contour guidance module;

[0073] S203: Input the contour feature image and the multi-scale feature image into the decoder, and output a building intensity map through the decoder.

[0074] Specifically, after inputting the remote sensing image into the encoder, a multi-scale feature image is generated. However, the multi-scale image currently lacks the ability to express contours or edges. Therefore, the multi-scale feature image is input into the contour guidance module. The output contour feature image can generate multiple contour feature images of different scales, the same depth, and representing different features, which can overcome the loss of low-level spatial information and the neglect of prior building shapes. The contour feature image and the multi-scale feature image are then input into the decoder, which outputs a building intensity map. The intensity map has the same size as the input image and a value range of [0, 1]. The larger the value, the higher the probability of a building. Generally, the image is polarized, with pixels belonging to buildings being much larger than 0.5, such as around 0.9, and pixels not belonging to buildings being much smaller than 0.5, such as around 0.1.

[0075] In one implementation, the training process of the contour-guided and structure-attention-based fully convolutional network is as follows: obtaining a training sample set, wherein the training sample set includes a number of training samples, and the training samples include training image data and at least one true image label corresponding to the training image data; inputting the training image data into a preset initial network model, and outputting a predicted building intensity image through the initial network model; obtaining a total loss function based on the predicted building intensity image and the true image label; and training the initial network model based on the total loss function to obtain a contour-guided and structure-attention-based fully convolutional network.

[0076] In this embodiment, a training dataset C is obtained and divided into a training sample set and a validation sample set according to a certain ratio. By default, 80% of the data is used as the training sample set, and 20% of the data is used as the validation sample set. The training sample set includes a number of training samples, each of which includes training image data and at least one true image label corresponding to the training image data. In this embodiment, the true image label can be a true image of a building, a true image of a building outline, or a true image of a building and a true image of a building outline. In practice, based on the true image of the building, an edge extraction algorithm can be used to automatically generate a true image of the building outline. If a true image of the building outline is already available, a morphological filling algorithm can also be used to automatically generate a true image of the building, i.e., a true image of the building area. Figure 5 represents remote sensing images, Figure 6 represents the corresponding building area ground truth image, Figure 7Represents the corresponding ground-truth image of the building outline. Remote sensing images consist of three bands: R, G, and B. The ground-truth image of the building area is a binary image, with 255 representing a building and 0 representing a non-building. The ground-truth image of the building outline is a binary image, with 255 representing the edge of a building and 0 representing the edge of a non-building. Training dataset C is unaugmented. Each training dataset consists of a 512×512 pixel three-band remote sensing image and the corresponding 512×512 pixel ground-truth image of the building area and the ground-truth image of the building outline. Then, the training image data is input into a preset initial network model, and a predicted building intensity image is outputted through the initial network model; then, a total loss function is obtained according to the predicted building intensity image and the true value image label; correspondingly, the total loss function obtained according to the predicted building intensity image and the true value image label includes the following steps: obtaining a weight matrix through the predicted building intensity image and the true value image label; constructing a weighted binary cross entropy loss function based on the weight matrix; wherein, the weighted binary cross entropy loss function is used to reflect the loss function of the building boundary; based on the A weight matrix is ​​used to construct a weighted intersection-over-union loss function; wherein the weighted intersection-over-union loss function is used to reflect the loss function of the overlap between buildings; a structural similarity loss function is obtained according to the predicted building intensity image and the true value image label, wherein the structural similarity loss function is used to characterize the similarity of building structures; the weighted binary cross entropy loss function, the weighted intersection-over-union loss function and the structural similarity loss function are added together to obtain several regional loss functions and edge loss functions, and the several regional loss functions and the edge loss functions are weightedly summed to obtain a total loss function.

[0077] Specifically, a weight matrix is ​​obtained by using the predicted building intensity image and the true value image label. The formula of the weight matrix W is:

[0078]

[0079] Where abs represents the absolute value operation; N rc represents the neighborhood of the pixel point (r,c) of the predicted building intensity image; g ij Represents the true image label at pixel (i, j), g rc Represents the true value image label at the pixel point (r, c). From the above formula, we can see that for any pixel point, w rc The range of w is [0,1]. rc The larger the value, the greater the difference between the pixel point (r, c) and its neighborhood. Then, based on the weight matrix, a weighted binary cross entropy loss function is constructed; wherein, the weighted binary cross entropy loss function is used to reflect the loss function of the building boundary; the weighted binary cross entropy loss function LwBCE It is constructed by the following formula:

[0080]

[0081] Where r, c represent the rows and columns of the image; H, W represent the height and width of the image; γ represents the hyperparameter for adjusting the proportion of difficult-to-classify pixels (i.e., pixels at the classification boundary); w rc represents the weight assigned to each pixel; I(.) is the indicator function; l∈{0,1} represents non-buildings and buildings; g r,c and p r,c Represents the true image label and predicted building intensity image at pixel point (r,c). Prob(p r,c = = l|ω) represents the predicted probability. Then, based on the weight matrix, a weighted intersection-over-union loss function is constructed; wherein the weighted intersection-over-union loss function is used to reflect the loss function of the overlap between buildings; the weighted intersection-over-union loss function is constructed by the following formula:

[0082]

[0083] Where r, c represent the rows and columns of the image; H, W represent the height and width of the image; γ represents the hyperparameter for adjusting the proportion of difficult-to-classify pixels (i.e., pixels at the classification boundary); w rc represents the weight assigned to each pixel; g r,c and p r,c Represents the true image label and predicted building intensity image at pixel point (r,c).

[0084] After obtaining the weighted intersection-over-union loss function, a structural similarity loss function is obtained according to the predicted building intensity image and the true value image label, wherein the structural similarity loss function is used to characterize the similarity of the building structure; accordingly, obtaining the structural similarity loss function according to the predicted building intensity image and the true value image label includes the following steps: obtaining a mask image corresponding to the predicted building intensity image; respectively intercepting pixel blocks of the same size from corresponding positions in the mask image and the true value image label; wherein the pixel block is composed of pixel points surrounded by a preset length and a preset width; and obtaining the structural similarity loss function based on the pixel values ​​in the pixel block.

[0085] Specifically, first obtain the mask image corresponding to the predicted building intensity image; then intercept pixel blocks of the same size from the corresponding positions in the mask image and the true value image label; wherein the pixel block is composed of pixel points surrounded by a preset length and a preset width; if p = {p i ,i=1,…,N 2} and g={g i,i=1,…,N 2} is the pixel value of two corresponding blocks intercepted from the mask image and the true value image label corresponding to the predicted building intensity image. Finally, the structural similarity loss function is obtained based on the pixel values ​​in the pixel blocks. The formula of the structural similarity loss function is as follows:

[0086]

[0087] Where, σ p ,σ g and μ p ,μ g Represents the standard deviation and mean of p and g respectively; σ pg is the covariance between them; C1=0.01 2 and C2 = 0.03 2 are two constants to prevent division by zero. SSIM The average value of represents the total structural similarity loss function (SSIM loss) of the entire predicted building intensity image.

[0088] After obtaining the weighted binary cross entropy loss function, the weighted intersection-over-union loss function and the structural similarity loss function, the above loss functions are added together to obtain several regional loss functions. Each regional loss function L i The expression is as follows:

[0089] L i =L wBCE +L SSIM +L wIoU

[0090] Marginal loss function The expression is as follows:

[0091]

[0092] Then, the weighted sum of the regional loss functions and the edge loss functions is performed to obtain the total loss function. total The expression is:

[0093]

[0094] Where w i is the weight of the multi-scale region output. Represents the regional loss function of the four region outputs, Indicates that there is only one edge loss function. The total loss function in the embodiment of the present invention hybridly calculates the regional loss function and the edge loss function of the four regional outputs, guiding the contour-guided and structure-attention-based full convolutional network to learn parameters from pixel-level similarity, local structural similarity, and global similarity, so that the contour-guided and structure-attention-based full convolutional network can achieve better building extraction results. After obtaining the total loss function, the initial network model can be trained based on the total loss function to obtain a contour-guided and structure-attention-based full convolutional network. In this embodiment, the initial network is trained using the total loss function and the Adam optimizer. When the total loss function value no longer decreases or the validation set accuracy no longer increases, the training is stopped. In other words, the weights of the initial network are learned using the training data set C. By learning the prior knowledge of various types of buildings in the training samples, the initial network can effectively select and combine deep features to generate building intensity images. When the initial network can correctly distinguish various building types, it means that the initial network has learned the building conditions in all training samples. At this time, the loss function reaches the minimum value or the accuracy in the validation set reaches the maximum value, that is, the training process is stopped, thus obtaining a contour-guided and structure-attention-based full convolutional network.

[0095] After obtaining the building strength map, you can execute Figure 1-2 The following steps are shown: S300, based on a threshold algorithm, extracting a binary image of the building intensity map to obtain a building extraction result.

[0096] Specifically, the value range of the binary image in the building intensity map is [0, 1]. The classification threshold is set to 0.5, and the value range greater than the classification threshold corresponds to buildings, and the value range less than the classification threshold corresponds to non-buildings, thereby obtaining the building extraction result: building or non-building.

[0097] pass Figure 8 and Figure 9 From the building extraction results on different data sets, it can be found that the embodiment of the present invention obtains the building extraction result that is closest to the actual situation.

[0098] In order to quantitatively analyze the final building results of the embodiment of the present invention, four indicators, namely Recall, Precision, Intersection over Union (IoU), and Edge F1 Score (BF-Score), are selected as the result evaluation indicators of building extraction. Assume that TP represents the number of pixels detected as buildings when the true value is buildings; FP represents the number of pixels detected as buildings when the true value is non-buildings, that is, the number of false positive pixels; FN represents the number of pixels detected as non-buildings when the true value is buildings, that is, the number of missed pixels. Then:

[0099] Recall = TP / (TP+FN) × 100%

[0100] Precision=TP / (TP+FP)×100%

[0101] IoU = TP / (TP+FP+FN) × 100%

[0102] F1=2*Recall*Precision / (Recall+Precision)×100%

[0103] The edge F1 score BF-Score calculates the F1 score within the specified buffer of the object boundary area to measure whether the points on the predicted boundary match the true boundary.

[0104] Table 2 further presents quantitative analysis metrics for the four building extraction results. The experimental data used are the publicly available WHU aerial building dataset and the INRIA dataset, denoted as training dataset A and training dataset B, respectively. Training dataset A contains 18,700 building objects and is divided into three parts: a training set (4736 images, 130,500 buildings), a validation set (1036 images, 14,500 buildings), and a test set (2416 images, 42,000 buildings). Each image is 512x512 pixels in size and has a resolution of 0.3 meters. The publicly available ground truth portion of training dataset B includes 180 images covering five cities, with 36 images per city. Each image has a spatial resolution of 0.3 meters and a size of 5000x5000 pixels. This dataset covers a wide range of representative terrain, landforms, and building types. The annotation quality of training dataset B is lower than that of training dataset A. The first five large images of each city are set as the test set, and the remaining images are used for training and validation. For ease of processing, each large image is divided into 512*512 pixel tiles. A total of 12555 tiles are generated for training and validation, and 2025 tiles are generated for testing. Each dataset includes remote sensing images and corresponding building ground truth. Figure 5-7 ,in, Figure 5 Represents remote sensing imagery; Figure 6 Represents the ground truth image of the building area; Figure 7 is the ground truth image of the building outline. To ensure fairness, the training strategies of all algorithms are consistent. The experimental results of the method U-Net correspond to Figure 8 (b); Experimental results of DeepLabv3+ method correspond to Figure 8 (c); Method MC-FCN represents the multi-constrained full convolution building contour extraction method, and the experimental results correspond to Figure 8(d); Method BR-Net represents the boundary-adjusted accurate building roof segmentation and contour extraction method, and the experimental results correspond to Figure 8 (e); Method Proposed represents the building extraction method CGSABENet based on contour guidance and structure attention proposed by this invention. The experimental results correspond to Figure 8 (f). Training data set B see Figure 9 , method sorting and Figure 8 In this embodiment, the training and application framework is as follows Figure 10 shown.

[0105] As shown in Table 2, the contour-guided and structure-attention-based fully convolutional network building extraction method CGSABENet, provided by an embodiment of the present invention, significantly outperforms other methods in building extraction results. Experiments show that the CGSABENet method consistently outperforms the comparison methods in building extraction results on two training datasets, A and B, in terms of Recall, IoU, and BF-Score. Therefore, the contour-guided and structure-attention-based encoder-decoder convolutional neural network building extraction method provided by an embodiment of the present invention can achieve superior building results, especially significant improvements in building edges, and has strong generalization and robustness.

[0106] Table 2 Quantitative analysis of building extraction results

[0107]

[0108]

[0109] Exemplary devices

[0110] like Figure 11 As shown in , an embodiment of the present invention provides a building extraction device based on a full convolutional network with contour guidance and structure attention, the device comprising a remote sensing image acquisition module 401, a building intensity map acquisition module 402, and a building extraction result acquisition module 403: the remote sensing image acquisition module 401 is used to acquire a remote sensing image;

[0111] A building intensity map acquisition module 402 is configured to input the remote sensing image into a trained contour-guided and structure-attention based fully convolutional network, and output a building intensity map through the contour-guided and structure-attention based fully convolutional network;

[0112] The building extraction result obtaining module 403 is used to extract the binary image of the building intensity map based on a threshold algorithm to obtain a building extraction result.

[0113] Based on the above embodiment, the present invention also provides an intelligent terminal, whose principle block diagram can be shown as follows: Figure 12As shown. The intelligent terminal includes a processor, a memory, a network interface, a display screen, and a temperature sensor connected through a system bus. Among them, the processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the intelligent terminal is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a fully convolutional network building extraction method based on contour guidance and structural attention is implemented. The display screen of the intelligent terminal can be a liquid crystal display screen or an electronic ink display screen, and the temperature sensor of the intelligent terminal is pre-set inside the intelligent terminal to detect the operating temperature of the internal equipment.

[0114] Those skilled in the art will understand that Figure 12 The schematic diagram is only a block diagram of a portion of the structure related to the solution of the present invention and does not constitute a limitation on the smart terminal to which the solution of the present invention is applied. The specific smart terminal may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0115] In one embodiment, a smart terminal is provided, comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors. The one or more programs include instructions for performing the following operations:

[0116] Acquisition of remote sensing images;

[0117] Inputting the remote sensing image into a trained contour-guided and structure-attention-based fully convolutional network, and outputting a building intensity map through the contour-guided and structure-attention-based fully convolutional network;

[0118] Based on a threshold algorithm, a binary image of the building intensity map is extracted to obtain a building extraction result.

[0119] Those skilled in the art will appreciate that all or part of the processes in the above-described embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described embodiments. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0120] In summary, the present invention discloses a method for extracting buildings using a fully convolutional network based on contour guidance and structure attention. The method comprises: acquiring a remote sensing image; inputting the remote sensing image into a trained fully convolutional network based on contour guidance and structure attention, outputting a building intensity map through the fully convolutional network based on contour guidance and structure attention; and extracting a binary image of the building intensity map based on a threshold algorithm to obtain a building extraction result. The fully convolutional network based on contour guidance and structure attention of an embodiment of the present invention can better focus on the edge information of buildings through contour guidance and structure attention, thereby improving the edge accuracy of building extraction and having good robustness and usability.

[0121] Based on the above embodiments, the present invention discloses a building extraction method based on a fully convolutional network with contour guidance and structure attention. It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, it can be improved or transformed according to the above description. All these improvements and transformations should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A fully convolutional network building extraction method based on contour guidance and structure attention, characterized in that: The method comprises: Acquisition of remote sensing images; Inputting the remote sensing image into a trained contour-guided and structure-attention-based fully convolutional network, and outputting a building intensity map through the contour-guided and structure-attention-based fully convolutional network; Extracting a binary image of the building intensity map based on a threshold algorithm to obtain a building extraction result; The contour-guided and structure-attention-based fully convolutional network includes an encoder, a contour-guided module, and a decoder, wherein the encoder downsamples the input image three times, the contour-guided module includes several residual modules, and the decoder is a symmetric module constructed based on the encoder; The training process of the fully convolutional network includes: The weight matrix is ​​obtained by predicting the building intensity image and the true value image label. The formula of the weight matrix is: Where abs represents the absolute value operation; N rc represents the neighborhood of the pixel point (r,c) of the predicted building intensity image; g ij Represents the true image label at pixel (i, j), g rc Represents the true image label at the pixel point (r, c); Based on the weight matrix, a weighted binary cross entropy loss function is constructed; wherein the weighted binary cross entropy loss function is used to reflect the loss function of the building boundary; Based on the weight matrix, a weighted intersection-over-union loss function is constructed; wherein the weighted intersection-over-union loss function is used to reflect the loss function of the overlap between buildings; Obtaining a structural similarity loss function according to the predicted building intensity image and the true value image label, wherein the structural similarity loss function is used to characterize the similarity of building structures; The weighted binary cross entropy loss function, the weighted intersection-over-union loss function, and the structural similarity loss function are added to obtain several regional loss functions and edge loss functions, and the several regional loss functions and the edge loss functions are weightedly summed to obtain a total loss function.

2. The method for extracting buildings using a fully convolutional network based on contour guidance and structure attention according to claim 1, wherein: Inputting the remote sensing image into a trained contour-guided and structure-attention-based full convolutional network, and outputting a building intensity map through the contour-guided and structure-attention-based full convolutional network includes: Inputting the remote sensing image into the encoder, and outputting a multi-scale feature image through the encoder; Inputting the multi-scale feature image into the contour guidance module, and outputting the contour feature image through the contour guidance module; The contour feature image is input into the decoder, and the decoder outputs a building intensity map.

3. The method for extracting buildings using a fully convolutional network based on contour guidance and structure attention according to claim 2, wherein: The training process of the fully convolutional network also includes: Acquire a training sample set, wherein the training sample set includes a plurality of training samples, and the training samples include training image data and at least one true value image label corresponding to the training image data; Inputting the training image data into a preset initial network model, and outputting a predicted building intensity image through the initial network model; Obtaining a total loss function based on the predicted building intensity image and the true image label; Based on the total loss function, the initial network model is trained to obtain a fully convolutional network based on contour guidance and structure attention.

4. The method for extracting buildings using a fully convolutional network based on contour guidance and structure attention according to claim 1, wherein: The method of obtaining a weight matrix by predicting the building intensity image and the true value image label includes: The neighborhood of each pixel point of the predicted building intensity image is obtained based on the neighborhood perception technology.

5. The method for extracting buildings using a fully convolutional network based on contour guidance and structure attention according to claim 1, wherein: Obtaining a structural similarity loss function according to the predicted building intensity image and the true value image label includes: Obtaining a mask image corresponding to the predicted building intensity image; Cutting out pixel blocks of the same size from corresponding positions in the mask image and the ground truth image label respectively; wherein the pixel blocks are composed of pixel points surrounded by a preset length and a preset width; A structural similarity loss function is obtained based on pixel values ​​in the pixel block.

6. The method for extracting buildings using a fully convolutional network based on contour guidance and structure attention according to claim 1, wherein: The step of extracting the binary image of the building intensity map based on a threshold algorithm to obtain a building extraction result includes: Based on a threshold algorithm, the binary image of the building intensity map is classified to obtain a building extraction result.

7. An intelligent terminal, characterized in that: The device comprises a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, including the one or more programs for executing the method according to any one of claims 1 to 6.

8. A non-transitory computer-readable storage medium, characterized in that When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 6.

Images