A method and system for semantic segmentation of buildings in remote sensing images

Abstract

The present invention provides a method and system for semantic segmentation of buildings in remote sensing images, comprising the following steps: step 1, obtaining a remote sensing image dataset; step 2, constructing a network model, wherein the network model comprises a feature extraction network and a feature recovery network, wherein the feature extraction network comprises a Resnet network and an FD-ASPP module; the feature recovery network comprises a transposed convolutional layer and an AGAP attention mechanism module, and the outputs of the Resnet network and the FD-ASPP module are both connected to the AGAP attention mechanism module; the output of the AGAP attention mechanism module is connected to a transposed convolutional layer; step 3, training the network model using the dataset obtained in step 1 to obtain a trained network model; step 4, segmenting the remote sensing image to be segmented using the trained network model; the present invention effectively overcomes the interference of factors such as roads, trees, and shadows, and obtains clearer building boundaries.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and relates to a method and system for semantic segmentation of remote sensing image buildings. Background Art

[0002] Building extraction is a classic problem in the field of automated information extraction from remote sensing images. Its primary goal is to identify and extract building areas from remote sensing images. The results of building extraction have widespread applications in many fields, such as military reconnaissance, environmental and fire monitoring, and meteorological and resource observation. Therefore, building extraction holds significant research value.

[0003] Most traditional methods segment specific targets by acquiring image features such as gradients, grayscale values, and texture information through manual or machine learning methods. These methods primarily include threshold-based, edge-based, and region-based segmentation methods. These methods fail to capture deep semantic features during image extraction, making effective feature extraction difficult and, consequently, struggling to achieve high-precision segmentation results.

[0004] In recent years, an increasing number of remote sensing image semantic segmentation methods have used deep learning techniques such as convolutional neural networks to achieve more accurate segmentation. However, accurate building segmentation in remote sensing imagery based on deep learning relies on the deep learning network's effective representation of the building image's feature information. Therefore, geometric features are crucial for the segmentation of remote sensing building images. However, due to the inherent interference of building data such as lighting, season, angle, and unclear boundaries, deep learning networks have difficulty truly learning their essential edge features, often resulting in blurred edges in building image segmentation. Summary of the Invention

[0005] The purpose of the present invention is to provide a method and system for semantic segmentation of buildings in remote sensing images, which solves the above-mentioned deficiencies in the prior art.

[0006] In order to achieve the above object, the technical solution adopted in the present invention is:

[0007] The present invention provides a method for semantic segmentation of buildings in remote sensing images, comprising the following steps:

[0008] Step 1: Obtain remote sensing image dataset;

[0009] Step 2: construct a network model, wherein the network model includes a feature extraction network and a feature recovery network, wherein the feature extraction network includes a Resnet network and an FD-ASPP module; the feature recovery network transposed convolution layer and an AGAP attention mechanism module, and the outputs of the Resnet network and the FD-ASPP module are both connected to the AGAP attention mechanism module; the output of the AGAP attention mechanism module is connected to the transposed convolution layer;

[0010] Step 3: Use the data set obtained in step 1 to train the network model to obtain a trained network model;

[0011] Step 4: Use the trained network model to segment the remote sensing image to be segmented.

[0012] Preferably, the Resnet network includes a convolutional layer, the output of the convolutional layer is connected to the maximum pooling layer, the output of the maximum pooling layer is connected to three convolutional layers respectively, and the outputs of the three convolutional layers are all connected to the FD-ASPP module; the output of the FD-ASPP module is sequentially connected to four convolutional layers, twenty-three convolutional layers and three convolutional layers.

[0013] Preferably, the FD-ASPP module includes five branches, each of which is composed of four dilated convolution sub-blocks and a global average pooling layer, wherein the output of each dilated convolution sub-block is connected to a DBC fractal dimension sub-block;

[0014] The output of each DBC fractal dimension sub-block and the global average pooling layer is connected to a concatenation layer; the output of the concatenation layer is connected to a convolution layer.

[0015] Preferably, the AGAP attention mechanism module includes a global average pooling layer, a fully connected layer and a Sigmoid activation function.

[0016] Preferably, the output of the AGAP attention mechanism module is connected to a classification function.

[0017] Preferably, in step 3, the network model is trained using the data set obtained in step 1 to obtain a trained network model, and the specific method is:

[0018] A multi-scale composite loss function combining Dice Loss and BCE Loss is used to train the constructed network model to obtain a trained network model.

[0019] Preferably, the expression of the multi-scale composite loss function is as follows:

[0020]

[0021] Among them, BCELossi is BCELoss, DiceLoss under scale i i is the DiceLoss under scale i, the value range of i is {0, 2, 3, ..., n}, and n is the number of selected scales.

[0022] A remote sensing image building semantic segmentation system, comprising:

[0023] Acquisition module, used to acquire remote sensing image datasets;

[0024] A construction module for constructing a network model, wherein the network model includes a feature extraction network and a feature recovery network, wherein the feature extraction network includes a Resnet network and an FD-ASPP module; the feature recovery network transposed convolution layer and an AGAP attention mechanism module, the outputs of the Resnet network and the FD-ASPP module are both connected to the AGAP attention mechanism module; the output of the AGAP attention mechanism module is connected to the transposed convolution layer;

[0025] A training module is used to train the network model using the obtained data set to obtain a trained network model;

[0026] The segmentation module is used to segment the remote sensing image to be segmented using the trained network model.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] The present invention provides a method for semantic segmentation of buildings in remote sensing images. The method uses a ResNet network as a feature extraction network, introduces a dilated spatial pyramid pooling module (FD-ASPP) fused with fractal priors in the encoding stage, utilizes fractal dimensions to capture the fractal features of remote sensing images, and enhances the geometric feature description capability of the ResNet network. In the decoding stage, a global average pooling attention fusion mechanism (AGAP) is proposed to effectively fuse high-level features with low-level features to obtain richer semantic information and location detail information of remote sensing images. A multi-scale composite loss function is used to train the constructed building extraction network, and the remote sensing image to be extracted is input into the trained network to obtain semantic segmentation results. Compared with existing semantic segmentation algorithms for remote sensing buildings, the proposed method has better segmentation accuracy, effectively overcomes interference from factors such as roads, trees, and shadows, and obtains clearer building boundaries. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic flow diagram of the present invention;

[0030] Figure 2 is a schematic diagram of a network structure provided by an embodiment of the present invention;

[0031] Figure 3 Schematic diagram of the FD-ASPP structure provided by an embodiment of the present invention;

[0032] Figure 4 Schematic diagram of the AGAP attention mechanism structure provided by an embodiment of the present invention;

[0033] Figure 5 This is a diagram showing the segmentation effect of an embodiment of the network constructed in the present invention; the first column on the left is the original image, the middle is the label, and the right column is the network segmentation result of the present invention;

[0034] Figure 6 A comparison chart of the segmentation results of an embodiment of the network model constructed in the present invention and FCN, SegNet and U-Net. DETAILED DESCRIPTION

[0035] The present invention is further described in detail below with reference to the embodiments.

[0036] See also Figure 1 As shown, the present invention provides a remote sensing image building semantic segmentation method, comprising the following steps:

[0037] Step 1: Enhance the data set: rotate, flip, blur, Gaussian filter, bilateral filter or add white noise to the original remote sensing data and the labeled image to obtain the enhanced data set.

[0038] Data enhancement methods are all existing technologies and will not be elaborated here.

[0039] Step 2: Construct a network structure, which includes a feature extraction network and a feature recovery network, wherein:

[0040] Feature extraction network, such as Figure 2 As shown in Figure 2, the feature extraction network includes the Resnet network and the FD-ASPP module, where:

[0041] The Resnet network includes a convolutional layer, the output of the convolutional layer is connected to the maximum pooling layer, the output of the maximum pooling layer is connected to three convolutional layers respectively, and the outputs of the three convolutional layers are all connected to the FD-ASPP module; the output of the FD-ASPP module is sequentially connected to four convolutional layers, twenty-three convolutional layers and three convolutional layers.

[0042] The FD-ASPP module consists of five branches, namely four dilated convolution sub-blocks and a global average pooling layer, where each dilated convolution sub-block is connected to a DBC fractal dimension sub-block.

[0043] The output of each DBC fractal dimension sub-block and the global average pooling layer is connected to the concatenation layer, and the output of the concatenation layer is connected to the convolution layer.

[0044] Specifically:

[0045] The ResNet network specifically includes a convolution kernel of 7×7 with a stride of 2 and an output channel of 64. The field of view is a maximum pooling layer of 3×3 with a stride of 2, and the output feature map result is recorded as F1.

[0046] This is followed by three convolution layers with kernels of 1×1, 3×3, and 1×1, each with a step size of 1 and output channels of 64, 64, and 256, respectively, and finally the output feature map F2 is obtained.

[0047] Pass F2 through the FD-ASPP module, which includes five parallel branches, each of which contains two parallel sub-blocks:

[0048] The convolution kernel is 3×3, the dilation rate a is 3, the dilated convolution sub-block and the sliding window size is 7.

[0049] The convolution kernel is 3×3 with a dilation rate of 5 and the DBC fractal dimension sub-block with a sliding window size of 11;

[0050] The convolution kernel is 3×3, the dilation rate is 11, and the DBC fractal dimension sub-block with a sliding window size of 23;

[0051] The convolution kernel is 3×3, the dilation rate is 15, the dilation convolution sub-block and the sliding window size is 31.

[0052] The last branch is global average pooling, which is used to fuse the depth information extracted by the dilated convolution sub-block and the fractal feature output into new features as the output result of a single branch.

[0053] Finally, the output results of the single branches are spliced ​​together, and then a 1×1 convolution is used to restore the number of channels to the same number of channels as the original feature map, and the multi-scale feature F2new that integrates the fractal prior can be obtained.

[0054] Then F2new is sent to four convolutions with kernels of 1×1, 3×3, and 1×1 with a step size of 1 and output channels of 128, 128, and 512 respectively to obtain F3.

[0055] Then F3 is sent to twenty-three convolution kernels with a step size of 1×1, 3×3, and 1×1, and the number of output channels is 256, 256, and 1024 respectively to obtain F4.

[0056] Then, F4 is fed into three convolutions with kernels of 1×1, 3×3, and 1×1 with a step size of 1 and output channels of 512, 512, and 2048 respectively to obtain F5.

[0057] A feature recovery network is constructed, wherein the feature recovery network includes a transposed convolution layer and an AGAP attention mechanism module, wherein the AGAP attention mechanism module includes a global average pooling layer, a fully connected layer and a sigmoid activation function.

[0058] The output of the AGAP attention mechanism module is connected to a classification function.

[0059] The input of the AGAP attention mechanism module is connected to the output of the Resnet network and the FD-ASPP module respectively, and the output of the AGAP attention mechanism module is connected to the transposed convolution layer.

[0060] See also Figure 4 As shown in Figure 1, F5 is restored to the same size as F4 through transposed convolution and recorded as a high-level feature map (H), and then fused using the AGAP attention mechanism module.

[0061] The specific method of using the AGAP attention mechanism for fusion is:

[0062] First, each channel of the high-level feature map H is globally average pooled to obtain a category information feature map (f1) of size 1×1×n (n is the number of channels);

[0063] Then, a fully connected layer is connected after f1 to generate weights for each feature channel through the parameter ω, which is learned to explicitly model the correlation between feature channels;

[0064] Next, the activation function f2 = Sigmoid (ω × f1) is used to map the features between 0 and 1 to represent the importance of the channel;

[0065] Finally, the feature map f2 after feature selection is weighted on the low-level feature map (L) channel by channel to complete the recalibration of the feature map L in the channel dimension, so that the feature map L also has the receptive field of the high-level feature map category information.

[0066] An attention fusion mechanism is connected between each high-level feature map and a low-level feature map of the same size. The network fuses the features of F5, F4, F3, F2new, and F1 through four layers of AGAP attention mechanisms of different scales. The final output feature map contains rich semantic information.

[0067] Step 3: Based on the training set, a multi-scale composite loss function combining Dice Loss and BCE Loss is used to train the constructed building extraction network;

[0068] The BCE Loss function is as follows:

[0069]

[0070] Among them, BCELoss i For scale i (i.e. the predicted probability map resolution is W / 2 i ×H / 2 i ) of BCELoss, the value range of i is {0,2,3,...,n}; G j is the true label of pixel j, G j =1 represents building pixels, G j =0 indicates non-building pixels; P j is the predicted probability that pixel j is a building pixel; N is the total number of pixels in an image.

[0071] The Dice Loss function is as follows:

[0072]

[0073] Among them, DiceLoss i For scale i (i.e. the resolution of the predicted probability map is W / 2 i ×H / 2 i ), the value range of i is {0, 2, 3, ..., n}; |G∩P| is the intersection of the true label and the predicted label, and |G| and |P| represent the sum of the pixel values ​​of the true label and the predicted label.

[0074] The expression of the multi-scale composite loss function is as follows:

[0075]

[0076] Among them, BCELoss i is BCELoss, DiceLoss under scale i i is the DiceLoss under scale i, the value range of i is {0, 2, 3, ..., n}, and n is the number of selected scales.

[0077] Step 4: Input the remote sensing image to be extracted into the trained building extraction network to obtain the building extraction result.

[0078] See also Figure 5As shown in the figure, the comparison of building segmentation in remote sensing images under tree interference shows that the proposed model is less affected by trees in remote sensing images and can better extract the complete edge information of buildings as a whole.

[0079] See also Figure 6 As shown in Figure 2, the buildings segmented by the FCN network have many misclassifications and omissions, such as Figure 6 As shown in (c), the small buildings in the first row are misclassified as background, and the top of the large buildings in the fourth row is missing. This missegmentation is primarily due to the FCN network's upsampling, which directly restores the fused feature maps to the input image size, resulting in feature information loss and a weaker representation of the FCN network. Figure 6 In (d), the Segnet network utilizes the codec structure and transfers the maximum pooling index to the decoder, which improves the segmentation resolution. Although most buildings can be separated from the background, the edge segmentation effect is not good and some details are lost. Figure 6 In (e), the ASPP structure is also used to improve the Deeplab v3 network of Resnet. Due to the lack of geometric details, the building edge segmentation effect is not good. For example, there is a sticking phenomenon in the segmentation of small buildings in the first row, and the edge smoothness above the large building in the third row is poor. In Figure (f), the U-Net network has a relatively good effect on building segmentation. It jumps and connects feature maps of the same size during the upsampling process and performs feature fusion. Because the 2x upsampling factor it uses is small, the features retained are relatively rich, and the small building segmentation effect is good. The building outline can also be basically segmented, but there are also some misclassification phenomena such as Figure 6 In (f), the container on the right side of the large building in row 5 is mistakenly divided into buildings. Figure 6 (g) is the segmentation result of the proposed model. As can be seen from the first and second rows of segmentation results, the proposed model can still further segment the edges of small buildings under the interference of tree backgrounds, has a certain anti-interference ability, and achieves good results in details. Due to the interference of the road, several classic network models failed to completely segment the large buildings in the fourth row, while the proposed model achieved more accurate segmentation and higher edge integrity, which effectively alleviated the phenomenon of losing detailed information. Among the segmentation results of all the compared models, the proposed model performed best in the segmentation of building edges, and the prediction results were also closer to the ground truth map. Therefore, the proposed model not only achieved better segmentation accuracy, but also achieved better results on the edges of buildings.

[0080] The proposed Resnet network model integrating fractal priors has added the FD-ASPP module, which makes more effective use of features of different scales and fractal feature information, thereby improving the segmentation accuracy of building edge information. Compared with several other comparison models, the proposed model has clear and smooth building edges, significantly reduces the mis-segmentation of distracting objects such as trees and roads, and has a good segmentation effect on irregular buildings, effectively reducing the blurred segmentation boundaries caused by the close distance of buildings. The segmentation results are better than those of FCN, Segnet, Deeplab V3, and U-net models.

Claims

1. A method for semantic segmentation of buildings in remote sensing images, characterized in that: The following steps are involved: Step 1: Obtain remote sensing image dataset; Step 2: construct a network model, wherein the network model includes a feature extraction network and a feature recovery network, wherein the feature extraction network includes a Resnet network and an FD-ASPP module; the feature recovery network transposed convolution layer and an AGAP attention mechanism module, and the outputs of the Resnet network and the FD-ASPP module are both connected to the AGAP attention mechanism module; the output of the AGAP attention mechanism module is connected to the transposed convolution layer; Step 3: Use the data set obtained in step 1 to train the network model to obtain a trained network model; Step 4, using the trained network model to segment the remote sensing image to be segmented; The FD-ASPP module includes five branches, each of which consists of four dilated convolution sub-blocks and a global average pooling layer. The output of each dilated convolution sub-block is connected to a DBC fractal dimension sub-block. The output of each DBC fractal dimension sub-block and the global average pooling layer is connected to a concatenation layer; the output of the concatenation layer is connected to a convolution layer.

2. The method for semantic segmentation of buildings in remote sensing images according to claim 1, characterized in that: The Resnet network includes a convolutional layer, the output of the convolutional layer is connected to the maximum pooling layer, the output of the maximum pooling layer is connected to three convolutional layers respectively, and the outputs of the three convolutional layers are all connected to the FD-ASPP module; the output of the FD-ASPP module is sequentially connected to four convolutional layers, twenty-three convolutional layers and three convolutional layers.

3. The method for semantic segmentation of buildings in remote sensing images according to claim 1, wherein: The AGAP attention mechanism module includes a global average pooling layer, a fully connected layer and a Sigmoid activation function.

4. The method for semantic segmentation of buildings in remote sensing images according to claim 1, wherein: The output of the AGAP attention mechanism module is connected to a classification function.

5. The method for semantic segmentation of buildings in remote sensing images according to claim 1, characterized in that: In step 3, the network model is trained using the data set obtained in step 1 to obtain a trained network model. The specific method is: A multi-scale composite loss function combining Dice Loss and BCE Loss is used to train the constructed network model to obtain a trained network model.

6. The method for semantic segmentation of buildings in remote sensing images according to claim 5, characterized in that: The expression of the multi-scale composite loss function is as follows: in, For scale Next , For scale Next , The value range of is { 0 , 2 , 3 , ... , }, is the number of selected scales.

7. A remote sensing image building semantic segmentation system, characterized by: Based on the segmentation method according to claim 1, the system comprises: Acquisition module, used to acquire remote sensing image datasets; A construction module for constructing a network model, wherein the network model includes a feature extraction network and a feature recovery network, wherein the feature extraction network includes a Resnet network and an FD-ASPP module; the feature recovery network transposed convolution layer and an AGAP attention mechanism module, the outputs of the Resnet network and the FD-ASPP module are both connected to the AGAP attention mechanism module; the output of the AGAP attention mechanism module is connected to the transposed convolution layer; A training module is used to train the network model using the obtained data set to obtain a trained network model; The segmentation module is used to segment the remote sensing image to be segmented using the trained network model.

Images