Building Extraction Method from Remote Sensing Image Based on Improved U-Net

Abstract

This invention presents a method for building extraction from remote sensing images based on an improved U-Net. First, high-resolution remote sensing images are acquired to construct a remote sensing image dataset. Then, the traditional U-Net is improved by introducing a global attention module and a dense connection module into the encoder, and a semantic analysis module between the encoder and decoder, resulting in a building extraction model. Finally, the building extraction model is trained using the remote sensing image dataset to obtain a trained building extraction model. This trained model is then used to extract buildings from the remote sensing images. Attention weights are used as transport cues in the dense connections, and the integrity of building feature extraction is ensured by fusing attention features from different attention modules. The semantic analysis module improves the utilization rate of feature information in the feature map extracted by the encoder, and also enhances the feature information by fusing the entire feature map, resulting in better segmentation performance.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of remote sensing technology and computer vision technology, specifically relating to a method for extracting buildings from remote sensing images based on an improved U-Net. Background Technology

[0002] With the development of modern remote sensing technology, high-resolution remote sensing images have made building extraction and recognition more feasible. Building extraction and recognition based on remote sensing images has been widely used in urban construction, urban planning, digital cities and other fields. Building extraction can be regarded as an image segmentation problem. Traditional methods mostly extract features manually, mainly based on the color, shape and texture of buildings. This method is cumbersome and has low accuracy.

[0003] In recent years, with the continuous development of deep learning technology, many researchers have applied neural networks to the extraction of buildings from remote sensing images, achieving good results and overcoming the limitations of the feature representation capabilities of manual extraction.

[0004] However, due to the generally large size of high-resolution remote sensing images, which contain rich spatial information and are characterized by complexity, diversity and uncertainty, existing deep learning methods are insufficient in extracting features of buildings. They often suffer from problems such as missing or missed detections when extracting small target buildings, holes in the segmentation of large-scale targets leading to incomplete extraction of target buildings, and overly coarse edge information leading to the loss of boundary information, ultimately resulting in low building extraction accuracy. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the technical problem this invention aims to solve is to provide a method for extracting buildings from remote sensing images based on an improved U-Net.

[0006] The technical solution adopted by the present invention to solve the aforementioned technical problem is as follows:

[0007] A method for extracting buildings from remote sensing images based on an improved U-Net, characterized by the following steps:

[0008] The first step is to acquire high-resolution remote sensing images and construct a remote sensing image dataset;

[0009] The second step involves improving the traditional U-Net by introducing a global attention module and a dense connection module into the encoder, and a semantic analysis module between the encoder and decoder, resulting in a building extraction model. The encoder comprises five feature extraction layers, with a global attention module connected after each of the second to fourth feature extraction layers. The attention weights generated by each global attention module are densely connected using the dense connection module. The attention weights generated by the three global attention modules are denoted as a1, a2, and a3. Attention weight a1 is passed to the second, third, and fifth global attention modules, respectively; attention weight a2 is passed to the third and fifth global attention modules, respectively; and attention weight a3 is passed to the fifth feature extraction layer, respectively. The values ​​are passed to the fifth feature extraction layer; the second global attention module averages the attention weights a1 and a2 and uses this average to update the global attention map of the second global attention module; the third global attention module averages the attention weights a1, a2, and a3 and uses this average to update the global attention map of the third global attention module; the fifth feature extraction layer averages the attention weights a1, a2, and a3 and multiplies this average with its own calculated feature map to obtain the output feature map of the fifth feature extraction layer;

[0010] The semantic analysis module includes four semantic analysis branches. The first semantic analysis branch includes a first feature fusion layer, a dual attention module, and a second feature fusion layer. The second semantic analysis branch includes a third feature fusion layer, a dual attention module, and a fourth feature fusion layer. The third semantic analysis branch includes a fifth feature fusion layer. The fourth semantic analysis branch includes a dual attention module.

[0011] The fourth semantic analysis branch extracts semantic features from the output feature map of the fifth feature extraction layer of the encoder. The output feature map of the fourth semantic analysis branch and the global attention map output by the third global attention module are the inputs to the first decoding layer of the decoder. The global attention maps output by the second and third global attention modules are the inputs to the fifth feature fusion layer. The output feature map of the fifth feature fusion layer is the input to the second decoding layer of the decoder. The global attention maps output by the first and second global attention modules are used as inputs to the third feature fusion layer. The output feature map of the third feature fusion layer is input to the dual attention module of the second semantic analysis branch. The output feature map of the dual attention module of the second semantic analysis branch and the output feature map of the fifth feature fusion layer are used as inputs to the fourth feature fusion layer. The output feature map of the fourth feature fusion layer is input to the second feature fusion layer and the third decoding layer of the decoder, respectively. The output feature map of the encoder's first feature extraction layer and the global attention map output by the first global attention module are input to the first feature fusion layer for fusion. The output feature map of the first feature fusion layer and the output feature map of the third feature fusion layer are concatenated and input to the dual attention module of the first semantic analysis branch. The output feature map of the dual attention module of the first semantic analysis branch and the output feature map of the fourth feature fusion layer are input to the second feature fusion layer for feature fusion. The output feature map of the second feature fusion layer is the input to the fourth decoding layer of the decoder.

[0012] The third step is to train the building extraction model using the remote sensing image dataset to obtain the trained building extraction model; then use the trained building extraction model to extract buildings from the remote sensing images.

[0013] Furthermore, the dual attention module includes a channel attention module and a spatial attention module. The channel attention module includes average pooling and max pooling operations. The input feature map of the channel attention module is subjected to average pooling and max pooling operations respectively, and the feature maps obtained by the two pooling operations are added together to obtain a channel attention map. Then, the channel attention map is input into the spatial attention module. First, average pooling and max pooling operations are performed along the channel axis of the feature map, and the features obtained by the two pooling operations are concatenated along the channel dimension to generate a feature descriptor. The feature descriptor is passed through a convolutional layer to generate a spatial attention map.

[0014] Furthermore, the decoder includes a first to a fourth decoding layer, each decoding layer including a concatenation operation, a convolutional layer, a normalization operation, and an activation function.

[0015] Furthermore, the specific operation of the global attention module is as follows: For the input feature map of the global attention module, the input feature map is passed through three different two-dimensional convolutional layers to obtain three different feature maps, denoted as q, k, and v respectively; feature map q is multiplied by the transpose of feature map k, and then divided by the square root of the dimension of feature map k. The result is then input into the softmax function to obtain the attention weights of the input feature map; the attention weights are multiplied by feature map v to obtain the global attention map output by the global attention module.

[0016] Compared with the prior art, the beneficial effects of the present invention are:

[0017] This invention improves upon the traditional U-Net by adding a semantic analysis module between the encoder and decoder. This module enhances the utilization of feature information in the feature maps extracted by the encoder and fuses the entire feature map to strengthen its feature information, resulting in better segmentation performance. A global attention module is introduced into the encoder to improve the model's ability to extract detailed features of buildings in remote sensing images, effectively improving the model's ability to extract building boundaries and preventing the loss of boundary information. The concept of dense connections is introduced into the encoder, using attention weights as transmission cues for dense connections. By fusing the attention features of different attention modules, rich semantic information is obtained, ensuring the integrity of building feature extraction. Furthermore, as the number of convolutional layers increases after the image passes through the encoder, some small target information may be lost. Since shallow features contain more texture information and deep features contain more semantic information, the semantic analysis module fuses shallow and deep features to enrich the information contained in the features, thereby improving the model's accuracy in detecting small target buildings and avoiding missed or undetected small target buildings. Attached Figure Description

[0018] Figure 1 This is an overall flowchart of the present invention;

[0019] Figure 2 This is a schematic diagram of the building extraction model of the present invention;

[0020] Figure 3 This is a schematic diagram of the encoder structure of the present invention;

[0021] Figure 4 This is a diagram comparing the extraction results of different models. Detailed Implementation

[0022] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments, but this does not limit the scope of protection of this application.

[0023] Figure 1This is a flowchart of the method for extracting buildings from remote sensing images in this embodiment, as follows: Figure 1 As shown, this invention provides a method for extracting buildings from remote sensing images based on an improved U-Net, comprising the following steps:

[0024] The first step is to acquire high-resolution remote sensing images and construct a remote sensing image dataset, which is then divided into a training set, a validation set, and a test set. Specifically, this embodiment uses the WHU Building Dataset, a database of remote sensing images of buildings that features both large scenes and high resolution. It contains 8188 images and corresponding pixel-level label images, with each image having a resolution of 512×512 pixels. 4736 images are used as the training set, 1036 as the validation set, and 2416 as the test set.

[0025] The second step involves improving the traditional semantic segmentation algorithm U-Net by introducing a global attention module and a dense connection module into the encoder, and a semantic analysis module between the encoder and decoder, thus obtaining a building extraction model. Therefore, the building extraction model consists of three parts: encoder, semantic analysis module, and decoder. The encoder is used to extract multi-scale features. To improve the utilization rate of information in the features extracted by the encoder, a semantic analysis module is introduced to extract richer and higher-level semantic features. The decoder includes four decoding layers, which progressively upsample the output feature map of the semantic analysis module and fuse it with low-level features during feature resolution recovery. Finally, bilinear interpolation is used to obtain a segmentation prediction of the same size as the original feature map, improving the accuracy of semantic segmentation.

[0026] The encoder consists of five feature extraction layers (Conv1 to Conv5), and the specific parameters are shown in Table 1. Each of the second to fourth feature extraction layers is followed by a global attention module, denoted as GCA1, GCA2, and GCA3, respectively. The purpose of the global attention module is to extract useful detail information from the feature maps and suppress invalid feature information. The operation process is as follows: For the input feature map of the global attention module, firstly, the input feature map is passed through three different two-dimensional convolutional layers to obtain three different feature maps, denoted as q, k, and v, respectively. Then, feature map q is multiplied by the transpose of feature map k, and then divided by the square root of the dimension of feature map k. This step is mainly to make the result of multiplying feature maps q and k satisfy a distribution with an expectation of 0 and a variance of 1, similar to a normalization operation. Finally, the result is input into the softmax function to obtain the attention weights of the input feature map. The attention weights are multiplied by feature map v to obtain the output of the global attention module, i.e., the global attention map. The introduction of the global attention module can enhance the learning ability of network features, improve the model's performance in handling details, and effectively make up for the defect of easy loss of boundary information when using traditional U-Net to extract buildings from remote sensing images.

[0027] Table 1 Feature Extraction Layer Parameters

[0028]

[0029]

[0030] To acquire denser multi-scale feature information, a dense connection module is introduced into the encoder. This dense connection method primarily involves sequentially and in parallel connecting the global attention modules downwards, and densely connecting the attention weights generated by each global attention module. By passing these attention weights, the semantic information of the feature map is enriched, enabling the encoder to generate a denser feature pyramid. Simultaneously, the encoder's output feature map also has a larger receptive field. The attention weights generated by the three global attention modules are denoted as a1, a2, and a3, respectively. Figure 3As shown, the attention weight a1 generated by the first global attention module GCA1 is passed to the second global attention module GCA2, the third global attention module GCA3, and the fifth feature extraction layer Conv5, respectively. The attention weight a2 generated by the second global attention module GCA2 is passed to the third global attention module GCA3 and the fifth feature extraction layer Conv5, respectively. The attention weight a3 generated by the third global attention module GCA3 is passed to the fifth feature extraction layer Conv5. During the process of updating the global attention map in the second global attention module GCA2, the attention weight a1 is added to the attention weight a2 it generates, the average is calculated, and the average is used to update the second global attention module GCA2. The output global attention map; similarly, during the process of updating the global attention map in the third global attention module GCA3, the attention weights a1, a2 and the attention weight a3 generated by itself are added together and averaged, and used to update the global attention map output by the third global attention module GCA3; during the process of updating the output feature map in the fifth feature extraction layer Conv5, the attention weights a1, a2, a3 are added together and averaged, and multiplied by the output feature map calculated by itself to obtain the updated output feature map of Conv5.

[0031] The semantic analysis module includes four semantic analysis branches. The first semantic analysis branch includes a first feature fusion layer Conv1-1, a dual attention module (CBAM), and a second feature fusion layer Conv1-2. The second semantic analysis branch includes a third feature fusion layer Conv2-1, a dual attention module, and a fourth feature fusion layer Conv2-2. The third semantic analysis branch includes a fifth feature fusion layer Conv3-1. The fourth semantic analysis branch includes a dual attention module. The feature fusion layer is used to fuse shallow and deep features, and the dual attention module is used to extract the channel attention map and spatial attention map of the input feature map.

[0032] The fourth semantic analysis branch extracts semantic features from the output feature map of the fifth feature extraction layer of the encoder. The output feature map of the fourth semantic analysis branch and the global attention map output by the third global attention module GCA3 are the inputs to the first decoding layer of the decoder. The global attention maps output by the second global attention module GCA2 and the third global attention module GCA3 are the inputs to the fifth feature fusion layer Conv3-1. The output feature map of the fifth feature fusion layer Conv3-1 is the input to the second decoding layer of the decoder. The global attention maps output by the first global attention module GCA1 and the second global attention module GCA2 are the inputs to the third feature fusion layer Conv2-1. The output feature map of the third feature fusion layer Conv2-1 is input to the dual attention module of the second semantic analysis branch. The output feature map of the dual attention module of the second semantic analysis branch and the output feature map of the fifth feature fusion layer Conv3-1 are the inputs to the fourth feature fusion layer Conv2-2. The output feature map of the fourth feature fusion layer Conv2-2 is input to the fifth feature extraction layer. The two feature fusion layers Conv1-2 and the third decoding layer of the decoder; the output feature map of the encoder's first feature extraction layer Conv1 and the global attention map output by the first global attention module GCA1 are input into the first feature fusion layer Conv1-1 for fusion. The output feature map of the first feature fusion layer Conv1-1 and the output feature map of the third feature fusion layer Conv2-1 are concatenated and input into the dual attention module of the first semantic analysis branch. The output feature map of the dual attention module of the first semantic analysis branch and the output feature map of the fourth feature fusion layer Conv2-2 are input into the second feature fusion layer Conv1-2 for feature fusion. The output feature map of the second feature fusion layer Conv1-2 is the input of the fourth decoding layer of the decoder.

[0033] Table 2 Feature Fusion Layer Parameters

[0034]

[0035] To improve the ability to extract multi-scale features and prevent the loss of building boundary information, the dual attention module includes a channel attention module and a spatial attention module. The channel attention module generates a channel attention map by utilizing the channel relationships of the feature map. Since each channel of a feature map can be considered a feature detector, channel attention focuses on meaningful content in the feature map. To efficiently compute channel attention and compress the spatial dimension of the input feature map, this paper proposes that max pooling collects another important clue about the features of iconic objects and uses this clue to infer finer-grained channel attention. Therefore, the channel attention module includes average pooling and max pooling operations. Average pooling and max pooling operations are performed on the feature map input to the channel attention module, and the feature maps obtained by the two pooling operations are then added together to obtain the channel attention map. Then, the obtained channel attention map is input into the spatial attention module, which generates a spatial attention map by utilizing the spatial relationships of the feature maps. Unlike channel attention, spatial attention focuses on "where" the information-rich parts of the feature map. This is complementary to channel attention. To compute spatial attention, average pooling and max pooling operations are first performed along the channel axis of the feature map. The features obtained from the two pooling operations are then concatenated along the channel dimension to generate an efficient feature descriptor. The feature descriptor is then passed through a convolutional layer to generate a spatial attention map. The spatial attention map encodes which regions in the feature map are highlighted or suppressed, extracting richer and higher-level semantic information and ensuring the overall integrity of the building extraction.

[0036] The decoder includes first to fourth decoding layers, each of which includes a concatenation operation (FF), a convolutional layer, a normalization operation, and an activation function. Figure 2 In this context, the convolutional layer, normalization operation, and activation function are denoted as CB.

[0037] The traditional semantic segmentation algorithm U-Net includes an encoder and a decoder, with VGG-16 as the backbone feature extraction network of the traditional U-Net.

[0038] The third step is to train the building extraction model using the remote sensing image dataset obtained in the first step, and then use the trained building extraction model to extract buildings from the remote sensing images.

[0039] Specifically, the training parameters were set as follows: batch size of 10, number of iterations of 50, and total downsampling factor of 16. The Adam optimization algorithm was used to optimize the model, followed by weight decay after updates. The initial learning rate was set to 0.001, and the weight decay rate was also set to 0.001. A periodic cosine annealing learning rate strategy was employed, where the learning rate decayed in a cosine function manner, with the initial learning rate as the maximum. After each decay to 0, the learning rate quickly returned to its initial value, thus avoiding getting trapped in local optima. A hybrid loss function, combining BCE and IOU loss functions, was used.

[0040] To verify the effectiveness of this invention, a test set of remote sensing image datasets was used to test the building extraction model using both the traditional U-Net and the present invention, and the results were as follows. Figure 4 The test results comparison chart is shown below. Figure 4 The comparison results in the first row show that traditional U-Net loses the features of the upper left corner of the target region, while the method of this invention can completely extract the buildings in the target region, ensuring the integrity of the buildings. This is due to two factors: firstly, a semantic analysis module is added between the encoder and decoder, which improves the utilization rate of feature information and enriches the semantic information of the feature map; secondly, a dense connection module is introduced into the encoder, which extracts multi-scale features and ensures the integrity of the extraction. Figure 4 The comparison results in the second row show that traditional U-Net loses some boundary information, while the method of this invention can clearly extract building boundaries. This is because the global attention module can enhance the network model's ability to extract detailed features of buildings, obtaining discriminative features, which allows the network to clearly distinguish the boundaries. The above results verify that the method of this invention can ensure the integrity of building extraction while preventing the loss of building boundary information.

[0041] Any aspects not covered in this invention are applicable to existing technologies.

Claims

1. A method for extracting buildings from remote sensing images based on an improved U-Net, characterized in that, The method includes the following steps: The first step is to acquire high-resolution remote sensing images and construct a remote sensing image dataset; The second step involves improving the traditional U-Net by introducing a global attention module and a dense connection module into the encoder, and a semantic analysis module between the encoder and decoder, resulting in a building extraction model. The encoder comprises five feature extraction layers, with a global attention module connected after each of the second to fourth feature extraction layers. The attention weights generated by each global attention module are densely connected using the dense connection module. These attention weights are denoted as a1, a2, and a3, and a1 is passed to the second, third, and fifth global attention modules, respectively. The attention weight a2 is passed to the third global attention module and the fifth feature extraction layer, respectively, and the attention weight a3 is passed to the fifth feature extraction layer. The second global attention module calculates the average of the attention weights a1 and a2 and uses it to update the global attention map of the second global attention module. The third global attention module calculates the average of the attention weights a1, a2, and a3 and uses it to update the global attention map of the third global attention module. The fifth feature extraction layer calculates the average of the attention weights a1, a2, and a3 and multiplies it with its own calculated feature map to obtain the output feature map of the fifth feature extraction layer. The semantic analysis module includes four semantic analysis branches. The first semantic analysis branch includes a first feature fusion layer, a dual attention module, and a second feature fusion layer. The second semantic analysis branch includes a third feature fusion layer, a dual attention module, and a fourth feature fusion layer. The third semantic analysis branch includes a fifth feature fusion layer. The fourth semantic analysis branch includes a dual attention module. The fourth semantic analysis branch extracts semantic features from the output feature map of the encoder's fifth feature extraction layer. The output feature map of the fourth semantic analysis branch and the global attention map output by the third global attention module serve as inputs to the decoder's first decoding layer. The global attention maps output by the second and third global attention modules serve as inputs to the fifth feature fusion layer, and the output feature map of the fifth feature fusion layer serves as input to the decoder's second decoding layer. The global attention maps output by the first and second global attention modules serve as inputs to the third feature fusion layer, and the output feature map of the third feature fusion layer is input to the dual attention module of the second semantic analysis branch. The output feature map of the force module and the output feature map of the fifth feature fusion layer are used as inputs to the fourth feature fusion layer. The output feature map of the fourth feature fusion layer is input to the second feature fusion layer and the third decoding layer of the decoder, respectively. The output feature map of the first feature extraction layer of the encoder and the global attention map output by the first global attention module are input to the first feature fusion layer for fusion. The output feature map of the first feature fusion layer and the output feature map of the third feature fusion layer are concatenated and input to the dual attention module of the first semantic analysis branch. The output feature map of the dual attention module of the first semantic analysis branch and the output feature map of the fourth feature fusion layer are input to the second feature fusion layer for feature fusion. The output feature map of the second feature fusion layer is the input to the fourth decoding layer of the decoder. The third step is to train the building extraction model using the remote sensing image dataset to obtain the trained building extraction model; then use the trained building extraction model to extract buildings from the remote sensing images.

2. The method for extracting buildings from remote sensing images based on improved U-Net according to claim 1, characterized in that, The dual attention module includes a channel attention module and a spatial attention module; The channel attention module includes average pooling and max pooling operations. The input feature map of the channel attention module is subjected to average pooling and max pooling operations respectively, and then the feature maps obtained by the two pooling operations are added together to obtain the channel attention map. Then, the channel attention map is input into the spatial attention module. First, average pooling and max pooling operations are performed along the channel axis of the feature map. The features obtained by the two pooling operations are concatenated along the channel dimension to generate a feature descriptor. The feature descriptor is then passed through a convolutional layer to generate the spatial attention map.

3. The method for extracting buildings from remote sensing images based on improved U-Net according to claim 1, characterized in that, The decoder includes a first to a fourth decoding layer, and each decoding layer includes a concatenation operation, a convolutional layer, a normalization operation, and an activation function.

4. The method for extracting buildings from remote sensing images based on improved U-Net according to claim 1, characterized in that, The specific operation of the global attention module is as follows: For the input feature map of the global attention module, the input feature map is passed through three different two-dimensional convolutional layers to obtain three different feature maps, denoted as q, k and v respectively; the transpose of feature map q and feature map k are multiplied, and then divided by the square root of the dimension of feature map k. The result is then input into the softmax function to obtain the attention weight of the input feature map; the attention weight is multiplied by feature map v to obtain the global attention map output by the global attention module.

Images