A satellite remote sensing image building extraction method
By using a backbone network and a head network architecture, combined with multi-scale feature extraction and fusion, the problem of low building extraction accuracy in satellite remote sensing images was solved, and high-precision building extraction was achieved.
Patent Information
- Application Number
- CN202210152637.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-18
- Publication Date
- 2026-01-23
- Estimated Expiration
- 2042-02-18
AI Technical Summary
Existing methods for extracting buildings from satellite remote sensing images suffer from low accuracy, especially when dealing with multi-scale, mixed-pixel, low-inter-class variance, and high-intra-class variance features of buildings, making it difficult to achieve high-precision extraction.
A backbone network and a head network architecture are used for building extraction. The backbone network is used for multi-scale feature extraction, while the head network performs feature fusion and upsampling through a splicing module, a channel dimension attention module, and an upsampling module, ultimately achieving high-precision building extraction.
It improves the accuracy of building extraction, effectively handles buildings of different sizes and shapes, takes into account the contextual information of the image and compresses the number of feature maps, and achieves high-precision extraction.
Smart Images
Figure CN116665033B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method for extracting buildings from satellite remote sensing images, belonging to the field of remote sensing image processing technology. Background Technology
[0002] The processing and interpretation of remote sensing imagery is a crucial task in surveying and mapping production and a key technology for achieving precise control of national spatial planning. Real-time and rapid building extraction is particularly important for urban planning, disaster emergency response, and updating of basic geographic information. However, current remote sensing imagery processing efficiency remains very low and cannot yet meet the demands of intelligent surveying and mapping in terms of dynamism, accuracy, and high automation. Therefore, how to effectively utilize remote sensing imagery information to support intelligent surveying and mapping in the era of big data remains a pressing issue.
[0003] Buildings generally have regular shapes and distribution patterns, and their spectral characteristics can distinguish them from other impermeable layers. Therefore, early researchers often manually designed features from a morphological perspective. For example, they combined right-angle point and right-angle side features to extract residential areas from high-resolution images. Starting from corner and edge information, they optimized building extraction methods based on conditional random fields and enhanced morphological building indices to distinguish buildings from impermeable physical layers and other features with large morphological differences. However, the above methods require manually designed feature operators, and the ability of manually designed feature operators to extract high-dimensional and heterogeneous features is limited, making the algorithms unable to effectively handle various building forms.
[0004] In recent years, deep learning networks capable of learning deep features of images have been introduced into satellite image recognition and segmentation tasks, such as building detection based on Fast-RCNN, direct image style transfer based on GAN, building classification based on FCN, and building contour recognition based on decoder-encoder structures. However, features such as multi-scale buildings, mixed pixels, low inter-class variance, and high intra-class variance limit the improvement of model performance. Therefore, researchers have improved deep learning models from different aspects. To address the limitation of image resolution on extraction accuracy, high-precision building contours can be extracted from low-resolution images. Considering the multi-scale features of buildings, fuzzy fusion methods are used to fuse multi-scale features at the decision level. To address the problem that existing networks cannot effectively detect small buildings, a parallel method is proposed to extract features at different scales, and an attention mechanism is used to adaptively combine the optimal features. In addition, fusing multi-source data is also a focus of researchers, such as fusing OSM data, SAR images, DSM data, and LiDAR. Although all of the above methods can improve the building extraction accuracy to some extent, the large differences in the scale of buildings in remote sensing images still affect the extraction accuracy. Summary of the Invention
[0005] The purpose of this invention is to provide a method for extracting buildings from satellite remote sensing images, so as to solve the problem of low accuracy in extracting buildings from current satellite remote sensing images.
[0006] To solve the above-mentioned technical problems, this invention provides a method for extracting buildings from satellite remote sensing images, the method comprising the following steps:
[0007] 1) Acquire satellite remote sensing image data;
[0008] 2) Input the acquired satellite remote sensing image data into the pre-trained network extraction model, and the pre-trained network extraction model extracts buildings from the input remote sensing images;
[0009] The network extraction model comprises a backbone network and a head network. The backbone network extracts multi-scale features from the input remote sensing image, while the head network fuses these multi-scale features and extracts buildings from the fused features. The head network includes a stitching module, a channel-dimensional attention module, and an upsampling module. The stitching module upsamples the multi-scale features extracted by the backbone network, converting them into features of the same scale, and stitches them together along the channel dimension. The channel-dimensional attention module averages the stitched features across each channel, applies fully connected and non-linear activation processing to the averaged features, multiplies the processed features element-wise with the stitched features output by the stitching module, and then stitches the result together with the stitched features. The stitched result is then subjected to feature extraction processing to obtain improved features. The upsampling module upsamples and extracts features from the improved features to obtain features of the original input size, and then performs feature extraction processing on the original input features to extract buildings.
[0010] This invention employs a backbone network plus a head network architecture for building extraction from satellite remote sensing imagery. The backbone network extracts multi-scale features from the remote sensing imagery, while the head network processes these features. First, the head network upsamples the multi-scale features to the same spatial resolution, then stitches the features together along the channel dimension. Next, a channel attention module performs feature fusion processing, demonstrating that multi-scale features have varying degrees of influence on the results and addressing the issue of low edge detection accuracy. Finally, an upsampling module further processes the features, ultimately achieving building extraction. The head network used in this invention enables the fusion of output features from different levels, resulting in better image representation, taking into account the contextual information of the image, and also compressing the number of feature maps to some extent. This facilitates the high-precision extraction of buildings of different sizes and shapes from remote sensing imagery.
[0011] Furthermore, the channel-dimensional attention module includes a first submodule that performs fully connected and nonlinear activation processing on the averaged features, wherein the first submodule includes at least one set of fully connected layers and nonlinear activation layers connected in sequence.
[0012] Furthermore, the channel-dimensional attention module uses a second sub-module for feature extraction processing. The second sub-module includes a set of sub-networks, each of which includes a normalization layer, a non-linear activation layer, and a convolutional layer connected in sequence.
[0013] Furthermore, the upsampling module uses a third sub-module for feature extraction processing. The third sub-module includes at least one set of sub-networks, each of which includes a normalization layer, a non-linear activation layer, and a convolutional layer connected in sequence.
[0014] Furthermore, the third sub-module includes two sets of sub-networks. Each set of sub-networks includes a normalization layer, a non-linear activation layer, and a convolutional layer connected in sequence. The upsampling module performs a first upsampling on the features output by the channel-dimensional attention module to obtain features of half the size of the original input image. After processing by the first set of sub-networks, it is upsampled again to obtain features of the same size as the original input image. The second set of sub-networks predicts the obtained features of the same size as the original input image to achieve the extraction of buildings.
[0015] Furthermore, the backbone network includes an image segmentation module and at least two sequentially connected feature extraction subnetworks; wherein, the image segmentation module is used to crop the input image into image blocks of a set size, the first-layer feature extraction subnetwork includes a linear mapping module and a shift transformer module, used to perform linear mapping and shift transformer operations on the input features; each subsequent layer of feature extraction subnetwork includes a sequentially connected image block merging module and a shift transformer module, used to merge and shift transformer operations on the corresponding size features output by the previous layer.
[0016] The backbone network described above crops the input image into image patches of a certain size through an image segmentation module. Subsequent processing is then performed based on the features of each image patch, ensuring the integrity of the input image features and facilitating subsequent processing. Furthermore, a multi-layered, sequentially connected feature extraction subnetwork is used to obtain image features at different scales, which is more conducive to subsequent building extraction and change detection. The aforementioned shifting transformer module performs transformer operations on the features within each window, introducing locally aggregated information and ensuring information exchange between different windows. The image patch merging module expands the spatial range of each feature to obtain greater scale information.
[0017] Furthermore, the shift transformer module is composed of two transformer-MSA modules connected together, wherein the first transformer-MSA module adopts the W-MSA mechanism and the second transformer-MSA module adopts the SW-MSA mechanism.
[0018] The shift transformer module uses two transformer modules. The first transformer-MSA module uses the W-MSA mechanism to perform MSA calculation on the features under each window, while the second transformer-MSA module uses the SW-MSA mechanism to perform MSA calculation on the features under the shift window, thus ensuring information exchange between different windows.
[0019] Furthermore, the input of the linear mapping module is connected to the output of the image segmentation module, and is used to perform linear mapping on the features of each image block output by the image segmentation module to obtain features of different dimensions of each image block.
[0020] The linear mapping module maps the features of each image block output by the image segmentation module to obtain the features of each image block in different dimensions. Attached Figure Description
[0021] Figure 1 This is a schematic diagram of the network model architecture used in the satellite remote sensing image building extraction method of the present invention;
[0022] Figure 2 This is a schematic diagram of the head network structure used in this invention;
[0023] Figure 3 This is a schematic diagram of the backbone network structure used in this invention;
[0024] Figure 4 This is a schematic diagram illustrating the principles of self-attention (SA) and multi-head self-attention (MSA) mechanisms;
[0025] Figure 5 This is a schematic diagram of the Transformer module structure for shifting windows in the head network used in this invention. Detailed Implementation
[0026] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.
[0027] This invention employs a backbone network plus a head network architecture for building extraction from remote sensing images, such as... Figure 1As shown, the backbone network is used to extract multi-scale features from the input remote sensing images to obtain features of different scales; the head network fuses the multi-scale features extracted by the backbone network and utilizes the fused features.
[0028] 1. Acquire remote sensing image data
[0029] This invention acquires satellite remote sensing image data of the target area. To ensure accuracy, the acquired images must be of high quality, meaning they must be clear and cloudless.
[0030] 2. Perform multi-scale feature extraction on the acquired remote sensing image data.
[0031] This invention inputs the acquired satellite remote sensing image data of the target area into a backbone network, and uses the backbone network to extract multi-scale features of the image, such as... Figure 3 As shown, the backbone network includes an image segmentation module and at least two sequentially connected feature extraction subnetworks. The image segmentation module crops the input image into image blocks of a set size. The first feature extraction subnetwork includes a linear mapping module and a shift transformer module, used to perform linear mapping and shift transformer operations on the input features. The input of the linear mapping module is connected to the output of the image segmentation module, used to perform linear mapping on the features of each image block output by the image segmentation module to obtain features of different dimensions for each image block. Subsequent feature extraction subnetworks each include a sequentially connected image block merging module and a shift transformer module, used to merge and shift transformer operations on features of corresponding sizes output by the previous layer. In other implementations, the number of layers in the feature extraction subnetwork can be set according to the feature complexity and extraction accuracy requirements of the actual image. For example, this embodiment uses a four-layer feature extraction subnetwork.
[0032] The specific process of multi-scale feature extraction is as follows:
[0033] (1) Image segmentation module
[0034] In the image segmentation module, the input large image is cropped into non-overlapping image blocks of a set size. Each image block is set to N×N pixels. If the input image is a three-band image (H×W×3), then each image is cropped into... There are 3 image patches. The feature vector (1, N×N×3) of each image patch can be determined, resulting in a total of 3 image patches. Each image has 128×128 feature vectors. Subsequent operations are performed on these feature vectors. In practice, the positional information of the image patches is usually also concatenated into the feature vectors. In this embodiment, the image patch size is set to 4×4 pixels. If the input image is 512×512 pixels, then 128×128 image patches can be obtained, along with 128×128 feature vectors. The image patch size can be determined based on the specific input image size.
[0035] (2) First-layer feature extraction subnetwork
[0036] The first-layer feature extraction subnetwork includes a linear mapping module and a shift transformer module. The feature vector of each image patch is input into the linear mapping module, which performs a linear mapping on the feature vector to obtain a feature vector (1, C) of different dimensions. Finally, we can obtain... The mapped feature vectors.
[0037] The mapped feature vectors are input into the shift transformer module for shift transformer operations. This shift transformer module employs two transformer-MSA modules. The first transformer-MSA module uses the W-MSA mechanism to perform MSA calculations on the features within each window, while the second transformer-MSA module uses the SW-MSA mechanism to perform MSA calculations on the features within the shift window, ensuring information exchange between different windows. For example, Figure 5 As shown, the first transformer-MSA module includes a regularization layer (Layer-norm), W-MSA, and MLP, while the second transformer-MSA module includes a regularization layer (Layer-norm), SW-MSA, a regularization layer (Layer-norm), and MLP. In the first transformer-MSA module, the mapped feature vector is first processed by the regularization layer and W-MSA to obtain the first layer of multi-head self-attention features. This feature, along with the initially input mapped features, is then processed by the regularization layer and MLP to obtain the latest features. These latest features, along with the first layer of multi-head self-attention features, are used as the input features for the second transformer-MSA module. This second layer of multi-head self-attention features is first processed by the regularization layer and SW-MSA to obtain the second layer of multi-head self-attention features. This second layer of multi-head self-attention features, along with the input features for the second transformer-MSA module, is then processed by the regularization layer and MLP to obtain the latest features, which are the extracted first-scale features of the image.
[0038] MSA is calculated based on SA operations, and the specific process is as follows:
[0039] The basic formula for the self-attention mechanism (SA) is:
[0040]
[0041] Where Q, K, and V are derived from the original feature X by weighting matrix W respectively. Q Q K Q V The mapping is obtained. The basic idea of the self-attention mechanism is to first calculate the similarity between Q and K, and after softmax normalization, obtain a result similar to weights. This weight is then used to sum the weights over V to obtain the final updated features.
[0042] Multi-head self-attention (MSA) mechanism utilizes different weight matrices W Q Q K Q V Perform SA operations on the initial features respectively, such as Figure 4 As shown, the results are stitched together, and the stitched structure is further mapped to obtain a matrix that integrates all self-attention information. Finally, the multi-scale features of the first temporal image and the multi-scale features of the second temporal image are obtained.
[0043] The specific process of the shift transformer operation is as follows:
[0044] First, the entire image is divided into windows of a set size, M×M, meaning each window contains M×M image blocks. For example, if there are 128×128 image blocks, each 4×4 pixels in size, and the window size is set to 8×8, then there will be 16×16 windows, each containing 8×8 image blocks, and each window itself will be 32×32 pixels. The window size can be determined based on the specific number of image blocks after cropping.
[0045] In each window, a transformer operation is performed on the mapped feature vector to obtain a new feature vector. Then, the window is shifted, and a transformer operation is performed within each shifted window according to the shift window size to introduce local aggregation information and obtain a new feature vector. Throughout this process, the number and dimension of the features remain unchanged. The shift window size can be determined based on the actual window size to ensure that the shift does not exceed the set window size.
[0046] (3) Other layer feature extraction subnetworks
[0047] There are three other layers in total. Each layer includes an image patch merging module and a transformer module for shifting windows. Feature extraction is performed at different spatial ranges (sizes) through the feature extraction subnetworks of other layers. In each layer, the image merging module first merges adjacent image patches in the image output by the previous layer's feature extraction subnetwork to obtain larger scale information; then, the merged image patches are subjected to a shifting transformer operation.
[0048] First, image patches are merged according to the set merging rules. The image patches are merged from N×N pixels in the first layer to 2N×2N pixels in the second layer, from 2N×2N pixels in the second layer to 4N×4N pixels in the third layer, and from 4N×4N pixels in the third layer to 8N×8N pixels in the fourth layer. For example, if each image patch is 4×4 pixels, the image patch merging module transforms it from 4×4 pixels to 8×8 pixels, from 8×8 pixels to 16×16 pixels, and from 16×16 pixels to 32×32 pixels. That is, the image patch size in the second feature extraction sub-network is 8×8 pixels, the image patch size in the third feature extraction sub-network is 16×16 pixels, and the image patch size in the fourth feature extraction sub-network is 32×32 pixels. In this way, the spatial range of the original image corresponding to each feature vector gradually increases, allowing for the acquisition of greater scale information.
[0049] After merging the image blocks in each layer, the above process (2) is used. The shift transformer module in the first layer feature extraction subnetwork performs transformer operation. First, the features output by the previous layer are divided into windows according to the set window size. Then, the features under each window are subjected to transformer operation to obtain a new feature vector. Then, the window is shifted. According to the shift window size, the transformer operation is performed in each shifted window to obtain a new feature vector.
[0050] Because the number and dimensionality of features remain constant within each feature extraction sub-network layer, this hinders the learning of multi-scale features and limits the accuracy of extracting buildings from remote sensing images. Therefore, the image patch merging module, which downsamples the image, effectively addresses this problem. Assume the original input image size is 224×224, and the window size is fixed at 7×7 patches throughout the network stages. In the first stage, the image patch size is 4×4, the total number of image patches is 56×56, and the number of windows is 8×8=64. The total number of image patches can also be calculated using the number of windows and the number of image patches contained in each window: 64×49. At the beginning of the second stage, the four surrounding image patches are merged together. At this point, the image patch size is 8×8, the total number of image patches is 28×28, and the number of windows is 4×4=16. The total number of image patches can also be calculated using the number of windows and the number of image patches contained in each window: 16×49. At the beginning of the third stage, further... First, the four surrounding image patches are stitched together. At this point, the image patch size is 16×16, the total number of image patches is 14×14, and the number of windows is 2×2=4. The total number of image patches can also be calculated by the number of windows and the number of image patches contained in each window: 4×49. At the beginning of the final stage e, the four surrounding image patches are stitched together again. This time, the image patch size is 32×32, the total number of image patches is 7×7, and the number of windows is 1×1=1. The total number of image patches can also be calculated by the number of windows and the number of image patches contained in each window: 1×49. Clearly, in the end, there is only one window on the entire image, and the window also contains 7×7 image patches, allowing the network to learn global features. When the input image size is 384, the fixed window size is generally 12. When the input image size is 512, the image patch and window division is similar, but some padding is needed to ensure an integer number of windows.
[0051] 3. Building extraction based on extracted multi-scale features
[0052] This invention employs a head network to extract buildings based on multi-scale features, wherein the structure of the head network is as follows: Figure 2As shown, the system includes a splicing module, a channel-dimensional attention module, and an upsampling module. The splicing module upsamples the multi-scale features extracted by the backbone network, converting them into features of the same scale, and splices them along the channel dimension. The channel-dimensional attention module averages the spliced features across each channel, performs fully connected and non-linear activation processing on the averaged features, multiplies the processed features element-wise with the spliced features output by the splicing module, and splices the multiplication result with the spliced features. The spliced result is then subjected to feature extraction processing to obtain the improved features. The upsampling module upsamples and extracts features from the improved features to obtain features of the original input size, and performs feature extraction processing on the features of the original input size to achieve the extraction of buildings.
[0053] The channel-dimensional attention module includes a first sub-module and a second sub-module. The first sub-module includes at least one set of fully connected layers and non-linear activation layers connected in sequence. In this embodiment, two sets can be used. The second sub-module includes a set of sub-networks, which includes a normalization layer, a non-linear activation layer, and a convolutional layer connected in sequence. The upsampling module uses a third sub-module for feature extraction processing. The third sub-module includes at least one set of sub-networks, which includes a normalization layer, a non-linear activation layer, and a convolutional layer connected in sequence.
[0054] The stitching module upsamples the multi-scale features output by the backbone network to a size equal to 1 / 4 of the original input and stitches them together along the channel dimension. The stitching module then inputs the stitched result into the fully connected layer of the first group of sub-networks in the first sub-module, where it is processed by the non-linear activation layer. This result is then input into the fully connected layer of the second group of sub-networks, where it undergoes non-linear activation. The final result is then element-wise multiplied with the output of the stitching module. This multiplication result is then stitched together again with the output of the stitching module, and the result is input into the upsampling module. The third sub-module of the upsampling module includes two groups of sub-networks, each consisting of a normalization layer, a non-linear activation layer, and a convolutional layer connected sequentially. The upsampling module first upsamples the features output by the channel-dimensional attention module to obtain features half the size of the original input image. After processing by the first group of sub-networks, it is upsampled again to obtain features of the same size as the original input image. The second group of sub-networks then predicts these features to extract buildings.
Claims
1. A method for extracting buildings from satellite remote sensing images, characterized in that, The method includes the following steps: 1) Acquire satellite remote sensing image data; 2) Input the acquired satellite remote sensing image data into the pre-trained network extraction model, and the pre-trained network extraction model extracts buildings from the input remote sensing images; The network extraction model comprises a backbone network and a head network. The backbone network extracts multi-scale features from the input remote sensing image, while the head network fuses these multi-scale features and extracts buildings from the fused features. The head network includes a stitching module, a channel-dimensional attention module, and an upsampling module. The stitching module upsamples the multi-scale features extracted by the backbone network, converting them into features of the same scale, and stitches them together along the channel dimension. The channel-dimensional attention module includes a first submodule and a second submodule. The first submodule averages the stitched features across each channel, performs fully connected and non-linear activation processing on the averaged features, and performs fully connected and non-linear activation processing on the output of the first submodule. It then performs element-wise multiplication of the processed features with the stitched features output by the stitching module, stitches the result with the stitched features, and performs feature extraction processing on the stitched result to obtain improved features. The upsampling module upsamples and extracts features from the improved features to obtain features of the original input size, and performs feature extraction processing on the original input features to extract buildings.
2. The method for extracting buildings from satellite remote sensing images according to claim 1, characterized in that, The first submodule includes at least one set of fully connected layers and nonlinear activation layers connected in sequence.
3. The method for extracting buildings from satellite remote sensing images according to claim 1, characterized in that, The second submodule includes a set of subnetworks, each subnetwork comprising a normalization layer, a nonlinear activation layer, and a convolutional layer connected in sequence.
4. The method for extracting buildings from satellite remote sensing images according to claim 1, characterized in that, The upsampling module uses a third sub-module for feature extraction. The third sub-module includes at least one set of sub-networks, which include a normalization layer, a non-linear activation layer, and a convolutional layer connected in sequence.
5. The method for extracting buildings from satellite remote sensing images according to claim 4, characterized in that, The third submodule includes two sets of subnetworks. Each set of subnetworks includes a normalization layer, a nonlinear activation layer, and a convolutional layer connected in sequence. The upsampling module performs a first upsampling on the features output by the channel dimension attention module to obtain features of half the size of the original input image. After processing by the first set of subnetworks, it is upsampled again to obtain features of the same size as the original input image. The second set of subnetworks predicts the features of the same size as the original input image to achieve building extraction.
6. The method for extracting buildings from satellite remote sensing images according to any one of claims 1-5, characterized in that, The backbone network includes an image segmentation module and at least two sequentially connected feature extraction subnetworks. The image segmentation module is used to crop the input image into image blocks of a set size. The first-layer feature extraction subnetwork includes a linear mapping module and a shift transformer module, which are used to perform linear mapping and shift transformer operations on the input features. Subsequent feature extraction subnetworks each include a sequentially connected image block merging module and a shift transformer module, which are used to merge and shift transformer operations on the corresponding size features output by the previous layer.
7. The method for extracting buildings from satellite remote sensing images according to claim 6, characterized in that, The shift transformer module consists of two interconnected transformer-MSA modules, wherein the first transformer-MSA module adopts the W-MSA mechanism and the second transformer-MSA module adopts the SW-MSA mechanism.
8. The method for extracting buildings from satellite remote sensing images according to claim 6, characterized in that, The input of the linear mapping module is connected to the output of the image segmentation module, and is used to perform linear mapping on the features of each image block output by the image segmentation module to obtain features of different dimensions of each image block.