A method for extracting buildings from optical remote sensing images with multi-layer feature decoupling

Abstract

This invention discloses a multi-layer feature decoupling method for extracting buildings from optical remote sensing images. First, a multi-layer feature decoupling network is used to extract and decompose multi-scale features of buildings, obtaining more stable semantic subject features and uncertain semantic boundary features. Then, based on the differences between the semantic subject features and uncertain semantic boundary features, a dual-stream semantic feature description network is used to gradually fuse them in different ways, deepening the semantic representation in deep features in strong semantic regions and preserving more detailed information in weak semantic regions. Finally, a multi-task supervision method is used to improve the accuracy of building edges while ensuring the integrity of the main body of the building, achieving high-performance extraction of buildings from high-resolution optical remote sensing images. This invention can significantly improve the extraction effect of buildings from high-resolution optical remote sensing images and can accurately extract buildings of different scales and spatial distributions in complex environments.

Description

Technical Field

[0001] This invention relates to the fields of remote sensing image processing and building extraction technology, specifically to a method for extracting buildings from optical remote sensing images with multi-layer feature decoupling. Background Technology

[0002] Building extraction plays a crucial role in urban planning, illegal construction monitoring, and geographic information surveying. With the rapid development of high-resolution optical remote sensing imagery, an increasing amount of data is available for building extraction. However, manually extracting buildings from massive amounts of optical remote sensing data is time-consuming and labor-intensive, necessitating an automated and efficient building extraction algorithm. Buildings exhibit different forms and spatial distributions in different environments. Furthermore, limitations in imaging conditions may result in low contrast between buildings and their surroundings, all of which pose challenges to accurate building extraction. Faced with increasingly complex building extraction tasks, traditional methods relying on the building's spectral, shape, color, texture, and shadow features are no longer sufficient. With the continuous development of deep learning, convolutional neural networks, possessing powerful feature extraction and generalization capabilities, are increasingly widely used in building extraction tasks, significantly improving the performance of automated building extraction algorithms.

[0003] To better extract buildings accurately from complex environments, many studies have improved building extraction performance by enhancing semantic feature description capabilities. For example, this involves increasing the receptive field by introducing spatial pyramid pooling modules into the encoder-decoder structure, and employing different convolutional forms such as asymmetric convolution and dense upsampling convolution to enhance feature description capabilities. Many studies focus on improving the accuracy of building edge extraction by introducing attention mechanisms into the U-Net network for better feature fusion, or by incorporating semantic edge information into feature fusion to further refine the boundaries of irregular buildings. Many studies further optimize extraction algorithms to address the problem of large differences in building volume. This is achieved by designing separate shape prediction branches to accurately extract building information at different scales, optimizing multi-scale feature fusion mechanisms through multi-scale attention models, or employing various forms of image post-processing operations to improve the completeness and accuracy of building extraction. However, with the continuous development of high-resolution optical remote sensing images, the amount of detail contained in images is constantly increasing, the inter-class differences in buildings are decreasing, and the intra-class differences are increasing. The above methods often result in over-extraction and under-extraction, leading to poor extraction accuracy. Summary of the Invention

[0004] In view of this, the present invention provides a method for extracting buildings from optical remote sensing images by multi-layer feature decoupling. By constructing a powerful semantic feature expression capability through multi-layer feature decoupling, the method improves the accuracy of building edges while ensuring the integrity of the building body, thereby improving the building extraction performance.

[0005] The method for extracting buildings from optical remote sensing images using multi-layer feature decoupling according to the present invention includes:

[0006] Step 1: Extract multi-scale features from the optical remote sensing image to obtain multi-layer feature maps at different scales;

[0007] Step 2: Perform feature decomposition on the feature map obtained in Step 1. Specifically, using adjacent shallow feature maps as references, calculate the offset of each feature in adjacent deep feature maps using the feature flow field; correct the deep feature map based on the offset to obtain more stable semantic main features in the strong semantic region representing the main part of the building; then use subtraction to obtain uncertain semantic boundary features in the weak semantic region representing the edge part of the building in the deep feature map.

[0008] Step 3: Fuse the multiple sets of more stable semantic subject features and uncertain semantic boundary features obtained in Step 2 layer by layer to obtain multiple pixel-level prediction maps; among them, strong semantic regions are fused from deep to shallow, and weak semantic regions are fused from shallow to deep.

[0009] Step four: Using a multi-task supervision method, supervised learning is performed on the multiple pixel-level prediction maps obtained in step three based on the multi-task joint loss function to complete the building extraction.

[0010] Preferably, in step one, AlexNet, VGGNet, ResNet, ResNeXt or DenseNet networks are used to extract features from the optical remote sensing images.

[0011] Preferably, step two specifically includes the following sub-steps:

[0012] S2.1 Feature preprocessing: Let F and F′ be the relatively shallow and relatively deep features of two adjacent feature layers; transform the deep feature to the same size as the shallow feature F, denoted as F″;

[0013] S2.2, Generate the characteristic flow field: Concatenate the deep feature F″ obtained in S2.1 with the shallow feature F and perform convolution to obtain the flow field δ;

[0014] S2.3, Generate more stable semantic subject features in the strong semantic region: Based on the flow field δ, obtain the offset corresponding to each feature in the deep feature map F″; correct the deep feature map F″ based on the offset to obtain more stable semantic subject features F′ constituting the strong semantic region. MainBody ;

[0015] S2.4, Generate uncertain semantic boundary features in weak semantic regions: Using a subtraction operation, obtain uncertain semantic boundary features F′ from the deep feature map F″. Uncerta in Boundary ;

[0016] S2.5, combine adjacent feature maps in pairs in the multi-layer feature maps from step one, and repeat S2.1 to S2.4 to generate N-1 sets of more stable semantic subjects and uncertain semantic boundaries; where N is the total number of layers of feature maps obtained in step one.

[0017] Preferably, in S2.1, a 1×1 convolution and upsampling operation are used to transform the deep features to the same size as the shallow features.

[0018] A better approach is to use feature flow distortion to correct deep feature maps.

[0019] Preferably, in step three, when the features of adjacent feature layers are fused, an attention mechanism is used to assign weights to the features of adjacent feature layers.

[0020] A better approach is to use the Sigmoid function as a gate to assign weights to features in adjacent feature layers.

[0021] Preferably, in step three, the fusion is performed in the following manner:

[0022]

[0023]

[0024]

[0025] in, and These represent the feature layers after applying the attention mechanism, Z represents the feature fusion result, and G represents the feature layers after applying the attention mechanism. X and G Y These represent the attention coefficients obtained from the selection gate,

[0026] G x =Sigmoid(conv) 1×1 (X))

[0027] G y =Sigmoid(conv) 1×1 (Y))

[0028] Where Sigmoid(·) represents performing the Sigmoid operation, and conv 1×1 (·) represents a 1×1 convolution.

[0029] Preferably, in step four, the cross-entropy loss function is used to perform supervised learning on each subtask.

[0030] Beneficial effects:

[0031] (1) This invention first utilizes a multi-layer feature decoupling network to extract and decompose multi-scale features of buildings, obtaining more stable semantic subject features and uncertain semantic boundary features. Then, based on the differences between semantic subject features and uncertain semantic boundary features, a dual-stream semantic feature description network is used to gradually fuse them in different ways, deepening the semantic representation in deep features in strong semantic regions and retaining more detailed information in weak semantic regions. Finally, a multi-task supervision method is used to improve the accuracy of building edges while ensuring the integrity of the main body of the building, achieving high-performance extraction of buildings from high-resolution optical remote sensing images. Compared with existing technologies, the method of this invention can significantly improve the extraction effect of buildings from high-resolution optical remote sensing images. Especially when facing buildings of different scales and spatial distributions in complex environments, this method can also ensure the accuracy of building edges while ensuring the integrity of the main body of the building, reducing over-extraction and under-extraction. This method is based on an encoder-decoder framework, utilizes multi-layer feature decoupling and a dual-stream semantic feature description network to improve semantic description capabilities, greatly improving building extraction performance and having good practical application value.

[0032] (2) The present invention obtains the offset of deep feature map features based on feature flow field, and then uses feature flow distortion operation to correct deep feature map. It can adaptively adjust and align deep features, improve feature localization ability, and obtain more stable semantic subject features.

[0033] (3) When performing feature fusion of adjacent feature layers, this invention considers the complementarity of adjacent feature layers and uses an attention mechanism to select and guide the fusion of complementary information of adjacent feature layers. This can significantly reduce the fusion of invalid feature information and make the fusion more efficient and reasonable. Attached Figure Description

[0034] Figure 1 This is a flowchart of the method of the present invention.

[0035] Figure 2 This is a schematic diagram of the feature structure module of the present invention.

[0036] Figure 3 This is the dual-stream semantic feature description framework of the present invention.

[0037] Figure 4 This is a schematic diagram of the component fusion module structure. Detailed Implementation

[0038] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0039] This invention provides a method for extracting buildings from optical remote sensing images using multi-layer feature decoupling. The method flowchart is shown below. Figure 1 As shown, the specific steps include the following:

[0040] Step 1: Perform multi-scale feature extraction on the high-resolution optical remote sensing image to obtain multi-layer feature maps at different scales.

[0041] This step can use feature extraction networks such as AlexNet, VGGNet, ResNet, ResNeXt, and DenseNet to achieve multi-scale feature extraction.

[0042] In this embodiment, a ResNet backbone network is used to extract features from the input high-resolution optical remote sensing image, such as... Figure 1 As shown in (a). Specifically, the original optical image is input into the ResNet feature extraction network, and after multiple convolution and pooling operations, deep feature information is obtained. The resulting four feature maps are denoted as F from shallow to deep. 1 ,F 2 ,F 3 ,F 4 .

[0043] Step 2: Decompose the feature map obtained in Step 1 to obtain more stable semantic subjects representing the main body of the building and uncertain semantic boundaries in weak semantic regions representing the edge of the building.

[0044] The continuous convolution operation in step one, while stabilizing deep semantic features, loses detailed building information and causes feature inaccuracies. Therefore, this invention introduces a feature flow field to align deep features with adjacent low-level features as references. The offset corresponding to each feature in the deep feature map obtained from the feature flow field is used to correct the deep feature map, resulting in a more stable semantic subject in a strong semantic region.

[0045] like Figure 1 As shown in (b), it can be divided into the following 5 sub-steps:

[0046] S2.1, Feature preprocessing: such as Figure 2As shown, let F and F′ represent the relatively shallow and relatively deep features of two adjacent feature layers. F′, as a deep feature, has more data channels and a smaller spatial size. To use the shallow feature F as a reference to decompose F′, a 1×1 convolution and upsampling operation are first used to transform the deep feature to the same size as the shallow feature F, denoted as F″, as shown in the following formula:

[0047] F″=Up(conv 1×1 (F′))

[0048] Where Up(·) represents the upsampling operation, conv 1×1 (·) represents a 1×1 convolution.

[0049] S2.2, Generating Feature Flow Field: Introducing a feature flow field allows the network to automatically learn feature misalignment information. First, the deep feature F″ obtained in sub-step (1) is concatenated with the shallow feature F, and then a 3×3 convolution is used to obtain the flow field δ. The flow field δ has two dimensions, representing the deviation direction of each feature point in the flow field.

[0050] δ=conv 3×3 (cat(F,F″))

[0051] Here, cat(·) represents a cascading operation, and conv 3×3 (·) represents a 3×3 convolution.

[0052] S2.3, Generating more stable semantic subject features in strong semantic regions: Based on the flow field δ, the offset corresponding to each feature in the relatively deep feature map F″ can be obtained. Therefore, by using the feature flow distortion operation to correct the deep feature map F″, more stable semantic subject features F′ constituting the strong semantic region are obtained. MainBody The formula is as follows:

[0053]

[0054] Where Ψ(·) represents the feature flow warping operation, ρ is a feature point in the deep feature map F″, N(ρ) represents the feature points surrounding the warped feature point ρ, and ω ρ It is the offset corresponding to ρ.

[0055] S2.4, Generating uncertain semantic boundary features in weak semantic regions: Using a subtraction operation, uncertain semantic boundary features F′ can be obtained from the deep feature map F″. Uncerta in Boundary .

[0056] F′ Uncerta in Boundary =F″-F′ MainBody

[0057] S2.5, Generate three sets of more stable semantic subjects and uncertain semantic boundaries: Combine the adjacent feature maps of the 4-layer feature maps obtained in step one in pairs, and repeat the above sub-steps S2.1 to S2.4 to generate three sets of more stable semantic subjects and uncertain semantic boundaries.

[0058] Step 3: Based on the component fusion module, the network structure is described using dual-stream semantic features. The three sets of more stable semantic subjects and uncertain semantic boundaries obtained in Step 2 are fused layer by layer. Specifically, features belonging to strong semantic regions and weak semantic regions are integrated into two parallel branches. In strong semantic regions, feature fusion is performed from deep to shallow to deepen the semantic representation in the deep features; in weak semantic regions, fusion is performed from shallow to deep to retain more detailed information.

[0059] The specific fusion method is as follows: First, upsampling and channel compression are used to ensure that feature layers of different depths have the same number of channels and spatial size. Then, the processed features of adjacent feature layers are fused together through a cascade operation. In this case, features from deep to shallow are fused for strong semantic regions, while features from shallow to deep are fused for weak semantic regions.

[0060] Furthermore, considering the complementarity of adjacent feature layers, this invention also introduces an attention mechanism during the fusion process to select and guide the fusion of complementary information. This not only significantly reduces the fusion of invalid feature information, but also makes the fusion more efficient and reasonable.

[0061] like Figure 3 As shown, this embodiment is specifically divided into the following 6 sub-steps:

[0062] S3.1 Before layer-by-layer feature fusion, operations such as upsampling, downsampling, and channel compression are used to adjust the feature maps with different numbers of channels and spatial sizes obtained in step two, so that they eventually have the same number of channels and spatial size.

[0063] S3.2 utilizes the designed component feature fusion module to efficiently fuse features from strong and weak semantic regions. For example... Figure 4 As shown, let X and Y be the adjacent feature layers after adjustment in sub-step S3.1, serving as the input to the component feature fusion module. Considering the complementarity of adjacent feature layers, an attention mechanism is used, employing the sigmoid function as a gate to select and guide the mutual fusion of complementary information. This process can be represented by the following formula:

[0064]

[0065]

[0066]

[0067] in, and Z represents the optimized feature layer, Z represents the output of the component feature fusion module, and G represents the optimized feature layer. X and G Y Let represent the attention coefficients obtained from the selection gate, which can be obtained from the following formula:

[0068] G x =Sigmoid(conv) 1×1 (X))

[0069] G y =Sigmoid(conv) 1×1 (Y))

[0070] Where Sigmoid(·) represents performing the Sigmoid operation, and conv 1×1 (·) represents a 1×1 convolution.

[0071] S3.3, repeat sub-step S3.2, sequentially fusing the features obtained in step two to obtain fused features of strong and weak semantic regions. The fusion of more stable semantic subject features in the strong semantic region follows a top-down fusion order, while the fusion of uncertain semantic boundary features in the weak semantic region follows a bottom-up order. The size of the fused feature space is 1 / 4 of the input feature map, and the number of channels is twice that of the input feature map.

[0072] In step S3.4, deep and parallel multi-rate dilated convolutions are performed on the fused strong and weak semantic regions obtained in sub-step S3.3, with dilation rates of 1, 2, and 5, respectively. Subsequently, 1×1 convolutions are used to refine the feature fusion, resulting in a more stable semantic subject and an uncertain semantic boundary.

[0073] S3.5, by adding the more stable semantic subject and the uncertain semantic boundary obtained in sub-step (4) together through point-by-point addition operation, the complete building features are obtained.

[0074] S3.6, using Figure 3 The prediction structure module shown yields three sets of pixel-level prediction maps. This structure can be represented by the following formula:

[0075] P = Up(ReLU(BN(conv) 1×1 (ReLU(BN(conv 3×3 (F)))))))

[0076] Where F represents the final feature obtained, P represents the predicted map obtained after the prediction structure module, Up(·) represents the upsampling operation, ReLU(·) represents the ReLU activation function, and conv 1×1(·) represents a 1×1 convolution, conv 3×3 (·) represents a 3×3 convolution, and BN(·) represents batch normalization.

[0077] Step 4: Using a multi-task supervision method, based on the cross-entropy loss function, supervise and optimize the three sets of prediction maps generated in Step 3: the more stable semantic subject, the uncertain semantic boundary, and the building. Figure 1 As shown in (d). Among them, the supervision of more stable semantic subjects and uncertain semantic boundaries is auxiliary supervision, which can help generate more complete building subjects and improve the accuracy of building boundaries.

[0078] Specifically, it consists of 3 sub-steps:

[0079] S4.1, use morphological image operations such as image erosion to generate building subject and edge label maps corresponding to the more stable semantic subject and uncertain semantic boundary in step three.

[0080] S4.2, supervised learning is performed on each subtask using the cross-entropy loss function, as shown in the following formula:

[0081]

[0082] Among them, L S L B and L E y represents the loss for complete building segmentation, the loss for more stable semantic subjects in strong semantic regions, and the loss for uncertain semantic boundaries in weak semantic regions, respectively; N represents the number of pixels in the image; y i ∈{0,1} represents whether pixel i belongs to a building, where i is the index; p i ∈[0,1] is the predicted probability of pixel i.

[0083] S4.3, using a multi-task joint loss function to optimize the network and improve building extraction performance, can be expressed by the following formula:

[0084] L total =λ1·L S +λ2·L B +λ3·L E

[0085] Among them, L total λ1, λ2, and λ3 are the total loss for multi-task supervision, and λ1, λ2, and λ3 are the loss weights for each task, which are set to 1, 1, and 20 in this method.

[0086] Through steps one through four, we finally obtained building extraction results with high accuracy and low false alarm rate.

[0087] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for extracting buildings from optical remote sensing images with multi-layer feature decoupling, characterized in that, include: Step 1: Extract multi-scale features from the optical remote sensing image to obtain multi-layer feature maps at different scales; Step 2: Perform feature decomposition on the feature map obtained in Step 1. Specifically, using adjacent shallow feature maps as references, calculate the offset of each feature in adjacent deep feature maps using the feature flow field; correct the deep feature map based on the offset to obtain more stable semantic main features in the strong semantic region representing the main part of the building; then use subtraction to obtain uncertain semantic boundary features in the weak semantic region representing the edge part of the building in the deep feature map. Step 3: Fuse the multiple sets of more stable semantic subject features and uncertain semantic boundary features obtained in Step 2 layer by layer to obtain multiple pixel-level prediction maps; among them, strong semantic regions are fused from deep to shallow, and weak semantic regions are fused from shallow to deep. Step four: Using a multi-task supervision method, supervised learning is performed on the multiple pixel-level prediction maps obtained in step three based on the multi-task joint loss function to complete the building extraction.

2. The method as described in claim 1, characterized in that, In step one, AlexNet, VGGNet, ResNet, ResNeXt, or DenseNet networks are used to extract features from optical remote sensing images.

3. The method as described in claim 1, characterized in that, Step two specifically includes the following sub-steps: S2.1 Feature preprocessing: Let F and F′ be the relatively shallow and relatively deep features of two adjacent feature layers; transform the deep feature to the same size as the shallow feature F, denoted as F″; S2.2, Generate the characteristic flow field: Concatenate the deep feature F″ obtained in S2.1 with the shallow feature F and perform convolution to obtain the flow field δ; S2.3, Generate more stable semantic subject features in the strong semantic region: Based on the flow field δ, obtain the offset corresponding to each feature in the deep feature map F″; correct the deep feature map F″ based on the offset to obtain more stable semantic subject features F′ constituting the strong semantic region. MainBody ; S2.4, Generate uncertain semantic boundary features in weak semantic regions: Using a subtraction operation, obtain uncertain semantic boundary features F′ from the deep feature map F″. UncertainBoundary ; S2.5, combine adjacent feature maps in pairs in the multi-layer feature maps from step one, and repeat S2.1 to S2.4 to generate N-1 sets of more stable semantic subjects and uncertain semantic boundaries; where N is the total number of layers of feature maps obtained in step one.

4. The method as described in claim 3, characterized in that, In S2.1, a 1×1 convolution and upsampling operation are used to transform the deep features to the same size as the shallow features.

5. The method as described in claim 1 or 3, characterized in that, Feature flow distortion is used to correct deep feature maps.

6. The method as described in claim 1, characterized in that, In step three, when the features of adjacent feature layers are fused, an attention mechanism is used to assign weights to the features of adjacent feature layers.

7. The method as described in claim 6, characterized in that, The Sigmoid function is used as a gate to assign weights to features in adjacent feature layers.

8. The method as described in claim 7, characterized in that, In step three, the fusion is performed in the following manner: in, and These represent the feature layers after applying the attention mechanism, Z represents the feature fusion result, and G represents the feature layers after applying the attention mechanism. X and G Y These represent the attention coefficients obtained from the selection gate, G x =Sigmoid(conv 1×1 (X)) G y =Sigmoid(conv 1×1 (Y)) Where Sigmoid(·) represents performing the Sigmoid operation, and conv 1×1 (·) represents a 1×1 convolution.

9. The method as described in claim 1, characterized in that, In step four, the cross-entropy loss function is used to perform supervised learning for each subtask.

Images