Remote sensing image building extraction method and system based on deep feature fusion network

Abstract

This invention relates to the field of intelligent interpretation of remote sensing images, and more particularly to a method and system for extracting buildings from remote sensing images based on a deep feature fusion network. The method first uses a depth information extraction module to extract depth information from remote sensing images. A deep feature fusion module then fuses this depth information with the corresponding remote sensing image to produce a fused feature map. Finally, the fused feature map is input into a U-Net network for building extraction from remote sensing images. This method improves the accuracy of building extraction using a model through depth information extraction and deep feature fusion. Furthermore, the method eliminates the need for additional multi-source registered remote sensing image data, thus addressing the limitations of traditional multi-source data fusion strategies, which typically require significant time and effort to acquire.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent interpretation of remote sensing images, and in particular to a method and system for extracting buildings from remote sensing images based on a deep feature fusion network. Background Art

[0002] Building extraction from remote sensing imagery is a challenging task in the field of intelligent remote sensing image interpretation. Early building extraction methods primarily relied on manually designed features, followed by edge detection, corner matching, region segmentation, and morphological shadow building index methods. However, these methods were complex, limited in applicability, low in accuracy, and costly. With the development of machine learning, data-driven machine learning methods combined with shallow features have been applied to building extraction. However, the poor expressive power of shallow features and the heavy reliance on prior knowledge of machine learning methods have resulted in low accuracy, high cost, poor generalization, and unsuitability for diverse scenarios. In recent years, deep learning semantic segmentation networks have been widely used for building extraction from remote sensing images due to their ability to automatically learn and extract building features from data. Researchers have improved the network structure to enhance building extraction accuracy. Furthermore, multi-source data fusion is a common strategy for improving building extraction accuracy. Multi-source remote sensing image fusion improves building extraction accuracy by providing additional information. However, acquiring registered multi-source remote sensing data is time-consuming and labor-intensive, resulting in high application costs. Summary of the Invention

[0003] The present invention aims to solve the problems existing in the prior art and proposes a remote sensing image building extraction method and system based on a deep feature fusion network. The method improves the accuracy of building extraction by the model through depth information extraction and deep feature fusion. In addition, it does not require additional multi-source registered remote sensing image data, which solves the limitations of traditional multi-source data fusion strategies. These additional multi-source registered remote sensing image data usually require a lot of time and effort to obtain.

[0004] In order to achieve the above purpose, the technical solutions adopted are:

[0005] A remote sensing image building extraction method based on a deep feature fusion network, comprising:

[0006] First, the depth information extraction module is used to extract the depth information of the remote sensing image;

[0007] Then, the depth feature fusion module is used to fuse the depth information and the corresponding remote sensing image to obtain the fused feature map;

[0008] Finally, the fused feature map is input into the U-Net network to extract buildings from remote sensing images.

[0009] According to the remote sensing image building extraction method based on the deep feature fusion network of the present invention, further, the depth information extraction module includes an encoder, using ViT as the encoder, ViT first evenly divides the image into 16×16 sub-image blocks, and then obtains the transformed visual token through the linear projection layer. ViT encodes the spatial position of the visual token into a feature vector and fuses it with the input visual token. The fused visual token outputs the extracted features after passing through several blocks.

[0010] According to the remote sensing image building extraction method based on the deep feature fusion network of the present invention, the ViT block further includes two linear projection layers, a multi-head self-attention layer, a multi-layer perceptron layer and two residual connections. The ViT is formally described as:

[0011]

[0012] z l '=MSA(LN(z l-1 ))+z l-1 ,l=1,2,…,L

[0013] z l =MLP(LN(z l '))+z′,l=1,2,…,L

[0014] Among them, x p is the segmented sub-image block, E is the position encoding feature vector, MSA is the multi-head self-attention layer, LN is the linear projection layer, MLP is the multi-layer perceptron layer, and z′ is the feature map extracted by multi-head self-attention.

[0015] According to the remote sensing image building extraction method based on the deep feature fusion network of the present invention, the depth information extraction module further includes a decoder, which includes two convolutional layers, two deconvolutional layers and an interpolation layer, wherein the convolutional layer is used to transform the feature map, the deconvolution layer is used to upsample the feature map, and the interpolation layer is used to upsample the output depth map to the original image size.

[0016] According to the remote sensing image building extraction method based on the deep feature fusion network of the present invention, the loss function of the depth information extraction module is further defined as follows:

[0017] L=L l +L u +L feat

[0018] Among them, L l 、L u and L featThey are supervised loss, unsupervised loss and feature consistency constraint.

[0019] According to the remote sensing image building extraction method based on deep feature fusion network of the present invention, further, the supervision loss L l The calculation formula is as follows:

[0020]

[0021] Among them, H and W are the height and width of the depth map respectively. and are the scaling and offset of the predicted and true results, respectively.

[0022] According to the remote sensing image building extraction method based on the deep feature fusion network of the present invention, CutMix is ​​further used to perform data enhancement on the unlabeled image when calculating the unsupervised loss. CutMix randomly selects a pair of images I a and I b , by interpolating the two images in proportion to mix the samples to obtain the enhanced image I ab ; The unsupervised loss is calculated in two parts based on a binary mask.

[0023] According to the remote sensing image building extraction method based on the deep feature fusion network of the present invention, further, the calculation formula of the feature consistency constraint is as follows:

[0024]

[0025] Among them, H f and W f are the height and width of the feature map, respectively, f i Represents the feature map output by the encoder to be trained, f i ′ represents the feature map output by the DINOv2 encoder.

[0026] According to the remote sensing image building extraction method based on the deep feature fusion network of the present invention, further, the deep feature fusion module includes two lightweight convolutional layers. First, the depth map is passed through the convolution layer to obtain a feature map. Similarly, the original remote sensing image is passed through the convolution layer to obtain a feature map. Then, the two obtained feature maps are spliced ​​or added to obtain a fused feature map.

[0027] Furthermore, the present invention also provides a remote sensing image building extraction system based on a deep feature fusion network, comprising a depth information extraction module, a deep feature fusion module and a building segmentation module, wherein:

[0028] Depth information extraction module, used to extract depth information from remote sensing images;

[0029] Deep feature fusion module, used to fuse depth information and corresponding remote sensing images to obtain fused feature maps;

[0030] The building segmentation module is used to input the fused feature map into the U-Net network to extract buildings from remote sensing images.

[0031] The beneficial effects achieved by adopting the above technical solution are:

[0032] The proposed method for extracting buildings from remote sensing images based on a deep feature fusion network utilizes a depth information extraction module to directly extract depth information from remote sensing images to obtain multi-source remote sensing data, eliminating the need for registration and resolving issues such as complex image registration. A deep feature fusion module is also designed to fully integrate depth information with the corresponding remote sensing image, alleviating issues such as information interference caused by the direct fusion of data from different modalities. The deep feature fusion network (DFF-Net) used in this invention improves building extraction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings of the embodiments of the present invention. The drawings are only used to illustrate some embodiments of the present invention, but not to limit all embodiments of the present invention thereto.

[0034] Figure 1 1 is an overall framework diagram of a remote sensing image building extraction method based on a deep feature fusion network according to an embodiment of the present invention;

[0035] Figure 2 is a structural diagram of a depth information extraction module according to an embodiment of the present invention;

[0036] Figure 3 is a structural diagram of a deep feature fusion module according to an embodiment of the present invention;

[0037] Figure 4 1 is an example diagram of the results of extracting buildings in the Massachusetts building dataset using DFF-Net of an embodiment of the present invention and a comparative method. DETAILED DESCRIPTION

[0038] The following will be combined with the accompanying drawings of specific embodiments of the present invention to clearly and completely describe the exemplary embodiments of the present invention. Unless otherwise defined, technical or scientific terms used in the present invention should be given the common meanings understood by people with ordinary skills in the relevant field.

[0039] like Figure 1 As shown, this embodiment discloses a remote sensing image building extraction method based on a deep feature fusion network, comprising the following steps:

[0040] Step S101: extracting depth information of a remote sensing image using a depth information extraction module.

[0041] The structure of the depth information extraction module is as follows Figure 2 As shown, it consists of an encoder and a decoder. This embodiment uses the classic ViT as the encoder. ViT first evenly divides the image into 16×16 sub-image blocks, and then obtains the transformed visual token through a linear projection layer. In order to take into account spatial information, ViT encodes the spatial position of the visual token into a feature vector and fuses it with the input visual token. The fused visual token outputs the extracted features after several blocks. The block of ViT includes two linear projection layers, a multi-head self-attention layer, a multi-layer perceptron layer and two residual connections. ViT is formally described as:

[0042]

[0043] z l '=MSA(LN(z l-1 ))+z l-1 ,l=1,2,…,L

[0044] z l =MLP(LN(z l '))+z′,l=1,2,…,L

[0045] Among them, x is the sub-image block after segmentation, E is the position encoding feature vector, MSA is the multi-head self-attention layer, LN is the linear projection layer, MLP is the multi-layer perceptron layer, and z′ is the feature map extracted by multi-head self-attention. The size of the input image is H×W×C, x p The size is N×(P×C), C is the number of channels, P is the patch size, and N=H×W / P.

[0046] Different versions of ViT are formed according to different parameters such as the number of block layers, hidden layer size, and the number of self-attention heads. The common parameter configurations of ViT-Small, ViT-Base, and ViT-Large are shown in Table 1.

[0047] Table 1 Specific parameters of the three versions of ViT

[0048]

[0049] The decoder consists of two convolutional layers, two deconvolutional layers, and an interpolation layer. Its final output is a depth map with the same size as the original image. The decoder parameter configuration is shown in Table 2. The convolutional layer is used to transform the feature map, the deconvolution layer is used to upsample the feature map, and the interpolation layer is used to upsample the output depth map to the original image size.

[0050] Table 2 Decoder structure

[0051]

[0052] The amount of labeled data for depth estimation from monocular images is relatively small. Compared to limited labeled data, massive open-source unlabeled data is easier to obtain. To improve the accuracy of depth estimation, feature consistency constraints are introduced. The loss function of the depth information extraction module is defined as follows:

[0053] L=L l +L u +L feat

[0054] Among them, L l 、L u and L feat They are supervised loss, unsupervised loss and feature consistency constraint.

[0055] Supervision loss L l The calculation formula is as follows:

[0056]

[0057] Among them, H and W are the height and width of the depth map respectively. and are the scale and offset of the predicted and true results, t(d) and s(d) are the parameters used to align the predicted and true results, and d i Represents the true depth value, and d represents the depth value in the disparity space, ranging from 0 to 1.

[0058] The present invention first uses labeled data to train a teacher network T based on a supervised loss function. Then T is applied to unlabeled images to obtain corresponding pseudo labels. On this basis, the student network S is trained using a combination of labeled and unlabeled data. Directly using pseudo labels to calculate the unsupervised loss has little effect on improving model performance. To address this issue, CutMix is ​​used to perform data augmentation on unlabeled images when calculating the unsupervised loss. By introducing CutMix to set a more challenging optimization target, the model performance can be improved. CutMix randomly selects a pair of images I a and I b , by interpolating the two images in proportion to mix the samples to obtain the enhanced image I ab for:

[0059] I ab =I a ⊙M+I b δ(1-M)

[0060] Where M represents a binary mask containing a rectangular area, and the rectangular area is marked as 1. The calculation of the unsupervised loss is divided into two parts according to the mask M, as follows:

[0061]

[0062] Among them, H and W are the height and width of the depth map respectively, H M 、W M are the height and width of the rectangular area in the mask M, T is the teacher network trained by labeled data, S is the student network to be trained, and ρ represents the affine invariant mean absolute error loss.

[0063] DINOv2 performs exceptionally well in tasks such as image classification, semantic segmentation, and object detection. To this end, during training, a feature consistency loss is used to guide the model's output depth information to remain consistent within the same object region. The feature consistency constraint is calculated as follows:

[0064]

[0065] Among them, H f and W f are the height and width of the feature map, respectively, f i Represents the feature map output by the encoder to be trained, f i ′ represents the feature map output by the DINOv2 encoder.

[0066] Compared to traditional deep learning models, the generalization capabilities of large language and vision models primarily stem from their greater number of parameters and a wider range of data. Therefore, to ensure the generalization of the deep information extraction module, we constructed training data containing 1.5M labeled images and 62M unlabeled images based on open-source data.

[0067] Step S102: Using a depth feature fusion module, the depth information and the corresponding remote sensing image are fused to obtain a fused feature map.

[0068] The structure of the deep feature fusion module is as follows Figure 3As shown in the figure. This module consists of two lightweight convolutional layers. Due to the differences in data from different modalities, direct fusion will cause information interference and thus fail to improve the accuracy of building extraction. Therefore, a deep feature fusion module is proposed and used to fully fuse depth information and the corresponding original remote sensing image. First, the depth map is processed through a 3×3 convolutional layer to obtain 64 512×512 feature maps. Similarly, the original remote sensing image is processed through a 3×3 convolutional layer to obtain 64 512×512 feature maps. Then, in order to fully retain the depth information features, the two feature maps are spliced ​​or added to obtain 128 512×512 fused feature maps or 64 512×512 fused feature maps. Splicing can increase the depth features describing the image itself. Addition can increase the amount of information under the depth features describing the image. Finally, the fused feature map is input into the U-Net network for subsequent building extraction. The whole process can be summarized by the following formula:

[0069] F i =[Leaky_relu(Conv(R),Leaky_relu(Conv(D))]

[0070] F i =<Leaky_relu(Conv(R),Leaky_relu(Conv(D))>

[0071] Among them, D represents the depth image, R is the original remote sensing image, and F i is the fused feature map, [, ] represents addition, and <, > represents concatenation.

[0072] Step S103: Input the fused feature map into the U-Net network to extract buildings from the remote sensing image.

[0073] This embodiment uses the U-Net training fusion feature map to obtain the final building segmentation map. Figure 1As shown, the U-Net has strong representational capabilities and reasonable generalization. Therefore, it was chosen as the backbone for extracting basic building features. The fused feature maps undergo four layers of downsampling and upsampling to obtain the final segmentation map. During the downsampling and upsampling processes, convolution operations and nearest neighbor interpolation are used, respectively, to obtain feature maps of the same resolution. The 512×512 feature map is first downsampled to 256×256, 128×128, 64×64, and 32×32 feature maps, which are then sequentially upsampled to 64×64, 128×128, 256×256, and 512×512 feature maps. This process fully exploits the rich details in high-resolution images. Skip connections are used to connect low-level and high-level features, achieving robustness and generalization in semantic segmentation. Similarly, as with a fully convolutional network (FCN), the fused feature maps undergo regression calculations in a 1×1 convolutional layer to obtain the predicted value for each pixel, resulting in the final building prediction map.

[0074] Corresponding to the above method, this embodiment also proposes a remote sensing image building extraction system based on a deep feature fusion network, which includes a depth information extraction module, a deep feature fusion module and a building segmentation module, wherein:

[0075] The depth information extraction module is used to extract the depth information of remote sensing images.

[0076] The deep feature fusion module is used to fuse the depth information and the corresponding remote sensing image to obtain the fused feature map.

[0077] The building segmentation module is used to input the fused feature map into the U-Net network to extract buildings from remote sensing images.

[0078] In order to verify the feasibility and effectiveness of the present invention, the present invention is further described in detail below with reference to experiments.

[0079] We conducted building extraction experiments on the public Massachusetts dataset. Four metrics were selected to evaluate the performance of DFF-Net on both datasets: precision, accuracy, F1 score, and Intersection over Union (IoU). These metrics are defined as follows (Pre. and Re. represent recall and precision, respectively):

[0080]

[0081] Among them, TP is the true positive example, which indicates the number of samples correctly classified as the positive class; FP is the false positive example, which indicates the number of samples incorrectly classified as the positive class; FN is the false negative example, which indicates the number of samples incorrectly classified as the negative class.

[0082] Table 3 Accuracy results of DDF-Net and comparison methods on the WHU dataset

[0083]

[0084] Figure 4 In the figure, a, b, c, d, and e represent the predicted results of comparison methods 1, 2, 3, and 4, and the method of the present invention, respectively. Among all the comparison methods, the method of the present invention achieved the highest accuracy in all four evaluation indicators. After adding the depth information extraction module and the depth feature fusion module, the accuracy of the present invention was significantly improved. Comparison methods 1, 2, 3, and 4 all missed or over-detected some small buildings and large buildings. The method of the present invention successfully identified relatively complete large buildings and small buildings that were easily missed.

[0085] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0086] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0087] The units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementation is not considered to be beyond the scope of the present invention.

[0088] Those skilled in the art will appreciate that all or part of the steps in the above method can be performed by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a magnetic disk, or an optical disk. Alternatively, all or part of the steps in the above embodiment can be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or software functional modules. The present invention is not limited to any specific combination of hardware and software.

[0089] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A remote sensing image building extraction method based on deep feature fusion network, characterized in that: Include: First, the depth information extraction module is used to extract the depth information of the remote sensing image. The depth information extraction module includes an encoder, using ViT as the encoder. ViT first evenly divides the image into 16×16 sub-image blocks, and then obtains the transformed visual token through a linear projection layer. ViT encodes the spatial position of the visual token into a feature vector and fuses it with the input visual token. The fused visual token is output as the extracted features after passing through several blocks. The depth information extraction module also includes a decoder, which includes two convolutional layers, two deconvolutional layers, and an interpolation layer, wherein the convolutional layer is used to transform the feature map, the deconvolution layer is used to upsample the feature map, and the interpolation layer is used to upsample the output depth map to the original image size; Then, a deep feature fusion module is used to fuse the depth information and the corresponding remote sensing image to obtain a fused feature map. The deep feature fusion module includes two lightweight convolutional layers. First, the depth map is passed through the convolutional layer to obtain a feature map. Similarly, the original remote sensing image is passed through the convolutional layer to obtain a feature map. The two feature maps are then spliced ​​or added together to obtain a fused feature map. Finally, the fused feature map is input into the U-Net network to extract buildings from remote sensing images.

2. The remote sensing image building extraction method based on deep feature fusion network according to claim 1 is characterized in that: The ViT block consists of two linear projection layers, a multi-head self-attention layer, a multi-layer perceptron layer, and two residual connections. ViT is formally described as: z l ‘=MSA(LN(z l-1 ))+z l-1 ,l=1,2,…,L With l =MLP(LN(z l '))+z′,l=1,2,…,L Among them, x p is the segmented sub-image block, E is the position encoding feature vector, MSA is the multi-head self-attention layer, LN is the linear projection layer, MLP is the multi-layer perceptron layer, and z′ is the feature map extracted by multi-head self-attention.

3. The remote sensing image building extraction method based on deep feature fusion network according to claim 1 is characterized in that: The loss function of the depth information extraction module is defined as follows: L=L l +L u +L feat Among them, L l , L u and L feat They are supervised loss, unsupervised loss and feature consistency constraint.

4. The remote sensing image building extraction method based on deep feature fusion network according to claim 3 is characterized in that: Supervision loss L l The calculation formula is as follows: Among them, H and W are the height and width of the depth map respectively. and are the scaling and offset of the predicted and true results, respectively.

5. The remote sensing image building extraction method based on deep feature fusion network according to claim 3 is characterized in that: When calculating the unsupervised loss, CutMix is ​​used to perform data augmentation on the unlabeled image. CutMix randomly selects a pair of images I a and I b , by interpolating the two images in proportion to mix the samples to obtain the enhanced image I ab ; The unsupervised loss is calculated in two parts based on a binary mask.

6. The remote sensing image building extraction method based on deep feature fusion network according to claim 3 is characterized in that: The calculation formula of feature consistency constraint is as follows: Among them, H f and W f are the height and width of the feature map, respectively, f i Represents the feature map output by the encoder to be trained, f i ′ represents the feature map output by the DINOv2 encoder.

7. A remote sensing image building extraction system based on deep feature fusion network, characterized in that: The system is used to implement the remote sensing image building extraction method based on the deep feature fusion network as described in any one of claims 1 to 6, comprising a depth information extraction module, a deep feature fusion module and a building segmentation module, wherein: Depth information extraction module, used to extract depth information from remote sensing images; Deep feature fusion module, used to fuse depth information and corresponding remote sensing images to obtain fused feature maps; The building segmentation module is used to input the fused feature map into the U-Net network to extract buildings from remote sensing images.

Images