Building contour extraction method based on double-encoder STR-Net

Abstract

The invention provides a building contour extraction method based on double encoders STR-Net, which belongs to the field of image processing, and comprises the following steps: acquiring a building remote sensing image, preprocessing the building remote sensing image to obtain an image block, determining the image block as input, and respectively sending the image block to a cross-scale encoder and a fine-grained encoder; global features in the image blocks are extracted according to a cross-scale encoder, local features in the image blocks are extracted based on a fine-grained encoder, and the cross-scale encoder and the fine-grained encoder work in parallel; integrating the global features and the local features of each feature layer based on a branch fusion module to obtain fusion features; and splicing the fusion features of each feature layer and the fusion features of the feature layer of the previous stage layer by layer according to a decoder, transmitting the fusion features to the feature layer of the previous stage according to convolution operation, and outputting a building profile diagram based on multi-layer decoding. According to the invention, the accuracy of building contour extraction can be improved.

Description

Technical Field

[0001] The present invention relates to image processing technology, and in particular to a building outline extraction method based on a dual encoder STR-Net. Background Art

[0002] With the rapid development of earth science observation technology and the rapid progress in aerospace, geographic information systems and sensor technology, the amount of high-resolution remote sensing image data has increased dramatically. As the main element of urban landscape, buildings are often the main objects of urban geospatial analysis. The operation and analysis of building complexes in cities is the basic task of applying cities as information carriers to all walks of life. Effective segmentation of buildings from remote sensing images is crucial for applications such as urban planning, damage detection and land use management. High-resolution remote sensing images contain rich geographic information, which provides a basis for related research tasks. Therefore, accurately extracting building outlines from high-resolution remote sensing images has always been an important research direction.

[0003] Methods based on deep learning have shown higher efficiency in the field of building contour extraction, and are particularly suitable for dealing with complex and diverse building types in urban and rural areas. Deep learning has become the mainstream research direction and application method for information extraction and image segmentation. This method has solved the problem of missing buildings when they are small in size and densely distributed to a certain extent, and improved the accuracy of building segmentation in remote sensing images. However, there is still the problem that deep features are not effectively utilized, which leads to the problem of blurred building boundaries not being well solved. Vision Transformer (ViT) effectively models the deep features of images through a multi-head attention mechanism. Although it can extract building contours more accurately and performs well in extracting global information, it also ignores key spatial detail contexts. Zheng et al. applied Transformer to semantic segmentation. Although they achieved excellent performance, they did not utilize information at multiple scales. The above problems all reduce the accuracy of building contour extraction.

[0004] Therefore, how to utilize deep features, enhance the processing capabilities of spatial detail context, and effectively utilize multi-scale information to improve the accuracy of building contour extraction has become a problem that needs to be solved urgently. Summary of the invention

[0005] The present invention provides a building outline extraction method based on a dual encoder STR-Net, which can make full use of deep features, enhance the processing capability of spatial detail context, and effectively utilize multi-scale information to improve the accuracy of building outline extraction.

[0006] A first aspect of the present invention provides a building outline extraction method based on a dual encoder STR-Net, comprising: Acquire a building remote sensing image, preprocess the building remote sensing image to obtain an image block, determine the image block as input, and send it to a cross-scale encoder and a fine-grained encoder respectively; Extracting global features in the image block according to the cross-scale encoder, and extracting local features in the image block based on the fine-grained encoder, wherein the cross-scale encoder and the fine-grained encoder work in parallel; Integrate the global features and the local features of each feature layer based on a branch fusion module to obtain a fusion feature; According to the decoder, the fusion features of each feature layer are concatenated layer by layer with the fusion features of the feature layer of the previous stage, and are transferred to the feature layer of the previous stage according to the convolution operation. The building outline map is output based on multi-layer decoding.

[0007] Optionally, in a possible implementation manner of the first aspect, acquiring a building remote sensing image, preprocessing the building remote sensing image to obtain an image block, determining the image block as input, and sending the image block to a cross-scale encoder and a fine-grained encoder respectively, including: Segmenting the building remote sensing image into a plurality of non-overlapping image blocks, determining each of the image blocks as an independent data item, and constructing a feature vector of each of the image blocks by concatenating the original RGB of all pixels in the image block; According to the linear connection layer, the feature vector of the image block is projected into the C-dimensional feature space, and the processed image block is determined as the input and sent to the cross-scale encoder and the fine-grained encoder respectively.

[0008] Optionally, in a possible implementation manner of the first aspect, extracting global features in the image block according to the cross-scale encoder, extracting local features in the image block based on the fine-grained encoder, and the cross-scale encoder and the fine-grained encoder working in parallel include: A cross-scale encoder is constructed based on a deep Swin transformer module, the image block is processed layer by layer according to the W-MSA unit and the SW-MSA unit, local and global features are extracted and integrated layer by layer, and global features are output based on the top layer, wherein each Swin transformer layer is composed of a W-MSA unit and a SW-MSA unit connected; A fine-grained encoder is constructed based on the ResNeXt module. The input feature matrix is ​​reduced in dimension based on the ResNeXt module. The reduced features are deeply extracted and refined based on group convolution. The number of channels is restored based on dimensionality increase. The processed features are fused with the short-circuit connection output to output local features.

[0009] Optionally, in a possible implementation manner of the first aspect, a cross-scale encoder is constructed based on a deep Swin transformer module, the image block is processed layer by layer according to the W-MSA unit and the SW-MSA unit, local and global features are extracted and integrated layer by layer, and global features are output based on the top layer, including: The operation process of the cross-scale encoder is as follows: For The feature layer is the input feature of the previous feature layer. As input, after LN processing, it is sent to the W-MSA unit for local feature extraction and integration to obtain the intermediate feature ,Will and Perform residual connection, and then process it through MLP and LN to obtain new intermediate features ; Will As input, after LN processing, it is sent to the SW-MSA unit for feature extraction and integration under the shift window to obtain the intermediate feature ,Will and Perform residual connection, and then go through an MLP and LN process to get the Output features of the layer ; Among them, for the top layer, its output feature The calculation process of is the same as above, but it is no longer used as the input of the next layer; The expression of the cross-scale encoder is as follows: ; ; ; ; ; in, Indicates Layer feature layer, represents the intermediate features after W-MSA and LN processing, Indicates Layer feature layer cross-scale encoder output features, Representation layer normalization, represents the self-attention mechanism under the conventional window configuration, Indicates The output features of the SW-Trans module, After SW-MSA and LN treatment and based on A further intermediate feature of Indicates Layer feature layer cross-scale encoder output features, Indicates Layer feature layer cross-scale encoder output features, Indicates The intermediate features of the layer feature layer, SW-MSA represents the window configuration scheme with shift operation, and SW-Trans module represents the Swin Transformer module.

[0010] Optionally, in a possible implementation of the first aspect, a dimensionality reduction process is performed on the input feature matrix based on a ResNeXt module, the features after dimensionality reduction are deeply extracted and refined according to grouped convolution, the number of channels is restored based on dimensionality increase, the processed features are fused with the short-circuit connection output, and local features are output, including: The input feature matrix is ​​reduced in dimension according to a 1×1 convolutional layer, which has a feature matrix with 256 channels as input and outputs a feature matrix with the number of channels reduced to 128; The reduced feature matrix is ​​processed based on a group convolution operation consisting of 32 independent 3×3 convolution kernels. Each 3×3 convolution kernel independently performs convolution operations on a local area of ​​the feature matrix to generate 32 intermediate feature matrices, each of which contains deep information extracted from the original features. The 32 intermediate feature matrices are merged and dimensionally upgraded according to another 1×1 convolutional layer. The convolutional layer combines the feature matrices of the 32 channels into one and restores the number of channels to 256 to generate a dimensionally upgraded feature matrix. The feature matrix after dimension increase is fused and added with the output feature matrix from the short-circuit connection to generate the local features of the final output.

[0011] Optionally, in a possible implementation manner of the first aspect, integrating the global features and the local features of each feature layer based on a branch fusion module to obtain a fused feature includes: Extract global features from the output of the cross-scale encoder according to the branch fusion module, process it through a 1×1 convolution layer, apply global average pooling in the channel direction to obtain a feature vector, pass the feature vector through two fully connected layers and ReLU and sigmoid activation functions, and obtain a channel attention weight containing channel-dependent features; Perform matrix multiplication and fusion on the channel attention weights and feature maps to generate the output of the cross-scale encoder; The output of the cross-scale encoder and the local features of the fine-grained encoder are fused according to the branch fusion module to generate a fused feature.

[0012] Optionally, in a possible implementation manner of the first aspect, extracting global features from the output of the cross-scale encoder according to the branch fusion module, processing through a 1×1 convolution layer, applying global average pooling in the channel direction to obtain a feature vector, and passing the feature vector through two fully connected layers and ReLU and sigmoid activation functions to obtain a channel attention weight containing channel-dependent features, including: Based on the feature map output from the cross-scale encoder by the branch fusion module The global features are obtained from the network and processed by a 1×1 convolution layer to obtain the processed feature map. , apply global average pooling in the channel direction to obtain the feature vector ; Eigenvector The calculation formula is as follows: in, represents the feature vector obtained after the global average pooling operation, represents the height of the feature map, represents the width of the feature map, Indicates the index in the vertical direction. Represents the horizontal index. Indicates the number of channels of the feature map, Indicates the new number of channels, represents the feature map output by the cross-scale encoder, Represents the processed feature map; The feature vector The channel attention weight containing channel-dependent features is obtained by sequentially passing through the fully connected layer, ReLU activation function, fully connected layer, and sigmoid activation function; The calculation formula of channel attention weight is as follows: in, represents the channel attention weight, represents the sigmoid activation function, represents the ReLU activation function, express, Represents the weight coefficient of the fully connected layer.

[0013] Optionally, in a possible implementation manner of the first aspect, fusing the output of the cross-scale encoder and the local features of the fine-grained encoder according to the branch fusion module to generate a fused feature includes: The specific formula for generating the fusion feature is as follows: ; ； in, represents the output of the cross-scale encoder, represents the local features of the fine-grained encoder, Represents fusion features.

[0014] Optionally, in a possible implementation of the first aspect, the method further includes: The Dice loss is calculated according to the Dice loss function, the cross entropy loss is calculated based on the cross entropy loss function, and the Dice loss and the cross entropy loss are weighted and summed according to the configuration ratio to obtain a combined loss function.

[0015] Optionally, in a possible implementation manner of the first aspect, a Dice loss is calculated according to a Dice loss function, a cross entropy loss is calculated based on a cross entropy loss function, and a weighted sum is performed on the Dice loss and the cross entropy loss according to a configuration ratio to obtain a combined loss function, including: The Dice loss calculation formula is as follows: in, represents the Dice loss, express and The intersection between express The number of elements, express The number of elements; The cross entropy loss calculation formula is as follows: in, represents the number of categories, represents a one-hot vector, Indicates that the predicted sample belongs to probability; The calculation formula of the combined loss function is as follows: in, represents the combined loss function.

[0016] A second aspect of the present invention provides a building outline extraction system based on a dual encoder STR-Net, comprising: A processing module is used to obtain a building remote sensing image, pre-process the building remote sensing image to obtain an image block, determine the image block as input, and send it to a cross-scale encoder and a fine-grained encoder respectively; An extraction module, configured to extract global features in the image block according to the cross-scale encoder, and extract local features in the image block based on the fine-grained encoder, wherein the cross-scale encoder and the fine-grained encoder work in parallel; A fusion module, used for integrating the global features and the local features of each feature layer based on the branch fusion module to obtain a fusion feature; The output module is used to splice the fusion features of each feature layer with the fusion features of the feature layer of the previous stage layer by layer according to the decoder, and transfer them to the feature layer of the previous stage according to the convolution operation, and output the building outline map based on multi-layer decoding.

[0017] The beneficial effect of the present invention is that the present invention designs a dual encoder structure of a cross-scale encoder based on a deep Swin transformer and a fine-grained encoder based on ResNeXt, and the two work in parallel to extract global context information and local detailed features from the image, thereby improving the understanding of the image content and the segmentation accuracy; at the same time, a branch fusion module is designed, which fuses global features and local features at each stage of the encoding process so as to extract discriminative features from remote sensing images and effectively distinguish highly similar ground objects; the building contour extraction method of the present invention achieves performance improvement on three building data sets, proving the effectiveness of the building contour extraction method of the present invention for building contour extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of a flow chart of a building outline extraction method based on a dual encoder STR-Net provided in an embodiment of the present invention; Figure 2 An overall architecture diagram of a dual encoder STR-Net provided in an embodiment of the present invention; Figure 3 A schematic diagram of a deep Swin converter structure provided by an embodiment of the present invention; Figure 4 A schematic diagram of the structure of a branch fusion module provided in an embodiment of the present invention; Figure 5 A graphical display diagram of evaluation indicators of different models on the WHU dataset provided by an embodiment of the present invention; Figure 6 A graphical display of evaluation indicators of different models on the INRIA dataset provided by an embodiment of the present invention; Figure 7 A graphical representation of evaluation indicators of different models on a Massachusetts dataset provided by an embodiment of the present invention; Figure 8 A schematic structural diagram of a building outline extraction system based on a dual encoder STR-Net provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] See also Figure 1 , is a flow chart of a building outline extraction method based on a dual encoder STR-Net provided in an embodiment of the present invention, Figure 1 The execution subject of the method shown may be a software and / or hardware device. The execution subject of the present application may include but is not limited to at least one of the following: user equipment, network equipment, etc. Among them, user equipment may include but is not limited to computers, smart phones, personal digital assistants (PDA) and the electronic devices mentioned above. Network equipment may include but is not limited to a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of computers or network servers, wherein cloud computing is a type of distributed computing, a super virtual computer composed of a group of loosely coupled computers. This embodiment does not limit this. It includes steps S1 to S4, as follows: S1, acquiring a building remote sensing image, preprocessing the building remote sensing image to obtain an image block, determining the image block as input, and sending the image block to a cross-scale encoder and a fine-grained encoder respectively.

[0021] See also Figure 2 , which is an overall architecture diagram of a dual encoder STR-Net provided in an embodiment of the present invention. A cross-scale encoder based on the Swin Transformer module and a fine-grained encoder based on the ResNeXt module are constructed. These two encoders work in parallel and can simultaneously extract the global context information and local fine features of the image, thereby improving the understanding of the image content and the segmentation accuracy. In addition, this embodiment also designs a branch fusion module, which fuses the global features extracted from the cross-scale encoder and the local features extracted from the fine-grained encoder at each stage of the encoding process, and then passes them to the next stage of the encoding process. In this way, the model can extract discriminative features from remote sensing images, thereby effectively distinguishing ground objects with high similarity.

[0022] Among them, building remote sensing images refer to remote sensing images containing buildings, image blocks refer to non-overlapping small blocks obtained after preprocessing building remote sensing images, cross-scale encoders are encoders constructed by multiple stacked Swin transformers, Swin transformers are Swin Transformer, a visual transformer model based on Transformer architecture, and fine-grained encoders refer to encoders constructed by ResNeXt modules. ResNeXt is a deep convolutional neural network architecture that combines and improves the advantages of VGG, ResNet, and Inception. ResNeXt adopts the "split-transform-aggregate" strategy in design, and each branch adopts the same topological structure, which makes the network structure simpler and reduces the number of hyperparameters. In addition, ResNeXt also introduces the concept of "cardinality", that is, the number of submodules in each module. By adjusting the size of the cardinality, the balance between model complexity and performance can be effectively controlled.

[0023] In some embodiments, step S1 may be implemented by the following steps: S11, segmenting the building remote sensing image into multiple non-overlapping image blocks, determining each image block as an independent data item, and constructing a feature vector of each image block by concatenating the original RGB of all pixels in the image block.

[0024] For the input remote sensing image, the network model uses the patch partitioning mechanism in the Vision Transformer (ViT), which first divides the RGB image into a series of non-overlapping image blocks, namely patches. In this process, each patch is regarded as an independent "token", that is, a data item, and its feature vector is constructed by concatenating the original RGB values ​​of all pixels in the patch. Among them, the data item refers to the processing unit.

[0025] In practical applications, the selected image size can be 512×512 pixel, and the patch size of 8×8 is used, so each patch contains 8×8=64 pixels, each pixel has 3 channels (R, G, B), so the feature dimension of each patch is 8×8×3=192.

[0026] This allows the model to process image information in a serialized manner, similar to words or characters in text.

[0027] S12, projecting the feature vector of the image block to the C-dimensional feature space according to the linear connection layer, determining the processed image block as input, and sending them to the cross-scale encoder and the fine-grained encoder respectively.

[0028] The linear connection layer is a neural network layer that can perform a linear transformation on the input feature vector. In this embodiment, the function is to project the feature vector of each patch to a new dimension C, which can be equal to 48. The processed image block can then be used as input and sent to the cross-scale encoder and the fine-grained encoder for in-and-out processing.

[0029] S2, extracting global features in the image block according to the cross-scale encoder, and extracting local features in the image block based on the fine-grained encoder, wherein the cross-scale encoder and the fine-grained encoder work in parallel.

[0030] Among them, global features refer to the results of feature extraction of the overall information of the image, including the global context information of the image; local features refer to the results of feature extraction of a local area of ​​the image, focusing on the details and local structures in the image.

[0031] Specifically, the cross-scale encoder is responsible for extracting global features, using a multi-scale feature extraction strategy to capture global context information in the image through convolution kernels and pooling operations. The fine-grained encoder is responsible for extracting local features, using convolution operations and network structures to capture local details and texture information in the image. The cross-scale encoder and the fine-grained encoder extract global features and local features in parallel, so that global context information and local fine features can be used simultaneously to improve the accuracy and precision of building contour extraction.

[0032] Based on the above embodiment, the specific implementation of step S2 may be: S21, build a cross-scale encoder based on the deep Swin transformer module, process the image block layer by layer according to the W-MSA unit and the SW-MSA unit, extract and integrate local and global features layer by layer, and output global features based on the top layer, wherein each Swin transformer layer is composed of a W-MSA unit and a SW-MSA unit connected.

[0033] See also Figure 3 , is a schematic diagram of a deep Swin converter structure provided by an embodiment of the present invention, from Figure 3 As can be seen in Figure 1, a W-MSA unit and a SW-MSA unit are connected to form a Swin transformer layer, and multiple Swin transformer layers are stacked to build a deep Swin transformer structure. W-MSA enables the Swin transformer to focus on local areas within the image, thereby reducing computational complexity, while capturing global dependencies by reorganizing local features. SW-MSA further enhances this capability by introducing a shift operation, which promotes interaction between adjacent windows and is very useful for processing spatial relationships in visual tasks.

[0034] In a deep cross-scale encoder, each feature layer can learn different features of the input data. Increasing the depth means that more levels of feature information can be captured, which can perform better when processing complex tasks. In addition, by increasing the depth, the cross-scale encoder can learn more details and contextual information about the data, which enables the model to have better generalization ability when processing new data. By incorporating the attention mechanism, the cross-scale encoder strikes a balance between efficiency and performance, making it very suitable for large-scale visual tasks, including remote sensing image footprint extraction.

[0035] Understandably, the standard Transformer module uses a multi-head self-attention sublayer (MSA) to calculate the global self-attention between all tokens, which leads to quadratic computational complexity of the number of tokens and limits its scope of application. In order to optimize modeling efficiency, Swin Transformer introduces a window self-attention mechanism (W-MSA) to replace the standard multi-head self-attention sublayer (MSA). In the design of W-MSA, two different window configuration strategies are adopted: one is the self-attention mechanism under the conventional window configuration, that is, the basic W-MSA; the other is a window configuration scheme with shift operation, that is, SW-MSA. These two variant structures together constitute the core components of Swin Transformer for efficient processing of visual information.

[0036] The operation process of the above cross-scale encoder is as follows: For The feature layer is the input feature of the previous feature layer. As input, after LN processing, it is sent to the W-MSA unit for local feature extraction and integration to obtain the intermediate feature ,Will and Perform residual connection, and then process it through MLP and LN to obtain new intermediate features ; Will As input, after LN processing, it is sent to the SW-MSA unit for feature extraction and integration under the shift window to obtain the intermediate feature ,Will and Perform residual connection, and then go through an MLP and LN process to get the Output features of the layer ; Among them, for the top layer, its output feature The calculation process of is the same as above, but it is no longer used as the input of the next layer; The expression of the cross-scale encoder is as follows: ; ; ; ; ; in, Indicates Layer feature layer, represents the intermediate features after W-MSA and LN processing, Indicates Layer feature layer cross-scale encoder output features, Representation layer normalization, represents the self-attention mechanism under the conventional window configuration, Indicates The output features of the SW-Trans module, After SW-MSA and LN treatment and based on A further intermediate feature of Indicates Layer feature layer cross-scale encoder output features, Indicates Layer feature layer cross-scale encoder output features, Indicates The intermediate features of the layer feature layer, SW-MSA represents the window configuration scheme with shift operation, and SW-Trans module represents the Swin Transformer module.

[0037] Through the above method, the cross-scale encoder can realize multi-level and cross-scale encoding of image features.

[0038] S22, builds a fine-grained encoder based on the ResNeXt module, performs dimensionality reduction processing on the input feature matrix based on the ResNeXt module, performs deep extraction and refinement on the reduced-dimensional features based on group convolution, restores the number of channels based on dimensionality increase processing, fuses the processed features with the short-circuit connection output, and outputs local features.

[0039] Specifically, the ResNeXt module will reduce the amount of information in the input feature matrix and the number of channels of the feature matrix through dimensionality reduction processing. Next, the ResNeXt module will divide the simplified feature matrix into multiple groups, each of which will perform independent and in-depth feature extraction. After grouping and extracting features, the ResNeXt module will recombine the results of all groups to restore the feature matrix to its original number of channels. Finally, the ResNeXt module will add the processed feature matrix to the original unchanged feature matrix and output the final feature matrix, which contains both the original information of the input feature matrix and the information obtained through in-depth extraction by grouping, and can better represent the characteristics of the image.

[0040] Based on the above embodiment, the specific implementation of step S22 may be: The input feature matrix is ​​reduced in dimensionality according to a 1×1 convolutional layer, which has a feature matrix with 256 channels as input and outputs a feature matrix with the number of channels reduced to 128.

[0041] The reduced feature matrix is ​​processed based on a grouped convolution operation consisting of 32 independent 3×3 convolution kernels. Each 3×3 convolution kernel independently performs a convolution operation on a local area of ​​the feature matrix to generate 32 intermediate feature matrices, each of which contains deep information extracted from the original features.

[0042] The 32 intermediate feature matrices are merged and dimensionally upgraded according to another 1×1 convolutional layer. The convolutional layer combines the feature matrices of 32 channels into 1 and restores the number of channels to 256 to generate a dimensionally upgraded feature matrix.

[0043] The feature matrix after dimension increase is fused and added with the output feature matrix from the short-circuit connection to generate the local features of the final output.

[0044] Specifically, when processing the input feature matrix, the ResNeXt module first uses a 1×1 convolution layer to reduce the channel dimension, reducing the original 256 channels to 128; then a group convolution operation containing 32 independent 3×3 convolution kernels is used to deeply extract and refine the features; then, another 1×1 convolution layer is used to increase the dimension of the feature matrix and restore its number of channels to 256; finally, the result calculated by this branch is fused and added with the output from the short-circuit connection to generate the final output feature, namely the local feature.

[0045] Through the above method, efficient extraction and fusion of image features can be achieved.

[0046] S3, integrating the global features and the local features of each feature layer based on a branch fusion module to obtain a fused feature.

[0047] The cross-scale encoder is divided into four feature extraction stages. The global features extracted in each stage are sent to the branch fusion module to be fused with the local features of the fine-grained encoder. The fused features are then input into the encoder of the next stage to complete the fusion of global and local features.

[0048] See also Figure 4 , which is a schematic diagram of the structure of a branch fusion module provided by an embodiment of the present invention. The main idea of ​​the branch fusion module is to extract channel-dependent features from the global features of the cross-scale encoder and then embed them into the local features obtained by the fine-grained encoder. In this way, the model can encode more global discriminative features to improve the segmentation accuracy of ground objects with high similarity in remote sensing images.

[0049] Based on the above embodiment, the specific implementation of step S3 may be: S31, extracting global features from the output of the cross-scale encoder according to the branch fusion module, processing it through a 1×1 convolution layer, applying global average pooling in the channel direction to obtain a feature vector, passing the feature vector through two fully connected layers and ReLU and sigmoid activation functions to obtain a channel attention weight containing channel-dependent features.

[0050] The 1×1 convolution layer is used to adjust the number of channels of the feature map. The global average pooling operation averages each feature map in the channel direction to obtain a vector with a length equal to the number of channels. The channel attention weight is calculated by passing the vector obtained by global average pooling through a fully connected layer, a ReLU activation function, another fully connected layer, and a sigmoid activation function. Among them, the fully connected layer is used to learn the complex relationship between channels and linearly transform the input vector through the weight coefficient. The ReLU activation function is used to introduce nonlinear characteristics to help the model learn more complex feature representations. The sigmoid activation function is used to map the output between 0 and 1, indicating the attention weight of each channel. These weights represent the importance of different channels for the current task.

[0051] Specifically, in some embodiments, step S31 includes the following steps: Based on the feature map output from the cross-scale encoder by the branch fusion module The global features are obtained from the network and processed by a 1×1 convolution layer to obtain the processed feature map. , apply global average pooling in the channel direction to obtain the feature vector ; Eigenvector The calculation formula is as follows: in, represents the feature vector obtained after the global average pooling operation, represents the height of the feature map, represents the width of the feature map, Indicates the index in the vertical direction. Represents the horizontal index. Indicates the number of channels of the feature map, Indicates the new number of channels, represents the feature map output by the cross-scale encoder, Represents the processed feature map; The feature vector The channel attention weight containing channel-dependent features is obtained by sequentially passing through the fully connected layer, ReLU activation function, fully connected layer, and sigmoid activation function; The calculation formula of channel attention weight is as follows: in, represents the channel attention weight, represents the sigmoid activation function, represents the ReLU activation function, express, Represents the weight coefficient of the fully connected layer.

[0052] S32, matrix multiplication and fusion of channel attention weights and feature maps are performed to generate the output of the cross-scale encoder.

[0053] S33, according to the branch fusion module, the output of the cross-scale encoder and the local features of the fine-grained encoder are fused to generate fused features.

[0054] The specific formula for generating the fusion feature is as follows: ; ； in, represents the output of the cross-scale encoder, represents the local features of the fine-grained encoder, Represents fusion features.

[0055] Through the branch fusion module, the network model can fully utilize the global discriminative features extracted by the cross-scale encoder and combine them with the local features extracted by the fine-grained encoder. This not only enhances the model's ability to model relationships between channel dimensions, but also improves its ability to distinguish highly similar objects in remote sensing images.

[0056] S4, according to the decoder, concatenates the fused features of each feature layer layer by layer with the fused features of the feature layer of the previous stage, and transfers them to the feature layer of the previous stage according to the convolution operation, and outputs the building outline map based on multi-layer decoding.

[0057] Specifically, in the decoder, the decoder output of each layer is concatenated and fused with the encoder output of the previous stage, and then input into the decoder of the previous layer through a convolution operation. After four layers of decoding, the final output is obtained.

[0058] In addition, based on the above embodiments, this solution also combines the loss functions of dice loss and cross entropy loss to supervise the model, so as to fully utilize the advantages of the two loss functions and improve the generalization ability and robustness of the model.

[0059] Specifically, the combined loss function generated by the loss functions of dice loss and cross entropy loss can be obtained in the following way, and the model can be supervised by the combined loss function: The Dice loss is calculated according to the Dice loss function, the cross entropy loss is calculated based on the cross entropy loss function, and the Dice loss and the cross entropy loss are weighted and summed according to the configuration ratio to obtain a combined loss function.

[0060] The Dice loss calculation formula is as follows: in, represents the Dice loss, express and The intersection between express The number of elements, express The number of elements. The numerator is multiplied by 2 to ensure that the denominator is in the range [0,1] after repeated calculations.

[0061] Dice Loss comes from Dice coefficient, which is a metric function used to evaluate the similarity between two samples. Its value range is between 0 and 1. The larger the value, the more similar they are.

[0062] The cross entropy loss calculation formula is as follows: in, represents the number of categories, represents a one-hot vector, Indicates that the predicted sample belongs to probability.

[0063] The element has only two values: 0 and 1. If the category is the same as the sample category, it takes 1, otherwise it takes 0.

[0064] The calculation formula of the combined loss function is as follows: in, Represents a combined loss function. 0.6 and 0.4 are the weight coefficients corresponding to Dice loss and cross entropy loss, respectively, which are used to balance the contribution between Dice loss and cross entropy loss.

[0065] In order to verify the accuracy of building extraction by the STR-UNet network, three different data sets are used in this embodiment, including the WHU building data set, the INRI building data set, and the Massachusetts data set, which can be used for image extraction training, verification, and testing.

[0066] The WHU building dataset is a complex and diverse remote sensing image database for building detection and ground cover identification. The dataset contains 8,189 high-resolution images of 512×512 pixels with a resolution of up to 0.3 meters per pixel. These images cover more than 220,000 building instances in an area of ​​approximately 450 square kilometers, fully reflecting the characteristics of buildings of different scales, shapes, and environmental backgrounds. The dataset is divided into a training subset (4,736 images), a validation subset (1,036 images), and a test subset (2,416 images) to meet the needs of model training, tuning, and performance evaluation.

[0067] The INRIA dataset is a remote sensing image resource with a spatial resolution of 0.3 meters per pixel, which is widely used in building detection tasks. It includes image data of multiple urban areas with significant geographical differences. The original dataset contains 180 images of 5000×5000 pixels with label information. In order to meet the experimental needs, each image was preprocessed in the study: first, they were enlarged to 5120×5120 pixels, and then cropped into small blocks of 512×512 pixels of the same size, generating a total of 18,000 image samples. Based on this, 14,400 of them were randomly sampled as a training set, and the remaining 3,600 constituted a validation set. This division method ensures that the model has good generalization capabilities in scenarios with a variety of regional styles and building types.

[0068] The Massachusetts dataset contains 151 aerial images, each with a resolution of 1500x1500 pixels and covering an area of ​​2.25 square kilometers. Therefore, the entire dataset covers an area of ​​approximately 340 square kilometers. The data is divided into a training set of 137 images, a test set of 10 images, and a validation set of 4 images. The target map is obtained by rasterizing the building footprints from the OpenStreetMap project. The dataset covers most urban and suburban areas, and the labels include buildings of various sizes, including individual houses and garages. All images have been rescaled to a resolution of 1 pixel per square meter. In order to improve the accuracy of the evaluation, the target maps in the test and validation parts of the dataset have been manually corrected.

[0069] The experimental evaluation indicators are: intersection over union, F1 score, precision and recall to evaluate the performance of the model. These indicators are widely used in the field of building extraction. The intersection over union is the ratio of the intersection and union between the predicted result of the specified target category and the true value label. The F1 score is an indicator that comprehensively considers precision and recall.

[0070] The experimental parameters are set as follows: the algorithm is implemented using Python 3.8 programming language and Pytorch 2.0.1 deep learning framework, the operating system is Ubuntu 20.04.6 LTS, and the graphics card is 4 Nvidia GeForce RTX 4090. The algorithm in this embodiment uses rotation and translation operations for data enhancement. The image size of the network input is 512×512. The SGD-M algorithm is used to optimize the network. The momentum parameter is set to 0.9, the learning rate is set to 0.0001, the batch size is 4, and the iterative training is 100 times.

[0071] In this example, the proposed STR-Net model is compared with the most advanced methods on three datasets to obtain performance comparison results, such as Figure 5 , Figure 6 and Figure 7As shown, the comparison models include Segmenter (2021), Swin-UNet (2022), DC-Swin (2022), BuildFormer (2022), DSAT-Net (2023), SPCL-Net (2023), IDD (2023), UANet (2023), ACMFNet (2024), MFCF-Net (2024) and TransC-GD-CD (2024). These network models are a hybrid of ViT and CNN. BuildFormer incorporates ViT modules in several stages of the CNN encoder, Swin-UNet is a UNet composed of pure Swin Transformer blocks, and DSAT-Net includes improved Transformer modules in all stages. For the STR-Net proposed in this embodiment, a cross-scale encoder with stacked Swin Transformers is added in the encoder stage and connected in parallel with the fine-grained encoder, so as to better learn image features and improve the accuracy of building contour extraction. Among them, Figure 5 A graphical representation of evaluation indicators of different models on the WHU dataset provided by an embodiment of the present invention, Figure 6 A graphical representation of evaluation indicators of different models on the INRIA dataset provided by an embodiment of the present invention, Figure 7 A graphical representation of evaluation indicators of different models on a Massachusetts dataset provided in an embodiment of the present invention.

[0072] For the WHU building dataset, the results are as follows Figure 5 As shown in Figure 2, the STR-Net model proposed in this example achieved the highest recall and F1 score indicators, which are 96.57% and 95.92% respectively. For the INRIA aerial image annotation dataset, the experimental results are as follows: Figure 6 As shown in the figure, the STR-Net proposed in this invention has the best overall performance, reaching the highest values ​​in IoU, recall rate and F1 score, which are 83.53%, 90.52% and 91.17% respectively. Figure 7 Evaluation metrics of different models on the Massachusetts buildings dataset are shown, including IoU, precision, recall, and F1 score. Figure 7 It shows that the STR-Net model proposed in this embodiment has achieved excellent results in these indicators, with an IoU of 81.53%, an accuracy of 88.79%, a recall rate of 87.12%, and an F1 score of 87.95%. Compared with other models, the STR-Net model proposed in this embodiment outperforms all other models in all indicators, indicating that it has higher pixel-level segmentation accuracy for buildings.

[0073] These comparative experiments fully verify the effectiveness and advancement of the model proposed in this embodiment. The uniqueness of the STR-Net model lies in its dual encoder design, in which one encoder captures global context features through a deep Swin transformer structure, and the other extracts local fine features through ResNeXt. In addition, STR-Net also designs BFM, which effectively fuses the global features extracted by the cross-scale encoder with the local features obtained by the fine-grained encoder. Through the position-sensitive channel attention mechanism, the model's understanding of the interaction between multi-channel features is enhanced, thereby achieving more accurate positioning and recognition of buildings.

[0074] See also Figure 8 , is a structural diagram of a building outline extraction system based on a dual encoder STR-Net provided by an embodiment of the present invention, and the building outline extraction system based on the dual encoder STR-Net includes: A processing module is used to obtain a building remote sensing image, pre-process the building remote sensing image to obtain an image block, determine the image block as input, and send it to a cross-scale encoder and a fine-grained encoder respectively; An extraction module, configured to extract global features in the image block according to the cross-scale encoder, and extract local features in the image block based on the fine-grained encoder, wherein the cross-scale encoder and the fine-grained encoder work in parallel; A fusion module, used for integrating the global features and the local features of each feature layer based on the branch fusion module to obtain a fusion feature; The output module is used to splice the fusion features of each feature layer with the fusion features of the feature layer of the previous stage layer by layer according to the decoder, and transfer them to the feature layer of the previous stage according to the convolution operation, and output the building outline map based on multi-layer decoding.

[0075] Figure 8 The apparatus of the embodiment shown can be used to perform Figure 1 The implementation principles and technical effects of the steps in the method embodiment shown are similar and will not be repeated here.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A building outline extraction method based on dual encoder STR-Net, characterized in that: include: Acquire a building remote sensing image, preprocess the building remote sensing image to obtain an image block, determine the image block as input, and send it to a cross-scale encoder and a fine-grained encoder respectively; Extracting global features in the image block according to the cross-scale encoder, and extracting local features in the image block based on the fine-grained encoder, wherein the cross-scale encoder and the fine-grained encoder work in parallel; Integrate the global features and the local features of each feature layer based on a branch fusion module to obtain a fusion feature; According to the decoder, the fusion features of each feature layer are concatenated layer by layer with the fusion features of the feature layer of the previous stage, and are transferred to the feature layer of the previous stage according to the convolution operation. The building outline map is output based on multi-layer decoding.

2. The method according to claim 1, characterized in that: Acquire a building remote sensing image, pre-process the building remote sensing image to obtain an image block, determine the image block as input, and send it to a cross-scale encoder and a fine-grained encoder respectively, including: Segmenting the building remote sensing image into a plurality of non-overlapping image blocks, determining each of the image blocks as an independent data item, and constructing a feature vector of each of the image blocks by concatenating the original RGB of all pixels in the image block; According to the linear connection layer, the feature vector of the image block is projected into the C-dimensional feature space, and the processed image block is determined as the input and sent to the cross-scale encoder and the fine-grained encoder respectively.

3. The method according to claim 1, characterized in that Extracting global features in the image block according to the cross-scale encoder, extracting local features in the image block based on the fine-grained encoder, the cross-scale encoder and the fine-grained encoder working in parallel, including: A cross-scale encoder is constructed based on a deep Swin transformer module, the image block is processed layer by layer according to the W-MSA unit and the SW-MSA unit, local and global features are extracted and integrated layer by layer, and global features are output based on the top layer, wherein each Swin transformer layer is composed of a W-MSA unit and a SW-MSA unit connected; A fine-grained encoder is constructed based on the ResNeXt module. The input feature matrix is ​​reduced in dimension based on the ResNeXt module. The reduced features are deeply extracted and refined based on group convolution. The number of channels is restored based on dimensionality increase. The processed features are fused with the short-circuit connection output to output local features.

4. The method according to claim 3, characterized in that: A cross-scale encoder is constructed based on the deep Swin transformer module, the image block is processed layer by layer according to the W-MSA unit and the SW-MSA unit, local and global features are extracted and integrated layer by layer, and global features are output based on the top layer, including: The operation process of the cross-scale encoder is as follows: For The feature layer is the input feature of the previous feature layer. As input, after LN processing, it is sent to the W-MSA unit for local feature extraction and integration to obtain the intermediate feature ,Will and Perform residual connection, and then process it through MLP and LN to obtain new intermediate features ; Will As input, after LN processing, it is sent to the SW-MSA unit for feature extraction and integration under the shift window to obtain the intermediate feature ,Will and Perform residual connection, and then go through an MLP and LN process to get the Output features of the layer ; Among them, for the top layer, its output feature The calculation process of is the same as above, but it is no longer used as the input of the next layer; The expression of the cross-scale encoder is as follows: ； ； ； ； ; in, Indicates Layer feature layer, represents the intermediate features after W-MSA and LN processing, Indicates Layer feature layer cross-scale encoder output features, Representation layer normalization, represents the self-attention mechanism under the conventional window configuration, Indicates The output features of the SW-Trans module, After SW-MSA and LN treatment and based on A further intermediate feature of Indicates Layer feature layer cross-scale encoder output features, Indicates Layer feature layer cross-scale encoder output features, Indicates The intermediate features of the layer feature layer, SW-MSA represents the window configuration scheme with shift operation, and SW-Trans module represents the Swin Transformer module.

5. The method according to claim 3, characterized in that: Based on the ResNeXt module, the input feature matrix is ​​reduced in dimension, and the reduced features are deeply extracted and refined based on group convolution. The number of channels is restored based on the dimensionality increase process, and the processed features are fused with the short-circuit connection output to output local features, including: The input feature matrix is ​​reduced in dimension according to a 1×1 convolutional layer, which has a feature matrix with 256 channels as input and outputs a feature matrix with the number of channels reduced to 128; The reduced feature matrix is ​​processed based on a group convolution operation consisting of 32 independent 3×3 convolution kernels. Each 3×3 convolution kernel independently performs convolution operations on a local area of ​​the feature matrix to generate 32 intermediate feature matrices, each of which contains deep information extracted from the original features. The 32 intermediate feature matrices are merged and dimensionally upgraded according to another 1×1 convolutional layer. The convolutional layer combines the feature matrices of the 32 channels into one and restores the number of channels to 256 to generate a dimensionally upgraded feature matrix. The feature matrix after dimension increase is fused and added with the output feature matrix from the short-circuit connection to generate the local features of the final output.

6. The method according to claim 3, characterized in that The global features and the local features of each feature layer are integrated based on the branch fusion module to obtain fusion features, including: Extract global features from the output of the cross-scale encoder according to the branch fusion module, process it through a 1×1 convolution layer, apply global average pooling in the channel direction to obtain a feature vector, pass the feature vector through two fully connected layers and ReLU and sigmoid activation functions, and obtain a channel attention weight containing channel-dependent features; Perform matrix multiplication and fusion on the channel attention weights and feature maps to generate the output of the cross-scale encoder; The output of the cross-scale encoder and the local features of the fine-grained encoder are fused according to the branch fusion module to generate a fused feature.

7. The method according to claim 6, characterized in that According to the branch fusion module, the global features are extracted from the output of the cross-scale encoder, processed by a 1×1 convolution layer, and a feature vector is obtained by applying global average pooling in the channel direction. The feature vector is passed through two fully connected layers and ReLU and sigmoid activation functions to obtain a channel attention weight containing channel-dependent features, including: Based on the feature map output from the cross-scale encoder by the branch fusion module The global features are obtained from the network and processed by a 1×1 convolution layer to obtain the processed feature map. , apply global average pooling in the channel direction to obtain the feature vector ; Eigenvector The calculation formula is as follows: in, represents the feature vector obtained after the global average pooling operation, represents the height of the feature map, represents the width of the feature map, Indicates the index in the vertical direction. Represents the horizontal index. Indicates the number of channels of the feature map, Indicates the new number of channels, represents the feature map output by the cross-scale encoder, Represents the processed feature map; The feature vector The channel attention weight containing channel-dependent features is obtained by sequentially passing through the fully connected layer, ReLU activation function, fully connected layer, and sigmoid activation function; The calculation formula of channel attention weight is as follows: in, represents the channel attention weight, represents the sigmoid activation function, represents the ReLU activation function, express, Represents the weight coefficient of the fully connected layer.

8. The method according to claim 7, characterized in that According to the branch fusion module, the output of the cross-scale encoder and the local features of the fine-grained encoder are fused to generate fused features, including: The specific formula for generating the fusion feature is as follows: ； ； in, represents the output of the cross-scale encoder, represents the local features of the fine-grained encoder, Represents fusion features.

9. The method according to claim 1, characterized in that: Also includes: The Dice loss is calculated according to the Dice loss function, the cross entropy loss is calculated based on the cross entropy loss function, and the Dice loss and the cross entropy loss are weighted and summed according to the configuration ratio to obtain a combined loss function.

10. The method according to claim 9, characterized in that The Dice loss is calculated according to the Dice loss function, the cross entropy loss is calculated based on the cross entropy loss function, and the Dice loss and the cross entropy loss are weighted and summed according to the configuration ratio to obtain a combined loss function, including: The Dice loss calculation formula is as follows: in, represents the Dice loss, express and The intersection between express The number of elements, express The number of elements; The cross entropy loss calculation formula is as follows: in, represents the number of categories, represents a one-hot vector, Indicates that the predicted sample belongs to probability; The calculation formula of the combined loss function is as follows: in, represents the combined loss function.

Images