A building contour extraction method based on double-encoder STR-Net

Abstract

This invention provides a building contour extraction method based on dual encoder STR-Net, belonging to the field of image processing. It involves acquiring remote sensing images of buildings, preprocessing these images to obtain image patches, and using these patches as input. These patches are then fed into a cross-scale encoder and a fine-grained encoder. The cross-scale encoder extracts global features from the image patches, while the fine-grained encoder extracts local features. The cross-scale encoder and fine-grained encoder operate in parallel. A branch fusion module integrates the global and local features from each feature layer to obtain fused features. The decoder then concatenates the fused features from each feature layer with the fused features from the previous stage, passing these features to the previous stage's feature layer via convolution operations. Finally, a multi-layer decoding method outputs the building contour map. This invention can improve the accuracy of building contour extraction.

Description

Technical Field

[0001] This invention relates to image processing technology, and more particularly to a method for extracting building contours based on dual encoder STR-Net. Background Technology

[0002] With the rapid development of Earth science observation technologies and the swift advancements in aerospace, geographic information systems, and sensor technologies, the amount of high-resolution remote sensing imagery data has increased dramatically. As a primary element of urban landscapes, buildings are often the main objects of urban geospatial analysis. Manipulating and analyzing building clusters within cities is a fundamental task in applying the city as an information carrier to various industries. Effectively segmenting buildings from remote sensing images is crucial for applications such as urban planning, damage detection, and land use management. High-resolution remote sensing images contain rich geographic information, providing a foundation for related research tasks. Therefore, accurately extracting building outlines from high-resolution remote sensing images has always been an important research direction.

[0003] Deep learning-based methods have demonstrated higher efficiency in building contour extraction, particularly suitable for handling the complex and diverse building types in urban and rural areas. Deep learning has become a mainstream research direction and application method for information extraction and image segmentation. This approach has, to some extent, solved the problem of missed extraction when buildings are small and densely distributed, improving the accuracy of building segmentation in remote sensing images. However, the problem of ineffective utilization of deep features remains, resulting in unresolved issues with blurred building boundaries. The Vision Transformer (VIT) effectively models the deep features of images through a multi-head attention mechanism, achieving more accurate building contour extraction and excelling in extracting global information, but it neglects crucial spatial detail context. Zheng et al. applied the Transformer to semantic segmentation, achieving excellent performance, but it did not utilize multi-scale information. All of these problems reduce the accuracy of building contour extraction.

[0004] Therefore, how to utilize depth features, enhance the ability to process spatial detail context, and effectively utilize multi-scale information to improve the accuracy of building outline extraction has become an urgent problem to be solved. Summary of the Invention

[0005] This invention provides a building contour extraction method based on dual encoder STR-Net, which can make full use of depth features, enhance the processing capability of spatial detail context, and effectively utilize multi-scale information to improve the accuracy of building contour extraction.

[0006] A first aspect of the present invention provides a method for extracting building contours based on a dual encoder STR-Net, comprising:

[0007] Acquire a remote sensing image of a building, preprocess the image to obtain image patches, determine the image patches as input, and feed them into a cross-scale encoder and a fine-grained encoder respectively;

[0008] The global features in the image patch are extracted based on the cross-scale encoder, and the local features in the image patch are extracted based on the fine-grained encoder. The cross-scale encoder and the fine-grained encoder work in parallel.

[0009] The global and local features of each feature layer are integrated based on the branch fusion module to obtain the fused features;

[0010] The decoder stitches together the fusion features of each feature layer with the fusion features of the previous stage feature layer, and then passes them to the previous stage feature layer according to the convolution operation, outputting the building outline map based on multi-layer decoding.

[0011] Optionally, in one possible implementation of the first aspect, acquiring a remote sensing image of a building, preprocessing the remote sensing image of the building to obtain image patches, determining the image patches as input, and feeding them into a cross-scale encoder and a fine-grained encoder respectively, includes:

[0012] The building remote sensing image is segmented into multiple non-overlapping image blocks, and each image block is determined as an independent data item. The feature vector of each image block is constructed by concatenating the original RGB values ​​of all pixels within the image block.

[0013] The feature vectors of the image patch are projected to the C-dimensional feature space by the linear connection layer, and the processed image patch is taken as input and fed into the cross-scale encoder and the fine-grained encoder respectively.

[0014] Optionally, in one possible implementation of the first aspect, global features in the image patch are extracted based on the cross-scale encoder, and local features in the image patch are extracted based on the fine-grained encoder, wherein the cross-scale encoder and the fine-grained encoder operate in parallel, including:

[0015] A cross-scale encoder is constructed based on the depth Swing transform module. The image patch is processed layer by layer according to the W-MSA unit and SW-MSA unit. Local and global features are extracted and integrated layer by layer. Global features are output based on the top layer. Each Swing transform layer is composed of W-MSA unit and SW-MSA unit connected together.

[0016] A fine-grained encoder is constructed based on the ResNeXt module. The input feature matrix is ​​dimensionality reduced based on the ResNeXt module. The dimensionality-reduced features are then extracted and refined in depth using grouped convolution. The number of channels is restored based on dimensionality increase. The processed features are then fused with the short-circuit connection output to output local features.

[0017] Optionally, in one possible implementation of the first aspect, a cross-scale encoder is constructed based on a depth Swin transform module, and the image patch is processed layer by layer according to the W-MSA unit and SW-MSA unit, extracting and integrating local and global features layer by layer, and outputting global features based on the top layer, including:

[0018] The operation process of the cross-scale encoder is as follows:

[0019] For the The feature layer takes the input features from the previous feature layer and... As input, after LN processing, it is fed into the W-MSA unit for local feature extraction and integration to obtain intermediate features. ,Will and Residual connections are performed, followed by MLP and LN processing to obtain new intermediate features. ;

[0020] Will As input, after LN processing, it is fed into the SW-MSA unit for feature extraction and integration under a shift window to obtain intermediate features. ,Will and Perform residual join, then pass through an MLP and LN process to obtain the first... Layer output features ;

[0021] Among them, for the top layer, its output features The calculation process is the same as the above process, but it is no longer used as the input for the next layer;

[0022] The expression for the cross-scale encoder is as follows:

[0023]

[0024]

[0025]

[0026]

[0027]

[0028] in, Indicates the first Feature layer, This represents the intermediate features after W-MSA and LN processing. Indicates the first The output features of the layer feature layer cross-scale encoder Representation layer normalization, This indicates the self-attention mechanism under normal window configuration. Indicates the first Output characteristics of the layer SW-Trans module This indicates that after SW-MSA and LN processing and based on Further intermediate features, Indicates the first The output features of the layer feature layer cross-scale encoder Indicates the first The output features of the layer feature layer cross-scale encoder Indicates the first The intermediate features of the layer feature layer, SW-MSA represents a window configuration scheme with shift operations, and SW-Trans module represents the Swing Transformer module.

[0029] Optionally, in one possible implementation of the first aspect, the input feature matrix is ​​dimensionality-reduced based on the ResNeXt module, and the dimensionality-reduced features are further extracted and refined using grouped convolution. The number of channels is restored based on dimensionality-up processing, and the processed features are fused with the output of the short-circuit connection to output local features, including:

[0030] The input feature matrix is ​​reduced in dimensionality using a 1×1 convolutional layer. This convolutional layer takes a feature matrix with 256 channels as input and outputs a feature matrix with 128 channels.

[0031] The dimensionality-reduced feature matrix is ​​processed by a grouped convolution operation consisting of 32 independent 3×3 convolution kernels. Each 3×3 convolution kernel independently performs convolution operations on local regions of the feature matrix to generate 32 intermediate feature matrices, each containing deep-level information extracted from the original features.

[0032] Another 1×1 convolutional layer merges and upscales the 32 intermediate feature matrices. This convolutional layer combines the feature matrices of the 32 channels into one and restores the number of channels to 256, generating the upscaled feature matrix.

[0033] The upgraded feature matrix is ​​fused and added to the output feature matrix from the short-circuit connection to generate the final output local features.

[0034] Optionally, in one possible implementation of the first aspect, the global features and local features of each feature layer are integrated based on the branch fusion module to obtain fused features, including:

[0035] The global features are extracted from the output of the cross-scale encoder by the branch fusion module, processed by a 1×1 convolutional layer, and global average pooling is applied in the channel direction to obtain the feature vector. The feature vector is then passed through two fully connected layers and ReLU and sigmoid activation functions to obtain channel attention weights containing channel-dependent features.

[0036] The channel attention weights and feature maps are fused by matrix multiplication to generate the output of the cross-scale encoder;

[0037] The branch fusion module fuses the output of the cross-scale encoder and the local features of the fine-grained encoder to generate fused features.

[0038] Optionally, in one possible implementation of the first aspect, global features are extracted from the output of the cross-scale encoder based on the branch fusion module, processed through a 1×1 convolutional layer, and global average pooling is applied in the channel direction to obtain a feature vector. The feature vector is then passed through two fully connected layers and ReLU and sigmoid activation functions to obtain channel attention weights containing channel-dependent features, including:

[0039] Based on the feature map output from the cross-scale encoder by the branch fusion module Global features are obtained from the first convolutional layer and then processed through a 1×1 convolutional layer to obtain the processed feature map. The feature vector is obtained by applying global average pooling in the channel direction. ;

[0040] Feature vector The calculation formula is as follows:

[0041]

[0042] in, This represents the feature vector obtained after global average pooling. Indicates the height of the feature map. Indicates the width of the feature map. Indicates the index in the vertical direction. Indicates the index in the horizontal direction. This represents the number of channels in the feature map. Indicates the new number of channels. This represents the feature map output by the cross-scale encoder. This represents the processed feature map;

[0043] The feature vector The channel attention weights, which contain channel-dependent features, are obtained by sequentially passing through a fully connected layer, a ReLU activation function, another fully connected layer, and a sigmoid activation function.

[0044] The formula for calculating channel attention weights is as follows:

[0045]

[0046] in, Indicates channel attention weights. This represents the sigmoid activation function. Represents the ReLU activation function. , This represents the weight coefficient of the fully connected layer.

[0047] Optionally, in one possible implementation of the first aspect, the fused features are generated by fusing the output of the cross-scale encoder and the local features of the fine-grained encoder according to the branch fusion module, including:

[0048] The specific formula for generating the fused features is as follows:

[0049]

[0050]

[0051] in, This represents the output of the cross-scale encoder. Represents the local features of a fine-grained encoder. This indicates the fusion feature.

[0052] Optionally, in one possible implementation of the first aspect, it also includes:

[0053] The Dice loss is calculated based on the Dice loss function, and the cross-entropy loss is calculated based on the cross-entropy loss function. The Dice loss and the cross-entropy loss are then weighted and summed according to the configuration ratio to obtain the combined loss function.

[0054] Optionally, in one possible implementation of the first aspect, the Dice loss is calculated based on the Dice loss function, the cross-entropy loss is calculated based on the cross-entropy loss function, and the Dice loss and cross-entropy loss are weighted and summed according to the configuration ratio to obtain a combined loss function, including:

[0055] The formula for calculating Dice loss is as follows:

[0056]

[0057] in, This indicates Dice's loss. express and The intersection between them express The number of elements, express The number of elements;

[0058] The formula for calculating cross-entropy loss is as follows:

[0059]

[0060] in, Indicates the number of categories. Represents a one-hot vector. Indicates that the predicted sample belongs to The probability of;

[0061] The formula for calculating the combined loss function is as follows:

[0062]

[0063] in, This represents the combined loss function.

[0064] A second aspect of the present invention provides a building contour extraction system based on a dual encoder STR-Net, comprising:

[0065] The processing module is used to acquire remote sensing images of buildings, preprocess the remote sensing images of buildings to obtain image blocks, determine the image blocks as input, and send them to the cross-scale encoder and the fine-grained encoder respectively.

[0066] The extraction module is used to extract global features from the image patch based on the cross-scale encoder and extract local features from the image patch based on the fine-grained encoder. The cross-scale encoder and the fine-grained encoder work in parallel.

[0067] The fusion module is used to integrate the global features and local features of each feature layer based on the branch fusion module to obtain fused features;

[0068] The output module is used to stitch together the fusion features of each feature layer by the decoder and the fusion features of the feature layer of the previous stage, and pass them to the feature layer of the previous stage according to the convolution operation, and output the building outline map based on multi-layer decoding.

[0069] The beneficial effects of this invention are as follows: It designs a dual-encoder structure consisting of a cross-scale encoder based on a depth Swin transform and a fine-grained encoder based on ResNeXt. These two encoders work in parallel to extract global contextual information and local detailed features from images, improving the understanding of image content and the accuracy of segmentation. Simultaneously, a branch fusion module is designed, which fuses global and local features at each stage of the encoding process to extract discriminative features from remote sensing images and effectively distinguish highly similar ground objects. The building contour extraction method of this invention achieves performance improvements on three building datasets, demonstrating the effectiveness of the building contour extraction method described in this invention for building contour extraction. Attached Figure Description

[0070] Figure 1 A flowchart illustrating a building contour extraction method based on dual encoder STR-Net provided in an embodiment of the present invention;

[0071] Figure 2 This invention provides an overall architecture diagram of a dual encoder STR-Net.

[0072] Figure 3 A schematic diagram of a depth Swing converter structure provided in an embodiment of the present invention;

[0073] Figure 4 This is a schematic diagram of the structure of a branch fusion module provided in an embodiment of the present invention;

[0074] Figure 5 A chart illustrating the evaluation metrics of different models on the WHU dataset, provided as an embodiment of the present invention;

[0075] Figure 6 A chart illustrating the evaluation metrics of different models on the INRIA dataset, provided as an embodiment of the present invention;

[0076] Figure 7 A chart illustrating the evaluation metrics of different models on the Massachusetts dataset, provided as an embodiment of the present invention;

[0077] Figure 8 This is a schematic diagram of a building contour extraction system based on dual encoder STR-Net, provided for an embodiment of the present invention. Detailed Implementation

[0078] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0079] See Figure 1 This is a flowchart illustrating a building contour extraction method based on dual encoder STR-Net provided in an embodiment of the present invention. Figure 1 The execution entity of the method shown can be a software and / or hardware device. The execution entity of this application can include, but is not limited to, at least one of the following: user equipment, network equipment, etc. User equipment can include, but is not limited to, computers, smartphones, personal digital assistants (PDAs), and the aforementioned electronic devices. Network equipment can include, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of computers or network servers. Cloud computing is a type of distributed computing, consisting of a super virtual computer composed of a group of loosely coupled computers. This embodiment does not limit this. Steps S1 to S4 are detailed as follows:

[0080] S1. Acquire a remote sensing image of the building, preprocess the remote sensing image of the building to obtain image blocks, determine the image blocks as input, and send them to the cross-scale encoder and the fine-grained encoder respectively.

[0081] See Figure 2 This diagram illustrates the overall architecture of a dual-encoder STR-Net provided in this embodiment of the invention. A cross-scale encoder based on the Swin Transformer module and a fine-grained encoder based on the ResNeXt module are constructed. These two encoders work in parallel, simultaneously extracting global contextual information and local fine features from the image, thereby improving the understanding of image content and segmentation accuracy. Furthermore, this embodiment also designs a branch fusion module, which fuses the global features extracted from the cross-scale encoder and the local features extracted from the fine-grained encoder at each stage of the encoding process, and then passes them to the next stage of the encoding process. In this way, the model can extract discriminative features from remote sensing images, thereby effectively distinguishing highly similar ground objects.

[0082] In this context, "building remote sensing image" refers to a remote sensing image containing buildings, "image patch" refers to a non-overlapping small patch obtained after preprocessing the building remote sensing image, "cross-scale encoder" is an encoder constructed from multiple stacked Swin Transformers (Swin Transformer, a visual transformer model based on the Transformer architecture), and "fine-grained encoder" refers to an encoder constructed from ResNeXt modules. ResNeXt is a deep convolutional neural network architecture that combines and improves upon the advantages of VGG, ResNet, and Inception. ResNeXt employs a "segmentation-transformation-aggregation" strategy, where each branch uses the same topology, resulting in a simpler network structure and a reduced number of hyperparameters. Furthermore, ResNeXt introduces the concept of "cardinality," which refers to the number of sub-modules within each module. By adjusting the cardinality, the balance between model complexity and performance can be effectively controlled.

[0083] In some embodiments, step S1 can be implemented through the following steps:

[0084] S11, the remote sensing image of the building is segmented into multiple non-overlapping image blocks, each image block is determined as an independent data item, and the feature vector of each image block is constructed by concatenating the original RGB values ​​of all pixels within the image block.

[0085] For the input remote sensing image, the network model uses the patch partitioning mechanism in Vision Transformer (ViT). First, the RGB image is divided into a series of non-overlapping image patches. In this process, each patch is treated as an independent "token," or data item, and its feature vector is constructed by concatenating the original RGB values ​​of all pixels within that patch. Here, a data item refers to a processing unit.

[0086] In practical applications, the selected image size can be 512×512 pixels, and an 8×8 patch size can be used. Therefore, each patch contains 8×8=64 pixels, and each pixel has 3 channels (R, G, B). Thus, the feature dimension of each patch is 8×8×3=192.

[0087] The above method enables the model to process image information in a serialized manner, similar to words or characters in text.

[0088] S12 projects the feature vectors of the image patch onto the C-dimensional feature space based on the linear connection layer, determines the processed image patch as input, and sends it to the cross-scale encoder and the fine-grained encoder respectively.

[0089] A linear connection layer is a type of neural network layer that performs a linear transformation on the input feature vector. In this embodiment, its role is to project the feature vector of each patch onto a new dimension C, where C can be equal to 48. The processed image patch can then be used as input and fed into both a cross-scale encoder and a fine-grained encoder for further processing.

[0090] S2, global features in the image patch are extracted based on the cross-scale encoder, and local features in the image patch are extracted based on the fine-grained encoder. The cross-scale encoder and the fine-grained encoder work in parallel.

[0091] Global features refer to the result of feature extraction on the overall information of an image, including the global context information of the image. Local features refer to the result of feature extraction on a local area of ​​an image, focusing on the details and local structure of the image.

[0092] Specifically, the cross-scale encoder is responsible for extracting global features, employing a multi-scale feature extraction strategy and using convolutional kernels and pooling operations to capture global contextual information in the image. The fine-grained encoder is responsible for extracting local features, using convolutional operations and network structures to capture local details and texture information in the image. The cross-scale encoder and the fine-grained encoder extract global and local features in parallel, thereby simultaneously utilizing global contextual information and fine local features, improving the accuracy and precision of building contour extraction.

[0093] Based on the above embodiments, step S2 can be implemented in the following ways:

[0094] S21, a cross-scale encoder is constructed based on the depth Swing transform module. The image block is processed layer by layer according to the W-MSA unit and SW-MSA unit. Local and global features are extracted and integrated layer by layer. Global features are output based on the top layer. Each Swing transform layer is composed of W-MSA unit and SW-MSA unit connected together.

[0095] See Figure 3 This is a schematic diagram of a depth Swing converter structure provided in an embodiment of the present invention. Figure 3 As can be seen, a W-MSA unit and an SW-MSA unit are connected to form a Swing transformer layer, and multiple Swing transformer layers are stacked to construct a deep Swing transformer structure. W-MSA enables the Swing transformer to focus on local regions within the image, thereby reducing computational complexity, while capturing global dependencies through the recombination of local features. SW-MSA further enhances this capability by introducing a shift operation, which facilitates the interaction between adjacent windows and is highly effective for handling spatial relationships in visual tasks.

[0096] In deep multiscale encoders, each feature layer learns different features of the input data. Increasing depth means capturing more layers of feature information, resulting in better performance when handling complex tasks. Furthermore, by increasing depth, multiscale encoders can learn more details and contextual information about the data, giving the model better generalization ability when processing new data. By incorporating attention mechanisms, multiscale encoders achieve a balance between efficiency and performance, making them well-suited for large-scale vision tasks, including footprint extraction from remote sensing images.

[0097] Understandably, the standard Transformer module uses a multi-head self-attention sublayer (MSA) to compute global self-attention among all tokens. This leads to quadratic computational complexity due to the number of tokens and limits its application scope. To optimize modeling efficiency, the Swin Transformer introduces a windowed self-attention mechanism (W-MSA) to replace the standard MSA. The W-MSA design employs two different window configuration strategies: one is a self-attention mechanism with a regular window configuration, i.e., the basic W-MSA; the other is a window configuration scheme with shift operations, i.e., SW-MSA. These two variant structures together constitute the core components of the Swin Transformer's efficient processing of visual information.

[0098] The computational process of the above-mentioned cross-scale encoder is as follows:

[0099] For the The feature layer takes the input features from the previous feature layer and... As input, after LN processing, it is fed into the W-MSA unit for local feature extraction and integration to obtain intermediate features. ,Will and Residual connections are performed, followed by MLP and LN processing to obtain new intermediate features. .

[0100] Will As input, after LN processing, it is fed into the SW-MSA unit for feature extraction and integration under a shift window to obtain intermediate features. ,Will and Perform residual join, then pass through an MLP and LN process to obtain the first... Layer output features .

[0101] Among them, for the top layer, its output features The calculation process is the same as the process described above, but it is no longer used as the input for the next layer.

[0102] The expression for the cross-scale encoder is as follows:

[0103]

[0104]

[0105]

[0106]

[0107]

[0108] in, Indicates the first Feature layer, This represents the intermediate features after W-MSA and LN processing. Indicates the first The output features of the layer feature layer cross-scale encoder Representation layer normalization, This indicates the self-attention mechanism under normal window configuration. Indicates the first Output characteristics of the layer SW-Trans module This indicates that after SW-MSA and LN processing and based on Further intermediate features, Indicates the first The output features of the layer feature layer cross-scale encoder Indicates the first The output features of the layer feature layer cross-scale encoder Indicates the first The intermediate features of the layer feature layer, SW-MSA represents a window configuration scheme with shift operations, and SW-Trans module represents the Swing Transformer module.

[0109] Using the methods described above, the cross-scale encoder can achieve multi-level, cross-scale encoding of image features.

[0110] S22, a fine-grained encoder is constructed based on the ResNeXt module, the input feature matrix is ​​dimensionality reduced based on the ResNeXt module, the dimensionality-reduced features are deeply extracted and refined based on grouped convolution, the number of channels is restored based on dimensionality increase, the processed features are fused with the short-circuit connection output, and the local features are output.

[0111] Specifically, the ResNeXt module reduces the amount of information in the input feature matrix and the number of channels by performing dimensionality reduction. Next, the ResNeXt module divides the simplified feature matrix into multiple groups, and each group undergoes independent and in-depth feature extraction. After grouping and extracting features, the ResNeXt module recombines the results of all groups to restore the feature matrix to its original number of channels. Finally, the ResNeXt module adds the processed feature matrix to the original, unmodified feature matrix and outputs the final feature matrix. This matrix contains both the original information of the input feature matrix and the information extracted through in-depth grouping, which can better represent the features of the image.

[0112] Based on the above embodiments, step S22 can be implemented in the following ways:

[0113] The input feature matrix is ​​reduced in dimensionality using a 1×1 convolutional layer. This convolutional layer takes a feature matrix with 256 channels as input and outputs a feature matrix with 128 channels.

[0114] The dimensionality-reduced feature matrix is ​​processed by a grouped convolution operation consisting of 32 independent 3×3 convolution kernels. Each 3×3 convolution kernel independently performs convolution operations on local regions of the feature matrix, generating 32 intermediate feature matrices, each containing deep-level information extracted from the original features.

[0115] Another 1×1 convolutional layer merges and upscales the 32 intermediate feature matrices. This convolutional layer combines the feature matrices of the 32 channels into one and restores the number of channels to 256, generating the upscaled feature matrix.

[0116] The upgraded feature matrix is ​​fused and added to the output feature matrix from the short-circuit connection to generate the final output local features.

[0117] Specifically, when processing the input feature matrix, the ResNeXt module first uses a 1×1 convolutional layer to reduce the channel dimension, decreasing the original 256 channels to 128. Then, it uses a set of grouped convolutional operations containing 32 independent 3×3 convolutional kernels to perform deep feature extraction and refinement. Next, it uses another 1×1 convolutional layer to increase the dimensionality of the feature matrix, restoring its channel count to 256. Finally, the result calculated in this branch is fused and added with the output from the short-circuit connection to generate the final output feature, i.e., the local feature.

[0118] The above methods can achieve efficient extraction and fusion of image features.

[0119] S3, Based on the branch fusion module, the global features and local features of each feature layer are integrated to obtain the fused features.

[0120] The cross-scale encoder is divided into four feature extraction stages. The global features extracted in each stage are sent to the branch fusion module to be fused with the local features of the fine-grained encoder. The fused features are then input into the encoder of the next stage to complete the fusion of global and local features.

[0121] See Figure 4 This is a schematic diagram of a branch fusion module provided in an embodiment of the present invention. The main idea of ​​the branch fusion module is to extract channel-dependent features from the global features of the cross-scale encoder and then embed them into the local features obtained by the fine-grained encoder. In this way, the model can encode more global discriminative features to improve the segmentation accuracy of highly similar ground objects in remote sensing images.

[0122] Based on the above embodiments, step S3 can be implemented in the following ways:

[0123] S31, global features are extracted from the output of the cross-scale encoder according to the branch fusion module, processed by a 1×1 convolutional layer, and global average pooling is applied in the channel direction to obtain feature vectors. The feature vectors are then passed through two fully connected layers and ReLU and sigmoid activation functions to obtain channel attention weights containing channel-dependent features.

[0124] A 1×1 convolutional layer is used to adjust the number of channels in the feature maps. Global average pooling averages each feature map along its channel direction, resulting in a vector of length equal to the number of channels. Channel attention weights are calculated by sequentially passing the vector obtained from global average pooling through a fully connected layer, a ReLU activation function, another fully connected layer, and a sigmoid activation function. The fully connected layer learns the complex relationships between channels by linearly transforming the input vector using weight coefficients. The ReLU activation function introduces non-linearity, helping the model learn more complex feature representations. The sigmoid activation function maps the output to 0 and 1, representing the attention weight for each channel. These weights indicate the importance of different channels to the current task.

[0125] Specifically, in some embodiments, step S31 includes the following steps:

[0126] Based on the feature map output from the cross-scale encoder by the branch fusion module Global features are obtained from the first convolutional layer and then processed through a 1×1 convolutional layer to obtain the processed feature map. The feature vector is obtained by applying global average pooling in the channel direction. ;

[0127] Feature vector The calculation formula is as follows:

[0128]

[0129] in, This represents the feature vector obtained after global average pooling. Indicates the height of the feature map. Indicates the width of the feature map. Indicates the index in the vertical direction. Indicates the index in the horizontal direction. This represents the number of channels in the feature map. Indicates the new number of channels. This represents the feature map output by the cross-scale encoder. This represents the processed feature map.

[0130] The feature vector By sequentially passing through a fully connected layer, a ReLU activation function, another fully connected layer, and a sigmoid activation function, channel attention weights containing channel-dependent features are obtained.

[0131] The formula for calculating channel attention weights is as follows:

[0132]

[0133] in, Indicates channel attention weights. This represents the sigmoid activation function. Represents the ReLU activation function. , This represents the weight coefficient of the fully connected layer.

[0134] S32 performs matrix multiplication and fusion of channel attention weights and feature maps to generate the output of the cross-scale encoder.

[0135] S33, The branch fusion module fuses the output of the cross-scale encoder and the local features of the fine-grained encoder to generate fused features.

[0136] The specific formula for generating the fused features is as follows:

[0137]

[0138]

[0139] in, This represents the output of the cross-scale encoder. Represents the local features of a fine-grained encoder. This indicates the fusion feature.

[0140] Through the branch fusion module, the network model can fully utilize the global discriminative features extracted by the cross-scale encoder and combine them with the local features extracted by the fine-grained encoder. This not only enhances the model's ability to model relationships between channel dimensions but also improves its ability to distinguish highly similar features in remote sensing images.

[0141] S4, based on the fusion features of each feature layer stitched together by the decoder and the fusion features of the feature layer of the previous stage, and based on the convolution operation passed to the feature layer of the previous stage, outputs the building outline map based on multi-layer decoding.

[0142] Specifically, in the decoder, the output of each layer is concatenated and fused with the output of the encoder in the previous stage, and then input into the decoder of the next layer after a convolution operation. After four layers of decoding, the final output is obtained.

[0143] Furthermore, based on the above embodiments, this scheme also combines the loss functions of Dice loss and cross-entropy loss to supervise the model, thereby making full use of the advantages of the two loss functions and improving the model's generalization ability and robustness.

[0144] Specifically, a combined loss function generated from the Dice loss and cross-entropy loss can be obtained in the following way, and this combined loss function can be used to supervise the model:

[0145] The Dice loss is calculated based on the Dice loss function, and the cross-entropy loss is calculated based on the cross-entropy loss function. The Dice loss and the cross-entropy loss are then weighted and summed according to the configuration ratio to obtain the combined loss function.

[0146] The formula for calculating Dice loss is as follows:

[0147]

[0148] in, This indicates Dice's loss. express and The intersection between them express The number of elements, express The number of elements. Multiplying the numerator by 2 is to ensure that the value of the denominator after repeated calculation is within the range of [0,1].

[0149] Dice Loss comes from the Dice coefficient, a metric function used to evaluate the similarity between two samples. Its value ranges from 0 to 1, with a larger value indicating greater similarity.

[0150] The formula for calculating cross-entropy loss is as follows:

[0151]

[0152] in, Indicates the number of categories. Represents a one-hot vector. Indicates that the predicted sample belongs to The probability of.

[0153] The element has only two values: 0 and 1. If the category is the same as the sample category, it takes the value 1; otherwise, it takes the value 0.

[0154] The formula for calculating the combined loss function is as follows:

[0155]

[0156] in, This represents the combined loss function. and These are the weight coefficients corresponding to the Dice loss and cross-entropy loss, respectively, used to balance the contributions between the Dice loss and cross-entropy loss.

[0157] To verify the accuracy of the STR-UNet network in extracting buildings, this embodiment uses three different datasets: the WHU building dataset, the INRI building dataset, and the Massachusetts dataset, which can be used for image extraction training, validation, and testing.

[0158] The WHU Buildings Dataset is a complex and diverse remote sensing image database used for building detection and land cover identification. This dataset contains 8189 high-resolution images, each 512×512 pixels in size, with a resolution of up to 0.3 meters per pixel. These images cover more than 220,000 building instances within an area of ​​approximately 450 square kilometers, fully demonstrating the characteristics of buildings of different sizes, shapes, and environmental contexts. The dataset is divided into a training subset (4736 images), a validation subset (1036 images), and a test subset (2416 images) to meet the needs of model training, optimization, and performance evaluation.

[0159] The INRIA dataset is a widely used remote sensing image resource with a spatial resolution of 0.3 meters per pixel for building detection tasks. It includes imagery data from several urban areas with significant geographical differences. The original dataset contains 180 labeled images of 5000×5000 pixels. To accommodate experimental needs, each image was preprocessed: first, they were enlarged to 5120×5120 pixels, then cropped into uniform 512×512 pixel patches, generating a total of 18,000 image samples. Based on this, 14,400 images were randomly selected as the training set, and the remaining 3,600 images constituted the validation set. This partitioning ensures that the model has good generalization ability across various regional styles and building types.

[0160] The Massachusetts dataset contains 151 aerial images, each with a resolution of 1500x1500 pixels, covering an area of ​​2.25 square kilometers. Therefore, the entire dataset covers an area of ​​approximately 340 square kilometers. The data is divided into a training set of 137 images, a test set of 10 images, and a validation set of 4 images. The target maps were obtained by rasterizing building outlines from the OpenStreetMap project. The dataset covers most urban and suburban areas, and the labels include buildings of various sizes, including detached houses and garages. All images have been rescaled to a resolution of 1 pixel per square meter. To improve the accuracy of the evaluation, the target maps in the test and validation portions of the dataset have been manually calibrated.

[0161] Experimental evaluation metrics include Intersection over Union (IoU), F1 score, precision, and recall to assess model performance. These metrics are widely used in building extraction. IoU is the ratio of the intersection to the union of predicted values ​​for a given target class with the ground truth labels. F1 score is a metric that comprehensively considers both precision and recall.

[0162] The experimental parameters were set as follows: the algorithm was implemented using the Python 3.8 programming language and the PyTorch 2.0.1 deep learning framework; the operating system was Ubuntu 20.04.6 LTS; and the graphics cards were four Nvidia GeForce RTX 4090s. In this embodiment, the algorithm used rotation and translation operations for data augmentation. The network input image size was 512×512. The SGD-M algorithm was used for network optimization, with a momentum parameter of 0.9, a learning rate of 0.0001, a batch size of 4, and 100 iterations of training.

[0163] This embodiment compares the proposed STR-Net model with state-of-the-art methods on three datasets, obtaining performance comparison results, such as... Figure 5 , Figure 6 and Figure 7 As shown, the comparison models include Segmenter (2021), Swin-UNet (2022), DC-Swin (2022), BuildFormer (2022), DSAT-Net (2023), SPCL-Net (2023), IDD (2023), UANet (2023), ACMFNet (2024), MFCF-Net (2024), and TransC-GD-CD (2024). These network models are hybrids of ViT and CNN. BuildFormer incorporates ViT modules in several stages of the CNN encoder, Swin-UNet is a UNet composed of pure Swing Transformer blocks, and DSAT-Net includes improved Transformer modules in all stages. For the STR-Net proposed in this embodiment, a cross-scale encoder with stacked Swing Transformers is added in the encoder stage and connected in parallel with the fine-grained encoder, thereby better learning image features and improving the accuracy of building contour extraction. Figure 5 This is a chart illustrating the evaluation metrics of different models on the WHU dataset, provided as an embodiment of the present invention. Figure 6 This is a chart illustrating the evaluation metrics of different models on the INRIA dataset, provided as an embodiment of the present invention. Figure 7 This is a chart illustrating the evaluation metrics of different models on the Massachusetts dataset, as provided in an embodiment of the present invention.

[0164] For the WHU building dataset, the results are as follows: Figure 5 As shown, the STR-Net model proposed in this embodiment achieved the highest values ​​in both recall and F1 score, at 96.57% and 95.92%, respectively. Experimental results for the INRIA aerial image annotation dataset are as follows... Figure 6 As shown, the STR-Net proposed in this invention has the best overall performance, achieving the highest values ​​in IoU, recall, and F1 score, which are 83.53%, 90.52%, and 91.17%, respectively. Figure 7 The evaluation metrics of different models on the Massachusetts Buildings dataset are shown, including IoU, precision, recall, and F1 score. Figure 7The results show that the STR-Net model proposed in this embodiment achieves excellent results on these metrics: IoU of 81.53%, precision of 88.79%, recall of 87.12%, and F1 score of 87.95%. Compared with other models, the STR-Net model proposed in this embodiment outperforms all other models on all metrics, indicating that it has higher pixel-level segmentation accuracy for buildings.

[0165] These comparative experiments fully validate the effectiveness and advancement of the model proposed in this embodiment. The unique feature of the STR-Net model lies in its dual-encoder design: one encoder captures global contextual features through a deep Swin transform structure, while the other extracts fine-grained local features through ResNeXt. Furthermore, STR-Net incorporates BFM, a module that effectively fuses global features extracted by the cross-scale encoder with local features obtained by the fine-grained encoder. Through a position-sensitive channel attention mechanism, the model's understanding of the interactions between multi-channel features is enhanced, thereby achieving more accurate building localization and recognition.

[0166] See Figure 8 This is a schematic diagram of a building contour extraction system based on dual encoder STR-Net provided in an embodiment of the present invention. The building contour extraction system based on dual encoder STR-Net includes:

[0167] The processing module is used to acquire remote sensing images of buildings, preprocess the remote sensing images of buildings to obtain image blocks, determine the image blocks as input, and send them to the cross-scale encoder and the fine-grained encoder respectively.

[0168] The extraction module is used to extract global features from the image patch based on the cross-scale encoder and extract local features from the image patch based on the fine-grained encoder. The cross-scale encoder and the fine-grained encoder work in parallel.

[0169] The fusion module is used to integrate the global features and local features of each feature layer based on the branch fusion module to obtain fused features;

[0170] The output module is used to stitch together the fusion features of each feature layer by the decoder and the fusion features of the feature layer of the previous stage, and pass them to the feature layer of the previous stage according to the convolution operation, and output the building outline map based on multi-layer decoding.

[0171] Figure 8 The apparatus of the illustrated embodiment can be used to perform corresponding actions. Figure 1 The steps in the method embodiments shown are implemented in a similar manner and have similar technical effects, and will not be repeated here.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A building contour extraction method based on a double-encoder STR-Net, characterized by, The method comprises the following steps: obtaining a building remote sensing image, pre-processing the building remote sensing image to obtain an image block, determining the image block as an input, and sending the image block into a cross-scale encoder and a fine-grained encoder respectively; extracting global features in the image block according to the cross-scale encoder, extracting local features in the image block based on the fine-grained encoder, and the cross-scale encoder and the fine-grained encoder working in parallel; integrating the global features and the local features of each feature layer based on a branch fusion module to obtain fusion features; splicing the fusion features of each feature layer with the fusion features of the previous stage feature layer according to a decoder, and transmitting to the feature layer of the previous stage according to convolution operation, and outputting a building contour map based on multiple layers of decoding; wherein, extracting global features in the image block according to the cross-scale encoder, extracting local features in the image block based on the fine-grained encoder, and the cross-scale encoder and the fine-grained encoder working in parallel, comprises: constructing a cross-scale encoder based on a deep Swin transformer module, processing the image block layer by layer according to a W-MSA unit and a SW-MSA unit, extracting and integrating local and global features layer by layer, and outputting global features based on the top layer, constructing a fine-grained encoder according to a ResNeXt module, performing dimension reduction processing on the input feature matrix based on the ResNeXt module, performing deep-level extraction and refinement on the dimension-reduced features according to grouped convolution, restoring the channel number based on dimension elevation processing, outputting fusion of the processed features and short circuit connection, and outputting local features; wherein, constructing a cross-scale encoder based on a deep Swin transformer module, processing the image block layer by layer according to a W-MSA unit and a SW-MSA unit, extracting and integrating local and global features layer by layer, and outputting global features based on the top layer, comprises: the operation process of the cross-scale encoder is as follows: For the first layer feature layer, the input features of the previous layer feature layer are processed by LN and sent to the W-MSA unit for local feature extraction and integration to obtain intermediate features As input, after LN processing, it is sent to the W-MSA unit for local feature extraction and integration to obtain intermediate features , and are connected in residual, and then processed by MLP and LN to obtain new intermediate features ; Will As input, after LN processing, it is sent to the SW-MSA unit for feature extraction and integration under the shift window to obtain the intermediate features , With Residual connection, and then through an MLP and LN processing, the output features of the first Layer are obtained ; wherein, for the top layer, its output features are computed in the same way as described above, but are no longer used as input for the next layer; The expression of the cross-scale encoder is as follows: ; ; ; ; ; wherein, represents the output feature of the layer feature layer, represents the further intermediate feature after the processing of the W-MSA and LN, represents the output feature of the layer feature layer cross-scale encoder, represents layer normalization, represents the self-attention mechanism under the regular window configuration, represents the output feature of the layer SW-Trans module, represents the further intermediate feature after the processing of the SW-MSA and LN and based on , wherein, represents the output feature of the layer feature layer cross-scale encoder, represents the output feature of the layer feature layer cross-scale encoder, represents the intermediate feature of the layer feature layer, and SW-MSA represents a window configuration scheme with a shift operation, and SW-Trans module represents a Swin Transformer module. wherein, integrating the global features and the local features of each feature layer based on a branch fusion module to obtain fusion features, comprises: extracting global features from the output of the cross-scale encoder according to the branch fusion module, processing through a 1x1 convolution layer, applying global average pooling in the channel direction to obtain a feature vector, passing the feature vector through two fully connected layers and ReLU, sigmoid activation functions to obtain channel attention weights containing channel-dependent features; performing matrix multiplication fusion on the channel attention weights and the feature map to generate the output of the cross-scale encoder; fusing the output of the cross-scale encoder and the local features of the fine-grained encoder according to the branch fusion module to generate fusion features.

2. The method of claim 1, wherein obtaining a building remote sensing image, pre-processing the building remote sensing image to obtain an image block, determining the image block as an input, and sending the image block into a cross-scale encoder and a fine-grained encoder respectively, comprises: segmenting the building remote sensing image into a plurality of non-overlapping image blocks, determining each image block as an independent data item, and constructing a feature vector of each image block from the original RGB of all pixel points in the image block. ​ According to the linear connection layer, the feature vector of the image block is projected to a C-dimensional feature space, and the processed image block is determined as the input and sent into the cross-scale encoder and the fine-grained encoder respectively.

3. The method of claim 1, wherein, Based on the ResNeXt module, the input feature matrix is processed in dimension reduction, the feature after dimension reduction is extracted and refined in deep level based on the grouped convolution, the channel number is recovered based on the dimension increasing processing, the processed feature is fused with the output of the short circuit connection, and the local feature is output, including: According to the 1*1 convolution layer, the input feature matrix is processed in dimension reduction, the convolution layer has a feature matrix with 256 channels as input, and outputs a feature matrix with channel number reduced to 128; Based on a group of grouped convolution operations containing 32 independent 3*3 convolution kernels, the feature matrix after dimension reduction is processed, each 3*3 convolution kernel independently convolves the local area of the feature matrix to generate 32 intermediate feature matrices, each matrix containing deep-level information extracted from the original feature; According to another 1*1 convolution layer, the 32 intermediate feature matrices are merged and processed in dimension increasing, the convolution layer combines the feature matrix with 32 channels into 1, and restores the channel number to 256 to generate the feature matrix after dimension increasing; The feature matrix after dimension increasing is fused and added with the output feature matrix from the short circuit connection to generate the final output local feature.

4. The method of claim 1, wherein, According to the branch fusion module, the global feature is extracted from the output of the cross-scale encoder, processed by a 1*1 convolution layer, and applied with global average pooling in the channel direction to obtain a feature vector, the feature vector is processed by two fully connected layers and ReLU, sigmoid activation functions to obtain channel attention weights containing channel-dependent features, including: Based on the feature map output from the cross-scale encoder by the branch fusion module The global feature is obtained from the middle, processed by a 1x1 convolutional layer, and a processed feature map is obtained Global average pooling is applied in the channel direction to obtain a feature vector ; eigenvector The calculation formula is as follows: ; wherein, represents a feature vector obtained after a global average pooling operation, represents a height of a feature map, represents a width of a feature map, represents an index in a vertical direction, represents an index in a horizontal direction, represents a number of channels of a feature map, represents a new number of channels, represents a feature map output by a cross-scale encoder, represents a processed feature map; The feature vector is obtained Passing through a full connection layer, a ReLU activation function, a full connection layer, and a sigmoid activation function in sequence, a channel attention weight containing channel-dependent features is obtained. The calculation formula of the channel attention weight is as follows: ; wherein, denotes a channel attention weight, denotes a sigmoid activation function, denotes a ReLU activation function, , denotes a fully connected layer weight coefficient.

5. The method of claim 4, wherein, According to the branch fusion module, the output of the cross-scale encoder and the local feature of the fine-grained encoder are fused to generate a fusion feature, including: The specific formula for generating the fusion features is as follows: ; ; wherein, represents an output of the cross-scale encoder, represents a local feature of the fine-grained encoder, represents a fused feature.

6. The method of claim 1, wherein, Further comprising: According to the Dice loss function, the Dice loss is calculated, based on the cross-entropy loss function, the cross-entropy loss is calculated, and according to the configuration ratio, the Dice loss and the cross-entropy loss are weighted and summed to obtain a combined loss function.

7. The method of claim 6, wherein, According to the Dice loss function, the Dice loss is calculated, based on the cross-entropy loss function, the cross-entropy loss is calculated, and according to the configuration ratio, the Dice loss and the cross-entropy loss are weighted and summed to obtain a combined loss function, including: The Dice loss calculation formula is as follows: ; wherein, denotes the Dice loss, denotes and the intersection between, denotes the number of elements, denotes the number of elements; The cross-entropy loss is calculated as follows: ; wherein, represents the number of classes, represents a one-hot vector, represents the probability that the predicted sample belongs to class c. The formula of the combined loss function is as follows: ; wherein, denotes the combined loss function.

Images