A building roof and facade extraction method and system based on remote sensing images

Abstract

The application discloses a kind of based on building roof and facade extraction method and system of remote sensing image.First, obtain building remote sensing image dataset, second, construct neural network framework NeSF-Net, the encoder of network adopts the ConvNeXt of performance and efficiency compromise.In order to enhance the spatial consistency of roof and facade, the application proposes a neighborhood relationship perception module, which extracts and sub-neighborhood prior perception by applying neighborhood prior to improve the class perception and spatial perception of deep features in a particular direction;Finally, a scale and frequency modulation decoder is constructed, multiple frequency components are implemented for feature enhancement on multiple scale branches to realize space-frequency domain information interaction, and foreground and background information flow is guided through feature aggregation block, finally the segmentation performance of the model is improved.Last, the training set and validation set after data preprocessing are input into the network framework, reasonable parameters are set for training, and the corresponding network model is obtained for realizing building roof and facade extraction.

Description

Technical Field

[0001] This invention belongs to the field of image information processing technology in computer vision, specifically relating to a method and system for extracting building roofs and facades based on remote sensing images. Background Technology

[0002] Buildings are primary human activity spaces, and timely updates to building information are crucial for smart city construction, improving urban management efficiency, and disaster early warning. Today, advanced Earth observation technologies enable researchers to easily acquire large amounts of high-quality remote sensing data. Compared to costly and limited-coverage LiDAR and SAR imagery, optical remote sensing imagery is widely used in building information extraction due to its ease of acquisition, low cost, and simple processing. In the past, cities primarily consisted of low-rise buildings, and traditional orthorectification effectively removed building tilt caused by terrain undulations. Therefore, it was typically assumed that building roofs and footprints completely overlapped, with research focusing primarily on extracting roof information. With accelerated urbanization, in urban areas with numerous high-rise buildings, buildings exhibit a tilt effect in imagery, meaning roofs and footprints are difficult to overlap. Currently, the common workflow involves improving the registration accuracy between roofs and the image, and then adaptively correcting the offset building vector data to achieve accurate building footprint extraction. However, even after registration updates, building vectors in traditional orthorectified imagery may still deviate from the actual footprint, especially when the tilt effect of high-rise buildings is significant. Another approach addresses the tilt effect of high-rise buildings by constructing roof, footprint, and roof-to-footprint offset vectors. However, the accuracy of offset vectors depends on high-quality digital surface models or digital building models, which may not be readily available or may be of limited quality. Furthermore, building facades reflect the displacement of building image points in satellite side-view imaging. Therefore, the new research paradigm directly uses facades as one of the extraction targets, which is not only widely applicable at low cost to various satellite imagery with different tilt angles, but also effectively expresses building height information and can further infer the building's footprint.

[0003] Deep learning methods have achieved remarkable results in building extraction, with mainstream approaches employing CNN models, Transformer models, and hybrid CNN-Transformer models. CNN models excel at capturing local details, such as building edges and corners. Transformer models are better at constructing long-range dependencies and are more suitable for large-scale buildings with complex roof materials. Hybrid models combine the strengths of both, balancing local and global information representation capabilities, but they also face challenges in model design and computational complexity. Currently, there are few deep learning network architectures specifically designed for building roof and facade extraction. Despite the encouraging results achieved by these methods, they are still limited by insufficient utilization of inter-pixel information and the limited spatial information recovery capabilities of the decoder, leading to difficulties in distinguishing building boundary areas and incomplete segmentation of roof and facade regions. Since roofs and facades often have similar materials, pixels in their adjacent areas also have similar feature information, which often leads to misclassification. Low spatial coherence affects the completeness of roof and facade region extraction, meaning that adjacent pixels with similar spatial features receive inconsistent predictions. Furthermore, partial invisibility and severe shading of facades further complicate the accurate differentiation of building information.

[0004] In summary, a method for extracting building roofs and facades based on remote sensing imagery is proposed, aiming to address the problems of insufficient utilization of inter-pixel information and limited spatial information recovery capability of the decoder. Summary of the Invention

[0005] To address the problems existing in the prior art, the present invention aims to provide a method for extracting building roofs and facades based on remote sensing imagery. This method can extract buildings with high precision from remote sensing imagery. This is crucial for applications such as urban planning, land use management, resource allocation, and public services.

[0006] To achieve the above objectives, this invention proposes a method for extracting building roofs and facades based on remote sensing imagery, comprising the following steps:

[0007] Step 1: Obtain the building dataset and perform preprocessing operations;

[0008] Step 2: Construct an end-to-end convolutional neural network, NeSF-Net. NeSF-Net includes an encoder, a neighborhood relationship-aware module, and a decoder. The encoder is used to extract features at different scales from the input image. The neighborhood relationship-aware module applies neighborhood prior extraction and sub-neighborhood prior perception to enhance the category and spatial perception of features in a specific direction, thus obtaining neighborhood relationship-aware features. The decoder takes the neighborhood relationship-aware features as input and concatenates them with the features from the previous level of the encoder, sequentially performing scale modulation, frequency modulation, and feature aggregation. The final output of the decoder is then processed by a semantic head to obtain the final output result.

[0009] Step 3: Design a reasonable loss function based on the characteristics of the NeSF-Net network structure, and train the constructed NeSF-Net using the preprocessed dataset from Step 1.

[0010] Step 4: Use the trained network model to extract the roof and facade from the image of the building to be tested.

[0011] Furthermore, the building dataset includes a training set, a validation set, and a test set, which are composed of image data and label data, respectively. All data in the training and validation sets are cropped into H×W pixels, where W and W represent the height and width of the image, respectively. Data augmentation is performed on the image data in the training and validation sets using methods such as translation, rotation, scaling, vertical mirroring, illumination adjustment, and adding salt and pepper noise points to obtain augmented image data.

[0012] Furthermore, the encoder section contains several ConvNeXt Blocks. For the input image I, after several ConvNeXt Blocks and downsampling operations, the output features e of each layer of the encoder are obtained. l , l represents the index of the layer in the encoder;

[0013] The decoder consists of multiple scale and frequency modulation decoders, each composed of a scale modulation module, a frequency modulation module, and a feature aggregation block. The input features of the decoder first need to be upsampled, then concatenated with the previous level features from the encoder, and finally fed into the scale and frequency modulation decoder after convolution. The input features are processed by multiple scale and frequency modulation decoders in the decoder, and finally the semantic head is used to obtain the final output of the network.

[0014] Furthermore, the specific implementation method for neighborhood prior extraction is as follows:

[0015] The output feature of the last layer of the encoder is defined as X, and neighborhood prior information is extracted; an 8-directional connectivity approach is used to process the input features. The channel information is compressed and embedded to obtain different categories of neighborhood prior information. N represents the number of categories; coarse supervision is applied to neighborhood prior information to impose connectivity constraints; specifically, N prior The output channels are 8×N, and are upsampled to the input size for monitoring purposes; then, N is compressed using a global average pooling operation. prior The ReLU activation function and fully connected convolution are used to map it into a vector with the same channel dimension as X; finally, the projected neighborhood prior vector is obtained by recoding and normalization through 1×1 convolution and the Sigmoid function. Each element in this neighborhood prior vector embeds pixel connectivity information in each direction, as expressed by the following formula:

[0016]

[0017] σ prior =γ(ω) 1×1 δ(ω 1×1 (GAP(N prior ))))

[0018] Where γ and δ represent the Sigmoid and ReLU activation functions, respectively, and ω 1×1 (·) represents a convolution operation with a kernel of 1, GAP represents a global average pooling operation, x represents the input, and i,j represent the number of rows and columns;

[0019] Next, X and σ are sliced ​​separately using channel slicing. prior Divided into 8 sub-parts and Where i∈[1,8], the purpose is to understand the category information and pixel neighborhood information in the entangled hidden layer.

[0020] Furthermore, the specific implementation of sub-neighborhood prior perception is as follows:

[0021] For each group of sub-path features x i Prior information of sub-neighborhood Cross-attention, global attention, and local attention are performed separately to guide the attention of each sub-path in different neighborhood directions. The specific methods are as follows:

[0022] (i): Cross-attention: First, from Generate query vector Q from x i The function generates a key vector K and a value vector V. Attention weights are obtained by calculating the dot product of Q and K and applying the Softmax function. Then, the attention weights are weighted and summed with V, and projected back into the original feature space. Finally, the function... The residuals are then reconnected to maintain feature integrity, thus obtaining the enhanced neighborhood features. The specific formula is as follows:

[0023]

[0024] Where d k It represents the dimension of the key vector, used to scale the dot product result; Proj indicates the projection transformation.

[0025] (ii): Global attention: First, W-Trans block is used to focus on the feature map x. i The window is partitioned into four windows, and the characteristics of the partitioned window are as follows: Then, vertical stripe convolution is applied to the features of each partition to create long-range dependencies between partitions; features with enhanced global information. It can be obtained through the following formula:

[0026]

[0027] Where Cat(·) represents channel concatenation, ξ represents batch normalization, and ω 3×3 For a regular 2D convolution with a kernel of 3, ω vs This indicates vertical stripe convolution, where the subscripts k×k, 1×k, and k×1 represent different settings in vertical stripe convolution.

[0028] (iii): Local attention: Pooling pyramids are used to represent multi-scale information of sub-path features. Multiple pooling operations are set up, and the multi-scale features are concatenated along the channel dimension. After convolution and batch normalization, the scale-enhanced features are obtained. The above process can be expressed by the following formula:

[0029] f BL (x i )=ω 3×3 (ω 1×1 (x i ))

[0030] f MS (x i ) = Cat(M 5×5 (f BL (x i )),M 9×9 (f BL (x i )),M 13×13 (f BL (x i )),x i )

[0031]

[0032] Where f BL (·) represents the Bottleneck layer, ω 1×1 For a regular 2D convolution with a kernel of 1, f MS (·) indicates multi-scale operations, M k×k This indicates a max pooling operation with kernel k, where k can be 5, 9, or 13.

[0033] (iv): For the three attention features obtained In other words, due to It contains neighborhood information in a specific direction. and The middle part contains enhanced category information. Element-wise multiplication can selectively highlight pixel correlations within the feature latent space. Therefore, the following formula is used to obtain the sub-neighborhood-aware features.

[0034]

[0035] Finally, all sub-neighborhood perceived features y i Channel stacking and recoding are performed to obtain neighborhood relationship-aware features Y, which are then used as input features for the decoder in subsequent decoding processes.

[0036] Furthermore, the scale modulation module sets up S parallel branches, each of which records the features learned using different scales;

[0037] For each s-scale branch, average pooling and convolution operations are first performed to adjust the scale and channels of the input feature F to improve computational efficiency, as shown in the following formula:

[0038]

[0039] Where 1≤s≤S, δ represents the ReLU activation function, and ξ represents layer normalization. Represents the convolution kernel. Indicates the step size. Indicates the number of fillers. The hole ratio is represented by W; W and GAP represent the convolution operation and global average pooling, respectively.

[0040] Immediately afterwards, The data is fed into the frequency modulation module and the feature aggregation block, where information interacts in the spatial and frequency domains to obtain the feature map. Among them, C s H s W s Representing features F respectively sThe number of channels, length, and width;

[0041] To balance the feature contributions from different scales, features from all scale branches are aggregated, their mean is calculated, and residual connections are made with the input features to obtain the final features. The specific formula is as follows:

[0042]

[0043] Where Z is the output of a single scale and frequency modulation decoder, and c, h, and w represent the number of channels, length, and width of feature Z, respectively.

[0044] Furthermore, the specific processing procedure of the frequency modulation module is as follows:

[0045] First, the output features of the S-th scale branch are... Divide into n sub-blocks according to the channel dimension. i∈[0,1,…,n-1], for each Assign a corresponding 2D DCT frequency component And calculate the weighted sum of the two, as shown in the following formula:

[0046]

[0047] Where (u k ,v k )express The corresponding frequency component exponents are determined using a top-K screening strategy. s Indicates the weighted result; frequency component It can be calculated using the following formula:

[0048]

[0049] Where h and w represent the length and width of the frequency component, respectively;

[0050] Subsequently, various pooling operations can be used to obtain... More refined channel information is obtained; the features obtained after average pooling, min pooling, and max pooling operations are sequentially passed through two fully connected layers and then element-wise summed; finally, the attention map after multi-frequency modulation is obtained through the Sigmoid function, and then multiplied with the original features for recalibration, as shown in the formula:

[0051]

[0052] in and This represents a fully connected layer, where δ represents the ReLU activation function and γ represents the Sigmoid activation function. Indicates to The results obtained using multiple pooling operations This indicates the characteristics of the frequency modulation module output.

[0053] Furthermore, the processing procedure for feature aggregation blocks is as follows:

[0054] Input features Spatial attention map of foreground information is obtained after 1×1 convolution and sigmoid function. Accordingly, background information is provided through The result is obtained by subtracting the matrix element by element. Then, two learnable parameters, α and β, are introduced to guide the information flow between the foreground and background, thereby reducing the risk of category confusion caused by various interfering factors. Finally, convolution and upsampling operations are used to unify the channels and sizes of the features, facilitating subsequent feature aggregation; thus, a feature map is obtained. The specific formula is described as follows:

[0055]

[0056] Among them, Up s Indicates upsampling, ω 3×3 This is a regular 2D convolution with a kernel of 3.

[0057] Furthermore, the overall loss function includes the OHEM loss with class weights, the Dice loss, and the neighborhood prior loss. By reasonably setting the weight ratio of each loss, the convergence of the network can be achieved.

[0058] The formula for calculating the overall loss function is as follows:

[0059]

[0060]

[0061] Seg,Conn=BV(N prior )

[0062]

[0063] An 8-directional connectivity approach is used to compress and embed the channel information in the output features of the last layer of the encoder to obtain the neighborhood prior information N for different categories. prior Due to the prior information N in the neighborhood prior Each set of 8 channels represents a connectivity feature of a category, and the bilateral voting module BV is used to select from N. prior Obtain the bidirectional connectivity probability graph, namely the semantic category probability graph Seg and the connectivity probability graph Conn; where L OHEM For the OHEM loss with class weights, LDice For Dice's loss, For neighborhood prior loss, w c p is the weight of category c, where C is the total number of categories. i,c It is the predicted value for category c, g i,c That is the real label. It's a hyperparameter.

[0064] The present invention also provides a building roof and facade extraction system based on remote sensing imagery, comprising:

[0065] The system includes a processor and a memory. The memory stores program instructions, and the processor calls the stored instructions in the memory to execute the method for extracting building roofs and facades based on remote sensing images, as described in the above technical solution.

[0066] The beneficial effects of this invention are:

[0067] This invention provides a method for extracting building roofs and facades based on remote sensing images. The network model proposed in this invention adopts a neighborhood relationship perception module and a scale and frequency modulation decoder, which effectively solves the problems of insufficient utilization of inter-pixel information and limited spatial information recovery capability of the decoder, ultimately improving the segmentation performance of the model.

[0068] This invention applies image recognition to building information detection, effectively improving the efficiency of building information acquisition and updating. Therefore, this invention solves the technical problem of improving the accuracy and efficiency of building roof and facade detection. Attached Figure Description

[0069] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0070] Figure 1 This is a flowchart illustrating the technical process of the method of the present invention.

[0071] Figure 2 This is a network structure diagram of the NeSF-Net proposed by the method of this invention;

[0072] Figure 3 This is a schematic diagram of the neighborhood relationship sensing module in the method of the present invention;

[0073] Figure 4 This is a schematic diagram of the scale modulation module in the method of the present invention;

[0074] Figure 5 This is a schematic diagram of the frequency modulation module in the method of the present invention;

[0075] Figure 6 This is a schematic diagram of the feature aggregation block in the method of the present invention;

[0076] Figure 7The figure shows the results of an embodiment of the method of the present invention. Detailed Implementation

[0077] To make the technical problems solved by this invention, the technical solutions adopted, and the technical effects achieved clearer, the technical solutions of this invention will be further described below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining this invention and not for limiting it. Furthermore, it should be noted that, for ease of description, only the parts related to this invention are shown in the accompanying drawings, not all of them.

[0078] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The terms "first position" and "second position" refer to two different positions.

[0079] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to fixed connections or detachable connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal connections between two components. Those skilled in the art can understand the specific meaning of these terms in this invention based on the specific circumstances.

[0080] This invention aims to update building information in a timely manner and actively apply it to tasks such as smart city construction, improving urban management efficiency, and disaster early warning. Exploring the use of image recognition and intelligent sensing technology to acquire large-scale remote sensing information can significantly reduce the manual costs associated with obtaining urban building information. Common building detection methods often rely on large amounts of manually labeled data; these time-consuming and labor-intensive methods cannot meet the demands of economic efficiency and real-time monitoring. Deep learning methods, however, provide faster and more accurate building detection results. Therefore, this invention proposes a method for extracting building roofs and facades based on remote sensing imagery, further improving the accuracy and efficiency of extracting roofs and facades of mid-to-high-rise buildings.

[0081] This invention relates to a method for extracting building roofs and facades based on remote sensing imagery, aiming to achieve accurate segmentation of building roofs and facades in high-resolution remote sensing images. First, a dataset of building remote sensing images is acquired, including a training set, a validation set, and a test set. Data augmentation is performed on the images in the training and validation sets, including cropping, translation, rotation, scaling, vertical mirroring, and illumination adjustment, and salt-and-pepper noise is added to enrich the diversity of the sample dataset. Second, this method constructs an end-to-end encoder-decoder neural network framework, NeSF-Net. The network's encoder uses ConvNeXt, which balances performance and efficiency. To enhance the spatial consistency of roofs and facades, this invention deeply mines the relationship between target pixels and neighboring pixels in the latent space and proposes a Neighborhood Relationship Awareness Module (NRAM). This module improves the category and spatial awareness of depth features in specific directions by applying neighborhood prior extraction and sub-neighborhood prior perception. Furthermore, this invention focuses on frequency information in the decoder to help the network overcome the effects of spatial information loss and shadow occlusion. A Scale & Frequency Modulation Decoder (SFMD) is constructed, applying multiple frequency components to multiple scale branches for feature enhancement to achieve spatial-frequency domain information interaction. A Feature Aggregation Block (FAB) guides the flow of foreground and background information, ultimately improving the model's segmentation performance. Finally, the preprocessed training and validation sets are input into the network framework, and reasonable parameters are set for training to obtain the corresponding network model. Predicting model performance using the test set demonstrates that the proposed network model has strong robustness and applicability. The NeSF-Net network proposed in this invention can achieve fine segmentation of building roofs and facades, providing valuable reference for high-resolution remote sensing image mapping in complex scenes.

[0082] like Figure 1 As shown in the figure, the method for extracting building roofs and facades based on remote sensing images provided by this invention includes the following steps:

[0083] Step 1: Obtain the building dataset and preprocess it. Construct an end-to-end convolutional neural network structure; the proposed network is called NeSF-Net. First, NeSF-Net mainly consists of an encoder, a neighborhood awareness module, and a scale-frequency modulation decoder. The data-augmented image is input into the network. The input image passes through the encoder, neighborhood awareness module, and scale-frequency modulation decoder in batches. Step 1 specifically includes the following:

[0084] (1): Obtain the building dataset and perform preprocessing operations on the obtained dataset. The building dataset includes a training set, a validation set, and a test set, which are composed of image data and label data, respectively. All data in the training set and the validation set are cropped to H×W pixels, which is set to 512×512 pixels in this embodiment. Data augmentation is performed on the image data in the training set and the validation set using methods such as translation, rotation, scaling, vertical mirroring, illumination adjustment, and adding salt and pepper noise points to obtain augmented image data.

[0085] (2): Construct the encoder for the backbone network, such as Figure 2 As shown. The backbone network module uses RGB images. As input, I consists of three channels with dimensions H×W, where H and W represent the height and width of the image, respectively. The encoder employs ConvNeXt and uses the pre-trained model ConvNeXt-Tiny to improve model learning efficiency. It is primarily composed of ConvNeXt Blocks. The encoder section contains four ConvNeXt Blocks. Each ConvNeXt Block replaces the commonly used 3×3 kernel convolution with a 7×7 kernel depthwise convolution, replaces Batch Normalization and ReLU operations in convolutional neural networks (CNNs) with Layer Normalization and GELU activation functions, and embeds two fully connected layers with 1×1 kernels to adjust the channel dimensions. After four ConvNeXt Block operations and downsampling, the input image I is processed to obtain the output features e of each layer of the encoder. l ,l∈[1,2,3,4].

[0086] Step Two: Feed the feature map from Step One, processed by the backbone network encoder, into the Neighborhood Relationship Awareness Module (NRAM). The NRAM effectively decouples and utilizes pixel connectivity features to enhance pixel correlations within the deep feature latent space. For example... Figure 3 As shown, this module includes Neighborhood Prior Extraction (NPE) (…). Figure 3 (a) and Sub-neighborhood Relationship Awareness (SNRA) Figure 3The algorithm consists of two parts, (b) and (c). First, the prior information of the building's neighborhood is extracted. Then, the deep features are deconstructed using the prior information. By effectively untangling the channel information of the deep features in different sub-neighborhood directions, the inter-pixel dependencies are enhanced. The feature map e4 of the last layer of the encoder is used as the input feature of the NRAM.

[0087] (1): Design of the NPE method. The output feature e4 of the last layer of the encoder is defined as X, and neighborhood prior information is extracted. This invention uses an 8-directional connectivity approach for the input features. The channel information is compressed (squeeze) and embedded to obtain neighborhood prior information for different categories. N represents the number of categories, which is set to 3 in this invention. Coarse supervision is applied to the output to impose connectivity constraints. Specifically, N... prior After a 3×3 and 1×1 convolution, the output has 8×N channels, which are then upsampled to the input size for supervision. Then, N is compressed using a global average pooling operation. prior The ReLU activation function and a fully connected 1×1 convolution are used to map it to a vector with the same channel dimension as X. Finally, the projected neighborhood prior vector is obtained by recoding and normalizing using a 1×1 convolution and the Sigmoid function. Each element in this vector embeds pixel connectivity information in each direction. As described in the following formula:

[0088]

[0089] σ prior =γ(ω) 1×1 δ(ω 1×1 (GAP(N prior ))))

[0090] Where γ and δ represent the Sigmoid and ReLU activation functions, respectively. ω 1×1 (·) represents a convolution operation with a kernel of 1, GAP represents a global average pooling operation, x represents the input, and i,j represent the number of rows and columns.

[0091] Next, we use channel-wise slicing to separate X and σ. prior Divided into 8 sub-parts and Where i∈[1,8]. The purpose is to understand the category information and pixel neighborhood information in the entangled hidden layer.

[0092] (2): Constructing the SNRA part. By using prior features from the neighborhood to guide the learning of category information of deep features in the neighborhood direction, highly distinguishable category information is achieved. This invention applies this method to each group of sub-path features x i Prior information of sub-neighborhood Cross-attention, global attention, and local attention are performed separately to guide the attention of each sub-path in different neighborhood directions. The specific method is as follows:

[0093] (i): Cross-attention (CA). To effectively fuse x i and This invention utilizes cross-attention to selectively focus on x i The neighborhood information is used to focus on neighborhood features by calculating the relevance of queries and keys within specific neighborhood regions. For example... Figure 3 As shown in (c), firstly, from Generate the query vector Q from x. i The key vector Q and value vector V are generated. Attention weights are obtained by calculating the dot product of Q and K and applying the Softmax function. Then, the attention weights are weighted and summed with V, and projected back into the original feature space. Finally, the... The residuals are then reconnected to preserve feature integrity. This yields the enhanced neighborhood features. The specific formula is as follows:

[0094]

[0095] Where d k `<key>` represents the dimension of the key vector, used to scale the dot product result. `Proj` represents the projection transformation. Therefore, this method of independently computing attention within each local neighborhood region through cross-attention can better capture fine-grained features and enhance the sensitivity of deep features in neighborhood directions. Furthermore, this local computation method can significantly reduce computational complexity, thereby improving model efficiency.

[0096] (ii): Global Attention (GA). To enhance the global dependencies of sub-path features, we designed a global attention module that leverages an improved Swin-Transformer block to achieve efficient representation of contextual information. For example... Figure 3 As shown in (d) above, firstly, the W-Trans block is used to process the feature map x. i Window partitioning is performed. This invention uses a 4-window approach, and the characteristics after partitioning are as follows: Then, vertical stripe convolutions are applied to the features of each partition to create long-range dependencies between partitions. By performing stripe convolutions in both horizontal and vertical directions, the directional features of buildings can be learned while reducing computational parameters. Features enhanced with global information. It can be obtained through the following formula:

[0097]

[0098] Where Cat(·) represents channel concatenation, ξ represents batch normalization, and ω 3×3 This is a regular 2D convolution with a kernel of 3. This represents a vertical stripe convolution with kernel k.

[0099] (iii): Local attention (LA). For example... Figure 3 As shown in (e), a pooling pyramid is used to represent the multi-scale information of sub-path features. Max pooling operations with kernels of 5×5, 9×9, and 13×13 are set, and the multi-scale features are concatenated along the channel dimension. After convolution and batch normalization, the scale-enhanced features are obtained. The above process can be expressed by the following formula:

[0100] f BL (x i )=ω 3×3 (ω 1×1 9x i ))

[0101] f MS (x i ) = Cat(M 5×5 (f BL (x i )),M 9×9 (f BL (x i )),M 13×13 (f BL (x i )),x i )

[0102]

[0103] Where f BL (·) represents the Bottleneck layer, ω 1×1 For a regular 2D convolution with a kernel of 1, f MS (·) indicates multi-scale operations, M k×k This represents a max pooling operation with kernel k.

[0104] (iv): For the three attention features obtained In other words, due to It contains neighborhood information in a specific direction. and The middle part contains enhanced category information, and element-wise multiplication can selectively highlight pixel correlations within the feature latent space. Therefore, the present invention can obtain sub-neighborhood perceived features by executing the following formula.

[0105]

[0106] Finally, all sub-neighborhood perceived features y i Channel stacking and recoding are performed to obtain neighborhood relationship-aware features Y, which are then used as input features for the decoder in subsequent decoding processes.

[0107] Step 3: Feed the output feature map from the neighborhood relationship perception module in Step 2 into the decoder. Construct a Scale & Frequency Modulation Decoder (SFMD) for the input feature d of the decoder. l For l∈[1,2,3], first, it needs to undergo an upsampling operation with a scale of 2. Then, it is compared with the previous level feature e from the encoder. l-1 Channel concatenation is performed. Finally, after a 3×3 convolution operation, the signals are sequentially fed into the Scale Modulation Module (SMM), Frequency Modulation Module (FMM), and Feature Aggregation Block (FAB). This invention performs the same process more than three times in the decoder, and finally passes through the semantic header (…). Figure 2 The SH in the text refers to the semantic header, which is used to obtain the final output of the network. The specific implementation of SFMD includes the following sub-steps:

[0108] (1): Design of SMM method. Considering the varying scales and large scale spans of buildings, multi-scale modulation can be used to better recover the spatial information of buildings in the decoder. For example... Figure 4 As shown, this invention establishes S parallel branches, each recording features learned at different scales. Input features It requires S scale branches. For each scale branch s (1≤s≤S), average pooling and convolution operations need to be performed first to adjust the scale and channels of F to improve computational efficiency. The specific formula is as follows:

[0109]

[0110] in Represents the convolution kernel. Indicates step size, Indicates the number of fillers. This represents the hole ratio. W and GAP represent the convolution operation and global average pooling, respectively. Following that, The data is fed into the FMM and FAB, where information interacts in the spatial and frequency domains to obtain feature maps.

[0111] To balance the feature contributions from different scales, this invention aggregates features from all scale branches, calculates the mean, and performs a residual connection with the input features to obtain the final feature. The specific formula is as follows:

[0112]

[0113] (2): Methods for designing FMMs. For example... Figure 5 As shown, the input features of the S-th scale branch (i.e., the S-th scale branch) are first... Divide into n sub-blocks according to the channel dimension. i∈[0,1,…,n-1], for each Assign a corresponding 2D DCT frequency component And calculate the weighted sum of the two, as shown in the following formula:

[0114]

[0115] Where (u k ,v k )express The corresponding frequency component index is determined using a top-K screening strategy in this invention. Frequency component It can be calculated using the following formula:

[0116]

[0117] Subsequently, various pooling operations can be used to obtain... More refined channel information is obtained. The features obtained through average pooling, min pooling, and max pooling operations are sequentially passed through two fully connected layers and then element-wise summed. Finally, a multi-frequency modulated attention map is obtained using the sigmoid function, and then multiplied with the original features for recalibration. As shown in the formula:

[0118]

[0119] in and This indicates a fully connected layer.

[0120] (3): A method for designing FABs. Features enhanced by frequency amplification can better understand the spatial distribution and semantic structure of buildings. Furthermore, this invention proposes using FABs in each scale branch to fully interact spatial-frequency domain information, thereby improving the inter-class discriminative power of each scale branch. For example... Figure 6 As shown, input features A spatial attention map of foreground information can be obtained by performing a 1×1 convolution and a sigmoid function. Accordingly, background information can be obtained through The result is obtained by subtracting the matrix element by element. Then, two learnable parameters, α and β, are introduced to guide the information flow between the foreground and background, thereby reducing the risk of category confusion caused by various interfering factors. Finally, convolution and upsampling operations are used to unify the channels and sizes of the features, facilitating subsequent feature aggregation. The feature map is then obtained. The specific formula is described as follows:

[0121]

[0122] Among them, Up s This indicates upsampling.

[0123] Step 4: Design a reasonable loss function based on the characteristics of the NeSF-Net network structure. This mainly uses the OHEM loss with class weights, the Dice loss, and the neighborhood prior loss. By appropriately setting the weight ratios of each loss function, network convergence is achieved. Step 4 specifically includes the following:

[0124] This invention primarily employs a hybrid loss mechanism to constrain network training, specifically by applying different methods to the neighborhood prior information N. prior The final semantic segmentation result is supervised. Considering the significant class imbalance between building facades and roofs and the background, this invention statistically analyzes the pixel proportion of each class in the training set. By using class proportion weights in the loss function, it further helps the model learn more facade samples. We employ OHEM loss with class weights and Dice loss to jointly supervise the semantic segmentation result, focusing on improving the attention to facade samples. Since N... prior Each set of 8 channels represents a connectivity feature of a category. This invention employs a bilateral voting module (BV) to select from N... prior A bidirectional connectivity probability map is obtained, namely the semantic category probability map Seg and the connectivity probability map Conn. Considering that obtaining the true connectivity labels would incur additional computational overhead, this invention only uses the semantic category probability map and the true labels for supervision, thereby indirectly learning pixel connectivity. Therefore, the hybrid loss is defined as follows:

[0125]

[0126]

[0127] Seg,Conn=BV)N prior )

[0128]

[0129] Where w c p is the weight of category c, where C is the total number of categories. i,c It is the predicted value for category c, g i,c That is the real label. These are hyperparameters, which are set to 1, 1, and 0.5 respectively in this invention.

[0130] Step 5: Finally, after multiple parameter adjustments, the optimal model is selected. The test set is used as the input image to predict the building extraction results, testing the model's performance and obtaining an evaluation of the prediction accuracy. Step 5 specifically includes the following:

[0131] First, the constructed NeSF-Net network framework is used to learn from building data of various types with different structures, shapes, and sizes. The training and validation sets obtained in step one are simultaneously fed into the NeSF-Net network. The training set is used to train the network to learn building features, while the validation set is used to verify the accuracy of the learned features. The results from the validation set are fed back into the network, allowing it to continuously update its parameters during learning to obtain the optimal learning parameters. In the experiment, the initial learning rate was set to 10e-4. The optimizer was the adaptive optimizer Adam, with a weight decay of 5e-4 and a batch size of 4. Through this iterative process, the optimal network model can be obtained.

[0132] Finally, the learning results of the optimal network model are tested using the test set from step one. Feeding the image data from the test set in step one into the optimal network model yields the network's predicted building extraction results. Comparing the prediction results with the labeled data in the test set provides the accuracy of the predictions.

[0133] In summary, this invention proposes a method and system for extracting building roofs and facades based on remote sensing imagery, achieving accurate segmentation of building roofs and facades in high-resolution remote sensing images. First, a dataset of building remote sensing images is acquired, including a training set, a validation set, and a test set. Data augmentation is performed on the images in the training and validation sets, including cropping, translation, rotation, scaling, vertical mirroring, and illumination adjustment, and salt-and-pepper noise is added to enrich the diversity of the sample dataset. Second, this method constructs an end-to-end encoder-decoder neural network framework, NeSF-Net. The encoder of the network adopts ConvNeXt, which balances performance and efficiency. To enhance the spatial consistency of roofs and facades, this invention deeply mines the relationship between target pixels and neighboring pixels in the latent space and proposes a Neighborhood Relationship Awareness Module (NRAM). This module improves the category and spatial awareness of depth features in specific directions by applying neighborhood prior extraction and sub-neighborhood prior perception. Furthermore, this invention focuses on frequency information in the decoder to help the network overcome the effects of spatial information loss and shadow occlusion. A Scale & Frequency Modulation Decoder (SFMD) is constructed, applying multiple frequency components to multiple scale branches for feature enhancement to achieve spatial-frequency domain information interaction. A Feature Aggregation Block (FAB) guides the flow of foreground and background information, ultimately improving the model's segmentation performance. Finally, the preprocessed training and validation sets are input into the network framework, and reasonable parameters are set for training to obtain the corresponding network model. Predicting model performance using the test set demonstrates that the proposed network model has strong robustness and applicability. The NeSF-Net network proposed in this invention can achieve fine segmentation of building roofs and facades, providing valuable reference for high-resolution remote sensing image mapping in complex scenes.

[0134] To evaluate the performance of the proposed network model, the following metrics are used: Overall accuracy (OA), Precision (P), Recall (R), F1 score, and Intersection over Union (IoU).

[0135]

[0136]

[0137]

[0138]

[0139] like Figure 7As shown, the input image size is 2048*2048 pixels, and the building prediction result image of the image can be obtained using the present invention.

[0140] Finally, quantitative evaluation showed that the proposed method achieved 85.06% accuracy, 81.27% recall, 83.04% F1 score, and 72.71% crossover-union ratio on the BANDON dataset. This demonstrates that the proposed NeSF-Net performs well and has strong generalization ability. NeSF-Net can accurately and completely extract building regions automatically.

[0141] In another aspect, embodiments of the present invention also provide a building roof and facade extraction system based on remote sensing imagery, comprising:

[0142] The system includes a processor and a memory. The memory stores program instructions, and the processor calls the stored instructions in the memory to execute the method for extracting building roofs and facades based on remote sensing images, as described in the above technical solution.

[0143] The above methods, in conjunction with the accompanying drawings, have described examples of the present invention. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art, under the guidance of the present invention, can make many other forms without departing from the spirit and scope of the claims, and all of these are within the protection scope of the present invention.

Claims

1. A method for extracting building roofs and facades based on remote sensing imagery, characterized in that, Includes the following steps: Step 1: Obtain the building dataset and perform preprocessing operations; Step 2: Construct an end-to-end convolutional neural network NeSF-Net. NeSF-Net includes an encoder, a neighborhood relationship awareness module, and a decoder. The encoder is used to extract features at different scales from the input image. The neighborhood relationship perception module applies neighborhood prior extraction and sub-neighborhood prior perception to enhance the category perception and spatial perception of features in a specific direction, thereby obtaining neighborhood relationship perception features; The decoder takes neighborhood relationship-aware features as input and concatenates them with the features from the previous level of the encoder, which are then processed through scale modulation, frequency modulation, and feature aggregation in sequence. The decoder's final output is then processed by the semantic header to obtain the final output result. Step 3: Design a reasonable loss function based on the characteristics of the NeSF-Net network structure, and train the constructed NeSF-Net using the preprocessed dataset from Step 1. Step 4: Use the trained network model to extract the roof and facade from the image of the building to be tested.

2. The method for extracting building roofs and facades based on remote sensing imagery as described in claim 1, characterized in that: The building dataset includes a training set, a validation set, and a test set, which are composed of image data and label data, respectively. All data in the training and validation sets are cropped into H×W pixels, where H and W represent the height and width of the image, respectively. Data augmentation is performed on the image data in the training and validation sets using methods such as translation, rotation, scaling, vertical mirroring, illumination adjustment, and adding salt and pepper noise points to obtain augmented image data.

3. The method for extracting building roofs and facades based on remote sensing imagery as described in claim 1, characterized in that: The encoder section contains several ConvNeXt Blocks. For the input image I, after several ConvNeXt Blocks and downsampling operations, the output features e of each layer of the encoder are obtained. l , l represents the index of the layer in the encoder; The decoder consists of multiple scale and frequency modulation decoders composed of scale modulation modules, frequency modulation modules, and feature aggregation blocks. The input features of the decoder first need to be upsampled, then concatenated with the previous level features from the encoder, and finally fed into the scale and frequency modulation decoder after convolution. The input features are processed by multiple scale and frequency modulation decoders in the decoder, and finally the semantic head is used to obtain the final output of the network.

4. The method for extracting building roofs and facades based on remote sensing imagery as described in claim 1, characterized in that: The specific implementation method of neighborhood prior extraction is as follows: The output feature of the last layer of the encoder is defined as X, and neighborhood prior information is extracted; an 8-directional connectivity method is used to process the input features. The channel information is compressed and embedded to obtain different categories of neighborhood prior information. N represents the number of categories; coarse supervision is applied to neighborhood prior information to impose connectivity constraints; specifically, N prior The output channels are 8×N, and are upsampled to the input size for monitoring purposes; then, N is compressed using a global average pooling operation. prior The ReLU activation function and fully connected convolution are used to map it into a vector with the same channel dimension as X; finally, the projected neighborhood prior vector is obtained by recoding and normalization through 1×1 convolution and the Sigmoid function. Each element in this neighborhood prior vector embeds pixel connectivity information in each direction, as expressed by the following formula: s prior =γ(ω 1×1 d(ω 1×1 (GAP(N prior )))) Where γ and δ represent the Sigmoid and ReLU activation functions, respectively, and ω 1×1 (·) represents a convolution operation with a kernel of 1, GAP represents a global average pooling operation, x represents the input, and i,j represent the number of rows and columns; Next, X and σ are sliced ​​separately using channel slicing. prior Divided into 8 sub-parts and Where i∈[1,8], the purpose is to understand the category information and pixel neighborhood information in the entangled hidden layer.

5. The method for extracting building roofs and facades based on remote sensing imagery as described in claim 4, characterized in that: The specific implementation of sub-neighborhood prior perception is as follows: For each group of sub-path features x i Prior information of sub-neighborhood Cross-attention, global attention, and local attention are performed separately to guide the attention of each sub-path in different neighborhood directions. The specific methods are as follows: (i): Cross-attention: First, from Generate query vector Q from x i The function generates a key vector K and a value vector V. Attention weights are obtained by calculating the dot product of Q and K and applying the Softmax function. Then, the attention weights are weighted and summed with V, and projected back into the original feature space. Finally, the function... The residuals are then reconnected to maintain feature integrity, thus obtaining the enhanced neighborhood features. The specific formula is as follows: Where d k It represents the dimension of the key vector, used to scale the dot product result; Proj indicates the projection transformation. (ii): Global attention: First, W-Trans block is used to focus on the feature map x. i The window is partitioned into four windows, and the characteristics of the partitioned window are as follows: j∈[1,4]; then apply vertical stripe convolution to the features of each partition to create long-range dependencies between partitions; features enhanced with global information It can be obtained through the following formula: Where Cat(·) represents channel concatenation, ξ represents batch normalization, and ω 3×3 For a regular 2D convolution with a kernel of 3, ω vs This indicates vertical stripe convolution, where the subscripts k×k, 1×k, and k×1 represent different settings in vertical stripe convolution. (iii): Local attention: Pooling pyramids are used to represent multi-scale information of sub-path features. Multiple pooling operations are set up, and the multi-scale features are concatenated along the channel dimension. After convolution and batch normalization, the scale-enhanced features are obtained. The above process can be expressed by the following formula: f BL (x i )=ω 3×3 (oh 1×1 (x i )) f MS (x i )=Cat(M 5×5 (f BL (x i )),M 9×9 (f BL (x i )),M 13×13 (f BL (x i )),x i ) Where f BL (·) represents the Bottleneck layer, ω 1×1 For a regular 2D convolution with a kernel of 1, f MS (·) indicates multi-scale operations, M k×k This indicates a max pooling operation with kernel k, where k can be 5, 9, or 13. (iv): For the three attention features obtained In other words, due to It contains neighborhood information in a specific direction. and The middle part contains enhanced category information. Element-wise multiplication can selectively highlight pixel correlations within the feature latent space. Therefore, the following formula is used to obtain the sub-neighborhood-aware features. Finally, all sub-neighborhood perceived features y i Channel stacking and recoding are performed to obtain neighborhood relationship-aware features Y, which are then used as input features for the decoder in subsequent decoding processes.

6. The method for extracting building roofs and facades based on remote sensing imagery as described in claim 3, characterized in that: The scaling module has S parallel branches, each of which records the features learned using different scales; For each s-scale branch, average pooling and convolution operations are first performed to adjust the scale and channels of the input feature F to improve computational efficiency, as shown in the following formula: Where 1≤s≤S, δ represents the ReLU activation function, and ξ represents layer normalization. Represents the convolution kernel. Indicates step size, Indicates the number of fillers. The hole ratio is represented by W; W and GAP represent the convolution operation and global average pooling, respectively. Immediately afterwards, The data is fed into the frequency modulation module and the feature aggregation block, where information interacts in the spatial and frequency domains to obtain the feature map. Among them, C s H s W s Representing features F respectively s The number of channels, length, and width; To balance the feature contributions from different scales, features from all scale branches are aggregated, their mean is calculated, and residual connections are made with the input features to obtain the final features. The specific formula is as follows: Where Z is the output of a single scale and frequency modulation decoder, and c, h, and w represent the number of channels, length, and width of feature Z, respectively.

7. The method for extracting building roofs and facades based on remote sensing imagery as described in claim 6, characterized in that: The specific processing procedure of the frequency modulation module is as follows: First, the output features of the S-th scale branch are... Divide into n sub-blocks according to the channel dimension. i∈[0,1,…,n-1], for each Assign a corresponding 2D DCT frequency component And calculate the weighted sum of the two, as shown in the following formula: Where (u k ,v k )express The corresponding frequency component exponents are determined using a top-K screening strategy. s Indicates the weighted result; frequency component It can be calculated using the following formula: Where h and w represent the length and width of the frequency component, respectively; Subsequently, various pooling operations can be used to obtain... More refined channel information is obtained; the features obtained after average pooling, min pooling, and max pooling operations are sequentially passed through two fully connected layers and then element-wise summed; finally, the attention map after multi-frequency modulation is obtained through the Sigmoid function, and then multiplied with the original features for recalibration, as shown in the formula: in and This represents a fully connected layer, where δ represents the ReLU activation function and γ represents the Sigmoid activation function. Indicates to The results obtained using multiple pooling operations This indicates the characteristics of the frequency modulation module output.

8. The method for extracting building roofs and facades based on remote sensing imagery as described in claim 7, characterized in that: The processing procedure for feature aggregation blocks is as follows: Input features Spatial attention map of foreground information is obtained after 1×1 convolution and sigmoid function. Accordingly, background information is provided through The result is obtained by subtracting the matrix element by element. Then, two learnable parameters, α and β, are introduced to guide the information flow between the foreground and the background, thereby reducing the risk of category confusion caused by various interfering factors. Finally, the channels and sizes of the features are unified through convolution and upsampling operations to facilitate subsequent feature aggregation. Obtain feature map The specific formula is described as follows: Among them, Up s Indicates upsampling, ω 3×3 This is a regular 2D convolution with a kernel of 3.

9. The method for extracting building roofs and facades based on remote sensing imagery as described in claim 1, characterized in that: The overall loss function includes the OHEM loss with class weights, the Dice loss, and the neighborhood prior loss. By reasonably setting the weight ratio of each loss, the convergence of the network can be achieved. The formula for calculating the overall loss function is as follows: Seg,Conn=BV(N prior ) An 8-directional connectivity approach is used to compress and embed the channel information in the output features of the last layer of the encoder to obtain the neighborhood prior information N for different categories. prior Due to the prior information N in the neighborhood prior Each set of 8 channels represents a connectivity feature of a category, and the bilateral voting module BV is used to select from N. prior Obtain the bidirectional connectivity probability graph, namely the semantic category probability graph Seg and the connectivity probability graph Conn; where L OHEM For the OHEM loss with class weights, L Dice For Dice's loss, For neighborhood prior loss, w c p is the weight of category c, where C is the total number of categories. i,c It is the predicted value for category c, g i,c η, θ, and θ are the real labels, while η, θ, and θ are hyperparameters.

10. A system for extracting building roofs and facades based on remote sensing imagery, characterized in that, include: The processor and memory, wherein the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the method for extracting building roofs and facades based on remote sensing images as described in any one of claims 1-9.

Images