Building roof segmentation method and system based on unmanned aerial vehicle oblique image

Abstract

The application discloses a building roof segmentation method and system based on unmanned aerial vehicle oblique images. The method comprises the following steps: acquiring a building oblique aerial image through an unmanned aerial vehicle; inputting the building oblique aerial image into a trained edge-guided coordinate attention mechanism network model to obtain a building roof segmentation result; wherein the edge-guided coordinate attention mechanism network model comprises an encoder and a decoder, the encoder comprises a plurality of cascaded encoding layers, each encoding layer comprises a coordinate attention U-shaped block, and the decoder comprises a plurality of cascaded decoding layers, each decoding layer comprises a coordinate attention U-shaped block and an edge-guided attention module. The application can extract a roof mask with complete geometric structure and consistent topology, and can restore a clear and sharp roof contour.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, system, storage medium, and electronic device for segmenting building roofs based on UAV tilted images. Background Technology

[0002] Accurately segmenting building rooftops to assess their photovoltaic power generation potential has become crucial for urban energy planning, and rooftop segmentation technology based on remote sensing imagery offers a feasible solution. Compared to orthophotos, oblique aerial imagery provides multi-dimensional geometric clues, making it an ideal data source for refined analysis. However, in reality, complex urban environmental interferences (such as shadows and distortions) and geometric distortions from the oblique perspective still pose significant challenges to high-precision rooftop segmentation in practical applications.

[0003] At present, semantic segmentation technology on two-dimensional images and roof extraction of buildings have made some progress. Traditional image processing and machine learning methods mostly rely on morphological operations (such as the watershed algorithm) or support vector machine (SVM) classifiers based on hand-designed descriptors (such as spectral and texture features), which rely on manual feature engineering and have limited feature expression capabilities. In practical applications, they are extremely sensitive to changes in lighting, differences in roof materials and multi-scale building patterns, and have poor generalization ability and robustness in complex urban scenes. In contrast, deep learning automatically extracts and learns semantic features through neural networks, which improves the accuracy and generalization ability of recognition. Considering the huge computational burden and difficulty in data annotation when directly using three-dimensional models for recognition and extraction in complex urban environments, the more operational implementation path is to use the semantic segmentation results of two-dimensional images to guide the automatic extraction of target objects from three-dimensional models. However, to achieve high-quality three-dimensional reconstruction through two-dimensional segmentation, there are still two major technical bottlenecks: (1) the problem of geometric integrity and multi-view aggregation under tilted perspective. In oblique aerial photography, a single image can only observe a part of the roof due to occlusion and projection effects. If the two-dimensional segmentation results lack geometric consistency, direct projection onto the model will form broken three-dimensional fragments. (2) Existing deep learning models often lack explicit modeling of long-range spatial dependencies and axial prior information, which makes it difficult to maintain the structural continuity in the horizontal and vertical directions when dealing with roofs with irregular shapes and complex structures. Summary of the Invention

[0004] This invention provides a method, system, storage medium, and electronic device for building roof segmentation based on UAV tilted imagery, which can extract a roof mask with complete geometric structure and consistent topology, and can recover a clear and sharp roof outline.

[0005] This invention provides a method for building roof segmentation based on UAV oblique imagery, comprising: Acquire aerial images of tilted buildings using drones; The tilted aerial image of the building is input into a trained edge-guided coordinate attention mechanism network model to obtain the building roof segmentation result; The edge-guided coordinate attention mechanism network model includes an encoder and a decoder. The encoder includes multiple cascaded encoding layers, each of which includes a coordinate attention U-shaped block. The decoder includes multiple cascaded decoding layers, each of which includes a coordinate attention U-shaped block and an edge-guided attention module.

[0006] Furthermore, according to the above-mentioned building roof segmentation method based on UAV oblique imagery, the edge-guided coordinate attention mechanism network model further includes a Laplacian pyramid sub-network; the processing procedure of the edge-guided coordinate attention mechanism network model includes: The tilted aerial image of the building is input into the encoder, and multiple cascaded coding layers output coded feature maps of different scales respectively; The tilted aerial image of the building is converted into a grayscale image, and the grayscale image is input into the Laplacian pyramid sub-network to obtain an edge feature map; The encoded feature map and the edge feature map are input into the decoder to obtain the building roof segmentation result.

[0007] Furthermore, according to the above-mentioned building roof segmentation method based on UAV tilted imagery, the coordinate attention U-shaped blocks in the first few coding layers of the encoder are ordinary CAUs with different sampling layers, and the coordinate attention U-shaped blocks in the last coding layer of the encoder are flattened CAUs. The processing procedure of the ordinary CAU includes: The input image or input feature map is fed into the initial convolution module for processing to obtain the first feature map; the initial convolution module is configured to perform convolution operation, batch normalization operation and non-linear activation function mapping in sequence. The first feature map is split into paths, wherein the first feature map of the first path is input to the coordinate attention module, and the first feature map of the second path is input to the multi-level downsampling processing module to obtain the second feature map; wherein the multi-level downsampling processing module has several downsampling units stacked consecutively, and each downsampling unit is configured to perform downsampling, convolution operation, batch normalization and activation function processing; The second feature map is input into the secondary convolution module for processing to obtain the third feature map; the secondary convolution module is configured to perform convolution operation, batch normalization operation and non-linear activation function mapping in sequence; The third feature map is input into the multi-level downsampling processing module to obtain the fourth feature map; wherein, the multi-level upsampling processing module has several upsampling units stacked consecutively, each upsampling unit is configured to perform upsampling, convolution operation, batch normalization and activation function processing, and each upsampling unit is also configured to receive the intermediate feature map output by the corresponding upsampling unit through skip connections; The fourth feature map and the first feature map are input into the coordinate attention module to obtain the output feature map of the ordinary CAU; The process of flattening the CAU includes: The input feature map is fed into a depthwise separable convolutional module for processing to obtain the fifth feature map; The sixth feature map is split into paths, wherein the fifth feature map of the first path is input into the coordinate attention module, and the fifth feature map of the second path is input into the multi-level depthwise separable dilated convolution module to obtain the sixth feature map. The sixth feature map and the fifth feature map are input into the coordinate attention module to obtain the output feature map of the flattened CAU.

[0008] Furthermore, according to the above-described method for building roof segmentation based on UAV tilted imagery, the processing procedure of the coordinate attention module includes: The input feature map is subjected to width-average pooling in the height direction to obtain the first pooled feature map, and the input feature map is subjected to height-average pooling in the width direction to obtain the second pooled feature map. The first pooling feature map and the second pooling feature map are concatenated and then subjected to convolution, batch normalization and activation function mapping operations to obtain nonlinear features; The nonlinear feature is split into a height orientation feature map and a width orientation feature map. The height orientation feature map and the height orientation feature map are respectively processed by a convolutional layer and an activation function to obtain height attention weights and width attention weights. The height attention weight and the width attention weight are applied to the input feature map through a broadcast mechanism to obtain the output feature map of the coordinate attention module.

[0009] Furthermore, according to the above-described method for building roof segmentation based on UAV tilted imagery, the edge-guided attention module includes multiple cascaded EGA blocks, each EGA block being connected to the corresponding coordinate attention U-shaped block in the decoding layer; the processing procedure of the decoding layer includes: The encoded feature map output by the corresponding encoding layer and the initial decoded feature map output by the coordinate attention U-shaped block from the previous decoding layer are fused and then input into the coordinate attention U-shaped block of the current decoding layer to obtain the initial decoded feature map of the current decoding layer. The edge feature map, the initial decoding feature map, and the segmentation prediction map output from the EGA block of the previous decoding layer are input into the EGA block of the current decoding layer to obtain the segmentation prediction map of the current decoding layer.

[0010] Furthermore, according to the above-described method for building roof segmentation based on UAV oblique imagery, the processing of the EGA block includes: The Laplacian operator is applied to the segmentation prediction map of the EGA block output from the previous decoding layer to obtain boundary attention and reverse attention; The initial decoded feature map is modulated by the edge feature map, the boundary attention, and the reverse attention respectively to obtain the edge modulation feature map, the boundary modulation feature map, and the reverse modulation feature map; The edge modulation feature map, the boundary modulation feature map, and the inverse modulation feature map are concatenated along the feature channel dimension and fused by convolution. Then, batch normalization and nonlinear activation are applied to obtain the fused feature. The fused features are used to generate a single-channel attention map through lightweight convolution; The fused features are weighted and adjusted based on the single-channel attention map to obtain optimized features; The optimized features are concatenated with the initial decoded feature map and then input into the CBAM module for recalibration of the channel dimension and spatial dimension to obtain the output of the EGA block.

[0011] Furthermore, according to the above-mentioned method for building roof segmentation based on UAV oblique imagery, the edge-guided coordinate attention mechanism network model is trained using a total loss function, which is:

[0012] in, For the total loss function, The loss is the segmentation prediction map output by each decoding layer. The loss is the segmentation prediction map output by the last decoding layer. and The weighting coefficient for each loss term; The loss of the segmentation prediction map is calculated using the following formula:

[0013] in, The loss of the segmentation prediction map output for each decoding layer, Represents the coordinates of pixels in an image. Indicates the height and width of the image. Indicates coordinates The pixel values ​​of the actual mask. Indicates coordinates The pixel probability values ​​of the model's predicted probability map.

[0014] The present invention also provides a building roof segmentation system based on UAV oblique imagery, comprising: Image acquisition module, used to acquire tilted aerial images of buildings via drones; The building roof segmentation module is used to input the tilted aerial image of the building into a trained edge-guided coordinate attention mechanism network model to obtain the building roof segmentation result; The edge-guided coordinate attention mechanism network model includes an encoder and a decoder. The encoder includes multiple cascaded encoding layers, each of which includes a coordinate attention U-shaped block. The decoder includes multiple cascaded decoding layers, each of which includes a coordinate attention U-shaped block and an edge-guided attention module.

[0015] The present invention also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute any of the above-described methods for building roof segmentation based on UAV oblique imagery.

[0016] The present invention also provides an electronic device, including a processor and a memory, wherein the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used in the steps of the building roof segmentation method based on UAV tilted imagery described in any of the preceding claims.

[0017] The present invention provides a method, system, storage medium, and electronic device for building roof segmentation based on UAV oblique imagery. The present invention has the following beneficial effects: (1) In the residual U-block of the U2net framework, a coordinate attention mechanism is introduced to design a coordinate attention U-block (CAU). This module enables the network model to explicitly capture geometric relationships. While maintaining accurate positioning information, it ensures that the geometrically complete and topologically consistent roof mask can still be extracted under the geometric distortion caused by tilted aerial photography.

[0018] (2) Based on the U2net framework, a multi-scale edge-guided attention (EGA) mechanism is introduced in the decoding stage. By utilizing explicit high-frequency edge cues based on the Laplacian operator and combining them with prediction-guided attention responses, the segmentation of the roof boundary region is gradually refined. Even in cases of shadow interference in oblique aerial photography, this invention can still recover a clear and sharp roof outline.

[0019] (3) The proposed Edge Guided Coordinate Attention Mechanism (ECAUs) network models are robust and data-applicable. They can capture hierarchical information of complex buildings through multi-scale feature fusion without the need for pre-training of backbone network models. This achieves efficient and automated end-to-end extraction, saving a significant amount of manpower and resources for city-level rooftop photovoltaic potential assessment and low-carbon transformation planning. Attached Figure Description

[0020] The technical solution and other beneficial effects of the present invention will become apparent from the following detailed description of specific embodiments of the invention, in conjunction with the accompanying drawings.

[0021] Figure 1 A flowchart of a building roof segmentation method based on UAV tilted imagery provided in an embodiment of the present invention.

[0022] Figure 2 This is a schematic diagram of the structure of the edge-guided coordinate attention mechanism network provided in an embodiment of the present invention.

[0023] Figure 3 This is a schematic diagram of the coding layer provided in an embodiment of the present invention.

[0024] Figure 4 This is a flowchart illustrating the processing of an EGA block provided in an embodiment of the present invention.

[0025] Figure 5 The oblique aerial images acquired are provided for embodiments of the present invention.

[0026] Figure 6 This is a diagram illustrating the segmentation effect on a drone-based tilted rooftop image dataset provided in an embodiment of the present invention.

[0027] Figure 7 The image shows a comparison of the segmentation effects of the two improvements provided in the embodiments of the present invention with the final model.

[0028] Figure 8 This is a schematic diagram of the structure of a building roof segmentation system based on UAV tilted imagery provided in an embodiment of the present invention.

[0029] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] This invention provides a method, system, storage medium, and electronic device for building roof segmentation based on UAV oblique imagery. The building roof segmentation system based on UAV oblique imagery provided by this invention can be integrated into an electronic device, which can be a terminal, server, or other device. The terminal can include a tablet computer, laptop computer, personal computer (PC), microprocessor box, or other devices.

[0032] Please see Figure 1 , Figure 1 The flowchart illustrates a method for building roof segmentation based on UAV oblique imagery, provided in an embodiment of the present invention. This method, applied in electronic devices, includes the following steps: S1 uses drones to acquire tilted aerial images of buildings.

[0033] S2. The tilted aerial image of the building is input into the trained Edge-Guided Coordinate Attention U-Square Network (ECAUs) model to obtain the building roof segmentation result.

[0034] The edge-guided coordinate attention mechanism network model includes an encoder and a decoder. The encoder includes multiple cascaded encoding layers, each of which includes a coordinate attention U-shaped block. The decoder includes multiple cascaded decoding layers, each of which includes a coordinate attention U-shaped block and an edge-guided attention module.

[0035] In one embodiment, the edge-guided coordinate attention mechanism network model further includes a Laplacian pyramid subnetwork. Figure 2 Please refer to the schematic diagram of the edge-guided coordinate attention mechanism network provided in the embodiment of the present invention. Figure 2 The processing steps of the edge-guided coordinate attention mechanism network model include: S21, the tilted aerial image of the building is input into the encoder, and multiple cascaded coding layers output coded feature maps of different scales respectively.

[0036] The coordinate attention U-blocks in the first few coding layers of the encoder are ordinary CAUs (Coordinate Attention-Based Residual Ublocks) with different sampling layers, while the coordinate attention U-blocks in the last coding layer of the encoder are flattened CAUs.

[0037] Figure 3 This is a schematic diagram of the coding layer structure provided in an embodiment of the present invention. Figure 3(a) in the diagram is a schematic diagram of the flattened CAU structure. Figure 3 (b) in the text represents a regular CAU. Figure 3 (c) in the diagram is a schematic diagram of the coordinate attention module (CA), as shown below. Figure 3 As shown, in a specific embodiment, the first four coding layers of the encoder and decoder are ordinary CAU (represented by CAU_L, where L is the number of sampling layers), and the last coding layer is flat CAU (represented by CAU_4F, which is a flat CAU_4 with 4 sampling layers).

[0038] It should be noted that the number of sampling layers for the ordinary CAU in the first four coding layers is different, at 7, 6, 5, and 4 respectively. The number of sampling layers for the ordinary CAU in the decoding layer is the same as that in the coding layer. The flattened CAU uses dilated convolutions with different dilation rates (d=2, 4, 8 in the figure) to replace the original downsampling and upsampling to expand the receptive field. Figure 3 In case (b) CAU_L is L=7, it performs 4 downsampling and upsampling operations. Different values ​​of L (L=4, 5, 6, 7) represent different depths and the number of downsampling and upsampling operations. If L=6, it performs 3 downsampling and upsampling operations, and so on.

[0039] The processing procedure for a standard CAU includes: S211, the input image or input feature map (the first coding layer processes the input building tilted aerial image, and the remaining coding layers process the feature map output by the previous coding layer) is input into the initial convolution module for processing to obtain the first feature map; the primary convolution module is configured to perform convolution operation, batch normalization operation and non-linear activation function (ReLU) mapping in sequence.

[0040] Specifically, the primary convolution module is configured to undergo two convolution + batch normalization + ReLU activation operations. The result obtained from the first operation is input into the coordinate attention module, and the result obtained from the second operation is input into the fifth upsampling unit.

[0041] S212, perform path splitting on the first feature map, wherein the first feature map of the first path is input to the coordinate attention module, and the first feature map of the second path is input to the multi-level downsampling processing module to obtain the second feature map; wherein the multi-level downsampling processing module has several downsampling units stacked consecutively, and each downsampling unit is configured to perform downsampling, convolution operation, batch normalization and activation function processing.

[0042] Specifically, there are five downsampling units and five upsampling units. The first feature map is processed step by step by the five downsampling units to obtain the second feature map. During the processing, the output of each downsampling unit is connected to the corresponding upsampling unit. For example, the input of the first downsampling unit is input to the fourth upsampling unit, the input of the second downsampling unit is input to the third upsampling unit, the input of the third downsampling unit is input to the second upsampling unit, the input of the second downsampling unit is input to the first upsampling unit, and the output of the first downsampling unit is input to the secondary convolutional module.

[0043] S213, the second feature map is input into the secondary convolution module for processing to obtain the third feature map; the secondary convolution module is configured to perform convolution operation, batch normalization operation and non-linear activation function mapping in sequence.

[0044] S214, the third feature map is input into the multi-level downsampling processing module to obtain the fourth feature map; wherein, the multi-level upsampling processing module has several upsampling units stacked consecutively, each upsampling unit is configured to perform upsampling, convolution operation, batch normalization and activation function processing, and each upsampling unit is also configured to receive the intermediate feature map output by the corresponding upsampling unit through skip connections.

[0045] S215, input the fourth feature map and the first feature map into the coordinate attention module to obtain the output feature map of the ordinary CAU.

[0046] The processing steps of the coordinate attention module include: S2151, perform width-average pooling on the input feature map in the height direction to obtain the first pooled feature map, and perform height-average pooling on the input feature map in the width direction to obtain the second pooled feature map.

[0047] The size is The input feature map (number of channels × height × width) is subjected to W Avg Pooling in the H (height) direction, preserving feature information in the W direction. The size of the output first pooled feature map becomes Perform H-Avg Pooling (height-average pooling) in the W (width) direction to preserve feature information in the H direction. The size of the output second pooled feature map becomes... In this way, the feature map not only retains channel information, but also preserves spatial features in the horizontal and vertical directions.

[0048] S2152, after concatenating the first pooling feature map and the second pooling feature map, nonlinear features are obtained through convolution operation, batch normalization processing and activation function mapping operation.

[0049] S2153 decomposes the nonlinear features into height-oriented feature maps and width-oriented feature maps. The height-oriented feature maps and width-oriented feature maps are then processed by convolutional layers and activation functions respectively to obtain height attention weights and width attention weights.

[0050] S2154 applies height and width attention weights to the input feature map through a broadcasting mechanism to obtain the output feature map of the coordinate attention module.

[0051] After concatenating the first and second pooling feature maps, a 2D convolution is applied for dimensionality reduction. Following batch normalization and a ReLU activation function, non-linear features are extracted and then split back into two feature maps for the height and width directions. Each map is then passed through its own Conv2d layer to restore the channel count to the original C. A Sigmoid activation function maps the values ​​to the range (0,1), generating attention weights for the height and width directions. Finally, these two attention weight maps are multiplied (using a broadcast mechanism) and applied to the original feature map. Input feature map.

[0052] The activation function mapping operation can be represented by the following formula:

[0053]

[0054] in, This represents the input value of the activation function. This represents the output value of the activation function. This indicates a maximum value operation, meaning that when the input is greater than 0, the output value is equal to 0. When the input is less than or equal to 0, the output is equal to 0. e represents a mathematical constant approximately equal to 2.71828.

[0055] The process of flattening CAU includes: S216, the input feature map is fed into the depthwise separable convolution module for processing to obtain the fifth feature map.

[0056] S217, the sixth feature map is split into paths, where the fifth feature map of the first path is input into the coordinate attention module, and the fifth feature map of the second path is input into the multi-level depthwise separable dilated convolution module to obtain the sixth feature map.

[0057] The multi-level depth-separable dilated convolution module includes seven cascaded depth-separable dilated convolution units, with the first three depth-separable dilated convolution units and the last three depth-separable dilated convolution units being connected in a skip connection.

[0058] S218, input the sixth and fifth feature maps into the coordinate attention module to obtain the output feature map of the flattened CAU.

[0059] S22, convert the tilted aerial image of the building into a grayscale image, and input the grayscale image into the Laplacian pyramid sub-network to obtain the edge feature map.

[0060] First, convert the aerial image X of the building's tilt into a single-channel grayscale image. Each Laplacian pyramid layer includes Gaussian filtering, double downsampling, and upsampling. The low-frequency image output from the k-th Laplacian pyramid layer and the high-frequency Laplacian residual are expressed as follows:

[0061]

[0062] Where G represents a Gaussian kernel of size 5×5, and These represent double downsampling and Gaussian upsampling, respectively.

[0063] The embodiments of the present invention use high-frequency parameters As an edge-feature, it is extracted directly from the input image by a fixed operator. It contains the most realistic high-frequency edge information that is least affected by the model's subjective judgment, providing the model with stable and reliable edge prior knowledge.

[0064] S23, input the encoded feature map and edge feature map into the decoder to obtain the building roof segmentation result.

[0065] In one embodiment, the edge-guided attention module includes multiple cascaded EGA blocks, each EGA block being connected to a corresponding coordinate attention U-shaped block in the decoding layer; the processing of the decoding layer includes: S231, the encoded feature map output by the corresponding encoding layer and the initial decoded feature map output by the coordinate attention U-shaped block of the previous decoding layer are fused and input into the coordinate attention U-shaped block of the current decoding layer to obtain the initial decoded feature map of the current decoding layer.

[0066] First, the input to the first decoding layer is described. Preferably, in the encoder, a CAU_4F identical to the last coding layer is introduced. After the last coding layer outputs the coded feature map, it is input into the introduced CAU_4F to obtain the coded feature map d6 used for the first decoder. Next, this coded feature map is fused with the coded feature map output by the last encoder layer and then input into the flattened CAU of the first decoding layer to obtain the initial decoded feature map output by the first decoding layer.

[0067] In the second decoding layer, the input of the ordinary CAU is the fusion feature of the initial encoded feature map output by the fourth encoding layer and the initial decoded feature map output by the flattened CAU in the first decoding layer. The input of subsequent decoding layers follows the same principle.

[0068] S232, input the edge feature map, the initial decoded feature map, and the segmentation prediction map output from the EGA block of the previous decoding layer into the EGA block of the current decoding layer to obtain the segmentation prediction map of the current decoding layer.

[0069] It should be noted that the "segmentation prediction map from the output of the EGA block of the previous decoding layer" as the input of the first layer EGA block is actually the encoded feature map d6 used for the first layer decoder.

[0070] The segmentation prediction map Pred from the EGA output of the previous decoding layer provides a clear optimization objective and contextual information, enabling the network to iterate layer by layer, continuously approaching the most accurate boundary. The initial decoded feature map Xid integrates deep semantic information (from the previous decoding layer) and shallow local details (from the corresponding encoding layer), but its segmentation of object boundaries may still be insufficiently accurate; the initial decoded feature map output by the coordinate attention U-shaped block of each decoder layer is the main enhancement target of the EGA module.

[0071] Figure 4 The flowchart of the EGA block processing provided in the embodiment of the present invention is as follows: Figure 4 As shown, the processing procedure for an EGA block includes: S2321, apply the Laplacian operator to the segmentation prediction graph of the EGA block output from the previous decoding layer to obtain boundary attention and reverse attention.

[0072] The segmentation prediction map from the EGA output of the previous decoding layer Transformed into a foreground probability map using the Sigmoid activation function. : .

[0073] Foreground probability map Applying the Laplacian operator to obtain boundary attention :

[0074] Among them, Gaussian filtering and scale reconstruction can remove low-frequency region information in the prediction map. The reconstruction residual mainly responds to the location where the foreground probability changes rapidly, that is, the contour region of the predicted target.

[0075] Reverse attention The calculation is as follows: This allows the model to re-examine areas previously classified as background, thereby identifying potential foreground areas that were missed at deeper levels.

[0076] S2322 modulates the initial decoded feature map using edge feature map, boundary attention, and reverse attention respectively, to obtain edge modulation feature map, boundary modulation feature map, and reverse modulation feature map.

[0077] The edge modulation feature map is obtained by multiplying the edge feature map and the initial decoded feature map element by element. The boundary modulation feature map is obtained by multiplying the boundary attention and the initial decoded feature map element by element. The inverse modulation feature map is obtained by multiplying the inverse attention and the initial decoded feature map element by element.

[0078] S2323 concatenates the edge modulation feature map, boundary modulation feature map, and reverse modulation feature map along the channel dimension of the feature map, and fuses them through (3×3) convolution. Then, it is processed by batch normalization and non-linear activation ReLU to obtain the fused feature.

[0079] S2324 generates a single-channel attention map by using lightweight convolution to fuse the features.

[0080] S2325, based on the single-channel attention map, the fused features are weighted and adjusted to obtain optimized features.

[0081] S2326, after concatenating the optimized features with the initial decoded feature map, inputs them into the CBAM module for recalibration of the channel dimension and spatial dimension, and obtains the output of the EGA block.

[0082] To suppress spurious responses and highlight structurally significant regions, lightweight convolutions are used to generate single-channel attention maps, which are then used to weight and adjust the fused feature representations. Residual connections are then used to combine the optimized features with the initial decoded feature map to maintain the integrity of the contextual information.

[0083] The CBAM module (Convolutional Block Attention Module, a lightweight attention mechanism module) includes channel attention and spatial attention mechanisms. The channel dimension and spatial dimension are recalibrated through the CBAM module to obtain the output of the EGA block.

[0084] Finally, the segmentation prediction map output by the EGA block of each decoding layer is upsampled to the input size, then concatenated by channels and subjected to convolution and Sigmoid activation function to obtain the final building roof segmentation result.

[0085] The training process of the edge-guided coordinate attention mechanism network model is described below: First, a dataset was created using tilted aerial images of buildings in urban scenes captured by drones. Figure 5 The oblique aerial images acquired in the embodiments of the present invention, such as Figure 5 As shown, the first row of images is an oblique aerial image of the building, and the second row is a pixel-level classification image separating the roof from the background. The roof portion of the image is labeled using dataset creation software to obtain pixel-level classification images, which are then used for subsequent model training. Next, the oblique aerial image of the building is input into the edge-guided coordinate attention mechanism network model to obtain the building roof segmentation result. Based on the building roof segmentation result, a total loss function is constructed, and the edge-guided coordinate attention mechanism network model is trained based on the total loss function. The total loss function is:

[0086] in, For the total loss function, The loss is the segmentation prediction map output by each decoding layer. The loss is the segmentation prediction map output by the last decoding layer. and The weighting coefficient for each loss term; The loss of the segmentation prediction map is calculated using the following formula:

[0087] in, The loss of the segmentation prediction map output for each decoding layer, Represents the coordinates of pixels in an image. Indicates the height and width of the image. Indicates coordinates The pixel values ​​of the actual mask. Indicates coordinates The pixel probability values ​​of the model's predicted probability map.

[0088] Figure 6 This is a diagram illustrating the segmentation effect on a drone-based tilted rooftop image dataset provided in an embodiment of the present invention. Figure 6 The models used in (a) to (h) are FPN, DeepLabV3+, UNet++, U2Net, UMamba, UNetFormer, SwinUNet, and ECAUs, respectively. GT represents the actual roof pixel classification map. Figure 6 The first and second columns of images are the segmentation results of building 1 after processing by various models. The first row, from left to right, consists of the real image and the actual roof pixel classification image, while the remaining rows represent the segmentation results obtained by the corresponding models. Figure 6The third and fourth columns of the image are the segmentation results of building 2 after processing by various models. The first row, from left to right, is the real image and the actual roof pixel classification image, and the remaining rows represent the segmentation results of the corresponding models. Figure 6 The fifth and sixth columns of the image are the segmentation results of building 3 after processing by various models. The first row, from left to right, is the real image and the actual roof pixel classification image, and the remaining rows represent the segmentation results of the corresponding models. Figure 6 The fifth and sixth columns of the image are the segmentation results of building 4 after processing by various models. The first row, from left to right, is the real image and the actual roof pixel classification image, and the remaining rows represent the segmentation results of the corresponding model processing. Figure 7 The image shows a comparison of the segmentation effects of the two improvements provided in the embodiments of the present invention with the final model. Figure 7 The first row and fourth row of the first column contain real images, while the second row and third row of the first column contain pixel-based classification images of the actual rooftops. Figure 7 Columns (a) to (d) in the figure represent the segmentation results obtained by using the base model U2net without any improvements, improving U-block CAUs by only introducing a coordinate attention mechanism, and introducing multi-scale edge-guided attention mechanisms EGAUs and ECAUs only in the decoding stage, respectively. Figure 6 and Figure 7 As can be seen, the ECAUs model combines the two improvements, resulting in more complete roof segmentation with fewer misclassifications of background as roof, and more refined boundary segmentation.

[0089] Based on the method described in the above embodiments, this embodiment will further describe the building roof segmentation system based on UAV oblique imagery. The building roof segmentation system based on UAV oblique imagery can be implemented as an independent entity or integrated into an electronic device. The electronic device can be a terminal, server, or other device. The terminal can include a tablet computer, laptop computer, personal computer (PC), microprocessor box, or other devices.

[0090] Please see Figure 8 , Figure 8 This invention specifically describes a building roof segmentation system based on UAV oblique imagery, applied in electronic devices. The system may include: Image acquisition module, used to acquire tilted aerial images of buildings via drones; The building roof segmentation module is used to input the tilted aerial image of the building into a trained edge-guided coordinate attention mechanism network model to obtain the building roof segmentation result; The edge-guided coordinate attention mechanism network model includes an encoder and a decoder. The encoder includes multiple cascaded encoding layers, each of which includes a coordinate attention U-shaped block. The decoder includes multiple cascaded decoding layers, each of which includes a coordinate attention U-shaped block and an edge-guided attention module.

[0091] In specific implementation, the above modules and / or units can be implemented as independent entities, or they can be arbitrarily combined and implemented as the same or several entities. For the specific implementation of the above modules and / or units, please refer to the previous method embodiments. For the specific beneficial effects that can be achieved, please also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0092] In addition, this embodiment of the invention also provides an electronic device, which may be a computer, tablet computer, or other similar device. This electronic device can implement the steps of any embodiment of the building roof segmentation method based on UAV oblique imagery provided by this invention. Therefore, it can achieve the beneficial effects that any of the building roof segmentation methods based on UAV oblique imagery provided by this invention can achieve, as detailed in the preceding embodiments, and will not be repeated here.

[0093] Figure 9 A specific structural block diagram of an electronic device provided in an embodiment of the present invention is shown. This electronic device can be used to implement the building roof segmentation method based on UAV oblique imagery provided in the above embodiments. The electronic device 500 can be a terminal, server, or other device. The terminal can include a tablet computer, laptop computer, personal computer (PC), microprocessor box, or other devices.

[0094] The memory 520 can be used to store software programs and modules, such as the program instructions / modules corresponding to those in the above embodiments. The processor 580 executes various functional applications and data processing by running the software programs and modules stored in the memory 520. The memory 520 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 520 may further include memory remotely located relative to the processor 580, and these remote memories can be connected to the electronic device 500 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0095] The input unit 530 can be used to receive input numeric or character information, and to generate a keyboard and mouse related to user settings and function control.

[0096] Display unit 540 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces, which can be composed of graphics, text, icons, video, and any combination thereof. Display unit 540 may include display panel 541, which may optionally be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or other similar forms.

[0097] Electronic device 500, through transmission module 570 (e.g., Wi-Fi module), can help users receive requests, send information, etc., providing users with wireless broadband internet access. Although transmission module 570 is shown in the figure, it is understood that it is not an essential component of electronic device 500 and can be omitted as needed without changing the essence of the invention.

[0098] The processor 580 is the control center of the electronic device 500. It connects to various parts of the phone via various interfaces and lines, and performs various functions and processes data of the electronic device 500 by running or executing software programs and / or modules stored in the memory 520, and by calling data stored in the memory 520, thereby providing overall monitoring of the electronic device. Optionally, the processor 580 may include one or more processing cores; in some embodiments, the processor 580 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 580.

[0099] Electronic device 500 also includes a power supply 590 (such as a battery) that supplies power to various components. In some embodiments, the power supply may be logically connected to processor 580 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 590 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0100] Although not shown, the electronic device 500 also includes cameras (such as front-facing cameras and rear-facing cameras), Bluetooth modules, etc., which will not be described in detail here. Specifically, in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors. One or more programs contain instructions for performing the following operations: Obtaining tilted aerial images of buildings using drones; The tilted aerial image of the building is input into a trained edge-guided coordinate attention mechanism network model to obtain the building roof segmentation result; The edge-guided coordinate attention mechanism network model includes an encoder and a decoder. The encoder includes multiple cascaded encoding layers, each of which includes a coordinate attention U-shaped block. The decoder includes multiple cascaded decoding layers, each of which includes a coordinate attention U-shaped block and an edge-guided attention module.

[0101] In practice, the above modules can be implemented as independent entities or combined in any way to be implemented as the same or several entities. For the specific implementation of the above modules, please refer to the previous method implementation examples, which will not be repeated here.

[0102] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. Therefore, embodiments of the present invention provide a storage medium storing multiple instructions that can be loaded by a processor to execute the steps of any embodiment of the building roof segmentation method based on UAV oblique imagery provided by the present invention.

[0103] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0104] Since the instructions stored in the storage medium can execute the steps in any embodiment of the building roof segmentation method based on UAV tilted imagery provided in the embodiments of the present invention, the beneficial effects that any building roof segmentation method based on UAV tilted imagery provided in the embodiments of the present invention can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0105] The foregoing has provided a detailed description of a method, system, storage medium, and electronic device for building roof segmentation based on UAV tilted imagery provided by embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for segmenting building roofs based on UAV oblique imagery, characterized in that, The method includes: Obtaining tilted aerial images of buildings using drones; The tilted aerial image of the building is input into a trained edge-guided coordinate attention mechanism network model to obtain the building roof segmentation result; The edge-guided coordinate attention mechanism network model includes an encoder and a decoder. The encoder includes multiple cascaded encoding layers, each of which includes a coordinate attention U-shaped block. The decoder includes multiple cascaded decoding layers, each of which includes a coordinate attention U-shaped block and an edge-guided attention module.

2. The method for building roof segmentation based on UAV oblique imagery according to claim 1, characterized in that, The edge-guided coordinate attention mechanism network model also includes a Laplace pyramid sub-network; The processing steps of the edge-guided coordinate attention mechanism network model include: The tilted aerial image of the building is input into the encoder, and multiple cascaded coding layers output coded feature maps of different scales respectively; The tilted aerial image of the building is converted into a grayscale image, and the grayscale image is input into the Laplacian pyramid sub-network to obtain an edge feature map; The encoded feature map and the edge feature map are input into the decoder to obtain the building roof segmentation result.

3. The method for building roof segmentation based on UAV oblique imagery according to claim 2, characterized in that, The coordinate attention U-shaped blocks in the first few coding layers of the encoder are ordinary CAUs with different sampling layers, and the coordinate attention U-shaped blocks in the last coding layer of the encoder are flattened CAUs. The processing procedure of the ordinary CAU includes: The input image or input feature map is fed into the initial convolution module for processing to obtain the first feature map; the initial convolution module is configured to perform convolution operation, batch normalization operation and non-linear activation function mapping in sequence. The first feature map is split into paths, wherein the first feature map of the first path is input to the coordinate attention module, and the first feature map of the second path is input to the multi-level downsampling processing module to obtain the second feature map; wherein the multi-level downsampling processing module has several downsampling units stacked consecutively, and each downsampling unit is configured to perform downsampling, convolution operation, batch normalization and activation function processing; The second feature map is input into the secondary convolution module for processing to obtain the third feature map; the secondary convolution module is configured to perform convolution operation, batch normalization operation and non-linear activation function mapping in sequence; The third feature map is input into the multi-level downsampling processing module to obtain the fourth feature map; wherein, the multi-level upsampling processing module has several upsampling units stacked consecutively, each upsampling unit is configured to perform upsampling, convolution operation, batch normalization and activation function processing, and each upsampling unit is also configured to receive the intermediate feature map output by the corresponding upsampling unit through skip connections; The fourth feature map and the first feature map are input into the coordinate attention module to obtain the output feature map of the ordinary CAU; The process of flattening the CAU includes: The input feature map is fed into a depthwise separable convolutional module for processing to obtain the fifth feature map; The sixth feature map is split into paths, wherein the fifth feature map of the first path is input into the coordinate attention module, and the fifth feature map of the second path is input into the multi-level depthwise separable dilated convolution module to obtain the sixth feature map. The sixth feature map and the fifth feature map are input into the coordinate attention module to obtain the output feature map of the flattened CAU.

4. The method for building roof segmentation based on UAV oblique imagery according to claim 3, characterized in that, The processing procedure of the coordinate attention module includes: The input feature map is subjected to width-average pooling in the height direction to obtain the first pooled feature map, and the input feature map is subjected to height-average pooling in the width direction to obtain the second pooled feature map. The first pooling feature map and the second pooling feature map are concatenated and then subjected to convolution, batch normalization and activation function mapping operations to obtain nonlinear features; The nonlinear feature is split into a height orientation feature map and a width orientation feature map. The height orientation feature map and the height orientation feature map are respectively processed by a convolutional layer and an activation function to obtain height attention weights and width attention weights. The height attention weight and the width attention weight are applied to the input feature map through a broadcast mechanism to obtain the output feature map of the coordinate attention module.

5. The method for building roof segmentation based on UAV oblique imagery according to claim 2, characterized in that, The edge-guided attention module includes multiple cascaded EGA blocks, each EGA block being connected to the corresponding coordinate attention U-shaped block in the decoding layer; the processing procedure of the decoding layer includes: The encoded feature map output by the corresponding encoding layer and the initial decoded feature map output by the coordinate attention U-shaped block from the previous decoding layer are fused and then input into the coordinate attention U-shaped block of the current decoding layer to obtain the initial decoded feature map of the current decoding layer. The edge feature map, the initial decoding feature map, and the segmentation prediction map output from the EGA block of the previous decoding layer are input into the EGA block of the current decoding layer to obtain the segmentation prediction map of the current decoding layer.

6. The method for building roof segmentation based on UAV oblique imagery according to claim 5, characterized in that, The processing procedure for the EGA block includes: The Laplacian operator is applied to the segmentation prediction map of the EGA block output from the previous decoding layer to obtain boundary attention and reverse attention; The initial decoded feature map is modulated by the edge feature map, the boundary attention, and the reverse attention respectively to obtain the edge modulation feature map, the boundary modulation feature map, and the reverse modulation feature map; The edge modulation feature map, the boundary modulation feature map, and the inverse modulation feature map are concatenated along the feature channel dimension and fused by convolution. Then, batch normalization and nonlinear activation are applied to obtain the fused feature. The fused features are used to generate a single-channel attention map through lightweight convolution; The fused features are weighted and adjusted based on the single-channel attention map to obtain optimized features; The optimized features are concatenated with the initial decoded feature map and then input into the CBAM module for recalibration of the channel dimension and spatial dimension to obtain the output of the EGA block.

7. The method for building roof segmentation based on UAV oblique imagery according to claim 6, characterized in that, The edge-guided coordinate attention mechanism network model is trained using a total loss function, which is: in, For the total loss function, The loss is the segmentation prediction map output by each decoding layer. The loss is the segmentation prediction map output by the last decoding layer. and The weighting coefficient for each loss term; The loss of the segmentation prediction map is calculated using the following formula: in, The loss of the segmentation prediction map output for each decoding layer, Represents the coordinates of pixels in an image. Indicates the height and width of the image. Indicates coordinates The pixel values ​​of the actual mask. Indicates coordinates The pixel probability values ​​of the model's predicted probability map.

8. A building roof segmentation system based on UAV oblique imagery, characterized in that, include: Image acquisition module, used to acquire tilted aerial images of buildings via drones; The building roof segmentation module is used to input the tilted aerial image of the building into a trained edge-guided coordinate attention mechanism network model to obtain the building roof segmentation result; The edge-guided coordinate attention mechanism network model includes an encoder and a decoder. The encoder includes multiple cascaded encoding layers, each of which includes a coordinate attention U-shaped block. The decoder includes multiple cascaded decoding layers, each of which includes a coordinate attention U-shaped block and an edge-guided attention module.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to execute the building roof segmentation method based on UAV oblique imagery as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, The device includes a processor and a memory, the processor being electrically connected to the memory, the memory being used to store instructions and data, and the processor being used to execute the steps in the building roof segmentation method based on UAV tilted imagery as described in any one of claims 1 to 7.

Images