Unmanned aerial vehicle image building classification method and system based on semantic segmentation

Abstract

The application discloses a kind of based on semantic segmentation's unmanned aerial vehicle image building classification method and system, belong to remote sensing image processing technical field, the technical problem to be solved: in prior art, direct classification is susceptible to background interference and class imbalance caused by long tail distribution;The technical scheme is as follows: S1 builds depth learning framework, including first stage building semantic segmentation model and second stage building classification model;S2 makes and labels multi-territory remote sensing image dataset, obtains original segmentation dataset;S3 extracts image and class, constructs classification dataset;S4 enhances image;S5 trains first stage building semantic segmentation model and second stage building classification model;S6 inputs the remote sensing image to be identified into first stage building semantic segmentation model, extracts segmentation mask and spatial coordinates;Cutting building instance region and inputting second stage building classification model, obtain classification result;S7 maps classification result and spatial coordinates, outputs identification result;It is applied to city planning, land management.

Description

Technical Field

[0001] This invention provides a method and system for classifying buildings in UAV images based on semantic segmentation, belonging to the fields of remote sensing image processing and computer vision technology. Background Technology

[0002] In fields such as urban planning, land management, disaster assessment, and rural census, it is necessary to accurately identify and classify buildings in remote sensing imagery. By automatically identifying the type, location, and distribution of buildings in remote sensing imagery, decision support can be provided to urban planning departments, monitoring tools can be provided to land management departments, and disaster assessment departments can provide a basis for disaster analysis.

[0003] Traditional building recognition methods primarily rely on manual visual interpretation. While this method boasts high accuracy, it is inefficient and costly, failing to meet the demands of large-scale remote sensing image processing. With the development of computer vision technology, deep learning-based building recognition methods have gradually become a research hotspot.

[0004] Existing deep learning-based methods for building recognition in remote sensing images suffer from the following technical problems:

[0005] First, single-stage "direct classification" models suffer from severe feature conflicts and background interference, making it difficult to balance high-precision contours and fine-grained classification. Existing technologies typically employ a single deep learning network (such as a single-stage instance segmentation model) to directly output building masks and specific categories simultaneously from a single remote sensing image containing complex backgrounds. However, pixel-level segmentation tasks heavily rely on local edge features, while fine-grained classification tasks depend more on the global texture features of rooftops. Coupled with these two tasks at the same network endpoint, this not only leads to gradient interference during the optimization process, but more importantly, under the direct classification mechanism, the extraction of classification features is easily contaminated by complex background pixels such as trees, roads, and shadows surrounding the building. This causes the model to mistakenly identify "background features" as "category features," ultimately resulting in coarse building boundaries or severe confusion between subcategories.

[0006] Second, the long-tail distribution of categories leads to a severe deficiency in the model's feature learning for rare categories. In real remote sensing scenarios, different building types naturally exhibit vastly different frequencies (e.g., bungalows and prefabricated steel houses are abundant, while photovoltaic rooftop houses and greenhouses are extremely rare). Existing conventional training methods rely on gradient descent dominated by majority class samples, resulting in blurred decision boundaries for rare categories such as "photovoltaic rooftops," making them prone to misjudgment and unable to support critical applications such as new energy census.

[0007] Third, the model has weak cross-regional generalization ability and suffers from a serious "domain shift" phenomenon. Most existing models are trained based on a single geographical region. The spatial layout of building clusters, the degree of aging of roof materials, and imaging conditions vary greatly in different regions. When encountering new regional data, the false negative rate and error rate of models trained in a single region increase significantly.

[0008] Fourth, the issue of missing context when cropping buildings at the edge. When a building is close to the edge of an image or slice, conventional cropping methods will truncate the part that exceeds the boundary, resulting in insufficient contextual content in that direction. Consequently, the input received by the classification model is incomplete, affecting classification accuracy.

[0009] Fifth, model selection is complex, especially in semantic segmentation and between CNN and Transformer architectures, making it difficult to make an accurate choice. Summary of the Invention

[0010] To address the technical problems existing in the background art, the present invention adopts the following technical solution: a building classification method for UAV images based on semantic segmentation, specifically including the following steps:

[0011] Step 1: Build a deep learning framework, which includes a first-stage building semantic segmentation model and a second-stage building classification model.

[0012] Step 2: Collect remote sensing images from multiple regions to create a multi-regional remote sensing image dataset, and label the categories of building instances within the multi-regional remote sensing image dataset to obtain the original segmentation dataset;

[0013] Step 3: Extract images and categories of individual building instances from the original segmentation dataset to construct a classification dataset;

[0014] Step 4: Enhance the images of building instances in the classification dataset to balance the distribution of building instances of different categories in the classification dataset;

[0015] Step 5: Train the first-stage building semantic segmentation model using the original segmentation dataset, and train the second-stage building classification model using the classification dataset processed in Step 4.

[0016] Step 6: Acquire the remote sensing image to be identified, input the remote sensing image to be identified into the first-stage building semantic segmentation model for building segmentation, and output pixel-level segmentation prediction results; extract the segmentation mask and spatial coordinates of each building instance from the pixel-level segmentation prediction results; use the context filling cropping strategy to crop the region of each building instance from the remote sensing image to be identified based on the spatial coordinates; input the cropped building instance region into the second-stage building classification model for category prediction, and obtain the classification result of each building instance;

[0017] Step 7: Map the classification results of each building instance to spatial information and output the recognition results of each building instance.

[0018] Furthermore, the first-stage building semantic segmentation model includes a Transformer-based semantic segmentation network and a convolutional block attention module. The Transformer-based semantic segmentation network contains a decoder, which includes a decoder feature fusion layer and a classification head. The second-stage building classification model includes a building classification model built on the YOLOv8 backbone network.

[0019] Furthermore, the semantic segmentation network based on Transformer adopts SegFormer, and the convolutional block attention module adopts the CBAM attention module.

[0020] Furthermore, step 5 specifically includes the following steps:

[0021] Step 5.1: Use SegFormer as the semantic segmentation network, set the CBAM attention module between the decoder feature fusion layer and the classification head, and train the first-stage building semantic segmentation model using the original segmentation dataset;

[0022] Step 5.2: Train the second-stage building classification model using the building classification model built on the YOLOv8 backbone network and the classification dataset processed in Step 4.

[0023] Furthermore, the CBAM attention module includes a channel attention submodule and a spatial attention submodule, which are connected in series.

[0024] Furthermore, step 5.1 specifically includes the following steps:

[0025] Step 5.1.1: Configure the first training parameters;

[0026] Step 5.1.2: Configure the first data augmentation strategy, which includes random horizontal flipping, random scaling, random cropping, color dithering, and random Gaussian blur.

[0027] Step 5.1.3: Configure the weights of the cross-entropy loss function;

[0028] Step 5.1.4: Configure the first early stop strategy and train the first-stage building semantic segmentation model.

[0029] Furthermore, step 5.2 specifically includes the following steps:

[0030] Step 5.2.1: Configure the second training parameters;

[0031] Step 5.2.2: Configure the second data augmentation strategy, which includes random rotation, random translation, random scaling, and horizontal flipping;

[0032] Step 5.2.3: Configure the second early stop strategy and train the second-stage building classification model.

[0033] Furthermore, the categories of building examples include tiled houses, bungalows, prefabricated steel houses, photovoltaic roof houses, greenhouses, urban buildings, and industrial plants.

[0034] Furthermore, the identification results of building instances include the location coordinates, outline, and category of the building instance.

[0035] Furthermore, in a system implementing the UAV image building classification method based on semantic segmentation as described above, the output of the data acquisition and preprocessing module is electrically connected to the input of the two-stage model training module, the output of the two-stage model training module is electrically connected to the input of the collaborative reasoning module, and the output of the collaborative reasoning module is electrically connected to the input of the result output module.

[0036] The advantages of this invention over the prior art are as follows: 1. High-precision building boundary segmentation is achieved through the SegFormer and CBAM semantic segmentation models.

[0037] 2. The CBAM attention module enhances the ability to represent building boundary features and identify rare categories.

[0038] 3. The problem of incomplete input for edge building classification was solved by using a context-filling clipping strategy.

[0039] 4. By using high-precision multi-regional data and labeled data, the model's cross-regional generalization ability is enhanced. Attached Figure Description

[0040] The present invention will be further described below with reference to the accompanying drawings:

[0041] Figure 1 This is a schematic diagram of the structure of the UAV image building classification system based on semantic segmentation in this invention;

[0042] Figure 2 This is a flowchart illustrating the UAV image building classification method based on semantic segmentation in this invention.

[0043] Figure 3 This is a schematic diagram illustrating the effect of the data augmentation strategy in this invention;

[0044] Figure 4 This is a comparison image before and after step 4 of the present invention;

[0045] Figure 5 This is a comparison chart of the classification effects at different stages in this invention;

[0046] Figure 6 This is a comparison chart of the accuracy of the method of the present invention with that of the prior art;

[0047] Figure 7 This is a comparison chart of classification accuracy under different schemes in this invention;

[0048] Figure 8 This is a comparison chart of the classification accuracy of various types of building examples in this invention. Detailed Implementation

[0049] like Figures 1 to 8 As shown, the present invention provides a method for classifying buildings in UAV images based on semantic segmentation, which specifically includes the following steps:

[0050] Step 1: Build a deep learning framework, which includes a first-stage building semantic segmentation model and a second-stage building classification model. The first-stage building semantic segmentation model includes a Transformer-based semantic segmentation network and a convolutional block attention module. The Transformer-based semantic segmentation network contains a decoder, which includes a decoder feature fusion layer and a classification head. The second-stage building classification model includes a building classification model built on the YOLOv8 backbone network. The Transformer-based semantic segmentation network uses SegFormer, and the convolutional block attention module uses the CBAM attention module. The CBAM attention module includes a channel attention submodule and a spatial attention submodule, which are connected in series.

[0051] Step 2: Collect remote sensing images from multiple regions to create a multi-regional remote sensing image dataset, and label the categories of building instances within the multi-regional remote sensing image dataset to obtain the original segmentation dataset. Divide the original segmentation dataset into a training set, a validation set, and a test set in a ratio of 7:1.5:1.5. The categories of building instances include tiled houses, bungalows, prefabricated steel houses, photovoltaic roof houses, greenhouses, urban buildings, and industrial plants.

[0052] Step 3: Extract images and categories of individual building instances from the original segmentation dataset to construct a classification dataset; this also includes the following steps:

[0053] Step 3.1: Based on the annotation information in the original segmentation dataset, crop the corresponding building instances from the multi-regional remote sensing image dataset;

[0054] Step 3.2: Organize the cropped building instances by category to construct a classification dataset.

[0055] Step 4: Enhance the images of building instances in the classification dataset to balance the distribution of building instances of different categories in the classification dataset.

[0056] Step 5: Train the first-stage building semantic segmentation model using the original segmentation dataset, and train the second-stage building classification model using the classification dataset processed in Step 4; specifically, this also includes the following steps:

[0057] Step 5.1: Using SegFormer as the semantic segmentation network, the CBAM attention module is placed between the decoder feature fusion layer and the classification head. The first-stage architectural semantic segmentation model is trained using the original segmentation dataset. This also includes the following steps:

[0058] Step 5.1.1: Configure the first training parameters: Set the input image size to 256×256~1024×1024, the batch size to 4~16, the number of training iterations to 50~200, use the AdamW optimizer to update the parameters, set the initial learning rate to 0.00001~0.0001, set the weight decay coefficient to 0.001~0.1, and use a cosine annealing learning rate decay strategy to dynamically adjust the learning rate during the training process;

[0059] Step 5.1.2: Configure the first data augmentation strategy, which includes random horizontal flipping, random scaling (scaling ratio range of 0.5~2.0), random cropping, color dithering, and random Gaussian blur;

[0060] Step 5.1.3: Configure the weights of the cross-entropy loss function: Use the cross-entropy loss function, assigning higher weights to the building category to address the class imbalance problem. The formula for the cross-entropy loss function is:

[0061] ;

[0062] Where L is the loss value; N is the total number of pixels; C is the total number of categories, C=8, including background category and 7 building categories; The one-hot encoding of the true label of the i-th pixel, when the pixel belongs to category c, =1, otherwise =0; The probability that the i-th pixel belongs to class c is predicted by the model and calculated using the Softmax function. The formula for the Softmax function is:

[0063] ;

[0064] in, For category indexing, and The output of the model is the first Each pixel belongs to the category and categories The original predicted score (logits value). Considering the small proportion of building pixels in remote sensing imagery, a weighted cross-entropy loss function is used:

[0065] ;

[0066] in For category c, the weight of the building category is set to 2-5 times the weight of the background category to balance the category distribution. The weights for each category are set as follows: Background category weight... =1.0, building category weight ~ =2.0~5.0.

[0067] Step 5.1.4: Configure the first early stopping strategy: Set the early stopping patience value to 10~30, that is, terminate training when the model validation accuracy has not improved for 10~30 consecutive rounds, effectively suppressing the overfitting phenomenon of the model; and then carry out the training of the first stage building semantic segmentation model.

[0068] Step 5.2: Train the second-stage building classification model using the building classification model built on the YOLOv8 backbone network and the classification dataset processed in Step 4; specifically, this includes the following steps:

[0069] Step 5.2.1: Configure the second training parameters: Set the input image size to 128×128~256×256, the batch size to 16~64, the number of training iterations to 50~200, use the AdamW optimizer to update the parameters, and set the initial learning rate to 0.0001~0.01;

[0070] Step 5.2.2: Configure the second data augmentation strategy, which includes random rotation, random translation, random scaling, and horizontal flipping;

[0071] Step 5.2.3: Configure the second early stopping strategy: Set the early stopping patience value to 10~30, that is, terminate training when the model validation accuracy has not improved for 10~30 consecutive rounds; and then carry out training of the second stage building classification model.

[0072] Step 6: Acquire the remote sensing image to be identified, input the image into the first-stage building semantic segmentation model for building segmentation, and output pixel-level segmentation prediction results; extract the segmentation mask and spatial coordinates of each building instance from the pixel-level segmentation prediction results; based on the spatial coordinates, use a context-filling cropping strategy to crop the region of each building instance from the remote sensing image to be identified; input the cropped building instance regions into the second-stage building classification model for category prediction, and obtain the classification result of each building instance; specifically including the following steps:

[0073] Step 6.1: Acquire the remote sensing image to be identified and perform preprocessing. For large-size images (greater than 2 times 1024), adopt the overlapping tiling strategy. The overlapping tiling strategy includes image size adjustment and normalization.

[0074] Step 6.2: Input the preprocessed remote sensing image to be identified into the first-stage building semantic segmentation model for building segmentation, and output pixel-level segmentation prediction results;

[0075] Step 6.3: Extract the segmentation mask and spatial coordinates of each building instance from the pixel-level segmentation prediction results, and input the segmentation mask of the corresponding building instance into the second-stage building classification model according to the spatial coordinates of the building instance.

[0076] Step 6.4: Calculate the clipping box based on the spatial coordinates of the building instance, and extend it outward by 15% of the maximum side length of the building as the context; when the clipping box exceeds the image boundary, use reflection padding to mirror and complete the out-of-bounds area; uniformly scale the building area containing the complete context to the input size of the classification model; input the second-stage building classification model, and output the classification result for each building instance. Specifically, this includes the following steps:

[0077] Step 6.4.1: Calculate the clipping frame; extract the outline of each building instance from Step 6.3:

[0078] ;

[0079] in, The total number of contour points. These are the x and y coordinates of the k-th sampling point in the contour, respectively; the spatial coordinates of the building instance are calculated using the minimum bounding rectangle algorithm to obtain the coordinates of the upper left corner of the clipping box. and width and height The calculation formula is:

[0080] ;

[0081] ;

[0082] Therefore, the initial trimming range is determined as follows: That is, the smallest axis-aligned rectangular area that can completely encompass the building instance.

[0083] Step 6.4.2: Maximum building side length and context expansion; take the larger value between the width and height of the building's circumscribed rectangle as the expansion reference scale, and determine the maximum building side length. The calculation formula is:

[0084] ;

[0085] Use 15% of the maximum side length as the four-way uniform expansion amount. The calculation formula is as follows:

[0086] ;

[0087] in, This is the context expansion scaling factor, with a value of 0.15. After expanding the initial clipping box outwards, the expanded clipping box... The calculation formula is:

[0088] ;

[0089] The theoretical size of the expanded cropping pattern is Using the maximum side length instead of expanding based on width and height separately ensures consistent contextual expansion in all four directions, avoiding non-uniform deformation of the clipped image. A 15% expansion ratio strikes a balance between preserving information about the building's surrounding environment (such as road distribution, adjacent building density, and auxiliary classification features like land cover type) and controlling computational load.

[0090] Step 6.4.3: Image Boundary Cross-Boundary Detection; Compare the expanded cropping box with the effective area boundary of the original remote sensing image to determine if any boundary crossing occurs. Let the width of the original image be... The height is The effective area boundary of the image is then defined as follows: The formulas for calculating the out-of-bounds amount in the four directions are as follows:

[0091] ;

[0092] ;

[0093] in, , , , These represent the out-of-bounds amounts for the left, top, right, and bottom sides, respectively. When the out-of-bounds amount in a certain direction is greater than 0, it indicates that there is an out-of-bounds issue in that direction and completion processing is required. Subsequently, the coordinates of the expanded cropping box are truncated to obtain the actual extractable range within the effective area of ​​the image. The truncated coordinates are... The calculation formula is:

[0094] ;

[0095] ;

[0096] in, These are the x-coordinate of the top left corner, the y-coordinate of the top left corner, the x-coordinate of the bottom right corner, and the y-coordinate of the bottom right corner, respectively; this effective area is extracted from the original image:

[0097] .

[0098] Step 6.4.4: Reflection filling mirror completion; when there is an out-of-bounds error in any direction (i.e. ,or ,or ,or When using the Reflect101 reflection fill mode, The out-of-bounds area is mirrored for completion. Reflection fill uses the image edge pixels as the axis of symmetry, mirroring the image content to the out-of-bounds position. The fill formula is:

[0099] ;

[0100] in, The coordinates of any pixel on the filled image. For the complete image after filling, For the effective region image, and These are its width and height, respectively. The Reflect101 reflection mapping function is defined as follows:

[0101] ;

[0102] The fill parameters for the four directions are as follows: top fill Rows, bottom fill Row, left fill Columns, right fill The Reflect101 mode is characterized by not copying boundary pixels (i.e., boundary pixels do not participate in mirroring), effectively avoiding texture duplication artifacts at fill seams. Compared to black edge interference caused by zero fill or edge stretching deformation caused by copied fill, reflection fill better maintains the texture continuity and ground feature realism of remote sensing image edge areas, thereby reducing interference with subsequent classification models. After reflection fill processing, the final output cropped image size is:

[0103] ;

[0104] Compared with the theoretical size of the expanded clipping frame in step 6.4.2 Consistent. The formula here illustrates the mapping logic within a single cycle. In actual engineering implementations, when the out-of-bounds amount exceeds the effective region size, the underlying library functions will automatically perform multi-cycle recursive reflection processing.

[0105] Step 6.4.5: Scale the image uniformly to the input size of the classification model; convert the building crop image with complete context to an RGB format PIL image and input it into the YOLO-based second-stage building classification model. The standard input size of the model is... Pixels. First, scale the cropped image proportionally along its longer side, scaling ratio... The calculation formula is:

[0106] ;

[0107] in, and These are the width and height of the cropped image after processing in step 6.4.4. (Image size after scaling) The calculation formula is:

[0108] ;

[0109] Since architectural cutouts are typically rectangles of varying widths and heights, they cannot be perfectly filled after being scaled proportionally. The input area is square, so a letterbox gray-edge fill strategy is used to fill the short side. The horizontal gray-edge fill amount... and vertical gray edge fill amount The calculation formulas are as follows:

[0110] ;

[0111] The final formula for constructing the input tensor is as follows (using the rounded padding amount for boundary determination):

[0112] ;

[0113] in, This is the gray fill value (usually 114 or the mean gray level). The input tensor is then normalized. The data is fed into a classification network for forward reasoning, and the output shows that the building instance is... probability distribution vector of building category The category with the highest probability is taken as the final classification result for the building instance:

[0114] ;

[0115] Step 7: Map the classification results of each building instance to spatial information and output the recognition results of each building instance. The recognition results of the building instance include the location coordinates, outline and category of the building instance.

[0116] In practical applications, SegFormer is NVIDIA's Transformer-based semantic segmentation network, available in five versions from B0 to B5. B2 is a mid-range model balancing accuracy and speed, and SegFormer-B2 is the preferred choice. In step 2 of this embodiment, remote sensing image data from multiple regions including Gulin, Ningxia, and Shanxi are collected, and the categories of building instances within the multi-regional remote sensing image dataset are labeled to construct the original segmentation dataset. The original segmentation dataset consists of 5751 images, including 3644 images for training, 1053 images for validation, and 1054 images for testing. In step 3, the classification dataset consists of 29260 building images, including 3420 tile-roofed houses, 9995 single-story houses, 13226 prefabricated steel houses, 292 photovoltaic roof houses, 465 greenhouses, 802 urban buildings, and 1060 industrial plants. In step 4, the enhancement process includes targeted oversampling enhancement: photovoltaic rooftop houses are oversampled 5 times, greenhouses are oversampled 3 times, and urban buildings are oversampled 2 times. The size of the oversampled classification dataset is approximately 35,000 building images to level the sample cardinality of building instances in each category.

[0117] like Figure 3 The image shown is a schematic diagram illustrating the effect of a data augmentation strategy. The first data augmentation strategy also includes HSV color space transformation to enhance robustness under different lighting conditions; Mosaic stitching enhancement to stitch four images into one, increasing scene complexity; random flipping, random rotation, and random scaling to improve robustness to geometric transformations; and random erasing: randomly erasing parts of the image and automatically selecting the optimal enhancement strategy to enhance occlusion robustness. Figure 4 As shown, in the category distribution before step 4, the sample numbers for bungalows and prefabricated steel houses were absolutely dominant, while the sample numbers for photovoltaic roofs, greenhouses, and urban buildings were extremely low. In the category distribution after step 4, the sample numbers for photovoltaic roofs, greenhouses, and urban buildings increased, improving the balance of the category distribution. Figure 5 As shown, from left to right in the image are the original image, the segmentation result image, and the classification result image. Figure 7 As shown, from left to right in the image are:

[0118] Option A (YOLO-seg+YOLO-cls): This option uses the YOLOv8 segmentation model for building semantic segmentation, extracts building instances, and then directly crops them before feeding them into the YOLO classification model for category identification. This is the original baseline option, using YOLO series models in both stages.

[0119] Option B (SF+YOLO-cls): The first-stage segmentation model is replaced with the SegFormer-B2 semantic segmentation model, while the second stage still uses the YOLO classification model. Compared to Option A, the introduction of a Transformer architecture segmentation model improves the pixel-level accuracy of building segmentation.

[0120] Option C (SF+CBAM+YOLO-cls): Based on Option B, CBAM (Convolutional Block Attention Module) is introduced into the decoder output of the SegFormer-B2 segmentation model to integrate channel attention and spatial attention mechanisms to enhance the expressive power of architectural features. The YOLO classification remains unchanged in the second stage.

[0121] Scheme D (SF+CBAM+YOLO-cls+Contextual Fill): Based on Scheme C, the extracted building instance cropping box is extended outward by a maximum side length of 15% of the context region as the classification input. When the cropping exceeds the image boundary, Reflect101 reflection filling is used to complete it, providing the classification model with more complete information about the building's surrounding environment.

[0122] A system for implementing the UAV image building classification method based on semantic segmentation as described above, wherein the output of the data acquisition and preprocessing module is electrically connected to the input of the two-stage model training module, the output of the two-stage model training module is electrically connected to the input of the collaborative reasoning module, and the output of the collaborative reasoning module is electrically connected to the input of the result output module.

[0123] In this embodiment, the data acquisition and preprocessing module is used to acquire and create a multi-regional remote sensing image dataset, label the categories of building instances within the dataset, and obtain the original segmentation dataset. From the original segmentation dataset, the image and category of a single building instance are extracted to construct a classification dataset. The images of building instances in the classification dataset are then augmented. The two-stage model training module is used to train the first-stage building semantic segmentation model and the second-stage building classification model. The collaborative inference module is used to input the remote sensing image to be identified into the first-stage building semantic segmentation model for building segmentation, outputting pixel-level segmentation prediction results. From the pixel-level segmentation prediction results, the segmentation mask and spatial coordinates of each building instance are extracted. Based on the spatial coordinates, a context-filled cropping strategy is used to crop the region of each building instance from the remote sensing image to be identified. The cropped building instance regions are then input into the second-stage building classification model for category prediction. The result output module is used to perform topological association between the category prediction output by the second-stage building classification model and the spatial information of the first-stage building semantic segmentation model, outputting building location, outline, and category information.

[0124] Example

[0125] Step 1: Collect remote sensing image data from multiple regions including Gulin, Taiyuan in Shanxi, and Jiaocheng in Shanxi. To ensure the regional diversity of the data, the images cover different seasons and lighting conditions.

[0126] Step 2: Label building instances in the multi-regional remote sensing image dataset. The labeling includes seven building types: tiled houses, bungalows, prefabricated steel houses, photovoltaic roof houses, greenhouses, urban buildings, and industrial plants. The labeling format uses the YOLO segmentation format, including normalized polygon coordinates.

[0127] Step S103: Construct the original segmentation dataset. The dataset consists of 5,751 images, divided into a training set of 3,644 images, a validation set of 1,053 images, and a test set of 1,054 images in a 70:15:15 ratio.

[0128] Step S104: Extract individual building regions from the labeled data and construct a classification dataset. Building regions are cropped based on polygon coordinates. The original size of the classification dataset is 29,260 building images, exhibiting a severe long-tail distribution (e.g., 13,226 images of prefabricated steel buildings and only 292 images of photovoltaic roofs).

[0129] Step S105: Perform targeted oversampling enhancement for rare categories. Perform 5x oversampling on photovoltaic rooftops, 3x oversampling on greenhouses, and 2x oversampling on urban buildings. The resulting classification dataset contains approximately 35,000 building images. This forces a rebalancing of sample weight distribution across categories, reversing the dominance of the training gradient.

[0130] Example 2: First-stage training of the architectural semantic segmentation model

[0131] In this embodiment, the first stage of architectural semantic segmentation model training is performed, and the specific steps are as follows:

[0132] Step S201: Select model architecture. SegFormer-B2 is used as the semantic segmentation backbone network. A CBAM attention module is inserted after the decoder feature fusion layer and before the classification head to construct the SegFormer-B2+CBAM segmentation model. The CBAM module includes a channel attention submodule and a spatial attention submodule. Channel attention aggregates channel information through adaptive average pooling and max pooling, while spatial attention aggregates spatial information through the mean and maximum values ​​of the channel dimensions. The two are concatenated to enhance the representation of building boundary features. ImageNet pre-trained weights are loaded for transfer learning.

[0133] Step S202: Configure training parameters. Image size 512×512, batch size 8, training 100 epochs, optimizer AdamW, initial learning rate 0.00006, cosine annealing decay, weight decay 0.01.

[0134] Step S203: Configure data augmentation strategies. These include random horizontal flipping, random scaling (0.5-2.0), random cropping, color dithering, and random Gaussian blur.

[0135] Step S204: Configure the loss function weights. Specific configurations are made to account for the small percentage of pixels representing remotely sensed buildings.

[0136] Step S205: Configure early stopping and training. Early stopping patience = 20, and convergence occurs after approximately 80 rounds during training.

[0137] Example 3: Second-stage building classification model training

[0138] In this embodiment, the second stage of building classification model training is performed, and the specific steps are as follows:

[0139] Step S301: Select model architecture. Use YOLOv8m-cls as the classification model and load pre-trained weights.

[0140] Step S302: Configure training parameters. Since the input is now a cropped pure building image, the size is reduced to 224×224, the batch size is increased to 32, training is performed for 100 epochs, the optimizer is AdamW, and the initial learning rate is 0.001.

[0141] Step S303: Configure lightweight data augmentation. Use random rotation (±10 degrees), translation, scaling, and horizontal flipping to preserve key texture features of the roof without excessive distortion.

[0142] Step S304: Early Stop and Training. With an early stop patience of 20, the system stabilizes and converges after approximately 60 rounds.

[0143] Example 4: Two-stage collaborative reasoning and output

[0144] In this embodiment, two-stage collaborative reasoning is performed, and the specific steps are as follows:

[0145] Step S401: Input the remote sensing image to be identified. For large-size images, an overlapping tiling strategy is adopted, with a tiling size of 512×512, an overlapping area of ​​128 pixels, and a step size of 384 pixels. Normalization preprocessing is performed on each tiling.

[0146] Step S402: First stage semantic segmentation and contour extraction. Input each slice into the SegFormer-B2+CBAM segmentation model, output pixel-level segmentation prediction, use a distance-weighted fusion strategy to smoothly stitch the prediction results of overlapping slices into a complete segmentation mask, and then extract the building contour from the mask.

[0147] Step S403: Context Fill Crop. Calculate the cropping box based on the building outline, and extend it outward by 15% of the building's maximum side length as the context environment. When the cropping box exceeds the image boundary, use reflection fill to mirror and complete the out-of-bounds area, ensuring that all buildings have a complete context environment input. This step achieves physical isolation of the background and complete preservation of context information.

[0148] Step S404: Second-stage fine-grained classification. Input the cropped image containing complete context into the classification model, and output the Top-1 class and its probability.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for classifying buildings from UAV imagery based on semantic segmentation, characterized in that, Specifically, the following steps are included: Step 1: Build a deep learning framework, which includes a first-stage building semantic segmentation model and a second-stage building classification model. Step 2: Collect remote sensing images from multiple regions to create a multi-regional remote sensing image dataset, and label the categories of building instances within the multi-regional remote sensing image dataset to obtain the original segmentation dataset; Step 3: Extract images and categories of individual building instances from the original segmentation dataset to construct a classification dataset; Step 4: Enhance the images of building instances in the classification dataset to balance the distribution of building instances of different categories in the classification dataset; Step 5: Train the first-stage building semantic segmentation model using the original segmentation dataset, and train the second-stage building classification model using the classification dataset processed in Step 4. Step 6: Acquire the remote sensing image to be identified, input the remote sensing image to be identified into the first-stage building semantic segmentation model for building segmentation, and output pixel-level segmentation prediction results; extract the segmentation mask and spatial coordinates of each building instance from the pixel-level segmentation prediction results; use the context filling cropping strategy to crop the region of each building instance from the remote sensing image to be identified based on the spatial coordinates; input the cropped building instance region into the second-stage building classification model for category prediction, and obtain the classification result of each building instance; Step 7: Map the classification results of each building instance to spatial information and output the recognition results of each building instance.

2. The UAV imagery building classification method based on semantic segmentation according to claim 1, characterized in that: The first-stage building semantic segmentation model includes a Transformer-based semantic segmentation network and a convolutional block attention module. The Transformer-based semantic segmentation network contains a decoder, which includes a decoder feature fusion layer and a classification head. The second-stage building classification model includes a building classification model built on the YOLOv8 backbone network.

3. The UAV imagery building classification method based on semantic segmentation according to claim 2, characterized in that: The semantic segmentation network based on Transformer uses SegFormer, and the convolutional block attention module uses the CBAM attention module.

4. The UAV imagery building classification method based on semantic segmentation according to claim 3, characterized in that: Step 5 also includes the following steps: Step 5.1: Use SegFormer as the semantic segmentation network, set the CBAM attention module between the decoder feature fusion layer and the classification head, and train the first-stage building semantic segmentation model using the original segmentation dataset; Step 5.2: Train the second-stage building classification model using the building classification model built on the YOLOv8 backbone network and the classification dataset processed in Step 4.

5. The UAV imagery building classification method based on semantic segmentation according to claim 4, characterized in that: The CBAM attention module includes a channel attention submodule and a spatial attention submodule, which are connected in series.

6. The UAV imagery building classification method based on semantic segmentation according to claim 5, characterized in that: Step 5.1 further includes the following steps: Step 5.1.1: Configure the first training parameters; Step 5.1.2: Configure the first data augmentation strategy, which includes random horizontal flipping, random scaling, random cropping, color dithering, and random Gaussian blur. Step 5.1.3: Configure the weights of the cross-entropy loss function; Step 5.1.4: Configure the first early stop strategy and train the first-stage building semantic segmentation model.

7. The UAV imagery building classification method based on semantic segmentation according to claim 6, characterized in that: Step 5.2 further includes the following steps: Step 5.2.1: Configure the second training parameters; Step 5.2.2: Configure the second data augmentation strategy, which includes random rotation, random translation, random scaling, and horizontal flipping; Step 5.2.3: Configure the second early stop strategy and train the second-stage building classification model.

8. The UAV imagery building classification method based on semantic segmentation according to claim 1, characterized in that: The categories of architectural examples include tiled houses, bungalows, prefabricated steel houses, photovoltaic roof houses, greenhouses, urban buildings, and industrial plants.

9. The UAV imagery building classification method based on semantic segmentation according to claim 1, characterized in that: The identification results of building instances include the location coordinates, outline, and category of the building instance.

10. A system for implementing the UAV imagery building classification method based on semantic segmentation as described in any one of claims 1-9, characterized in that: The output of the data acquisition and preprocessing module is electrically connected to the input of the two-stage model training module. The output of the two-stage model training module is electrically connected to the input of the collaborative reasoning module. The output of the collaborative reasoning module is electrically connected to the input of the result output module.

Images