Unmanned aerial vehicle image building extraction method and system fusing DINOv2 and global attention

Abstract

The invention relates to the technical field of image segmentation, in particular to an unmanned aerial vehicle image building extraction method and system fusing DINOv2 and global attention, and the method comprises the steps: constructing a building segmentation network based on DINOv2 and global attention; the building segmentation network comprises an encoder network used for extracting an input image feature map and carrying out encoding output and a decoder network used for decoding the encoding output and segmenting and predicting an input image building, and the encoder network adopts a DINOv2 visual model as a backbone network for extracting the image feature map; the decoder network captures image space semantic information by using a global attention mechanism and a feature enhancement mechanism so as to perform semantic segmentation prediction; and training the building segmentation network based on the target loss function by using the sample data set, so as to segment the unmanned aerial vehicle image building by using the trained model. According to the method, image building features can be more stably obtained, network performance is improved, and building extraction with rich high-resolution unmanned aerial vehicle image semantics and complex background information is met.

Description

Technical Field

[0001] The present invention relates to the technical field of image segmentation, and in particular to a method and system for extracting buildings from drone images by integrating DINOv2 and global attention. Background Art

[0002] As the main component of modern cities, buildings are the main carriers of human production and life and social and economic development. The automatic extraction of buildings using remote sensing images has important application value in the fields of military, urban planning, disaster monitoring, and land and resources management. In recent years, the extraction of buildings using drone images has attracted widespread attention from researchers, but due to its rich texture features and complex ground information, the accuracy of building extraction is poor. Therefore, how to accurately and efficiently automatically extract buildings from high-resolution drone images has become a hot topic in remote sensing image interpretation research.

[0003] In recent years, deep learning technology has developed rapidly, and automatic building extraction based on deep learning has become the mainstream method for building extraction in remote sensing images. Most methods usually use convolutional neural networks (CNN) for feature extraction, but since CNN has difficulty in obtaining global features of images, its segmentation accuracy is often poor. Summary of the invention

[0004] To this end, the present invention provides a method and system for extracting buildings from drone images by integrating DINOv2 and global attention, so as to solve the problem of insufficient accuracy of existing drone image building extraction tasks.

[0005] According to the design scheme provided by the present invention, on the one hand, a method for extracting buildings from drone images by integrating DINOv2 and global attention is provided, comprising:

[0006] Collect remote sensing image data, pre-process the remote sensing image data, and obtain a sample data set containing several image blocks with building annotation labels;

[0007] A building segmentation network is constructed based on DINOv2 and global attention, wherein the building segmentation network includes: an encoder network for extracting a feature map of an input image and encoding an output and a decoder network for decoding the encoded output and segmenting and predicting buildings in the input image, wherein the encoder network uses the DINOv2 visual model as a backbone network for extracting image feature maps, and the decoder network uses a global attention mechanism and a feature enhancement mechanism to capture image spatial semantic information for semantic segmentation prediction;

[0008] Set a target loss function for model training, train the building segmentation network based on the target loss function and using the sample data set to obtain a building segmentation target model;

[0009] The drone image to be processed is input into the building segmentation target model, and the building segmentation target model is used to predict the segmentation of the buildings in the drone image to be processed.

[0010] As a method for extracting buildings from drone images by integrating DINOv2 and global attention, the remote sensing image data is further preprocessed, including:

[0011] Fill the remote sensing image and crop it into image blocks of specified size;

[0012] Performing data enhancement processing on each image block using data enhancement technology, wherein the data enhancement technology includes random image scaling, image flipping and image rotation;

[0013] The image patches are annotated with building labels and converted into binary images containing only building areas and non-building areas.

[0014] As a method for extracting buildings from drone images that integrates DINOv2 and global attention in the present invention, further, the encoder network includes: an embedding layer for converting an image block into an embedding vector, and a plurality of stacked encoding layers for encoding the embedding vector to obtain an encoding feature vector of the image block, each encoding layer includes an attention mechanism and a feedforward neural network, and the output of each encoding layer is used as the input of the next encoding layer, so as to utilize the stacked multiple encoding layers to capture the spatial information and global features of the image block.

[0015] As a drone image building extraction method that integrates DINOv2 and global attention in the present invention, further, the encoder network adopts DINOv2-giant as the backbone network, and the terminal encoding layer of the backbone network is connected to the decoder network using a jump connection to maximize the image understanding ability of the backbone network pre-trained parameters, wherein the jump connection uses linear transformation and downsampling operations to obtain deep global features of image blocks.

[0016] As a drone image building extraction method that integrates DINOv2 and global attention in the present invention, further, the decoder network includes several global attention layers and feature enhancement layers, wherein each global attention layer adopts two parallel branch structures of global branch and local branch to capture the spatial details and semantic information of the image block, and the feature enhancement layer uses channel feature enhancement branch and spatial feature enhancement branch to reduce the semantic gap between features.

[0017] As a drone image building extraction method that integrates DINOv2 and global attention in the present invention, further, the global branch obtains the global context information of the feature vector based on the multi-head attention of the cross window, and the local branch extracts the local context information of the feature vector through parallel convolution layers, normalization and fusion operations, and for the global context information and the local context information, the global attention layer obtains the global-local context information through aggregation, convolution and normalization, and the global-local context information includes the spatial details and semantic content of the image block.

[0018] As a drone image building extraction method that integrates DINOv2 and global attention in the present invention, further, the channel feature enhancement branch uses a global average pooling layer to generate image block channel features and uses two convolution layers to reduce the channel dimension and restore it to the original dimension to obtain image block channel enhancement features, and the spatial feature enhancement branch uses deep convolution to generate image block spatial enhancement features, and for the image block feature vector, channel enhancement feature and spatial enhancement feature, the feature enhancement layer performs feature fusion through a summation operation, and uses convolution operations and upsampling to obtain the final output building segmentation image.

[0019] As a drone image building extraction method that integrates DINOv2 and global attention in the present invention, further, the process of feature fusion through summation operation is expressed as: FF=f·VF-+(1-f)GF, wherein f is a fusion factor, VF is a feature after the channel enhancement feature and the spatial enhancement feature are aggregated, and GF is a feature map encoding representation obtained by the encoder network.

[0020] As a method for extracting buildings from drone images that integrates DINOv2 and global attention in the present invention, further, the target loss function adopts a main and auxiliary loss function that combines a cross entropy-dice mixed loss as a main loss function and a cross entropy loss as an auxiliary loss function to train the model, wherein the main loss function is connected to the feature enhancement mechanism of the decoder network during the model training process to train parameters related to the feature enhancement mechanism, and the auxiliary loss function is connected to the global attention mechanism of the decoder network to train parameters related to the global attention mechanism.

[0021] On the other hand, the present invention also provides a drone image building extraction system integrating DINOv2 and global attention, comprising: a sample collection model, a model construction module, a model training module and a target extraction module, wherein:

[0022] The sample collection module is used to collect remote sensing image data, pre-process the remote sensing image data, and obtain a sample data set containing a number of image blocks with building annotation labels;

[0023] A model construction module is used to construct a building segmentation network based on DINOv2 and global attention, wherein the building segmentation network includes: an encoder network for extracting a feature map of an input image and encoding an output and a decoder network for decoding the encoded output and segmenting and predicting buildings in the input image, wherein the encoder network uses the DINOv2 visual model as a backbone network for extracting image feature maps, and the decoder network uses a global attention mechanism and a feature enhancement mechanism to capture image spatial semantic information for semantic segmentation prediction;

[0024] A model training module is used to set a target loss function for model training, and train a building segmentation network based on the target loss function and using a sample data set to obtain a building segmentation target model;

[0025] The target extraction module is used to input the unmanned aerial vehicle image to be processed into the building segmentation target model, and use the building segmentation target model to perform segmentation prediction on the buildings in the unmanned aerial vehicle image to be processed.

[0026] Beneficial effects of the present invention:

[0027] The present invention introduces DINOv2 with strong feature extraction capability as the backbone network in the encoder module, and performs structural adaptation and transfer learning on it, so as to obtain image building features more robustly. Then, through the decoder module that takes into account global attention, the spatial details and contextual semantic information of the image are effectively captured, the network performance is improved, and finally the high-precision extraction of buildings from drone remote sensing images is achieved. And further experimental data show that the various evaluation indicators on LoveDA, WHU-Buildings and self-made data sets are optimal, among which the IOU of this case solution reaches 73.75%, 92.27% and 88.54% respectively, which are 4.43%, 0.88% and 3.34% higher than the existing building extraction methods. The solution of this case has good superiority and effectiveness in extracting buildings from drone images, and can meet the segmentation of buildings with rich semantics and complex background information in high-resolution drone images, and has good application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a schematic diagram of the process of extracting buildings from drone images by integrating DINOv2 and global attention in the embodiment;

[0029] Figure 2 This is a schematic diagram of the building segmentation network structure in the embodiment;

[0030] Figure 3 This is a schematic diagram of the DINOv2 model structure in the embodiment;

[0031] Figure 4This is a schematic diagram of the jump connection dimension transformation in the embodiment;

[0032] Figure 5 It is a schematic diagram of the working principle of the global attention mechanism in the embodiment;

[0033] Figure 6 It is a schematic diagram of the working principle of the feature enhancement mechanism in the embodiment;

[0034] Figure 7 This is a comparison diagram of the LoveDA dataset before and after Mask processing in the embodiment;

[0035] Figure 8 This is a comparison diagram of the visualization results of the building extraction scheme in the embodiment;

[0036] Fig. 9 Schematic comparison of the visualization results of the semantic segmentation scheme in the embodiment. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the present invention clearer and more understandable, the present invention is further described in detail below in conjunction with the accompanying drawings and technical solutions.

[0038] Since high-resolution drone images are semantically rich and have complex background information, traditional building extraction methods often perform poorly. Figure 1 As shown, a method for extracting buildings from drone images by integrating DINOv2 and global attention is provided, including:

[0039] S101, collecting remote sensing image data, preprocessing the remote sensing image data, and obtaining a sample data set containing a number of image blocks with building annotation labels.

[0040] Specifically, the preprocessing of remote sensing image data can be designed to include:

[0041] Fill the remote sensing image and crop it into image blocks of specified size;

[0042] Performing data enhancement processing on each image block using data enhancement technology, wherein the data enhancement technology includes random image scaling, image flipping and image rotation;

[0043] The image patches are annotated with building labels and converted into binary images containing only building areas and non-building areas.

[0044] Through preprocessing such as cropping, normalization, data augmentation, label annotation, and position encoding, the input image is divided into image blocks of fixed size to improve the quality and availability of sample data and ensure that the model can better learn and optimize.

[0045] S102. Construct a building segmentation network based on DINOv2 and global attention, wherein the building segmentation network includes: an encoder network for extracting a feature map of an input image and encoding an output and a decoder network for decoding the encoded output and segmenting and predicting buildings in the input image, and the encoder network uses the DINOv2 visual model as a backbone network for extracting image feature maps, and the decoder network uses a global attention mechanism and a feature enhancement mechanism to capture image spatial semantic information for semantic segmentation prediction.

[0046] Specifically, the encoder network includes: an embedding layer for converting an image block into an embedding vector, and a plurality of stacked encoding layers for encoding the embedding vector to obtain an image block encoding feature vector, each encoding layer includes an attention mechanism and a feedforward neural network, and the encoding layer output of each layer is used as the input of the next encoding layer, so as to utilize the stacked multiple encoding layers to capture the spatial information and global features of the image block.

[0047] like Figure 2 As shown in the figure, the building segmentation network mainly consists of three parts: feature encoding, feature decoding, and metric learning. First, in the encoding stage, the DINOv2 model is used as the feature extraction network for feature extraction; then, the extracted features are enhanced using global attention and feature enhancement; finally, it is trained based on the main and auxiliary loss functions to obtain the optimal model parameters.

[0048] DINOv2 is a visual basic model based on the ViT structure with powerful image feature extraction capabilities. Compared with the traditional CNN network, DINOv2 abandons the way of obtaining image features through convolution operations, but adds a large number of attention modules to the model to obtain richer spatial information and global features. In addition, because DINOv2 uses a larger and more extensive data set during training, it can learn more and higher-quality features, has better generalization capabilities, and can adapt to more scenarios and data distributions.

[0049] The DINOv2 model is as follows Figure 3 As shown in the figure, first, the input image passes through a two-dimensional convolution layer with a convolution kernel of 14×14 and a stride of 14, and a normalization layer to obtain a Patch of size 14×14. Then the obtained Patch is input to the ViTBlocks module. After ViT Blocks, a feature matrix of dimension C×D is obtained, where C represents the number of channels of the tensor and D represents the feature length. Then, after passing through the normalization layer, a feature vector of size 1×n is output. Finally, the Head module is flexibly selected according to the image task.

[0050] According to different configurations, DINOv2 has four models of different sizes, namely DINOv2-small, DINOv2-base, DINOv2-large and DINOv2-giant. Among them, DINOv2-giant has the largest number of parameters and stronger feature extraction capabilities. In this case, the encoder network can use DINOv2-giant as the backbone network. The stacked multiple coding layers have a level greater than 4, and the end coding layer of the backbone network is connected to the decoder network using a jump connection to maximize the backbone network's pre-trained parameter image understanding ability. The jump connection uses linear transformation and downsampling operations to obtain deep global features of image blocks. Crop the Head module and Layer Norm module in the DINOv2-giant model, and set a "skip connection" at the end of the backbone network output, such as Figure 4 As shown in the figure, the output of the last four layers of ViT Blocks are selected as the input of the "skip connection" in order to maximize the image understanding ability of the pre-trained parameters of the DINOv2 model. In addition, during the training of the model, in order to better obtain parameters suitable for the building extraction task, the last four layers are unfrozen for training. In each layer of "skip connection", linear transformation and downsampling operations are designed to obtain deeper global features.

[0051] In the decoding stage, in order to capture global features, the current mainstream methods focus on adding a single attention module at the end of the model or directly introducing the Transformers module as a decoder. However, the former cannot capture multi-scale global features, and the latter greatly increases the complexity of the network and loses spatial detail information. Therefore, in the embodiment of this case, a global local attention module and a feature enhancement module are added to the decoder to help the decoder effectively utilize global features at multiple scales and effectively capture the spatial details and contextual semantic information of the image.

[0052] Specifically, the decoder network includes several global attention layers and feature enhancement layers, wherein each global attention layer adopts two parallel branch structures of global branch and local branch to capture the spatial details and semantic information of image blocks, and the feature enhancement layer uses channel feature enhancement branch and spatial feature enhancement branch to reduce the semantic gap between features.

[0053] like Figure 2 The decoder shown is mainly composed of a global attention block (GAB) and a feature enhancement block (FEB).

[0054] The GAB module consists of a global-local attention mechanism, a normalization layer, and convolution operations, such as Figure 5As shown in the figure. GAB uses two parallel branches. The global branch uses multi-head attention based on cross-window to obtain global context information. First, a 1×1 convolution operation is used to convert the input two-dimensional feature vector into a one-dimensional sequence. Then, the one-dimensional sequence is input into the multi-head attention using a window partitioning operation. After the cross-window context interaction module, the global context information is output; the local branch uses two parallel convolution layers to extract local context information. The convolution kernel sizes are 1×1 and 3×3 respectively. Then, the local context information is obtained after normalization and fusion operations. The final stage of GAB will further aggregate the global context and the local context to generate a global-local context, and obtain more detailed global-local context information through deep convolution, normalization and 1×1 convolution. Through the two parallel structures of global branch and local branch, GAB can efficiently capture the spatial details and semantic content of the image and improve the accuracy of semantic segmentation. In addition, the Batch Norm operation, which is more suitable for image processing, is used in GAB to ensure that the same features between different samples can be compared with each other.

[0055] The FEB module is used to narrow the semantic gap between features to further improve accuracy. FEB constructs a channel feature enhancement branch and a spatial feature enhancement branch, respectively. Figure 6 As shown in the figure, the channel feature enhancement branch first uses a global average pooling layer to generate channel features, then reduces the channel dimension by 4 times through two 1×1 convolution layers, and then expands it to the original dimension. The spatial feature enhancement branch uses deep convolution to generate spatial features. The fused features are first enhanced by these two branches, and then the features are fused through summation operations. Finally, the segmented image is obtained through convolution operations and upsampling. In addition, residual connections are introduced in FEB to alleviate network degradation.

[0056] In the feature aggregation stage of the decoder, the model performs a weighted sum operation on the semantic features extracted by DINOv2 and the features obtained by GAB according to their contribution to the segmentation extraction accuracy, thereby obtaining a more generalized fusion feature. The weighted sum formula is:

[0057] FF=f·VF+(1-f)·GF (1)

[0058] In the formula, FF is the fused feature, VF is the semantic feature vector extracted by ViT, GF is the feature vector generated by the GAB module, and f is the fusion factor, which is calculated by the initialization weight through the ReLU function and normalization, and can be taken as 0.5.

[0059] S103, setting a target loss function for model training, training the building segmentation network based on the target loss function and using the sample data set to obtain a building segmentation target model.

[0060] Specifically, the target loss function can be used to train the model by combining a main and auxiliary loss function with a cross entropy-dice mixed loss as the main loss function and a cross entropy loss as the auxiliary loss function, wherein the main loss function is connected with the feature enhancement mechanism of the decoder network during the model training process to train the relevant parameters of the feature enhancement mechanism, and the auxiliary loss function is connected with the global attention mechanism of the decoder network to train the relevant parameters of the global attention mechanism.

[0061] In order to better obtain the learning parameters in the model, the embodiment of this case uses a main and auxiliary loss function that combines the cross entropy-dice mixed loss function (Cross Entropy-Dice Loss) and the cross entropy loss function (Cross Entropy Loss) to train the model. Among them, the main loss function is a mixture of the cross entropy loss function and the dice loss function. The mixed loss function considers both the pixel-level binary classification problem and the similarity evaluation. The specific expression of the mixed loss function is as follows:

[0062]

[0063] L m =α1×L1+α2×L2 (4)

[0064] Where, L m is a mixed loss function, L1 is the cross entropy loss function, L2 is the dice loss function, N is the number of samples, T is the number of categories, α1 and α2 are weight parameters, and α1 and α2 are set to 1 during training.

[0065] The auxiliary loss function is the cross entropy loss function, which is expressed as:

[0066]

[0067] Therefore, the total objective loss function used can be expressed as:

[0068] L=L m +β×L dice (6)

[0069] In the formula, β is the adjustment factor for adjusting the influence of the main loss function and the auxiliary loss function, and the default value can be set to 0.4.

[0070] During the training process, the main loss function is connected to the FEB in the decoder to train the parameters in the FEB module. The auxiliary loss function is connected to the three GAB modules to train the parameters in the GAB module. The loss function is controlled by adjusting the factor β to regulate the model training.

[0071] S104: Input the unmanned aerial vehicle image to be processed into a building segmentation target model, and use the building segmentation target model to perform segmentation prediction on the buildings in the unmanned aerial vehicle image to be processed.

[0072] Furthermore, based on the above method, an embodiment of the present invention also provides a drone image building extraction system integrating DINOv2 and global attention, comprising: a sample collection model, a model construction module, a model training module and a target extraction module, wherein:

[0073] The sample collection module is used to collect remote sensing image data, pre-process the remote sensing image data, and obtain a sample data set containing a number of image blocks with building annotation labels;

[0074] A model construction module is used to construct a building segmentation network based on DINOv2 and global attention, wherein the building segmentation network includes: an encoder network for extracting a feature map of an input image and encoding an output and a decoder network for decoding the encoded output and segmenting and predicting buildings in the input image, wherein the encoder network uses the DINOv2 visual model as a backbone network for extracting image feature maps, and the decoder network uses a global attention mechanism and a feature enhancement mechanism to capture image spatial semantic information for semantic segmentation prediction;

[0075] A model training module is used to set a target loss function for model training, and train a building segmentation network based on the target loss function and using a sample data set to obtain a building segmentation target model;

[0076] The target extraction module is used to input the unmanned aerial vehicle image to be processed into the building segmentation target model, and use the building segmentation target model to perform segmentation prediction on the buildings in the unmanned aerial vehicle image to be processed.

[0077] In order to verify the effectiveness of this solution, the following is a further explanation based on experimental data:

[0078] To ensure the objectivity of the test results, all experiments were conducted in a consistent hardware and software environment, and the training platform was Nvidia RTX 4090. In the experiment, the sample training batch was set to 32, the number of training epochs was set to 50, and the initial learning rate was set to 6e-4. In order to speed up convergence, the AdamW optimizer was used for training, and the cosine annealing strategy was used to dynamically adjust the learning rate.

[0079] The datasets were preprocessed. First, the images in each dataset were padded and cropped to 504×504 images. Then, during the training process, the images in each dataset were randomly scaled, flipped, and rotated, and data augmentation operations were performed. During the testing process, the test set was horizontally flipped and vertically flipped.

[0080] The UAVid dataset, LoveDA dataset, WHU-Building dataset and self-made UAV dataset (ZZULIUAV) were selected to carry out relevant experiments.

[0081] The LoveDA dataset contains 5987 high-resolution remote sensing images from three regions of China, all with a size of 1024×1024, and densely annotated in 7 categories. In the data preprocessing stage, the original 7-category annotated color images are converted into binary images containing only building areas (white) and non-building areas (black), such as Figure 7 shown.

[0082] The original aerial image data of the WHU-Building dataset comes from the New Zealand Land Information Service. The dataset contains more than 220,000 independent buildings extracted from Christchurch aerial images. After processing, the ground resolution of most buildings in the dataset (about 187,000 buildings) was reduced from 0.075m to 0.3m and cropped into 8189 512×512 images, including a training set (about 130,500 buildings), a validation set (14,500 buildings) and a test set (42,000 buildings).

[0083] The ZZULIUAV dataset is derived from the oblique photography 3D model of the Zhengzhou University of Light Industry area. The dataset is manually annotated and divided into two categories: buildings and non-buildings. There are 1575 images in total, all with a size of 512×512.

[0084] The UAVid dataset is a high-resolution UAV image semantic segmentation dataset, focusing on urban street scenes. 300 images are densely annotated in 8 categories for semantic segmentation tasks. The dataset has two different spatial resolutions, 3840×2140 and 4096×2160, with a total of 422 sequences and 420 images.

[0085] In order to quantitatively analyze the performance of the solution, the experiment selected four indicators: overall classification accuracy (OA), intersection over union (IoU), precision (Precision) and F1 as evaluation indicators. The specific formula is as follows:

[0086]

[0087] Where TP is the correct prediction of the segmented object, TN is the correct prediction of the non-segmented object, FP is the wrong prediction of the non-segmented object, and FN is the wrong prediction of the segmented object.

[0088] A series of mainstream segmentation methods are selected for comparative analysis. For example, lightweight semantic segmentation networks based on CNN: ABCNet, U-Net and DeepLabv3+; attention mechanism networks based on CNN: DANet; networks based on hybrid encoder-decoder structures of Transformer and CNN: SwinUperNet, DC-Swin, BANet; and networks based entirely on Transformer: SegFormer and Segmenter.

[0089] 1. Building extraction experiment

[0090] Experiments were conducted based on the LoveDA dataset, WHU-Building dataset, and ZZULIUAV dataset, and the above mainstream methods were selected for comparison with the ViTFormer solution in this case. The detailed experimental results are shown in Table 1.

[0091] Table 1 Test results of building extraction using ViTFormer and other methods

[0092]

[0093] Note: The bold part represents the solution for this case. If there is no other explanation, the following table has the same meaning.

[0094] From the test results in Table 1, we can see that compared with the mainstream methods, the scheme of this case has achieved the best indicators on the three data sets. On the LoveDA dataset, IoU increased by 4.43%, Precision increased by 2.1%, and F1 increased by 3.01%; on the ZZULIUAV dataset, compared with UnetFormer, the IoU of this case increased by 3.34%, Precision increased by 1.59%, and F1 increased by 1.84%; on the WHU-Building dataset, IoU increased by 0.88%, Precision increased by 2.49%, and F1 increased by 0.49%. From the precision evaluation indicators, it can be quantitatively analyzed that the scheme of this case, ViTFormer, has greatly improved the precision and accuracy of building extraction compared with the mainstream methods.

[0095] In addition, in order to more intuitively illustrate the effectiveness and superiority of this solution, some images in the dataset are randomly selected for building segmentation and extraction experiments. Figure 8As shown in the figure. From the visualized segmentation results, we can conclude that: (1) ViTFormer performs better in identifying buildings. For example, the road in the image in row (a) and the small container in the image in row (b) are accurately distinguished by ViTFormer and classified as non-buildings; (2) ViTFormer can effectively extract buildings at the edge of the image. For example, the conjoined buildings at the edge of the image in row (c) and the independent buildings at the edge of the image in row (d) can be accurately extracted by ViTFormer; (3) ViTFormer can suppress the interference of noise and accurately extract the building area. For example, the building in row (e) is surrounded by vegetation. Affected by the vegetation, UnetFormer fails to completely segment and extract it, while ViTFormer can effectively suppress the interference of vegetation noise and accurately extract the building.

[0096] Compared with mainstream methods, ViTFormer has greatly improved the building extraction accuracy and precision, especially in accurately extracting small buildings, edge buildings and buildings affected by noise in the image. This is mainly due to the strong feature extraction capability of DINOv2, which can obtain high-quality features; the GAB module in the decoder adopts a dual-path global-local branching structure, which can efficiently capture the spatial details and semantic content of the image and improve the segmentation extraction accuracy. The FEB module can narrow the semantic gap between features and further improve the building extraction accuracy. In addition, the "jump connection" of this solution captures deep global features, which helps to improve the extraction accuracy.

[0097] 2. Semantic Segmentation Experiment

[0098] A semantic segmentation experiment was conducted based on the UAVid dataset. The experimental results are shown in Table 2.

[0099] As can be seen from Table 2, the proposed solution ViTFormer outperforms the mainstream network in all indicators, with an improvement of 4.95% in mIoU, 2.14% in OA, and 3.13% in F1. From a quantitative perspective, it can be seen that the proposed solution ViTFormer performs well in the semantic segmentation task, which fully confirms that the DINOv2-based encoder used by the proposed solution ViTFormer has strong feature extraction capabilities and can extract various different image features.

[0100] Table 2 Semantic segmentation test results of this solution and other methods

[0101]

[0102]

[0103] In addition, the visual segmentation results of the most advanced segmentation network UnetFormer on the UAVid dataset are selected for qualitative comparative analysis, such as Fig. 9 As shown in the figure, from the visualization results, we can conclude that: (1) the ViTFormer solution in this case has more advantages in the recognition and segmentation of small objects, such as in rows (a) and (c), the model can accurately identify different objects and perform effective segmentation; (2) the ViTFormer solution in this case can more completely extract distant objects in the image, such as in row (b), the model can more completely extract the segmentation of distant buildings in the image.

[0104] From the quantitative and qualitative analysis of the semantic segmentation experiment, it can be concluded that the solution ViTFormer in this case has higher extraction accuracy than the mainstream method, and can efficiently segment and extract buildings and other objects. The reason is that the encoder introduces the DINOv2 backbone network with strong extraction capabilities. DINOv2 contains a large number of attention modules, which can effectively capture the global information in the image and then establish long-range dependencies, thereby obtaining the overall features of the image and accurately identifying and extracting different objects in the image; the global local attention module and feature enhancement module are added to the decoder, and multi-scale information is aggregated through jump connections to help capture the spatial details and contextual semantic information of the image.

[0105] 3. Cross-validation experiment

[0106] Cross-validation experiments were conducted on different datasets, and the models trained on the WHU-Building dataset were tested on the LoveDA dataset and the ZZULIUAV dataset. The most advanced segmentation network UnetFormer was selected as the comparison network for the experiment, and the detailed results are shown in Table 3.

[0107] Table 3 Cross-validation test results of ViTFormer and other methods

[0108]

[0109] As can be seen from the table, the performance of ViTFormer in this case is significantly better than UnetFormer in the cross-validation test. On the LoveDA dataset, IoU is improved by 25.69%, Precision is improved by 20.94%, and F1 is improved by 24.98%; on the ZZULIUAV dataset, IoU is improved by 28.36%, Precision is improved by 2.41%, and F1 is improved by 30.68%. From a quantitative perspective, it can be seen that ViTFormer in this case performs significantly better than UnetFormer and has good generalization ability. This is mainly due to the DINOv2 backbone network introduced in the encoder part of ViTFormer in this case. DINOv2 has many learning parameters and is trained based on ultra-large-scale datasets. It has extremely strong generalization ability. Fine-tuning its learning parameters can adapt well to downstream tasks.

[0110] 4. Fine-tuning ablation experiment

[0111] According to different configurations, DINOv2 has four models of different sizes, namely DINOv2-small, DINOv2-base, DINOv2-large and DINOv2-giant. The specific parameters of the four models are shown in Table 4.

[0112] Table 4 Comparison of different model parameters

[0113]

[0114] In order to explore the impact of different unfreezing layers on the experimental results during model training, and taking into account the training cost, the experiment selected DINOv2-base as the backbone network, and set the number of ViT Blocks unfreezing layers of the DINOv2-base backbone network to 0, 2, 4, 6, 8, 10, and 12 layers for experiments. The detailed experimental results are shown in Table 5.

[0115] Table 5 Effect of different thawing layers on extraction accuracy

[0116]

[0117] Note: The bold part indicates the best.

[0118] From the test results in Table 5, we can see that as the number of unfreezing layers increases, the IoU, Precision and F1 of ViTFormer have not improved significantly, but have decreased in some indicators. This shows that DINOv2-base has learned different features in the image well in the pre-training stage and has good feature extraction and expression capabilities in downstream tasks. Considering the impact of the three indicators on the building extraction results, the model setting of 4 unfreezing layers has the best effect.

[0119] The above experimental results show that the proposed scheme has good superiority and effectiveness in extracting buildings from drone images, and can meet the requirements of extracting and identifying buildings from high-resolution drone images with rich semantics and complex background information.

[0120] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0121] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0122] The units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementation is not considered to be beyond the scope of the present invention.

[0123] Those skilled in the art will appreciate that all or part of the steps in the above method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk. Optionally, all or part of the steps in the above embodiment can also be implemented using one or more integrated circuits, and accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or in the form of software function modules. The present invention is not limited to any specific form of combination of hardware and software.

[0124] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for extracting buildings from drone images by integrating DINOv2 and global attention, characterized in that: Include: Collect remote sensing image data, pre-process the remote sensing image data, and obtain a sample data set containing several image blocks with building annotation labels; A building segmentation network is constructed based on DINOv2 and global attention, wherein the building segmentation network includes: an encoder network for extracting a feature map of an input image and encoding an output and a decoder network for decoding the encoded output and segmenting and predicting buildings in the input image, wherein the encoder network uses the DINOv2 visual model as a backbone network for extracting image feature maps, and the decoder network uses a global attention mechanism and a feature enhancement mechanism to capture image spatial semantic information for semantic segmentation prediction; Set a target loss function for model training, train the building segmentation network based on the target loss function and using the sample data set to obtain a building segmentation target model; The unprocessed UAV image is input into the building segmentation target model, and the building segmentation target model is used to predict the segmentation of the buildings in the unprocessed UAV image.

2. The method for extracting buildings from drone images by integrating DINOv2 and global attention according to claim 1, characterized in that: Preprocess remote sensing image data, including: Fill the remote sensing image and crop it into image blocks of specified size; Performing data enhancement processing on each image block using data enhancement technology, wherein the data enhancement technology includes random image scaling, image flipping and image rotation; The image patches are annotated with building labels and converted into binary images containing only building areas and non-building areas.

3. The method for extracting buildings from drone images by integrating DINOv2 and global attention according to claim 1, characterized in that: The encoder network comprises: an embedding layer for converting an image block into an embedding vector, and a plurality of stacked encoding layers for encoding the embedding vector to obtain an encoding feature vector of the image block, each encoding layer comprises an attention mechanism and a feedforward neural network, and the encoding layer output of each layer is used as the input of the encoding layer of the next layer, so as to capture the spatial information and global features of the image block by using the stacked multiple encoding layers.

4. The method for extracting buildings from drone images by integrating DINOv2 and global attention according to claim 3 is characterized in that: The encoder network uses DINOv2-giant as the backbone network, and the terminal encoding layer of the backbone network is connected to the decoder network using a skip connection to maximize the image understanding ability of the backbone network pre-trained parameters, wherein the skip connection uses linear transformation and downsampling operations to obtain deep global features of image blocks.

5. The method for extracting buildings from drone images by integrating DINOv2 and global attention according to claim 1, characterized in that: The decoder network includes several global attention layers and feature enhancement layers, wherein each global attention layer adopts two parallel branch structures of global branch and local branch to capture the spatial details and semantic information of image blocks, and the feature enhancement layer uses channel feature enhancement branch and spatial feature enhancement branch to reduce the semantic gap between features.

6. The method for extracting buildings from drone images by integrating DINOv2 and global attention according to claim 5, characterized in that: The global branch obtains the global context information of the feature vector based on cross-window multi-head attention, and the local branch extracts the local context information of the feature vector through parallel convolution layers, normalization and fusion operations. For the global context information and local context information, the global attention layer obtains the global-local context information through aggregation, convolution and normalization, and the global-local context information includes the spatial details and semantic content of the image blocks.

7. The method for extracting buildings from drone images by integrating DINOv2 and global attention according to claim 5, characterized in that: The channel feature enhancement branch uses a global average pooling layer to generate image block channel features and uses two convolution layers to reduce the channel dimension and restore it to the original dimension to obtain image block channel enhancement features. The spatial feature enhancement branch uses deep convolution to generate image block spatial enhancement features. For the image block feature vector, channel enhancement feature and spatial enhancement feature, the feature enhancement layer performs feature fusion through a summation operation, and uses convolution operations and upsampling to obtain the final output building segmentation image.

8. The method for extracting buildings from drone images by integrating DINOv2 and global attention according to claim 7, characterized in that: The process of feature fusion through summation operation is expressed as: FF=f·VF-+(1-f)GF, where f is the fusion factor, VF is the feature after the channel enhancement feature and the spatial enhancement feature are aggregated, and GF is the feature map encoding representation obtained by the encoder network.

9. The method for extracting buildings from drone images by integrating DINOv2 and global attention according to claim 1, characterized in that: The target loss function adopts a main and auxiliary loss function combined with a cross entropy-dice mixed loss as the main loss function and a cross entropy loss as the auxiliary loss function to train the model. The main loss function is connected with the feature enhancement mechanism of the decoder network during the model training process to train the relevant parameters of the feature enhancement mechanism, and the auxiliary loss function is connected with the global attention mechanism of the decoder network to train the relevant parameters of the global attention mechanism.

10. A drone image building extraction system integrating DINOv2 and global attention, characterized in that: It includes: sample collection model, model construction module, model training module and target extraction module, among which, The sample collection module is used to collect remote sensing image data, pre-process the remote sensing image data, and obtain a sample data set containing a number of image blocks with building annotation labels; A model construction module is used to construct a building segmentation network based on DINOv2 and global attention, wherein the building segmentation network includes: an encoder network for extracting a feature map of an input image and encoding an output and a decoder network for decoding the encoded output and segmenting and predicting buildings in the input image, wherein the encoder network uses the DINOv2 visual model as a backbone network for extracting image feature maps, and the decoder network uses a global attention mechanism and a feature enhancement mechanism to capture image spatial semantic information for semantic segmentation prediction; A model training module is used to set a target loss function for model training, and train a building segmentation network based on the target loss function and using a sample data set to obtain a building segmentation target model; The target extraction module is used to input the unmanned aerial vehicle image to be processed into the building segmentation target model, and use the building segmentation target model to perform segmentation prediction on the buildings in the unmanned aerial vehicle image to be processed.

Images