Building remote sensing image feature extraction method, feature extraction model, system and storage medium

Abstract

The invention relates to the technical field of deep learning, in particular to a building remote sensing image feature extraction method, a feature extraction model, a system and a storage medium. Designing a multi-scale wavelet transform convolution feature fusion network, and controlling a misclassification phenomenon under the conditions of different sizes, different resolutions, complex building distribution conditions and more similar objects; a multi-scale wavelet transform convolution module is designed on an encoder of the multi-scale wavelet transform convolution feature fusion network, and multi-scale feature learning is carried out on high-frequency information and low-frequency information in the image; a space and channel attention synergistic effect mechanism module used for enhancing the extraction capability of global information of a model is arranged at the tail end of an encoder, and the problem of parameter dramatic increase caused by multi-scale convolution is avoided while the capture capability of local context information is enhanced; in addition, a feature fusion extraction module is designed on the decoder and used for fusing features of different scales. The objective of the invention is to solve the problem of how to reduce the misclassification phenomenon in the building remote sensing image recognition process and save the model computing power overhead.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a feature extraction method, feature extraction model, system and storage medium for building remote sensing images. Background Art

[0002] With the rapid development of remote sensing technology and unmanned aerial vehicle photogrammetry technology, the research on efficient and accurate automatic extraction of buildings from high-resolution images has attracted widespread attention from scholars at home and abroad. The traditional feature extraction method of building remote sensing images is to use manual visual interpretation to identify and extract shallow features such as building spectrum and texture in the image based on the spectral information, geometric information and texture of the building, and then use machine learning methods such as random forest and support vector machine based on these features to further extract the building. This type of method has a strong dependence on the researcher's experience and knowledge, the classification results will vary due to personal experience, and the extraction efficiency is low. In addition, since these methods can only judge from the perspective of shallow features, it is difficult to mine deep semantic features with richer semantic information, resulting in a certain limit on their performance upper limit.

[0003] In recent years, deep learning has become the mainstream method in this research field due to its ability to automatically learn features and its superior efficiency and performance. At present, there are two main research directions for building extraction methods based on high-resolution remote sensing images based on deep learning: building feature extraction methods based on CNN (Convolutional Neural Network) models and building feature extraction methods based on CNN+Transformer hybrid models.

[0004] Although the CNN model has a great advantage in extracting local features of images, traditional convolution is not good enough in capturing contextual information. Increasing the receptive field by increasing the convolution kernel will be severely limited by parameters, resulting in its performance ceiling being constrained. In addition, in urban areas, due to the complex distribution of buildings and other land objects, the frequency distribution of images will also change greatly. When faced with such complex high and low frequency changes, many models often lack better universal performance, resulting in obvious misclassification.

[0005] Although the Transformer model has obvious advantages in processing global contextual information, it is often difficult for the Transformer to pay attention to the multiple and complex local feature changes in complex and changeable building images. In addition, the Transformer itself has a high number of parameters and computational complexity, making it difficult for the model to balance performance and efficiency.

[0006] In view of this, a building feature extraction model is needed that can control the misclassification phenomenon in cases of different sizes, different resolutions, complex building distribution, and a large number of similar objects, while also avoiding the high model computing power overhead caused by the sharp increase in parameters brought about by multi-scale convolution.

[0007] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0008] The main purpose of this application is to provide a feature extraction method for building remote sensing images, aiming to solve the problem of how to reduce the misclassification phenomenon in the process of building remote sensing image recognition while saving model computing power overhead.

[0009] To achieve the above purpose, the present application provides a method for extracting features from a building remote sensing image, the method comprising:

[0010] Obtain remote sensing images of buildings collected by drones;

[0011] Inputting the building remote sensing image into a pre-trained model including a multi-scale wavelet transform convolutional attention network, wherein the multi-scale wavelet transform convolutional attention network includes an encoder and a decoder;

[0012] The encoder includes a multi-scale wavelet transform convolution module for feature learning of high-frequency information and low-frequency information in the building remote sensing image, and a spatial and channel attention synergy mechanism module arranged at the end of the encoder for enhancing the extraction capability of the model's global information;

[0013] Wherein, the decoder comprises a feature fusion sampling module for weighted fusion of feature maps of corresponding sizes in the encoder and upsampling layer by layer to restore the resolution;

[0014] Obtain the building image remote sensing recognition result output by the pre-trained model.

[0015] Optionally, the step of processing the building remote sensing image by the multi-scale wavelet transform convolutional attention network includes:

[0016] Based on a multi-scale wavelet transform convolution module, the building remote sensing image is decomposed by wavelet to obtain a low-frequency feature map and a high-frequency feature map, and the low-frequency feature map, the high-frequency feature map and the original feature map are subjected to multi-scale convolution;

[0017] The low-frequency feature map, the high-frequency feature map and the original feature map after multi-scale convolution are respectively averaged and fused to obtain a multi-scale convolution fusion feature map; and the low-frequency feature map and the high-frequency feature map are scaled, inversely transformed and restored to the original resolution and spliced ​​together to obtain a spliced ​​feature map;

[0018] The multi-scale convolution fusion feature map and the splicing feature map are fused and output.

[0019] Optionally, the step of performing multi-scale convolution on the low-frequency feature map, the high-frequency feature map and the original feature map comprises:

[0020] Convolution kernels of sizes 3*3, 4*4, and 7*7 are used respectively to perform wavelet transform convolution on the low-frequency feature map, the high-frequency feature map, and the original feature map.

[0021] Optionally, the pre-trained model includes a global channel and spatial attention module, the global channel and spatial attention module includes a self-attention-channel attention module and a spatial attention mechanism module, and the global channel and spatial attention module performs the following steps:

[0022] Obtaining the channel global attention weight of the deepest feature map through the self-attention-channel attention module, and obtaining the spatial global attention weight of the deepest feature map through the spatial attention mechanism module;

[0023] Multiplying the channel global attention weight by the input feature map to obtain a channel attention weight feature map; and multiplying the spatial global attention weight by the input feature map to obtain a spatial attention weight feature map;

[0024] The channel attention weight feature map and the spatial attention weight feature map are multiplied to fuse and enhance the features extracted by the wavelet transform convolution in the encoder.

[0025] Optionally, the self-attention-channel attention module uses global average pooling to reduce the deepest feature map to 1x1 and retain channel information, and then obtains the Querry value, Key value and Value value corresponding to the deepest feature map through a convolutional layer, and performs a self-attention mechanism operation to obtain the channel global attention weight.

[0026] Optionally, the spatial attention mechanism module performs maximum pooling and average pooling on the channel of the deepest feature map to reduce the channel to 1 and retain the spatial position information, and then adds and fuses the feature map after maximum pooling with the feature map after average pooling at the channel level to obtain the average eigenvalue and maximum eigenvalue of each position, and inputs the average eigenvalue and the maximum eigenvalue into the group convolution module for feature learning, and finally obtains the spatial global attention weight through the activation function.

[0027] Optionally, the decoder performs the following steps:

[0028] Upsampling the feature map output by the encoder to transform the size to the same scale as the feature map of the corresponding skip connection;

[0029] Performing weighted fusion on the upsampled feature map in the decoder and the feature map of the encoder for skip connection of the same scale, and performing feature enhancement on the fused feature map using depthwise separable convolution;

[0030] The feature map after depth-wise separable convolution is output as the input of the next upsampling, and the feature map after upsampling is used as the feature fusion of the output layer.

[0031] In addition, to achieve the above purpose, the present application also provides a building feature extraction model, characterized in that it includes a multi-scale wavelet transform convolutional attention network, and the multi-scale wavelet transform convolutional attention network includes:

[0032] An encoder, comprising a multi-scale wavelet transform convolution module for feature learning of high-frequency information and low-frequency information in the remote sensing image of the building, and a spatial and channel attention synergy mechanism module arranged at the end of the encoder for enhancing the extraction capability of the model's global information;

[0033] The decoder includes a feature fusion sampling module for weighted fusion of feature maps of corresponding sizes in the encoder and upsampling layer by layer to restore the resolution.

[0034] In addition, to achieve the above-mentioned purpose, the present application also provides a building remote sensing image recognition system, which includes: a memory, a processor, and a feature extraction program for building remote sensing images stored in the memory and executable on the processor, wherein the feature extraction program for building remote sensing images implements the steps of the feature extraction method for building remote sensing images as described in any of the above items when executed by the processor.

[0035] In addition, to achieve the above-mentioned purpose, the present application also provides a computer-readable storage medium, on which a feature extraction program for building remote sensing images is stored. When the feature extraction program for building remote sensing images is executed by a processor, the steps of the feature extraction method for building remote sensing images as described in any of the above items are implemented.

[0036] This application has at least the following beneficial effects:

[0037] 1. A multi-scale wavelet transform convolutional feature fusion network (MWTCFFNet) is proposed, which can control the misclassification phenomenon and maintain high accuracy in the cases of different sizes, different resolutions, complex building distribution and many similar objects.

[0038] 2. A multi-scale wavelet transform convolution module is designed to perform multi-scale feature learning on the high-frequency information and low-frequency information in the image respectively. In addition, while enhancing the ability to capture local context information, it can also avoid the problem of dramatic increase in parameters caused by multi-scale convolution.

[0039] 3. The FFU module is proposed to fuse features of different scales, thereby focusing on the feature map information of different scales and further refining the learned building features. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of the architecture of the hardware operating environment of the building remote sensing image recognition system involved in the embodiment of the present application;

[0041] Figure 2 A schematic diagram of a flow chart of a first embodiment of a method for extracting features from a building remote sensing image of the present application;

[0042] Figure 3 A frequency distribution diagram of different images of buildings collected by the drone involved in the embodiment of the present application;

[0043] Figure 4 This is a schematic diagram of the architecture of a multi-scale wavelet transform convolutional attention network involved in an embodiment of the present application;

[0044] Figure 5 This is a schematic diagram of the architecture of the self-attention-channel attention module involved in the embodiment of the present application;

[0045] Figure 6 This is a schematic diagram of the spatial attention mechanism module architecture involved in an embodiment of the present application;

[0046] Figure 7 This is a comparison chart of the segmentation results of the WHU dataset involved in the embodiment of the present application;

[0047] Figure 8A comparison diagram of segmentation results of a typical Chinese urban building example dataset involved in an embodiment of the present application;

[0048] Fig. 9 A comparison diagram of the segmentation results of the Massachusetts Buildings dataset involved in the embodiment of the present application;

[0049] Fig.10 This is a comparison chart of the parameters of the IOU indicators of various models involved in the embodiments of the present application on the WHU dataset.

[0050] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0051] In order to better understand the above technical solution, exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0052] As an implementation plan, Figure 1 This is a schematic diagram of the architecture of the hardware operating environment of the building remote sensing image recognition system involved in the embodiment of the present application.

[0053] like Figure 1 As shown, the building remote sensing image recognition system may include: a processor 1001, such as a CPU, a memory 1005, a user interface 1003, a network interface 1004, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0054] Those skilled in the art will understand that Figure 1 The architecture of the building remote sensing image recognition system shown in the figure does not constitute a limitation on the building remote sensing image recognition system, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0055] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a feature extraction program for building remote sensing images. The operating system is a program for managing and controlling the hardware and software resources of the building remote sensing image recognition system, the feature extraction program for building remote sensing images, and the operation of other software or programs.

[0056] exist Figure 1 In the building remote sensing image recognition system shown, the user interface 1003 is mainly used to connect to the terminal and communicate data with the terminal; the network interface 1004 is mainly used for the background server and communicates data with the background server; the processor 1001 can be used to call the feature extraction program of the building remote sensing image stored in the memory 1005.

[0057] In this embodiment, the building remote sensing image recognition system includes: a memory 1005, a processor 1001, and a feature extraction program of the building remote sensing image stored in the memory and executable on the processor, wherein:

[0058] When the processor 1001 calls the feature extraction program of the building remote sensing image stored in the memory 1005, the following operations are performed:

[0059] Obtain remote sensing images of buildings collected by drones;

[0060] Inputting the building remote sensing image into a pre-trained model including a multi-scale wavelet transform convolutional attention network, wherein the multi-scale wavelet transform convolutional attention network includes an encoder and a decoder;

[0061] The encoder includes a multi-scale wavelet transform convolution module for feature learning of high-frequency information and low-frequency information in the building remote sensing image, and a spatial and channel attention synergy mechanism module arranged at the end of the encoder for enhancing the extraction capability of the model's global information;

[0062] Wherein, the decoder comprises a feature fusion sampling module for weighted fusion of feature maps of corresponding sizes in the encoder and upsampling layer by layer to restore the resolution;

[0063] Obtain the building image remote sensing recognition result output by the pre-trained model.

[0064] When the processor 1001 calls the feature extraction program of the building remote sensing image stored in the memory 1005, the following operations are performed:

[0065] Obtain remote sensing images of buildings collected by drones;

[0066] Inputting the building remote sensing image into a pre-trained model including a multi-scale wavelet transform convolutional attention network, wherein the multi-scale wavelet transform convolutional attention network includes an encoder and a decoder;

[0067] The encoder includes a multi-scale wavelet transform convolution module for feature learning of high-frequency information and low-frequency information in the building remote sensing image, and a spatial and channel attention synergy mechanism module arranged at the end of the encoder for enhancing the extraction capability of the model's global information;

[0068] Wherein, the decoder comprises a feature fusion sampling module for weighted fusion of feature maps of corresponding sizes in the encoder and upsampling layer by layer to restore the resolution;

[0069] Obtain the building image remote sensing recognition result output by the pre-trained model.

[0070] When the processor 1001 calls the feature extraction program of the building remote sensing image stored in the memory 1005, the following operations are performed:

[0071] Based on a multi-scale wavelet transform convolution module, the building remote sensing image is decomposed by wavelet to obtain a low-frequency feature map and a high-frequency feature map, and the low-frequency feature map, the high-frequency feature map and the original feature map are subjected to multi-scale convolution;

[0072] The low-frequency feature map, the high-frequency feature map and the original feature map after multi-scale convolution are respectively averaged and fused to obtain a multi-scale convolution fusion feature map; and the low-frequency feature map and the high-frequency feature map are scaled, inversely transformed and restored to the original resolution and spliced ​​together to obtain a spliced ​​feature map;

[0073] The multi-scale convolution fusion feature map and the splicing feature map are fused and output.

[0074] When the processor 1001 calls the feature extraction program of the building remote sensing image stored in the memory 1005, the following operations are performed:

[0075] Convolution kernels of sizes 3*3, 4*4, and 7*7 are used respectively to perform wavelet transform convolution on the low-frequency feature map, the high-frequency feature map, and the original feature map.

[0076] When the processor 1001 calls the feature extraction program of the building remote sensing image stored in the memory 1005, the following operations are performed:

[0077] Obtaining the channel global attention weight of the deepest feature map through the self-attention-channel attention module, and obtaining the spatial global attention weight of the deepest feature map through the spatial attention mechanism module;

[0078] Multiplying the channel global attention weight by the input feature map to obtain a channel attention weight feature map; and multiplying the spatial global attention weight by the input feature map to obtain a spatial attention weight feature map;

[0079] The channel attention weight feature map and the spatial attention weight feature map are multiplied to fuse and enhance the features extracted by the wavelet transform convolution in the encoder.

[0080] When the processor 1001 calls the feature extraction program of the building remote sensing image stored in the memory 1005, the following operations are performed:

[0081] Upsampling the feature map output by the encoder to transform the size to the same scale as the feature map of the corresponding skip connection;

[0082] Performing weighted fusion on the upsampled feature map in the decoder and the feature map of the encoder for skip connection of the same scale, and performing feature enhancement on the fused feature map using depthwise separable convolution;

[0083] The feature map after depth-wise separable convolution is output as the input for the next upsampling, and the feature map after upsampling is used as the feature fusion of the output layer.

[0084] Based on the hardware architecture of the building remote sensing image recognition system using the above-mentioned deep learning technology, an embodiment of the feature extraction method of the building remote sensing image of the present application is proposed.

[0085] First embodiment

[0086] Reference Figure 2 In a first embodiment, the feature extraction method of the building remote sensing image comprises the following steps:

[0087] Step S10, obtaining a remote sensing image of a building collected by a drone;

[0088] In this embodiment, see Figure 3 The frequency distribution diagram of different images of buildings collected by drones is shown. In urban areas, due to the complex distribution of buildings and other objects, the frequency distribution of images will also change greatly. It can be seen that the image frequency displayed in the densely populated area on the left side of the figure is higher, and the image frequency displayed in the sparsely populated area on the right side of the figure is lower. This feature will cause many models to lack better universal performance when facing such complex high and low frequency changes, resulting in obvious misclassification.

[0089] Step S20, inputting the building remote sensing image into a pre-trained model including a multi-scale wavelet transform convolutional attention network, wherein the multi-scale wavelet transform convolutional attention network includes an encoder and a decoder;

[0090] In this embodiment, based on the characteristics of building remote sensing images, a pre-training model including a Multiscale Wavelet Transform Convolutional Feature Fusion Network (MWTCFFNet) is proposed.

[0091] For example, refer to Figure 4 The architecture diagram of the multi-scale wavelet transform convolutional attention network shown in Figure 1 is a schematic diagram of the architecture of the multi-scale wavelet transform convolutional attention network. This architecture adopts the common U-shaped structure of the encoder and decoder ( Figure 4 The encoder includes a multi-scale wavelet transform convolution module (Multi-Scale Wavelet Transform Convolution, MWTConv) for performing feature learning on high-frequency information and low-frequency information in the building remote sensing image, namely, the VGG16_(MSWTConv)_Encoder part in the figure.

[0092] It should be noted that, in this embodiment, the input feature map is decomposed into sub-bands of different frequencies through wavelet transform, and multi-scale convolution is performed on the sub-bands of different frequencies respectively, and then depthwise separable convolution is added, which can effectively avoid the problem of sudden increase in parameters caused by parallel processing of convolution kernel branches of different scales.

[0093] Further and optionally, reference is made to Figure 4 , the end part of the encoder ( Figure 4 (shown in the upper right part of the figure) is also provided with a spatial and channel attention synergy mechanism module (Global Channel and Spatial Attention, GCASA) for enhancing the model's ability to extract global information. In GCASA, in this embodiment, the self-attention-channel attention module (Self-Channel Attention, SCA) is used to obtain the channel global attention weight of the deepest feature map, and the spatial attention mechanism (Spatail Attention, SA) is used to obtain the spatial global attention weight of the deepest feature map. The attention weights are multiplied with the input feature map to obtain a feature map with attention weights. Finally, the channel and spatial attention feature maps are multiplied to fuse and enhance the attention feature map.

[0094] Optionally, to effectively obtain channel-level context information, refer to Figure 5 The schematic diagram of the self-attention-channel attention module architecture is shown. The self-attention-channel attention module uses global average pooling (Pool) to reduce the deepest feature map to 1x1 and retain the channel information, and then obtains the Querry value, Key value and Value value corresponding to the deepest feature map through the convolution layer (Conv), and performs self-attention mechanism operation to obtain the channel global attention weight (Matmul) and output it.

[0095] Optionally, to obtain global information at the spatial level, refer to Figure 6 The schematic diagram of the spatial attention mechanism module architecture is shown. The SA mechanism module performs maximum pooling (Max Pool) and average pooling (Average Pool) on the channel of the deepest feature map to reduce the channel to 1 and retain the spatial position information. Then, the feature map after maximum pooling and the feature map after average pooling are added and fused at the channel level to obtain the average eigenvalue and maximum eigenvalue of each position. The average eigenvalue and the maximum eigenvalue are input into the group convolution module (Grpups Conv2d) for feature learning, and finally the spatial global attention weight is obtained through the activation function (Sigmoid).

[0096] Also refer to Figure 4 The decoder part includes a feature fusion sampling module for weighted fusion of feature maps of corresponding sizes in the encoder and upsampling layer by layer to restore the resolution. Through feature fusion sampling (Feature Fused Upsampling, FFU), the input feature map is jump-connected with the MSWTConv feature map corresponding to the encoder and upsampled to restore the same resolution as the input image.

[0097] Optionally, deep features have rich semantic information, while shallow features have rich detail information. Inspired by the Feature Pyramid Network (FPN), this embodiment introduces the FFU module in the decoder to solve the problem that frequent upsampling leads to increased computational complexity when fusing features of different scales, and the accuracy of simple fusion strategies is limited, and enables the decoder to perform the following steps during the decoding process:

[0098] Step S100, upsampling the feature map output by the encoder to transform the size to the same scale as the feature map of the corresponding skip connection;

[0099] Step S200, performing weighted fusion on the upsampled feature map in the decoder and the feature map of the encoder for skip connection of the same scale, and performing feature enhancement on the fused feature map using depthwise separable convolution;

[0100] Step S300, output the feature map after the depth-wise separable convolution as the input of the next upsampling, and use the feature map after the upsampling as the feature fusion of the output layer.

[0101] Step S30, obtaining the building image remote sensing recognition result output by the pre-training model.

[0102] Second embodiment

[0103] Based on the first embodiment, in this embodiment, taking into account that traditional convolution will cause a sharp increase in computational complexity when the convolution kernel is increased, in this embodiment, wavelet transform convolutions with convolution kernels of 3*3, 5*5, and 7*7 are used to learn multi-scale local features of feature maps containing attention information, so as to reduce the influence of the convolution kernel size on the parameter quantity.

[0104] In addition, wavelet transform has multi-resolution analysis capability and can effectively separate signals of different frequencies, while multi-scale convolution can extract spatial features of different scales. In this embodiment, the two are combined to improve the sensitivity of the model to different features.

[0105] Third embodiment

[0106] Based on any of the above embodiments, the feature extraction method of the building remote sensing image proposed in the above embodiments and the feature extraction effect of the building feature extraction model are exemplarily described in this embodiment:

[0107] (I) Configuration environment

[0108] In this embodiment, the hardware configuration is NVIDIA RTX 2080*2 (22G) graphics card, the processor is Intel(R) Xeon(R) Platinum 8336C CPU@2.30GHz, the running memory is 45G, the operating system is Linux system, the Pytorch deep learning framework is used, the Python version is 3.8, and the CUDA version is 11.1.

[0109] (II) Dataset Selection

[0110] In this embodiment, the WHU dataset, the Chinese typical city building instance dataset and the Massachusetts Buildings dataset are selected for comparative examples.

[0111] (1) WHU dataset

[0112] The WHU dataset consists of aerial images and satellite images. The aerial image dataset includes about 220,000 buildings in New Zealand, with a spatial resolution of 0.3m and a coverage of 450km2; the satellite image dataset includes urban images in East Asia acquired by remote sensing satellites such as QuickBird, Worldview series, IKONOS, and ZY-3, covering an area of ​​550km2. This study uses the high-resolution aerial images in the WHU dataset as the implementation object, mainly including 4736 training sets, 1036 validation sets, and 2416 test sets with pixels of 512×512, with a total of about 187,000 buildings. After data cleaning, the dataset consists of 4317 images in the training set, 737 images in the validation set, and 1371 images in the test set. The ratio of training set, validation set, and test set is 6:1:2.

[0113] (2) China’s Typical Urban Building Example Dataset

[0114] The dataset of typical urban buildings in China contains 7260 samples of 500×500 pixel image areas, with a total of 63,886 buildings distributed in four cities: Beijing, Shanghai, Shenzhen and Wuhan. Since it mainly focuses on buildings in urban areas, it is challenging to extract buildings in densely distributed and diverse urban areas. In this embodiment, the dataset is divided into 4356 training sets, 1452 validation sets and 1452 test sets in a ratio of 6:2:2. After data cleaning, the training set of the dataset consists of 4098 images, the validation set consists of 1367 images, and the test set consists of 1366 images. The ratio of training set, validation set and test set is 3:1:1.

[0115] (3) Massachusetts Buildings Dataset

[0116] The Massachusetts Buildings dataset contains approximately 340 square kilometers of building image samples in the Boston area. The dataset consists of 151 aerial images with a size of 1500×1500 and a spatial resolution of 1m. Since the images cover a large area of ​​densely populated small buildings in urban and suburban areas, it is also challenging for small building detection. We expand the original training set to 411 images through data augmentation (flipping). During the training phase, we randomly cropped the images and labels to 1024×1024 pixels as input. During the validation and testing phases, both the images and labels were padded to a size of 1536×1536 pixels to ensure that it is divisible by 32. The padding part is ignored when calculating the evaluation metrics.

[0117] (III) Hyperparameter Setting

[0118] In this embodiment, the Adam optimizer and the cosine annealing (CosineAnnealingLR) learning rate adjustment algorithm are used to train 100 epochs for the above three data sets, and a simple cross entropy loss function is used to guide model optimization. In addition, in order to ensure the fairness of the comparison, no pre-trained model is used in all comparative experiments.

[0119] (IV) Accuracy evaluation indicators

[0120] In this embodiment, four indicators, namely, intersection over union (IoU), precision, recall and F1 score, are used to evaluate the accuracy of the model.

[0121] IoU measures the ratio of the intersection and union between the model's predicted segmentation mask and the true segmentation mask. The calculation formula is shown in formula (1):

[0122]

[0123] In formula (1), IntersectionArea is the intersection area of ​​the target actual area and the predicted area; Union Area is the union area of ​​the target actual area and the predicted area; the higher the IoU value, the higher the model accuracy.

[0124] Precision indicates how many of the samples classified as positive examples are true positive examples. The calculation formula is shown in formula (2):

[0125]

[0126] In formula (2), TP represents the number of positive samples correctly predicted by the model as positive samples; FP represents the number of negative samples incorrectly predicted by the model as positive samples.

[0127] Recall indicates how many samples of all actual positive categories are correctly predicted as positive by the model. The calculation formula is shown in formula (3):

[0128]

[0129] In formula (3), TN represents the number of negative samples correctly predicted by the model as negative samples; FN represents the number of positive samples incorrectly predicted by the model as negative samples.

[0130] The F1 score is the harmonic mean of Precision and Recall, and is an indicator that comprehensively considers both precision and recall. The calculation formula is shown in formula (4):

[0131]

[0132] (V) Analysis of experimental results

[0133] In this embodiment, comparative experiments are conducted on three datasets: the WHU dataset, the Chinese typical urban building instance dataset, and the MassachusettsBuildings dataset, with nine classic models and recent SOTA models, including UNet, TransUNe, DeepLabV3, BANe, Buildformer, C3Net, STTNe, CFENet, and UANet.

[0134] exist Figure 7 Comparison of WHU dataset segmentation results, Figure 8 Comparison of segmentation results of typical urban building datasets in China and Fig. 9 In the comparison chart of the segmentation results of the Massachusetts Buildings dataset, several models are selected to show the image comparison of the building extraction results. The white part represents the building, the black part represents the background, the red part represents the part where the model mistakenly judges other objects as buildings, and the green part represents the part that the model misses.

[0135] The experimental results are shown in the following table:

[0136] Table 1. Index scores of the WHU dataset in each model

[0137]

[0138]

[0139] Table 2. Index scores of typical urban building example datasets in China in various models

[0140]

[0141] Table 3. Indicator scores of the Massachusetts Buildings dataset in each model

[0142]

[0143] It can be seen that for the WHU dataset, the building feature extraction model (Ours) proposed in this embodiment has the highest precision, F1, and IOU. Figure 7 It can be seen that our model performs significantly better than other models in maintaining the integrity of buildings, which is partly due to the enhanced ability to extract local context information brought by multi-scale wavelet transform convolution.

[0144] For the typical urban building dataset in China, the building feature extraction model is higher than other models in terms of F1 and IOU (see Table 2). It is worth mentioning that although CFENet has the highest precision, its recall value is very low, which means that it is easy to miss points. Figure 4 Relatively speaking, our building feature extraction model takes into account the excellent performance of both Precision and Recall, and effectively copes with the phenomenon of misclassification and omission.

[0145] Our model also performs well on the Massachusetts Buildings dataset. Figure 5 As can be seen from the figure, our model has significantly fewer misclassifications (red part) than other models, and also achieves the best performance in Precision, F1, and IOU (Table 3).

[0146] On the other hand, in order to verify the performance improvement brought by different modules, this embodiment further conducts an ablation experiment on the building feature extraction model proposed by us on the Massachusetts Buildings dataset. The experimental results are shown in Table 4:

[0147] Table 4. Ablation experiment results

[0148]

[0149] It can be seen that the multi-scale wavelet transform convolution module (MSWTConv), the spatial and channel attention synergy mechanism module (GCASA) and the feature fusion sampling module (FFU) of the building feature extraction model all have good scores in the four indicators.

[0150] On the other hand, in order to better reflect the relationship between the performance and complexity of different models, this embodiment also visualizes the IOU index on the WHU dataset through the Params function. Fig.10 We can make an intuitive comparison with the SOTA (State-Of-The-Art) methods of other current models. As can be seen from the figure, when the IOU value reaches the highest, the number of parameters of our proposed model is relatively low.

[0151] In addition, as an implementation scheme, the present application also proposes a building feature extraction model, including a multi-scale wavelet transform convolutional attention network, wherein the multi-scale wavelet transform convolutional attention network includes:

[0152] An encoder, comprising a multi-scale wavelet transform convolution module for feature learning of high-frequency information and low-frequency information in the remote sensing image of the building, and a spatial and channel attention synergy mechanism module arranged at the end of the encoder for enhancing the extraction capability of the model's global information;

[0153] The decoder includes a feature fusion sampling module for weighted fusion of feature maps of corresponding sizes in the encoder and upsampling layer by layer to restore the resolution.

[0154] In addition, it can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the building remote sensing image recognition system to implement the process steps of the embodiment of the above method.

[0155] Therefore, the present application also provides a computer-readable storage medium, which stores a feature extraction program for building remote sensing images. When the feature extraction program for building remote sensing images is executed by a processor, the various steps of the feature extraction method for building remote sensing images described in the above embodiment are implemented.

[0156] The computer-readable storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which may store program codes.

[0157] It should be noted that since the storage medium provided in the embodiment of the present application is the storage medium used to implement the method of the embodiment of the present application, based on the method introduced in the embodiment of the present application, the person skilled in the art can understand the specific structure and deformation of the storage medium, so it is not repeated here. All storage media used in the method of the embodiment of the present application belong to the scope of protection of this application.

[0158] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0159] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0160] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0161] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0162] It should be noted that in the claims, any reference signs placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claim. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The present application may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0163] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0164] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A feature extraction method for building remote sensing images, characterized in that: The method comprises the following steps: Obtain remote sensing images of buildings collected by drones; Inputting the building remote sensing image into a pre-trained model including a multi-scale wavelet transform convolutional attention network, wherein the multi-scale wavelet transform convolutional attention network includes an encoder and a decoder; The encoder includes a multi-scale wavelet transform convolution module for feature learning of high-frequency information and low-frequency information in the building remote sensing image, and a spatial and channel attention synergy mechanism module arranged at the end of the encoder for enhancing the extraction capability of the model's global information; Wherein, the decoder comprises a feature fusion sampling module for weighted fusion of feature maps of corresponding sizes in the encoder and upsampling layer by layer to restore the resolution; Obtain the building image remote sensing recognition result output by the pre-trained model.

2. The method according to claim 1, characterized in that The multi-scale wavelet transform convolutional attention network processes the building remote sensing image in the following steps: Based on a multi-scale wavelet transform convolution module, the building remote sensing image is decomposed by wavelet to obtain a low-frequency feature map and a high-frequency feature map, and the low-frequency feature map, the high-frequency feature map and the original feature map are subjected to multi-scale convolution; The low-frequency feature map, the high-frequency feature map and the original feature map after multi-scale convolution are respectively averaged and fused to obtain a multi-scale convolution fusion feature map; and the low-frequency feature map and the high-frequency feature map are scaled, inversely transformed and restored to the original resolution and spliced ​​together to obtain a spliced ​​feature map; The multi-scale convolution fusion feature map and the splicing feature map are fused and output.

3. The method according to claim 2, characterized in that The step of performing multi-scale convolution on the low-frequency feature map, the high-frequency feature map and the original feature map comprises: Convolution kernels of sizes 3*3, 4*4, and 7*7 are used respectively to perform wavelet transform convolution on the low-frequency feature map, the high-frequency feature map, and the original feature map.

4. The method according to claim 1, characterized in that The pre-trained model includes a global channel and a spatial attention module, wherein the global channel and the spatial attention module includes a self-attention-channel attention module and a spatial attention mechanism module, and the global channel and the spatial attention module performs the following steps: Obtaining the channel global attention weight of the deepest feature map through the self-attention-channel attention module, and obtaining the spatial global attention weight of the deepest feature map through the spatial attention mechanism module; Multiply the channel global attention weight by the input feature map to obtain a channel attention weight feature map; And, multiplying the spatial global attention weight by the input feature map to obtain a spatial attention weight feature map; The channel attention weight feature map and the spatial attention weight feature map are multiplied to fuse and enhance the features extracted by the wavelet transform convolution in the encoder.

5. The method according to claim 4, characterized in that The self-attention-channel attention module uses global average pooling to reduce the deepest feature map to 1x1 and retain the channel information, and then obtains the Querry value, Key value and Value value corresponding to the deepest feature map through the convolution layer, and performs self-attention mechanism operation to obtain the channel global attention weight.

6. The method according to claim 4, characterized in that The spatial attention mechanism module performs maximum pooling and average pooling on the deepest feature map on the channel to reduce the channel to 1 and retain the spatial position information, and then adds and fuses the feature map after maximum pooling and the feature map after average pooling at the channel level to obtain the average eigenvalue and maximum eigenvalue of each position, and inputs the average eigenvalue and the maximum eigenvalue into the group convolution module for feature learning, and finally obtains the spatial global attention weight through the activation function.

7. The method according to claim 1, characterized in that The decoder performs the following steps: Upsampling the feature map output by the encoder to transform the size to the same scale as the feature map of the corresponding skip connection; Performing weighted fusion on the upsampled feature map in the decoder and the feature map of the encoder for skip connection of the same scale, and performing feature enhancement on the fused feature map using depthwise separable convolution; The feature map after depth-wise separable convolution is output as the input for the next upsampling, and the feature map after upsampling is used as the feature fusion of the output layer.

8. A building feature extraction model, characterized in that: Comprising a multi-scale wavelet transform convolutional attention network, the multi-scale wavelet transform convolutional attention network comprising: An encoder, comprising a multi-scale wavelet transform convolution module for feature learning of high-frequency information and low-frequency information in the remote sensing image of the building, and a spatial and channel attention synergy mechanism module arranged at the end of the encoder for enhancing the extraction capability of the model's global information; The decoder includes a feature fusion sampling module for weighted fusion of feature maps of corresponding sizes in the encoder and upsampling layer by layer to restore the resolution.

9. A building remote sensing image recognition system, characterized in that: The building remote sensing image recognition system includes: a memory, a processor, and a feature extraction program for building remote sensing images stored in the memory and executable on the processor. When the feature extraction program for building remote sensing images is executed by the processor, the steps of the feature extraction method for building remote sensing images as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a feature extraction program for building remote sensing images, and when the feature extraction program for building remote sensing images is executed by a processor, the steps of the feature extraction method for building remote sensing images according to any one of claims 1 to 7 are implemented.

Images