Building change detection method based on resnet34 remote sensing image

Abstract

The present application relates to a kind of ResNet34-based remote sensing image building change detection method, belong to remote sensing image information change detection technical field.The method is: the training dataset is preprocessed and data is enhanced;Input ResNet34 encoder, obtain the feature map of different scales of each stage of two images;The image before change and the image after change are input into GFSN module, and building-related texture features are extracted in multiple scales and directions;The features extracted in GFSN module are fused with layered encoder, and integrated into Siamese network composed of 4 pairs of multistage down-sampling-transform block;The feature weight of the multi-layer feature after fusion is redistributed through ratio attention module, and the image difference is obtained through difference module simultaneously;The final prediction map is obtained by layer-by-layer fusion of decoder upsampling operation.This application solves the problems of building boundary blur and void in detection results in building change detection.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of remote sensing image information change detection, and relates to a remote sensing image building change detection method based on ResNet34. BACKGROUND

[0002] The purpose of change detection is to detect the changes of remote sensing images in different stages in the same area, and the change detection of remote sensing images is realized by using the binarization method. Remote sensing image change detection can be divided into land change detection, building change detection, water body change detection and the like. The remote sensing images of the same area at different times are affected by factors such as illumination, contrast, imaging angle, and the complex scene relationship and background complexity of the remote sensing images themselves, which leads to limited detection accuracy of building changes.

[0003] From the early stage of remote sensing data sets, the pixel-based data collection method has been widely studied. These methods usually take pixels as the basic unit of image processing and are widely used in low-resolution remote sensing images. However, with the continuous development of earth observation technology, large-scale high-resolution multispectral images have posed challenges to previous traditional research methods. The main reason is that with the continuous improvement of spatial resolution, the increase of internal variability of ground objects and the complexity of ground object boundaries have become major obstacles. With the rapid development of convolutional neural networks in deep learning in recent years, these obstacles have been greatly overcome. Through convolutional neural networks, image features are extracted, and the similarity of these features is calculated to obtain a change map. The current popular convolutional neural networks mainly include end-to-end fully convolutional neural networks such as fully convolutional networks, U-Net and its variants. Inspired by DenseNet and NestedNet, Siamese Nested U-Net for Change Detection (SNUNet) uses dense skip connections between the encoder and the decoder, thereby realizing high-resolution and fine-grained representation. After that, in order to better utilize the scene information, researchers began to add attention mechanisms to the change detection network, such as channel attention, spatial attention, self-attention and the like.

[0004] In addition to the above research on network structure optimization, some scholars have also studied semi-supervised and weakly supervised change detection methods. Peng et al. designed a CD semi-supervised convolutional network based on generative adversarial networks (GANs). Some scholars have studied unsupervised change detection, such as pre-training convolutional neural networks to extract features from multi-temporal images, unsupervised deep learning frameworks to extract difference representations of the same or different source image pairs, or using the results of traditional methods to train deep neural networks, so that the entire change detection process does not require human intervention.

[0005] Although the above model has made progress in building change detection in high-resolution remote sensing images, there are still problems to be solved, such as poor extraction of building change edge information, building boundary blurring in the results, weak global context information retention capability, and occasional holes in the extraction results. SUMMARY

[0006] Therefore, the present application aims to provide a ResNet34-based remote sensing image building change detection method to solve the building boundary blurring problem in complex urban scenes and the hole problem in the extraction results of the existing building change detection method.

[0007] To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0008] The ResNet34-based remote sensing image building change detection method comprises the following steps:

[0009] S1: Divide the obtained remote sensing building data set image into a training set, a validation set and a test set, crop the training set image in a sliding window manner, process the labeled image accordingly, and then perform data augmentation on the training set image;

[0010] S2: Input the data augmented training set image in S1 into an extraction model with a residual network, the extraction model extracts features from two remote sensing images at different times, and obtains first stage to fourth stage, four stage feature maps of different scales;

[0011] S3: Introduce a multi-scale Gabor filter in the S2 layered encoder, introduce a pair of Gabor filters with different scales and kernels in each stage, and obtain 4 pairs of Gabor feature maps with different sizes and resolutions;

[0012] S4: The feature extraction module DAM aims to combine the down-sampled Gabor feature map extracted by GFSN in S3 with the features of different layers in the ResNet encoder, here a pair of Gabor features are selected for mapping, here we connect the low-frequency Gabor features that capture larger patterns in the image in GFSN with the smaller resolution ResNet features, in order to retain meaningful information after the down-sampling operation.

[0013] S5: The fusion features of multiple stages obtained in S4, the feature maps of four stages with different scales, are obtained by upsampling and ratio attention modules to obtain more refined results. The result feature map reduces useless information and highlights the features of the changing part. The weight distribution of the feature maps of different layers is re-performed to obtain a feature map with rich semantic information and spatial information at different resolutions of each stage. Finally, the difference value features of different layers of the two images are calculated to obtain the difference value features.

[0014] S6: The difference value features of the first stage to the fourth stage obtained in S5 are fused by the decoder to obtain a final prediction map, and the final prediction map is taken as the final building extraction result.

[0015] S7: The prediction data set is put into the model trained by the training set in S6 for testing, and finally the relevant evaluation indicators of the entire model are obtained.

[0016] Further, in S1, the labeled image is processed, specifically: the pixels of the non-building area in the label image are set to 0, and the pixels of the building area are set to 1.

[0017] Further, in S1, the data enhancement specifically includes: random size scaling, random horizontal flipping, vertical flipping, random Gaussian noise, limited contrast adaptive histogram equalization, and random cropping of the image.

[0018] Further, the random size scaling uses a scaling ratio of 0.5-2.0, the random horizontal flipping, vertical flipping, random Gaussian noise, and limited contrast adaptive histogram equalization use a probability of 0.5 to perform data enhancement on the image, and the random cropping has a cropping size of 256x256.

[0019] Further, in S2, the deep residual neural network extraction model has four stages, each stage contains a plurality of residual blocks, wherein the first stage contains 3, the second stage contains 4, the third stage contains 6, and the fourth stage contains 3. These residual blocks are connected to each other through skip connection, allowing the network to directly learn residual mapping.

[0020] Further, in S2, the Gabor filter has the ability to mimic the receptive field of the mammalian visual cortex and is widely used in image analysis and texture feature extraction, especially for capturing repeated visual patterns in images. We believe that this ability is suitable for our building change detection task, because we need to perceive the repeated texture of a cluster of buildings in an area.

[0021] Therefore, in each stage, the Gabor filter feature extraction network (GFSN) is introduced, each stage includes a pair of different scale size filters, and the pre-change and post-change input images are convolved with multiple filters to generate a set of feature maps of different frequencies and directions. The size and direction scale of each layer of filter should be considered in order to capture the repeated texture generated by the building well. Here, the direction scale is set to 4, which can cover the horizontal, vertical and diagonal directions. Through experiments, it is found that when the size is 1, the smaller building can be clearly captured, when the frequency increases, the filter will respond to larger objects, and when the size is greater than 6, the large building will also be very blurred and cannot extract any meaningful texture of the building, so the filter size is set to 6 at most.

[0022] Further, in S2 and S3, different feature maps are obtained through the filter network and the ResNet encoder, and through the feature fusion module (DAM), a pair of Gabor feature maps are selected to be fused with the ResNet features of different layers. When the fusion is performed, the low-frequency features of the larger modules captured in the Gabor are fused with the smaller resolution ResNet features, so as to retain the meaningful information after the downsampling operation. For example, when the filter size is 6, only the larger building can be captured, and the small building is ignored. At this time, it is fused with the smallest spatial resolution H / 32xW / 32 in the ResNet feature, so as to retain more complete information.

[0023] Further, in S5, the fused multi-layer features are re-allocated feature map weights through the ratio attention module, reducing the attention of useless information and making the module pay more attention to useful information.

[0024] The ratio attention module processes the data of two features respectively to obtain a ratio data in the range of (0, 1). A large weight is added to the data with a small ratio. T(x) = 1-ratio is proposed. The smaller the ratio is, the more important the result of multiplying the original data by 1 minus the ratio is. The same ratio is larger, and 1-ratio is used to make the model pay less attention to these contents.

[0025] The ratio operation on the input two image images is to divide two matrices to obtain a feature map with a value range of 0-1. Two branches are made on the feature map. The left branch is through the channel attention mechanism, which allocates importance between different features according to the characteristics of different tasks. The right branch is processed through the spatial attention mechanism, and the most important part of the network is processed. Finally, the attention of the layer is obtained as follows:

[0026] T i =Div1 i +Ti (1)

[0027] T i =Div2 i +T i (2)

[0028] Diff i =|X i -Y i |(3)

[0029] In the formula: T i x and y in the input image, i represents the current layer number, Div1 i , Div2 i respectively represent the attention weights obtained by the current i layer of the two feature maps; Diff i represent the difference value of the i layer of the two feature maps; X i , Y i respectively represent the feature maps obtained after formula (1) and formula (2).

[0030] Further, the S4 is specifically: the multi-scale feature maps fused and refined in the four stages of the first stage to the fourth stage output by the decoder in S3 are respectively used for dense upsampling to the same size as the feature map of the first stage, and the shape is all 64x64x256;Then, the four feature maps are spliced in the channel dimension using the cascade operation, and the spliced shape is 64x64x1024, then the spliced feature map is fused using 3x3 convolution, and a feature map of 64x64x256 is obtained;The fused result is classified using 1x1 convolution and up-sampled to the original image size, and finally a prediction map is obtained.

[0031] The beneficial effects of the present application are:

[0032] First, the high-resolution optical remote sensing image building change detection framework based on multi-scale fusion proposed by the present application can perform multi-scale context feature extraction, fully capture the local and long-range dependency relationships between features in the image, and improve the extraction capability of large buildings.

[0033] Second, the present application introduces a traditional Gabor filter into the convolution network to generate a filter network, which helps the module to better capture the repeated texture of building clusters and solves the problem of boundary blur that may occur in building change detection.

[0034] Thirdly, the application proposes a feature fusion module (DAM) to fuse the features of the four stage layers of the traditional residual network ResNet34 and the features of the Gabor filter network in high and low frequencies, thereby retaining the global context information and to a certain extent solving the problem of holes in building change detection.

[0035] Fourthly, the application proposes to combine the ratio method in the traditional method into the channel attention and spatial attention to form a new attention mechanism, so that the model pays more attention to the features of the change part, and further solves the problem of boundary blur in building change detection.

[0036] Other advantages, objects, and features of the application will be set forth in part in the following specification, and in part will become apparent to those skilled in the art from a consideration of the following specification, or can be learned from practice of the application. The objects and other advantages of the application can be realized and attained by the methods and instrumentalities particularly pointed out in the specification. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to make the objects, technical solutions and advantages of the application clearer, the preferred detailed description of the application will be combined with the drawings to describe the application, in which:

[0038] Figure 1 The specific step flowchart of the application is shown in the figure;

[0039] Figure 2 The model structure in the application is shown in the figure;

[0040] Figure 3 The diff attention module is shown in the figure;

[0041] Figure 4 The building extraction effect diagram is shown in the figure. DETAILED DESCRIPTION

[0042] The embodiments of the application are described below through specific examples, and those skilled in the art can easily understand other advantages and effects of the application from the disclosure of the specification. The application can also be implemented or applied in different specific embodiments, and the details in the specification can be modified or changed based on different views and applications without departing from the spirit of the application. It should be noted that the diagrams provided in the following examples only illustrate the basic concept of the application in a schematic manner, and the following examples and features in the examples can be combined with each other without conflict.

[0043] Among them, the drawings are only used for illustrative description, and the representation is only a schematic diagram, not a physical diagram, and cannot be understood as a limitation on the present application; in order to better illustrate the embodiments of the present application, some components of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0044] The same or similar reference numerals in the drawings of the embodiments of the present application correspond to the same or similar components; in the description of the present application, it should be understood that if the terms "upper", "lower", "left", "right", "front", "back" and the like indicate the orientation or positional relationship shown in the drawings, only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore the positional relationship described in the drawings is only used for illustrative description, and cannot be understood as a limitation on the present application, for those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0045] Please refer to Figures 1-4 Taking the building change detection of the LEVIR-CD dataset as an example, the remote sensing image building change detection method based on ResNet34 is as follows:

[0046] S1: The acquired remote sensing building dataset image is cropped in a sliding window manner, and the labeled image is processed, and the dataset is divided into a training set, a validation set and a test set, and the training set image is subjected to data enhancement;

[0047] Specifically, the pixels of the non-building area in the label image of the WHU and LEVIR-CD dataset are set to 0, and the pixels of the building area are set to 1. The dataset is divided into three parts, there are 137 images in the training set, 4 images in the validation set, and 10 images in the test set, and the original image is cropped into a picture with a size of 256x256 with an overlap ratio of 0.5%.

[0048] In the training process, the training set is subjected to data enhancement, including horizontal flipping, vertical flipping, random size scaling, random Gaussian noise, limited contrast adaptive histogram equalization and random cropping methods. The random size scaling uses a scaling ratio of 0.5-2.0 by default. The horizontal flipping, vertical flipping, random Gaussian noise and limited contrast adaptive histogram equalization method uses a probability of 0.5 to enhance the data of the image. Finally, random cropping is performed, and the default cropping size is 256x256.

[0049] All the methods are combined together to enhance the original image.

[0050] S2: input the training set data into the ResNet34 encoder to obtain the first stage to the fourth stage, and the feature maps of the four stages are of different scales.

[0051] S3: introduce a Gabor filter feature extraction network (GFSN) in each stage of the hierarchical encoder in S2, each stage includes a pair of filters of different scale sizes, and the input image before and after the change is convolved with a plurality of filters to generate a set of feature maps of different frequencies and directions. Here, the size and direction scale of each layer of filter are considered to capture the repeated texture generated by the building well. Here, the direction scale is set to 4, which can cover the horizontal, vertical and diagonal directions. The filter size is obtained by experiment, and when the size is 1, the smaller building can be clearly captured. When the frequency increases, the filter will respond to larger objects. When the size is greater than 6, large buildings will also be very blurred and cannot extract any meaningful texture of the building, so the filter size is set to a maximum of 6.

[0052] S4: in S2 and S3, different feature maps are obtained through the filter network and the ResNet encoder, and through a feature fusion module (DAM), a pair of Gabor feature maps are selected to be fused with the ResNet features of different layers. When the fusion is performed, we choose to fuse the low-frequency features of the larger modules captured in Gabor with the smaller resolution ResNet features to retain meaningful information after the downsampling operation. For example, when the filter size is 6, only larger buildings can be captured, and small buildings are ignored. At this time, it is fused with the smallest spatial resolution H / 32x W / 32 in the ResNet feature, so as to retain more complete information.

[0053] S5: in S5, the fused multi-layer features are re-allocated feature map weights through the ratio attention module to reduce the attention of useless information and make the module pay more attention to useful information.

[0054] The ratio attention module mainly performs ratio processing on the data of two features, and obtains a ratio data range of (0, 1). We know that the smaller the ratio, the greater the change may be, so we need to add a large proportion to the data with a small ratio, so that the model can pay attention to this part of the content. Based on this idea, T(x) = 1-ratio, that is, the smaller the ratio, the more important the result of multiplying the original data by 1 minus the ratio, and the greater the same ratio, the smaller the change, and we also use 1-ratio to make the model pay less attention to these contents.

[0055] Here we do the ratio operation of the input two image images, that is, we get a feature map with a value range of 0-1 by dividing two matrices and taking the absolute value. We make two branches for this feature map, the left branch mainly passes through the channel attention mechanism, and the importance between different features is taken. According to the characteristics, it is allocated to different tasks, and the right branch passes through the spatial attention mechanism, and the most important part in the network is processed. Finally, the attention obtained in this layer is represented as.

[0056] T i =Div1 i +T i (1)

[0057] T i =Div2 i +T i (2)

[0058] Diff i =|X i -Y i |(3)

[0059] In the formula, T i x and y in the input image; i represents the current layer; Div1 i , Div2 i respectively represent the attention weights obtained by the current i layer of the two feature maps; Diff i represents the difference value of the i layer of the two feature images; X i , Y i respectively represent the feature maps obtained after formula (1) and formula (2);

[0060] S6: The first stage to the fourth stage of the decoder output in S5 is fused, and the result after processing the fused feature map of each stage is obtained. A final prediction map is obtained, and the final prediction map is taken as the final building extraction result.

[0061] Specifically, the multi-scale feature maps of the first stage to the fourth stage of the decoder output in S5 are fused and refined, and the multi-scale feature maps are respectively used for dense upsampling to the same size as the feature map of the first stage. The shape is 64x64x256.

[0062] Then, the four feature maps are spliced in the channel dimension using the cascade operation, and the spliced shape is 64x64x1024. Then, 3x3 convolution is used to fuse the spliced feature map, and a feature map of 64x64x256 is obtained.

[0063] The fused result is classified using 1x1 convolution and up-sampled to the original image size, and finally a prediction map is obtained.

[0064] S7: input the test data image into the trained model for testing, and select IOU, OA, F1-Score, Precision, and recall as evaluation indexes.

[0065] Specifically, the evaluation index OA is the global accuracy rate, and the equation is: (TP+TN) / (TP+TN+FP+FN);

[0066] The evaluation index Precision is the precision rate, and the equation is: (TP) / (TP+FP);

[0067] The evaluation index Recall is the recall rate, and the equation is: (TP) / (TP+FN);

[0068] The evaluation index F1-Score is the harmonic mean of the precision rate and the recall rate, and the equation is: (2×Presicion×Recall) / (Precision+Recall);

[0069] The evaluation index Iou is the intersection over union, and the equation is: (TP) / (TP+FP+FN);

[0070] Wherein, TP, TN, FP, and FN in the equation respectively represent:

[0071] TP is the number of samples correctly predicted as positive class;

[0072] TN is the number of samples correctly predicted as negative class;

[0073] FP is the number of samples incorrectly predicted as positive class;

[0074] FN is the number of samples incorrectly predicted as negative class.

[0075] Test using test set data, and select a picture to export the final building extraction result, as shown in Figure 4

[0076] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present application can be modified or replaced equivalently without departing from the purpose and scope of the technical solutions, which should be covered in the scope of the claims of the present application.​

Claims

1. A method for detecting building changes in remote sensing images based on ResNet34, characterized in that: The method includes the following steps: S1: Divide the acquired remote sensing building dataset images into training set, validation set and test set. Crop the training set images using a sliding window method, process the labeled images accordingly, and then perform data augmentation on the training set images. S2: The data-enhanced training set images from S1 are input into a hierarchical encoder extraction model with a residual network. The hierarchical encoder extracts features from two remote sensing images taken at different times, obtaining feature maps at four different scales from the first to the fourth stage. The hierarchical encoder is built based on ResNet34, and each stage contains several residual blocks. The residual maps are learned through skip connections. The hierarchical encoder extraction model with a residual network is a four-stage model, each containing several residual blocks. The first stage contains 3, the second stage contains 4, the third stage contains 6, and the fourth stage contains 3. These residual blocks are interconnected through skip connections, allowing the network to directly learn the residual maps. A Gabor filter feature extraction network GFSN is introduced in each stage. Each stage includes a pair of filters of different scales. The input images before and after the change are convolved with multiple filters to generate a set of feature maps with different frequencies and orientations. The size and orientation scale of each filter layer are taken into account to capture the repetitive textures generated by buildings. The orientation scale is set to 4 to cover the horizontal, vertical, and diagonal directions. The maximum filter size is set to 6. S3: In the hierarchical encoder of S2, a multi-scale Gabor filter feature extraction network GFSN is introduced. Each stage includes a pair of Gabor filters of different scales. The two input images are convolved with the filters to generate feature maps of different frequencies and directions. The direction scale is set to 4, and the maximum filter size is 6. S4: The feature fusion module DAM combines the downsampled Gabor feature maps extracted by GFSN in S3 with features from different layers in the ResNet encoder. It selects a pair of Gabor features for mapping, connecting the low-frequency Gabor features of larger patterns in the captured image in GFSN with the ResNet features of smaller resolution, so as to retain meaningful information after the downsampling operation. S5: The fused feature maps of the four stages obtained in S4 are processed by upsampling and ratio attention module, the feature map weights are redistributed, the features of the changing parts are highlighted, and finally the difference between the features of different layers of the two input images is calculated to obtain the difference features. S6: The difference features from the first to fourth stages obtained in S5 are fused and upsampled through the decoder to obtain a final prediction map, and the final prediction map is used as the final building extraction result; S7: Input the test set images into the trained model for testing and obtain evaluation metrics, including global accuracy (OA), precision (Precision), recall (Recall), F1-Score, and intersection-union ratio (Iou). In S2 and S3, the high and low frequency features of the filter network and encoder are fused by the feature fusion module to enhance the contextual semantic information and solve the problem of voids inside the building. Different feature maps were obtained through the filter network and ResNet encoder. The feature fusion module DAM selected to fuse a pair of Gabor feature maps with features from different layers of ResNet. During the fusion, the low-frequency features of the larger module of the captured image in Gabor were selected to be fused with the features of the smaller resolution ResNet to retain the information after the downsampling operation. In S5, the ratio attention module performs a ratio operation on the features of two input images, i.e., remote sensing images at different times, and calculates the feature map obtained through the ratio operation, i.e., the absolute value of the division of two matrices, with a value range of 0 to 1; then the weights are redistributed through channel attention and spatial attention branches. Channel attention branch: (1) Spatial attention branch: (2) in Div 1 i and Div2 i The first i Layer channel and spatial attention weights, Indicates the feature difference; T i Indicates the first i Feature map of the layer; i Indicates the current floor level; X i , Y i These represent the feature maps obtained using formulas (1) and (2), respectively. Finally, the attention results are integrated to enhance the change features.

2. The method for detecting building changes in remote sensing images based on ResNet34 according to claim 1, characterized in that: In step S1, the labeled image is processed by setting the pixels in the non-building areas of the label image to 0 and the pixels in the building areas to 1.

3. The method for detecting building changes in remote sensing images based on ResNet34 according to claim 1, characterized in that: In S1, data augmentation specifically includes: random scaling of the image, random horizontal flipping, vertical flipping, random Gaussian noise, contrast-limited adaptive histogram equalization, and random cropping. The random scaling uses a scaling ratio of 0.5-2.

0. The random horizontal flip, vertical flip, random Gaussian noise, and contrast-limited adaptive histogram equalization are applied to the image with a probability of 0.5 for data enhancement. The random cropping size is 256×256.

4. The method for detecting building changes in remote sensing images based on ResNet34 according to claim 1, characterized in that: Specifically, S6 involves: taking the first to fourth stages output by the decoder in S3, and the multi-scale feature maps after fusion and refinement of the four stages, and performing dense upsampling to the same size as the feature map of the first stage, with a shape of 64×64×256; then using a concatenation operation to stitch the four feature maps together along the channel dimension, resulting in a shape of 64×64×1024; next, using a 3×3 convolution to fuse the stitched feature maps, resulting in a 64×64×256 feature map; finally, using a 1×1 convolution to classify the fused result and upsampling it to the original image size to obtain the prediction map.

Images