An automatic extraction method of out-of-range building patches based on high-view satellite images

Abstract

The present invention relates to the technical field related to remote sensing applications, and discloses a method for automatically extracting over-range building patches based on high-view satellite images, including: remote sensing image preprocessing; sample data set preparation; construction of a convolutional neural network; training a network model using samples; and model inference prediction. The present invention mainly automatically extracts building land through an improved U-Net model, uses resnet50 as the encoder part of the U-Net model, and uses its powerful feature extraction capability to better capture feature information of different scales. The CBAM hybrid attention mechanism is introduced so that the model can adaptively adjust the importance of different channels and spatial positions in the feature map, so as to better focus on the key areas of the building target. At the same time, the FPN feature pyramid structure is added to the model, and by fusing multi-scale feature information at different levels, the model can better obtain the contextual information of the building target, thereby enhancing the robustness and accuracy of building land extraction.

Description

Technical Field

[0001] The present invention relates to the technical field related to remote sensing applications, and in particular to a method for automatically extracting out-of-range building patches based on high-view satellite images. Background Art

[0002] With the rapid development of urbanization, various types of land use have been converted to construction land, and dynamic monitoring of construction land is needed to better provide fast and specific information on changes in construction land after approval for land use control, and optimize and improve the monitoring and evaluation system. The new generation of sensors can obtain higher-resolution images, among which the spatial resolution of high-view satellite images can reach 0.5m, which greatly enhances the ground feature information and positioning information, and can clearly display the spatial distribution, boundary information and construction status of out-of-range construction areas. However, the pixels of a single image also increase exponentially, and the structure of construction land is relatively complex, the spectral characteristics are diverse, and there is a large amount of interference information and similar ground feature information. It is difficult to extract boundary information of different categories of construction land from high-resolution images.

[0003] Many scholars have conducted a lot of research on the extraction of building land information, which can be summarized into three types according to the extraction principle: First, based on the geometric characteristics of the building land itself, the external contour of the building land is identified by relying on edge information and corner information to further extract the construction land. Second, with the help of auxiliary information, the building land is extracted by combining the auxiliary information such as position and elevation obtained from laser radar data, synthetic aperture radar, digital surface model and other data. Third, based on the segmentation method, the image is segmented into each unit, and information is extracted according to the characteristics within the unit. However, the above has the following shortcomings: First, the traditional method relies on the potential features of the object itself, such as texture, shape and shadow, and has weak expression ability for building land information. It is impossible to extract building land based on high-level semantic features such as spatial context information, and there are omissions and errors. Second, the segmentation effect can be enhanced to a certain extent with the help of auxiliary information, but there are certain limitations due to the high cost of obtaining auxiliary information, and the extraction of building land is easily limited by the accuracy of auxiliary information. 3. In high spatial resolution images, spatial relationships and intra-category differences increase, the segmentation scale is difficult to determine, and there are problems such as under-segmentation and over-segmentation, resulting in fragmented and continuous extraction of building land. The segmentation parameters must be set according to the characteristics of the specific area, which involves subjective factors and poor generalization ability. Summary of the invention

[0004] The purpose of the present invention is to provide a method for automatically extracting out-of-range building patches based on high-view satellite images, and to extract building information using an improved U-Net model, so as to overcome the above-mentioned defects in the prior art and improve the accuracy of building land extraction.

[0005] The present invention is achieved through the following technical solutions.

[0006] The present invention provides a method for automatically extracting out-of-range building spots based on high-view satellite images, comprising:

[0007] Step S1: remote sensing image preprocessing;

[0008] Step S2: Sample data set preparation;

[0009] Step S3: Construct a convolutional neural network, including:

[0010] S3.1: Improved based on the U-Net network structure, using resnet50 as the encoding part of the U-Net network;

[0011] S3.2: Introduce the CBAM hybrid attention mechanism in the improved U-Net network encoder part;

[0012] S3.3: Introduce the FPN feature pyramid structure in the improved U-Net network decoder;

[0013] Step S4: training the network model using the samples, including:

[0014] S4.1: Data augmentation of manually delineated datasets;

[0015] S4.2: Divide all the augmented data sets into training sets, validation sets, and test sets;

[0016] S4.3: Operating environment settings;

[0017] S4.4: Network parameter settings;

[0018] Step S5: Model inference prediction, including:

[0019] S5.1: Set the optimal weights of the model and input the test set into the network for inference and prediction;

[0020] S5.2: Application of extraction of out-of-range building patches, using the method of expansion, cropping and splicing to extract out-of-range building patches.

[0021] A further technical solution includes image radiation correction, orthorectification, image fusion and light and color uniformity in the remote sensing image preprocessing of step S1.

[0022] As a further technical solution, in the sample data set preparation of step S2, ArcGIS is used to annotate the building targets through the Gaojing-1 satellite image, and the original image and label are cropped respectively using 512×512 pixels to generate a remote sensing sample data set in tiff format.

[0023] According to a further technical solution, the step S3.1 in the step S3 specifically includes:

[0024] Replace the feature layers corresponding to the U-Net network encoder with the first layer output feature map Efeat1 of stage0 of the resnet50 network, the output feature map Efeat2 of stage1, the output feature map Efeat3 of stage2, the output feature map Efeat4 of stage3, and the output feature map Efeat5 of stage4.

[0025] According to a further technical solution, the step S3.2 in the step S3 specifically includes:

[0026] The CBAM hybrid attention mechanism is added after the feature maps Efeat1 and Efeat5 respectively to make the network focus more on the key areas.

[0027] According to a further technical solution, the step S3.3 in the step S3 specifically includes:

[0028] First, the output feature maps Dfeat1, Dfeat2, Dfeat3, and Dfeat4 of each layer of the decoder are fused from top to bottom to obtain the feature map Dfeat5. Then, Dfeat5 is upsampled 8 times and fused with Dfeat4 to obtain Dfeat6. Finally, Dfeat6 is upsampled 2 times and output.

[0029] According to a further technical solution, the data augmentation process in S4.1 mainly includes random rotation, horizontal flipping, vertical flipping and diagonal mirroring operations;

[0030] According to a further technical solution, the data set described in S4.2 is divided in a ratio of 8:1:1.

[0031] A further technical solution is that the operating environment setting described in S4.3 specifically uses the Windows operating system, the Pytorch deep learning framework and the VScode language compiler.

[0032] A further technical solution is that the network parameter setting described in S4.4 specifically uses the Adam optimizer, selects the combination of cross entropy loss and dice loss as the loss function of the network, sets the initial learning rate to 0.0007, and selects the poly learning strategy and ReLu activation function.

[0033] According to a further technical solution, the step S5.1 in the step S5 specifically includes:

[0034] First, the optimal weights of the model are set, and then the test set is input into the network for inference and prediction. In order to evaluate the actual learning ability of the network, the precision (Precision), recall rate (Recall), F1 value (F1-score) and single-class intersection-over-union (Iou) of the test set are calculated.

[0035] According to a further technical solution, the step S5.2 in the step S5 specifically includes:

[0036] In the application process of extracting out-of-range building patches, the remote sensing images of the 500m buffer zone of the project red line are selected. Since the network training size is 512×512, the remote sensing images to be predicted should be cropped to a fixed size and then spliced, and the expansion cropping and splicing method should be used to extract out-of-range building patches to avoid obvious splicing marks in the inference prediction results.

[0037] Beneficial effects of the present invention:

[0038] The present invention is aimed at monitoring of out-of-range building spots in construction projects in land and space use control, to solve the problem of difficult and slow detection in monitoring of out-of-range building spots in construction projects. The surface conditions of out-of-range building spots are complex and changeable, and the texture features of the identified targets on high-resolution remote sensing images are very complex, which puts high demands on the detail extraction and edge detection capabilities of the deep learning model. The present invention uses the improved U-NET deep learning model to accurately identify the location and segment the boundaries of building targets, which has the following benefits:

[0039] (1) Resnet50 is used as the encoder part of the U-Net model, taking advantage of its powerful feature extraction capability to better capture feature information of different scales.

[0040] (2) The introduction of the CBAM hybrid attention mechanism enables the model to adaptively adjust the importance of different channels and spatial positions in the feature map, thereby better focusing on the key areas of the building target.

[0041] (3) The FPN feature pyramid structure is added to the model. By fusing multi-scale feature information at different levels, the model can better obtain the contextual information of building targets, thereby enhancing the robustness and accuracy of building land extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0044] Figure 1 It is a flow chart of a method for automatically extracting out-of-range building patches based on high-view satellite images of the present invention;

[0045] Figure 2 This is a network diagram of introducing resnet50 in the process of building a deep learning model;

[0046] Figure 3 This is a network diagram of the introduction of the CBAM hybrid attention mechanism in the process of building a deep learning model;

[0047] Figure 4 It is a schematic diagram of the improved U-Net network in the present invention;

[0048] Figure 5 It is a schematic diagram of comparison of building spot extraction of Gaojing-1 image in the present invention;

[0049] Figure 6 It is a schematic diagram of the building patch extraction within the 500m buffer zone of the red line of the approved project;

[0050] Figure 7 This is a schematic diagram comparing the effects of expansion, cutting and splicing. DETAILED DESCRIPTION

[0051] Figure 1 The flowchart of the method provided by the present invention is shown in the figure. The present invention is a method for automatically extracting out-of-range building spots based on high-view satellite images. The specific working steps are as follows:

[0052] Step S1: remote sensing image preprocessing;

[0053] Step 1.1: Radiation correction; During the remote sensing imaging process, the sensor itself has a certain absolute error. In addition, the sky light formed by atmospheric scattering and cloud reflection will enter the remote sensing detector together with the radiation energy of the ground target, resulting in distortion of remote sensing radiation, decreased image contrast, and reduced quality of remote sensing images. Radiation correction is used to eliminate errors caused by sensors and atmospheric effects.

[0054] Step 1.2: Orthorectification; During the imaging process, the terrain, camera geometry, and errors caused by the sensor will cause significant geometric distortion and the ground elevation will cause parallax. The extraction of building land information requires high positioning accuracy and contour accuracy. Therefore, orthorectification of the image is performed to eliminate the above effects.

[0055] Step 1.3: Image fusion: In order to obtain multi-channel high-resolution images, the panchromatic sensor image and the multispectral sensor image are fused by image fusion to retain the color and spectral information of the multispectral data while enhancing the details and resolution of the image.

[0056] Step 1.4: Even light and color; During the image acquisition process, due to the imaging time and various environmental factors, there are many differences in color, brightness and other aspects between the images of each flight strip. Therefore, the fused images are evenly processed to make the overall accuracy of each image consistent, the tone is uniform, the texture is clear, the contrast is moderate, and the color transition is natural.

[0057] Step S2: Sample data set preparation: Use ArcGIS software to manually outline the building targets in the Gaojing-1 image, assign the target object attribute to 255, assign other objects to 0, and convert them into a raster image in tiff format. Then, crop the original image and label according to 512×512 pixels to generate a remote sensing building sample data set.

[0058] Step S3: construct a convolutional neural network;

[0059] Step 3.1: Use resnet50 as the encoding part of the U-Net network, and replace the feature layers corresponding to the U-Net network encoder with the first layer output feature map Dfeat1 of stage0 of the resnet50 network, the output feature map Dfeat2 of stage1, the output feature map Dfeat3 of stage2, the output feature map Dfeat4 of stage3, and the output feature map Dfeat5 of stage4. The core idea of ​​the resnet50 network is to design a residual structure, whose residual block consists of three convolutional layers of 1×1, 3×3 and 1×1. The mathematical expression of the residual structure is as follows:

[0060] x i+1 =x i +F(x i ,W i )

[0061] Among them, x i is the input feature, x i+1 is the output feature, x i and x i+1 The number of channels is the same, F(x i ,W i ) is the residual part, W i Represents a convolution operation.

[0062] Step 3.2: Introduce the CBAM hybrid attention mechanism in the improved U-Net network encoder part, and add the CBAM hybrid attention mechanism after the feature map Dfeat1 and the feature map Dfeat5 respectively, so that the network can focus more on the key areas. Figure 3 The figure shows the network diagram of the CBAM hybrid attention mechanism. The mathematical expression of the CBAM hybrid attention mechanism is as follows:

[0063] A c (x i )=σ(W1(W0(x iavg ))+W1(W0(x imax ))

[0064] A s (x i )=σ(W 7×7 (x iavg ⊙x imax ))

[0065]

[0066] Among them, x i is the input feature, x i+1 is the output feature, W0 and W1 represent the convolution operation using ReLu activation function and sigmoid activation function respectively, x iavg and x imax Represents average pooling and maximum pooling, A c (x i ) represents the channel attention function, A s (x i ) represents the spatial attention function, σ is the sigmoid activation function, ⊙ represents the concatenation operation, Represents element-wise multiplication.

[0067] Step 3.3: Introduce the FPN feature pyramid structure in the decoder part of the improved U-Net network. First, fuse the output feature maps Efeat1, Efeat2, Efeat3, and Efeat4 of each layer of the decoder from top to bottom to obtain the feature map Efeat5. Then, upsample Efeat5 by 8 times and fuse it with Dfeat4 to obtain Efeat6. Finally, upsample Efeat6 by 2 times and output it. Figure 4 Shown is a schematic diagram of the improved U-Net network with the FPN feature pyramid structure added.

[0068] Step S4: training the network model using the samples;

[0069] Step 4.1: Perform data augmentation on the manually drawn dataset, and perform random rotation of 0-360°, horizontal flipping, vertical flipping, and diagonal mirroring on the original image and label to increase the richness of the data.

[0070] Step 4.2: Divide all the augmented data sets into training set, validation set and test set in a ratio of 8:1:1.

[0071] Step 4.3: Set up the running environment; select the Windows operating system, Pytorch deep learning framework and VScode language compiler, 64G running memory, and NVIDIA GeForce RTX 3060 graphics card.

[0072] Step 4.4: Network parameter setting; Adam optimizer is selected. Adam absorbs the advantages of adaptive learning rate gradient descent algorithm and momentum gradient descent algorithm, can adapt to sparse gradients, and can alleviate gradient oscillation. The combination of cross entropy loss and dice loss is selected as the hybrid loss function of the network. This type of hybrid function can not only focus on the difference between the output result and the actual result, but also take into account the type imbalance problem in the sample. The initial learning rate is set to 0.0007, the poly learning strategy and ReLu activation function are selected, the batchsize is set to 2, and the epoch is set to 300. The mathematical expressions of Adam optimizer and hybrid loss function are shown below.

[0073] Adam optimizer mathematical expression:

[0074] m t =β1m t-1 +(1-β1)g t

[0075] v t =β2m t-1 +(1-β2)g t 2

[0076]

[0077] Among them, β1 is the exponentially weighted average of the gradient, β2 is the exponentially weighted average of the square of the gradient, and g t represents the parameter gradient, m t is the first-order moment estimate at time t, v t is the second-order moment estimate at time t, and is the deviation-corrected moving average of the corresponding gradient, θ t+1 After updating the parameter, ε is a constant close to 0.

[0078] Mathematical expression of the mixed loss function:

[0079] L cross =-(y*log(p)+(1-y)*log(1-p))

[0080]

[0081] L loss =L cross +L dice

[0082] Among them, y represents the label value of the input network, p represents the predicted probability value, A represents the prediction map, and B represents the label map.

[0083] Step S5: Model inference prediction.

[0084] Step 5.1: When the loss function curve tends to be stable, that is, converged, select the last weight in the convergence stage as the prediction weight, use the test set for inference prediction, and calculate the Precision, Recall, F1-score and Iou of the test set to quantitatively test the actual learning ability of the network. In order to verify the performance of the algorithm proposed in this patent, the classic U-Net and the U-Net based on the resnet50 backbone network (hereinafter referred to as R50_U-Net) are selected for comparative analysis. Figure 5 The results of three algorithms extracting Gaojing No. 1 building spots on the test set are shown. Each row from left to right represents the original image, the label image, the improved U-Net network building spot extraction result of the patent of this invention, the R50_U-Net network building spot extraction result, and the U-Net network building spot extraction result. It can be seen that the method proposed in the patent of this invention is closer to the label image in segmentation results, and the phenomenon of missing questions and wrong questions is significantly reduced. In addition, the outline of the extracted building spots is clear, and the segmentation effect is good.

[0085] Step 5.2: In order to reflect the role of the method proposed in this patent in the monitoring of buildings beyond the scope of construction projects in the land use control, several approved project red lines are selected and a 500m buffer zone is set for image cropping. In order to avoid obvious splicing marks in the inference prediction results, thereby reducing the accuracy of building extraction, this patent uses the expansion cropping and splicing method to perform inference prediction on the 500m buffer zone image of the approved project red line (such as Figure 6 ), and the conventional cropping and splicing method was used for comparison (as shown in Figure 7 shown).

[0086] The specific working steps of expansion cropping and splicing prediction are as follows:

[0087] Step 5.3.1: Get the width and height of the image to be predicted, set the sliding step size to 256, divide the width and height by 256 and take the remainder. If the remainder is less than 256, fill the width and height to 256.

[0088] Step 5.3.1: Set the sliding window to 512 and perform sliding cropping on the predicted image according to the sliding step size.

[0089] Step 5.3.2: Perform inference prediction on the image after sliding cropping to obtain the prediction result map of the corresponding image.

[0090] Step 5.3.3: Take a 256×256 image of the center position of each prediction result image and stitch them together in order.

[0091] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology in this field to understand the content of the present invention and implement it, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for automatically extracting out-of-range building patches based on high-view satellite images, characterized in that: include: Step S1: remote sensing image preprocessing; Step S2: Sample data set preparation; Step S3: Construct a convolutional neural network, including: S3.1: Improved based on the U-Net network structure, using resnet50 as the encoding part of the U-Net network; S3.2: Introduce the CBAM hybrid attention mechanism in the improved U-Net network encoder part; S3.3: Introduce the FPN feature pyramid structure in the improved U-Net network decoder; Step S4: training the network model using the samples, including: S4.1: Data augmentation of manually delineated datasets; S4.2: Divide all the augmented data sets into training sets, validation sets, and test sets; S4.3: Operating environment settings; S4.4: Network parameter settings; Step S5: Model inference prediction, including: S5.1: Set the optimal weights of the model and input the test set into the network for inference and prediction; S5.2: Application of extracting out-of-range building patches, using the expansion, cropping and splicing method to extract out-of-range building patches; In the remote sensing image preprocessing of step S1, image radiation correction, orthorectification, image fusion and uniform light and color are included; In the sample data set preparation of step S2, ArcGIS is used to annotate the building targets through the Gaojing-1 satellite image, and the original image and label are cropped using 512×512 pixels to generate a remote sensing sample data set in tiff format; The step S3.1 in the step S3 specifically includes: Replace the feature layers corresponding to the U-Net network encoder with the first layer output feature map Efeat1 of stage0 of the resnet50 network, the output feature map Efeat2 of stage1, the output feature map Efeat3 of stage2, the output feature map Efeat4 of stage3, and the output feature map Efeat5 of stage4; The step S3.2 in the step S3 specifically includes: The CBAM hybrid attention mechanism is added after the feature maps Efeat1 and Efeat5 respectively, so that the network can focus more on the key areas; The step S3.3 in the step S3 specifically includes: First, the output feature maps Dfeat1, Dfeat2, Dfeat3, and Dfeat4 of each layer of the decoder are fused from top to bottom to obtain the feature map Dfeat5, and then Dfeat5 is upsampled 8 times and fused with Dfeat4 to obtain Dfeat6, and finally Dfeat6 is upsampled 2 times and output; The data augmentation processing in S4.1 mainly includes random rotation, horizontal flipping, vertical flipping and diagonal mirroring operations; The division ratio of the data set described in S4.2 is 8:1:1; The operating environment setting described in S4.3 specifically uses the Windows operating system, the Pytorch deep learning framework, and the VScode language compiler; The network parameter setting described in S4.4 specifically uses the Adam optimizer, selects the combination of cross entropy loss and dice loss as the network's loss function, sets the initial learning rate to 0.0007, and selects the poly learning strategy and ReLu activation function.

2. The method for automatically extracting out-of-range building spots based on high-view satellite images according to claim 1, characterized in that: The step S5.1 in the step S5 specifically includes: First, the optimal weights of the model are set, and then the test set is input into the network for inference and prediction. In order to evaluate the actual learning ability of the network, the precision (Precision), recall rate (Recall), F1 value (F1-score) and single-class intersection-over-union (Iou) of the test set are calculated.

3. The method for automatically extracting out-of-range building spots based on high-view satellite images according to claim 1, characterized in that: The step S5.2 in the step S5 specifically includes: In the application process of extracting out-of-range building patches, the remote sensing images of the 500m buffer zone of the project red line are selected. Since the network training size is 512×512, the remote sensing images to be predicted should be cropped to a fixed size and then spliced, and the expansion cropping and splicing method should be used to extract out-of-range building patches to avoid obvious splicing marks in the inference prediction results.

Images