A method for ground object extraction from medium and high resolution satellite remote sensing images

Abstract

The present invention discloses a method for extracting ground objects from medium- and high-resolution satellite remote sensing images. The method extracts features from an input image through HRNet, and then fuses the features with context features extracted through a pyramid pooling module to obtain a rough segmentation result. Some difficult-to-learn points are selected from the rough segmentation result, and a multi-layer perceptron is used to learn again on the fused features. The rough segmentation result is refined using the re-learning result to obtain a refined segmentation result. The Loss of the entire training process consists of CE_loss, Points_loss of some points, and SSIM_loss. The extraction effect is improved by fusing high-resolution feature representation with context features, and the boundary extraction effect is improved by combining re-learning of the rough segmentation result with structural similarity loss. The specific performance is as follows: the context information of the image is easy to obtain, the image boundary segmentation result is fine, the processing result is clear, and the performance is excellent on small-size targets and multi-category remote sensing data sets; the generalization ability of different data is strong.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image processing and deep learning technology, and in particular to a method for extracting ground objects from medium- and high-resolution satellite remote sensing images. Background Art

[0002] Conventional semantic extraction faces three problems: (1) In the process of downsampling and extracting image features, the resolution of typical convolutional neural networks such as UNet will become lower and lower. This is beneficial for large-size images and can remove some useless information, but it is not beneficial for small-size images and will lose a lot of important information in the image; (2) The context information of the image is not easy to obtain. The success of PSPNet and Transformer has proved that context information is very important for classification; (3) The image boundary segmentation results are not precise. The erroneous pixels in the image segmentation task are usually located at the boundary between two categories.

[0003] Remote sensing image segmentation is more difficult than conventional image segmentation tasks: (1) The original image resolution is lower. At present, the object size of civilian high-resolution satellite remote sensing images has increased significantly compared to other remote sensing images other than medium and low resolution ones. However, compared with Pascal VOC, Cityscape and other datasets used for conventional image segmentation tasks, the object size in the image is still much smaller; (2) The clarity of remote sensing images is poor. The shooting of remote sensing images is greatly affected by atmospheric conditions. Compared with Pascal VOC, Cityscape and other datasets used for conventional image segmentation tasks, a lot of noise is added to the image. These two problems cause the segmentation performance of conventional extraction methods on remote sensing images to drop significantly. Summary of the invention

[0004] In order to solve the above technical problems, the present invention improves the extraction effect by fusing high-resolution feature representation with contextual features, and improves the boundary extraction effect by relearning the rough segmentation results and combining them with structural similarity loss. The purpose is to provide a ground object extraction method for medium and high-resolution satellite remote sensing images.

[0005] The present invention protects a method for extracting ground objects from medium- and high-resolution satellite remote sensing images, which specifically includes the following steps:

[0006] S1, remote sensing images obtained through satellite;

[0007] S2, the input remote sensing image is used to extract high-resolution features of the remote sensing image through HRNet;

[0008] S3, the high-resolution features of the remote sensing image extracted by HRNet are fused (Concat) with the context features of the remote sensing image extracted by the pyramid pooling (PPM) module to obtain a rough segmentation result;

[0009] S4, select some difficult-to-learn points from the rough segmentation results for boundary optimization, use the multi-layer perceptron (MLP) to learn again on the fusion features, and use the results of the re-learning to refine the rough segmentation results to obtain refined segmentation results. The Loss of the entire training process consists of CE_loss, Points_loss of some points, and SSIM_loss.

[0010] Further, in step S2, high-resolution feature extraction of remote sensing images: HRNet connects high-resolution and low-resolution subnets in parallel, and continuously upsamples and fuses with the main path features during the downsampling process, so that the main path maintains a high-resolution representation, and the entire feature extraction is divided into four stages, each stage performs a convolution operation on the main path, and when the first stage is about to end, the next stage of downsampling is started to give the second path feature, and then at the end of each stage, in addition to downsampling, a new path is given for the next stage, and it is fused with other path features after upsampling and downsampling until the end of the fourth stage to obtain four-path features of decreasing size, and in the segmentation task, the features on the four paths are concatenated (Concat) as the final output feature;

[0011] Furthermore, in step S3, context feature extraction of remote sensing images: pyramid pooling (PPM) extracts the features of the 720 channels output by HRNet again using four convolution kernels of sizes 1, 2, 3, and 6, and outputs 256 feature channels. The features extracted by each convolution kernel are concatenated with the features of HRNet after passing through 1 convolution layer, 1 batch normalization layer, 1 ReLu activation layer, and 1 Dropout layer, and then upsampled to restore the image size. The feature channels after splicing become 1174.

[0012] Furthermore, in step S4, the boundary optimization consists of two parts, one part comes from the Pointrend network, the result output by PPM is used as the rough segmentation result, the feature extracted by PPM is taken as the input feature, the most difficult point to learn is selected from the rough segmentation result, and the multi-layer perceptron (MLP) is used to learn the features of each point again for the most difficult point to learn, and the rough segmentation result is used to provide context information for the most difficult point to learn, and the prediction result of the most difficult point to learn in the rough segmentation result is updated by the MLP team with the prediction result of the most difficult point to learn, so that the rough segmentation result is refined. From the visualization results, it can be seen that the points predicted incorrectly in the rough segmentation result are mainly concentrated on the boundary, so the optimization effect of this part is also mainly on the boundary; the other part is to add an SSIM_loss to the training process. SSIM was originally an indicator for measuring the similarity of two images from a combination of three different factors: brightness, contrast and structure. SSIM_loss=1-SSIM is taken to provide an additional boundary optimization loss function for the training process.

[0013] Furthermore, the loss function in the network structure consists of three parts. The first part calculates the difference between the rough segmentation result and the true value, using the cross entropy loss function (CE_loss); the second part calculates the difference between the predicted results of the points selected in the boundary optimization after MLP and the points in the true value (Points_loss); the third part calculates the structural similarity loss (SSIM_loss) between the rough output result and the true value. The total loss function loss = CE_loss + Points_loss + λ*SSIM_loss, where λ is the weight of SSIM_loss, which can be adjusted as needed.

[0014] Furthermore, the most difficult-to-learn point selected from the rough segmentation result accounts for 50% of the probability of each category of the pixel.

[0015] Compared with the existing technology, the present invention has the following beneficial effects:

[0016] The method of the present invention extracts features from the input image through HRNet, and then fuses and concats the features with the context features extracted by the PPM module to obtain a rough segmentation result, selects some difficult-to-learn points from the rough segmentation result, uses the multi-layer perceptron MLP to learn again on the fused features, and refines the rough segmentation result with the result of the re-learning to obtain a refined segmentation result. The Loss of the entire training process consists of CE_loss, Points_loss and SSIM_loss of some points. The extraction effect is improved by fusing high-resolution feature representation with context features, and the boundary extraction effect is improved by re-learning the rough segmentation result and combining it with the structural similarity loss. The specific performance is as follows: the context information of the image is easy to obtain, the image boundary segmentation result is fine, the processing result is clear, and it performs well on small-size targets and multi-category remote sensing data sets; it has strong generalization ability on different data. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of the method of the present invention;

[0018] Figure 2 This is the structure diagram of HRNet. DETAILED DESCRIPTION

[0019] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Example 1

[0021] A method for extracting ground objects from medium- and high-resolution satellite remote sensing images specifically comprises the following steps:

[0022] S1, remote sensing images obtained through satellite;

[0023] S2, the input remote sensing image is extracted through HRNet to extract the high-resolution features of the remote sensing image; High-resolution feature extraction of remote sensing images: HRNet connects the high-resolution and low-resolution subnets in parallel, and continuously upsamples and fuses with the main path features during the downsampling process, so that the main path maintains a high-resolution representation. The entire feature extraction is divided into four stages, and each stage performs a convolution operation on the main path. When the first stage is about to end, the next stage of downsampling is started to give the second path feature. Later, at the end of each stage, in addition to downsampling, a new path is given to the next stage, and it is fused with the other path features after upsampling and downsampling until the end of the fourth stage to obtain four-way features of decreasing size. In the segmentation task, the features on the four paths are concatenated (Concat) as the final output feature (see the attached HRNet structure diagram for details) Figure 2 );

[0024] S3, the high-resolution features of the remote sensing image extracted by HRNet are fused (Concat) with the contextual features of the remote sensing image extracted by the Pyramid Pooling (PPM) module to obtain a rough segmentation result; Contextual feature extraction of remote sensing images: Pyramid pooling (PPM) extracts the features of the 720 channels output by HRNet again using four convolution kernels of sizes 1, 2, 3, and 6, and outputs 256 feature channels. The features extracted by each convolution kernel are concatenated with the features of HRNet after 1 convolution layer, 1 batch normalization layer, 1 ReLu activation layer, and 1 Dropout layer, and then upsampled to restore the image size. The feature channels after concatenation become 1174;

[0025] S4, select some difficult-to-learn points from the rough segmentation results for boundary optimization, use multi-layer perceptron (MLP) to learn again on the fusion features, and use the re-learning results to refine the rough segmentation results to obtain refined segmentation results. The Loss of the entire training process consists of CE_loss, Points_loss and SSIM_loss of some points; the boundary optimization consists of two parts, one part comes from the Pointrend network, the result of PPM output is used as the rough segmentation result, and the features extracted by PPM are taken as input features. 8096 most difficult to learn points (accounting for 50% of the probability of each category of the pixel) are selected from the rough segmentation results. For this most difficult to learn point, a multi-layer perceptron (MLP) is used to learn again to obtain the features of each point. At the same time, the rough segmentation result is used to provide context information for this most difficult to learn point, and the prediction result of the most difficult to learn point in the rough segmentation result is updated by the MLP team. The prediction result of the most difficult to learn point is used to update the prediction result of the most difficult to learn point in the rough segmentation result, so that the rough segmentation result is refined. From the visual result It can be seen from the results that the points with incorrect predictions in the rough segmentation results are mainly concentrated on the boundaries, so the optimization effect of this part is also mainly on the boundaries; the other part is to add a SSIM_loss to the training process. SSIM was originally an indicator that measures the similarity of two images from a combination of three different factors: brightness, contrast and structure. SSIM_loss=1-SSIM is taken to provide an additional boundary optimization loss function for the training process; the loss function in the network structure consists of three parts. The first part calculates the difference between the rough segmentation result and the true value, using the cross entropy loss function (CE_loss); the second part calculates the difference between the predicted results of the points selected in the boundary optimization after MLP and these points in the true value (Points_loss); the third part calculates the structural similarity loss (SSIM_loss) between the rough output result and the true value. The total loss function loss=CE_loss+Points_loss+λ*SSIM_loss, where λ is the weight of SSIM_loss, which can be adjusted as needed.

[0026] Example 2

[0027] Select 3 data sets to compare the extraction accuracy of the invention and the prior art

[0028] 1. The ecological resource extraction dataset of Qilian County, Haibei Prefecture was set through domestic high-resolution satellite images such as GF1, GF6, and ZY3. After preprocessing, mosaicking, and labeling, the water area, road, and cultivated land datasets were obtained. The extraction results are shown in Table 1 below:

[0029] Table 1. Satellite image extraction results of Qilian County, Haibei

[0030] Model Waters other background QUR PSPNet 30.52 96.52 88.72 71.92 HRNet 63.77 98.08 96.02 85.96 Our 70.93 98.57 99.54 89.68 Model the way other background QUR PSPNet 5.74 98.16 95.06 66.32 HRNet 57.37 99.02 96.54 84.31 Our 60.02 99.08 98.48 85.86 Model arable land other background QUR PSPNet 73.96 88.93 / 81.45 Our 80.66 92.25 / 86.45

[0031] From the results in Table 1, it can be seen that the segmentation effect of the convolutional neural network PSPNet with ResNet as the backbone network drops sharply on small-sized targets such as water areas and roads. The method of the present invention has a significant improvement on small-sized targets, with an improvement of nearly 40% on water areas and nearly 54% on roads. For larger-sized targets such as cultivated land, the method of the present invention also has an improvement of nearly 7%. The method of the present invention performs well on this data.

[0032] 2. The open-source remote sensing image segmentation dataset (CCF Dataset, CCFD) in the CCF Big Data and Computational Intelligence Competition has a total of about 140,000 samples, including 8 categories. 120,000 samples are used as training sets, 13,000 samples are used as validation sets, and 13,000 samples are used as test sets. They are compared with conventional image extraction networks such as DANet, PSPNet, OCRNet, Deeplab V3, and Deeplab V3+. The results are shown in Table 2 below.

[0033] Table 2 CCFD extraction results

[0034] network PSPNet DANet OCRNet DeeplabV3+ DeeplabV3 Our architecture 76.9 77.9 77.73 73.47 72.94 79.95 arable land 88.55 88.91 88.86 87.39 84.88 90.07 woodland 84.55 84.95 85.22 83.66 82.56 86.28 Water 90.48 91.11 91.12 90.11 88.53 91.92 the way 38.26 31.55 35.22 24.89 16.99 37.53 grassland 21.37 24.73 22.54 16.00 17.38 30.59 other 65.63 68.20 67.22 63.27 60.05 72.27 background 97.96 98.60 98.75 97.59 6.82 98.28 mIou 70.46 70.24 70.83 67.05 53.77 73.36

[0035] The results show that the performance of conventional extraction networks on remote sensing datasets has indeed dropped significantly, and the performance differences of each network are mainly in the two rare sample classes of road and grass. Compared with the best average OCRNet, the method of the present invention improves mIoU by about 2.5%, and achieves the best results in most categories, especially in the rare sample class of grass, which is improved by about 8%. The performance on the rare sample class of road is close to the best accuracy, reaching 37.53%. The method of the present invention performs well on multiple categories of remote sensing datasets.

[0036] 3. Use the open source dataset (NaicDataset, NaicD) of the remote sensing image segmentation track in the 2020 National Artificial Intelligence Competition, which contains about 100,000 samples in 8 categories. Take 90,000 samples as the training set and 10,000 samples as the validation set. The extraction results are shown in Table 3.

[0037] Table 3 NaicD extraction results

[0038]

[0039]

[0040] The results show that the proposed method is about 3.7% and 3.4% better than PSPNet and OcrNet networks, respectively, proving that the proposed method has a strong generalization ability on different data.

[0041] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for extracting ground objects from medium- and high-resolution satellite remote sensing images, characterized in that: The specific steps include: S1, remote sensing images obtained through satellite; S2, the input remote sensing image is obtained and the high-resolution features of the remote sensing image are extracted by HRNet; the high-resolution feature extraction of the remote sensing image: HRNet connects the high-resolution and low-resolution subnets in parallel, and continuously upsamples and fuses with the main path features during the downsampling process, so that the main path maintains a high-resolution representation, and the entire feature extraction is divided into four stages, each stage performs a convolution operation on the main path, and the next stage of downsampling is started at the end of the first stage to give the second path feature. Later, at the end of each stage, in addition to downsampling, a new path is given for the next stage, and it is fused with the other path features after upsampling and downsampling until the end of the fourth stage to obtain four-path features with decreasing sizes. In the segmentation task, the features on the four paths are concatenated as the final output features; S3, concatenates the high-resolution features of the remote sensing image extracted by HRNet with the context features of the remote sensing image extracted by the pyramid pooling PPM module to obtain a rough segmentation result; the context feature extraction of the remote sensing image: the pyramid pooling PPM extracts the features of the 720 channels output by HRNet again using four convolution kernels of sizes 1, 2, 3, and 6, and outputs 256 feature channels. The features extracted by each convolution kernel are concatenated with the features of HRNet after passing through 1 convolution layer, 1 batch normalization layer, 1 ReLu activation layer, and 1 Dropout layer, and then upsampled to restore the image size. The concatenated feature channels become 1174; S4, select some difficult-to-learn points from the rough segmentation results for boundary optimization, use the multi-layer perceptron MLP to learn again on the fusion features, and use the re-learning results to refine the rough segmentation results to obtain refined segmentation results. The Loss of the entire training process is composed of CE_loss, Points_loss and SSIM_loss of some points; the boundary optimization consists of two parts, one part comes from the Pointrend network, the result of PPM output is used as the rough segmentation result, the features extracted by PPM are taken as the input features, and the most difficult to learn points are selected from the rough segmentation results. For this most difficult to learn point, The multi-layer perceptron MLP is used to learn again to obtain the features of each point, and the rough segmentation result is used to provide context information for the most difficult point to learn. The prediction result of the most difficult point to learn in the rough segmentation result is updated by the MLP prediction result of the most difficult point to learn, so that the rough segmentation result is refined; the other part is to add a SSIM_loss to the training process, taking SSIM_loss=1-SSIM, and providing an additional boundary optimization loss function for the training process; the loss function in the network structure consists of three parts. The first part calculates the difference between the rough segmentation result and the true value, using the cross entropy loss function CE_loss; The second part calculates the difference between the predicted results of the points selected in the boundary optimization after MLP and the true values ​​of these points Points_loss; The third part calculates the structural similarity loss SSIM_loss between the rough output result and the true value. The total loss function loss = CE_loss + Points_loss + λ*SSIM_loss, where λ is the weight of SSIM_loss, which can be adjusted as needed.

2. The method for extracting ground objects from medium- and high-resolution satellite remote sensing images according to claim 1, characterized in that: The most difficult point to learn selected from the rough segmentation result accounts for 50% of the probability of each category of the pixel.

Images