A remote sensing image building extraction and segmentation result regularization method and system

Abstract

The application provides a remote sensing image building extraction and segmentation result regularization method, which comprises the following steps: acquiring a remote sensing image, wherein the full-color spatial resolution of the remote sensing image is greater than 1 m; cutting the image picture to 512*512 as input data; extracting a building probability map in the image picture by using an improved FCN based on DeepLabV3+; and regularizing the probability map by using a trained generative adversarial network, wherein the trained generative adversarial network comprises a generator and a discriminator, and the generator comprises an encoder and a decoder; the full convolutional neural network is trained by using the generative adversarial method, is used for post-processing of a semantic segmentation result of a building in a remote sensing image, and is used for regularizing the semantic segmentation result; compared with a traditional morphological method, the influence of errors in semantic segmentation on subsequent regularization results is reduced, so that regular building contours are generated.

Description

Technical Field

[0001] This invention relates to the field of remote sensing imagery, and in particular to a method and system for regularizing the extraction and segmentation results of buildings in remote sensing images. Background Technology

[0002] Current fully convolutional neural networks (FCNs) achieve high accuracy in semantic segmentation. However, when used to extract buildings from remote sensing images, noise in the images can cause irregular edges in the segmentation results, making them difficult to apply to mapping tasks.

[0003] The process of extracting buildings from remote sensing images can be divided into two steps: building segmentation and polygon prediction.

[0004] Building partitioning:

[0005] Previous building segmentation methods used cross-entropy, DICE, and Focal loss for training, achieving high scores in IoU, recall, and accuracy. However, they generated irregular building outlines that were neither aesthetically pleasing nor suitable for most mapping applications. A typical problem with semantic segmentation networks and instance segmentation networks is their inability to delineate straight building walls and sharp corners in the presence of ground noise (such as occluded building corners).

[0006] Polygonization:

[0007] Semantic segmentation and instance segmentation networks can generate accurate segmentation results, but mapping applications in remote sensing require segmentation data in vector format, not raster segmentation results. Therefore, the segmentation results need to be regularized. Current methods can be divided into two categories, but either they are computationally expensive and errors in the segmentation results accumulate during the entire polygonization process, affecting the quality of the final polygon, or they require multiple iterations and cannot handle complex building shapes.

[0008] The first type of method involves post-processing regularization of the segmentation results. This method uses FCN to generate building segments; then, morphological methods are used to regularize the segmentation results. This type of method decomposes the building extraction and polygonization problem into smaller tasks that can be solved individually. However, most of these methods are computationally intensive, and their hyperparameters must be carefully tuned to achieve the desired results. Most importantly, because they consist of a series of subtasks, errors accumulate during the process, thus degrading the quality of the final polygons.

[0009] The second type of method involves directly learning vector representations. A polygon is initialized for each building object instance in the remote sensing image, and vertex positions are optimized using networks such as the Transformer. These methods directly process polygon parameters, but they are generally more difficult to train and require multiple iterations during inference. Furthermore, they struggle with complex building shapes, such as those with curved edges or openings. Summary of the Invention

[0010] The main objective of this invention is to overcome the aforementioned deficiencies in the prior art and propose a method for regularizing the extraction and segmentation results of buildings in remote sensing images. A fully convolutional neural network is trained using a generative adversarial method for post-processing of semantic segmentation results of buildings in remote sensing images, thereby regularizing the semantic segmentation results. Compared with the traditional morphological methods, this reduces the impact of errors in semantic segmentation on subsequent regularization results, thus producing regular building outlines.

[0011] The present invention adopts the following technical solution:

[0012] A method for regularizing the extraction and segmentation results of buildings from remote sensing images, comprising:

[0013] Acquire remote sensing images, wherein the panchromatic space resolution of the remote sensing images is greater than 1m;

[0014] Crop the image to 512*512 as input data;

[0015] An improved FCN based on DeepLabV3+ is used to extract building probability maps from image images;

[0016] The probabilistic graph is regularized using a trained generative adversarial network, wherein the trained generative adversarial network includes a generator and a discriminator, and the generator includes an encoder and a decoder.

[0017] Specifically, the improved DeepLabV3+ based FCN replaces the DeepLabV3+ encoder portion with a RegNetY040 network.

[0018] Specifically, the generator uses FCN to perform regularization G:{X,Z}→Y; where X is the irregular segmentation result, Y is the regular segmentation result, and Z is the image.

[0019] The generator includes encoder E G With decoder F, encoder E G Feature information is extracted from {X,Z}, and a regularized segmentation result is generated using the decoder F. Therefore, the entire process can be viewed as a combination of encoder and decoder: G(x,z)=F(E) G (x,y)), x i∈X, y i ∈Y, z i ∈Z, i=1,2,3...N, where N is the total number of images.

[0020] Specifically, the discriminator receives the regularized segmented image or the actual segmentation label generated by the generator after passing through the encoder E. G The output of the decoder F is compared with the output of the decoder, and it is determined whether the image was generated by the generator.

[0021] Specifically, the loss functions include: adversarial loss, reconstruction loss, Potts loss, and normalized cut loss.

[0022] Combat losses:

[0023] The formula for adversarial loss is as follows:

[0024] L GAN (G,D)=E x,z [log(1-D(G(x,z))]

[0025] The discriminator is used to distinguish whether the segmentation result was generated by the generator, and its loss function is as follows:

[0026] L D (G,R,D)=E y [log(1-D(R(y))]+E x,z [logD(G(x,z))]

[0027] Where R(y) = F(E) R (y) represents the actual label after passing through encoder E. G The output of decoder F is G(x,z)=F(E) G (x,z) represents the regularization of the original irregular segmented image.

[0028] Reconstruction losses:

[0029] The reconstruction loss uses cross-entropy loss;

[0030]

[0031] Among them, w c y is the weight, M is the number of categories, and y is the weight. c One-hot vectors take only 0 and 1 values; if the class and the sample class are the same, the value is 1; otherwise, it is 0. c To predict the probability that a sample belongs to c;

[0032] w c The weights are shown in the formula below, where N represents the total number of pixels. C This represents the total number of pixels occupied by category C in the actual situation.

[0033]

[0034] Potts vs. normalized cut loss:

[0035] The Potts loss formula is as follows:

[0036]

[0037] The formula for normalized cut loss is as follows:

[0038]

[0039] Where S = G(x, z) is a k-way softmax mask generated by the network, and k describes the vectorization of its k-th channel; W and It is a pairwise discontinuous cost matrix, where each item describes the weight between two nodes or pixels, and T is a parameter value.

[0040] Another embodiment of the present invention provides a system for regularizing the extraction and segmentation results of buildings from remote sensing images, comprising:

[0041] Image acquisition unit: acquires remote sensing images, wherein the panchromatic resolution of the remote sensing images is greater than 1m;

[0042] Preprocessing unit: Crops the image to 512*512 pixels as input data;

[0043] Extraction Unit: An improved FCN based on DeepLabV3+ is used to extract building probability maps from image images;

[0044] Result regularization unit: Regularizes the probabilistic graph using a trained generative adversarial network, wherein the trained generative adversarial network includes a generator and a discriminator, and the generator includes an encoder and a decoder.

[0045] In another aspect, the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method for regularizing the extraction and segmentation results of remote sensing images of buildings.

[0046] In another aspect of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the steps of the above-described method for regularizing the extraction and segmentation results of buildings from remote sensing images.

[0047] As can be seen from the above description of the present invention, compared with the prior art, the present invention has the following beneficial effects:

[0048] This invention provides a method for extracting and regularizing building segmentation results from remote sensing images, comprising: acquiring remote sensing images with a panchromatic space resolution greater than 1m; cropping the image to 512*512 as input data; extracting building probability maps from the image images using an improved DeepLabV3+-based FCN; and regularizing the probability maps using a trained generative adversarial network, wherein the trained generative adversarial network includes a generator and a discriminator, and the generator includes an encoder and a decoder. This invention uses a generative adversarial method to train a fully convolutional neural network for post-processing of semantic segmentation results of buildings in remote sensing images, thereby regularizing the semantic segmentation results. Compared with traditional morphological methods, this reduces the impact of errors in semantic segmentation on subsequent regularization results, thus generating regular building outlines. Attached Figure Description

[0049] Figure 1 This is a generative adversarial network architecture diagram for segmentation probabilistic graph regularization provided in an embodiment of the present invention;

[0050] Figure 2 This is a structural diagram of the encoder provided in an embodiment of the present invention;

[0051] Figure 3 This is a decoder structure diagram provided in an embodiment of the present invention;

[0052] Figure 4 This is a structural diagram of the discriminator provided in an embodiment of the present invention;

[0053] Figure 5 These are images provided in embodiments of the present invention;

[0054] Figure 6 This is a segmentation result diagram provided in an embodiment of the present invention;

[0055] Figure 7 This is a regularized segmentation result image provided in an embodiment of the present invention;

[0056] Figure 8 These are actual images provided in the embodiments of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0058] The present invention adopts the following technical solution:

[0059] A method for regularizing the extraction and segmentation results of buildings from remote sensing images, comprising:

[0060] s101: Acquire remote sensing imagery, wherein the panchromatic resolution of the remote sensing imagery is greater than 1m;

[0061] s102: Crop the image to 512*512 as input data;

[0062] s103: An improved FCN based on DeepLabV3+ is used to extract building probability maps from image images;

[0063] The improved FCN based on DeepLabV3+ replaces the DeepLabV3+ encoder with a RegNetY040 network, which can improve the accuracy of the extracted building segmentation results.

[0064] s104: Regularize the probabilistic graph using a trained generative adversarial network, wherein the trained generative adversarial network includes a generator and a discriminator, and the generator includes an encoder and a decoder;

[0065] like Figure 1 This is a generative adversarial network architecture diagram for segmentation probabilistic graph regularization provided in an embodiment of the present invention; the generator uses FCN to perform regularization G:{X,Z}→Y; where X is the irregular segmentation result, Y is the regular segmentation result, and Z is the image;

[0066] The generator is used to regularize the segmentation results. For the training data... in Where y i In the network, the generator learns the mapping from the original irregular segmentation result X to the ideal regular segmentation result Y. To further improve the generator's output, it also receives the segmented image (RGB image). Where z i ∈Z, used to further constrain the generated results;

[0067] The generator includes encoder E G With decoder F, encoder E G Feature information is extracted from {X,Z}, and a regularized segmentation result is generated using the decoder F. Therefore, the entire process can be viewed as a combination of encoder and decoder: G(x,z)=F(E) G (x,y)), x i ∈X, y i ∈Y, z i ∈Z, i=1,2,3...N, where N is the total number of images.

[0068] The encoder structure is as follows: Figure 2 As shown, it includes, in sequence: 3*3 convolution + ReLU + BN layer, 3*3 convolution + ReLU + BN layer, max pooling layer, 3*3 convolution + ReLU + BN layer, 3*3 convolution + ReLU + BN layer, max pooling layer;

[0069] The structure of the decoder is as follows Figure 3 As shown, it includes, in sequence: 8 consecutive residual blocks, 2*2 upsampling layer, 3*3 convolution + ReLU + BN layer, 3*3 convolution + ReLU + BN layer, 3*3 convolution + ReLU + BN layer, 2*2 upsampling layer, 3*3 convolution + ReLU + BN layer, 3*3 convolution + ReLU + BN layer, 3*3 convolution + ReLU + BN layer, 1*1 convolution + sigmoid;

[0070] The encoder-decoder structure provided in this embodiment of the invention is simple, reduces training difficulty, and the encoder can be easily replaced.

[0071] Specifically, the discriminator receives the regularized segmented image or the actual segmentation label generated by the generator after passing through the encoder E. G The output of the decoder F is compared with the output of the decoder, and it is determined whether the image was generated by the generator.

[0072] The reason why the discriminator does not directly receive the real segmentation label is that the real segmentation label is one-hot encoded and only contains two values, 0 and 1. The result value from the decoder output is between 0 and 1. In this case, the discriminator can easily distinguish whether the image comes from the generator, so the training purpose cannot be achieved.

[0073] The specific structure of the discriminator is as follows: Figure 4 As shown, the layers are arranged in sequence as follows: 3*3 convolution + ReLU + BN layer, 3*3 convolution + ReLU + BN layer, 2*2 max pooling layer, 3*3 convolution + ReLU + BN layer, 3*3 convolution + ReLU + BN layer, 2*2 max pooling layer, 3*3 convolution + ReLU + BN layer, 3*3 convolution + ReLU + BN layer, 2*2 max pooling layer, 3*3 convolution + ReLU + BN layer, 3*3 convolution + ReLU + BN layer, 2*2 max pooling layer, 3*3 convolution + ReLU + BN layer, 3*3 convolution + ReLU + BN layer, 1*1 convolution + sigmoid.

[0074] Specifically, the loss functions include: adversarial loss, reconstruction loss, Potts loss, and normalized cut loss.

[0075] Combat losses:

[0076] Adversarial loss is used to enable the generator to learn the mapping from the original irregular segmentation result X to the ideal regular segmentation result Y. It aims to constrain the geometric edges of buildings in the segmentation result. The adversarial loss formula is as follows:

[0077] L GAN (G,D)=E x,z [log(1-D(G(x,z))]

[0078] The discriminator is used to distinguish whether the segmentation result was generated by the generator, and its loss function is as follows:

[0079] L D (G,R,D)=E y [log(1-D(R(y))]+E x,z [logD(G(x,z))]

[0080] Where R(y) = F(E) R (y) represents the actual label after passing through encoder E. G The output of decoder F is G(x,z)=F(E) G (x,z) represents the regularization of the original irregular segmented image.

[0081] Reconstruction losses:

[0082] The reconstruction loss uses cross-entropy loss;

[0083]

[0084] Among them, w c y is the weight, M is the number of categories, and y is the weight. c One-hot vectors take only 0 and 1 values; if the class and the sample class are the same, the value is 1; otherwise, it is 0. c To predict the probability that a sample belongs to c;

[0085] w c The weights are shown in the formula below, where N represents the total number of pixels. C This represents the total number of pixels occupied by category C in the actual situation.

[0086]

[0087] The reconstruction loss is used to make the generator's output closer overall to the original irregularly segmented image input to the generator. The reconstruction loss formula for the generator path is as follows:

[0088] L recG (G)=-E x,z [x·logG(x,z)]

[0089] The reconstruction loss for the true label reconstruction route is as follows:

[0090] L recR (R)=-E y [y·logR(y)]

[0091] Potts vs. normalized cut loss:

[0092] Information from the RGB image is used to further improve the regularization results. The Potts method and normalized cut are graph clustering algorithms used for image segmentation. These two loss functions further align the regularized segmentation results with the building boundaries observed in the intensity image.

[0093] The Potts loss formula is as follows:

[0094]

[0095] The formula for normalized cut loss is as follows:

[0096]

[0097] Where S = G(x, z) is a k-way softmax mask generated by the network, and k describes the vectorization of its k-th channel; W and It is a pairwise discontinuous cost matrix, where each item describes the weight between two nodes or pixels, and T is a parameter value.

[0098] In summary, the complete loss function is as follows:

[0099]

[0100] Training process:

[0101] The optimizer used is AdamW, with an initial learning rate of 0.0002 and no weight decay. The batch size is set to 16, and cosine annealing with preheating is selected for learning rate tuning.

[0102] The generator and discriminator are combined and trained using the aforementioned loss function. Data is input into the neural network for training. During training, the generator and discriminator produce prediction results, the loss function calculates the error between the prediction and the actual result, and the optimizer guides the loss function to update the parameter weights of appropriate sizes in the correct direction. Both the generator and discriminator are trained simultaneously in each training round. After one round of training, predictions can be made on the validation set. The above steps are repeated until the error on the validation set no longer shows a significant decreasing trend, at which point training ends. The resulting generator can then be used to regularize the segmentation results.

[0103] Prediction phase:

[0104] After the training phase is complete, the segmentation results of FCN are input into the generator obtained from the training to obtain regularized building segmentation results.

[0105] like Figure 5 These are images provided in embodiments of the present invention; Figure 6 This is a segmentation result diagram provided in an embodiment of the present invention; Figure 7 This is a regularized segmentation result image provided in an embodiment of the present invention; Figure 8 These are actual images provided in the embodiments of the present invention.

[0106] Another embodiment of the present invention provides a system for regularizing the extraction and segmentation results of buildings from remote sensing images, comprising:

[0107] Image acquisition unit: acquires remote sensing images, wherein the panchromatic resolution of the remote sensing images is greater than 1m;

[0108] Preprocessing unit: Crops the image to 512*512 pixels as input data;

[0109] Extraction Unit: An improved FCN based on DeepLabV3+ is used to extract building probability maps from image images;

[0110] Result regularization unit: Regularizes the probabilistic graph using a trained generative adversarial network, wherein the trained generative adversarial network includes a generator and a discriminator, and the generator includes an encoder and a decoder.

[0111] In another aspect, the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method for regularizing the extraction and segmentation results of remote sensing images of buildings.

[0112] Since the electronic device described in this embodiment is the device used to implement the embodiments of the present invention, those skilled in the art can understand the specific implementation methods and various variations of the electronic device in this embodiment based on the methods described in the embodiments of the present invention. Therefore, how the electronic device implements the methods in the embodiments of the present invention will not be described in detail here. Any device used by those skilled in the art to implement the methods in the embodiments of the present invention is within the scope of protection of the present invention.

[0113] In another aspect of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the steps of the above-described method for regularizing the extraction and segmentation results of buildings from remote sensing images.

[0114] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0115] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0116] A generative adversarial approach is used to train a fully convolutional neural network for post-processing of semantic segmentation results of buildings in remote sensing imagery. This regularizes the semantic segmentation results, reducing the impact of errors in semantic segmentation on subsequent regularization results compared to traditional morphological methods. Furthermore, training is simpler than methods that directly learn vector representations, and multiple iterations are not required during inference.

[0117] Regularizing the semantic segmentation results of buildings in remote sensing images produces regular building outlines that are both aesthetically pleasing and suitable for most mapping applications.

[0118] A generative adversarial approach is used to train a fully convolutional neural network for post-processing of semantic segmentation results of buildings in remote sensing imagery. This regularizes the semantic segmentation results, reducing the impact of errors in semantic segmentation on subsequent regularization results compared to traditional morphological methods. Furthermore, training is simpler than methods that directly learn vector representations, and multiple iterations are not required during inference.

[0119] Regularizing the semantic segmentation results of buildings in remote sensing images produces regular building outlines that are both aesthetically pleasing and suitable for most mapping applications.

[0120] The above are merely specific embodiments of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial modifications made to the present invention using this concept shall be considered as infringing upon the protection scope of the present invention.

Claims

1. A method for regularizing the extraction and segmentation results of buildings from remote sensing images, characterized in that, include: Acquire remote sensing images, wherein the panchromatic space resolution of the remote sensing images is greater than 1m; Crop the image to 512*512 as input data; An improved FCN based on DeepLabV3+ is used to extract building probability maps from image images; The probabilistic graph is regularized using a trained generative adversarial network, wherein the trained generative adversarial network includes a generator and a discriminator, and the generator includes an encoder and a decoder; The generator uses FCN to perform rule-making. ;in Irregular segmentation results The result of the rule-based segmentation. Images; The generator includes an encoder. With decoder encoder from Extract feature information and use the decoder The process generates regularized segmentation results, thus the entire process can be viewed as a combination of an encoder and a decoder: , , , i = 1, 2, 3... N, where N is the total number of images; Loss functions include: adversarial loss, reconstruction loss, Potts loss, and normalized cut loss; Combat losses: The formula for adversarial loss is as follows: ； The discriminator is used to distinguish whether the segmentation result was generated by the generator, and its loss function is as follows: ； in For real labels passed through the encoder With decoder The output after that, To regularize the original irregular segmented image; Reconstruction losses: The reconstruction loss uses cross-entropy loss; ； in, Here, M represents the weight, and M represents the number of categories. One-hot vectors can only take the values ​​0 and 1. A value of 1 is used if the category matches the sample's category, and 0 otherwise. To predict the probability that a sample belongs to c; The weights are given by the formula shown below, where N represents the total number of pixels. This represents the total number of pixels occupied by category C in the actual situation. ； Potts vs. normalized cut loss: The Potts loss formula is as follows: ； The formula for normalized cut loss is as follows: ； Where S = G(x, z) is a k-way softmax mask generated by the network, and k describes the vectorization of its k-th channel; and It is a pairwise discontinuous cost matrix, where each item describes the weight between two nodes or pixels, and T is a parameter value.

2. The method for regularizing the extraction and segmentation results of remote sensing images of buildings according to claim 1, characterized in that, The improved FCN based on DeepLabV3+ replaces the DeepLabV3+ encoder portion with a RegNetY040 network.

3. The method for regularizing the extraction and segmentation results of remote sensing images of buildings according to claim 1, characterized in that, The discriminator receives the regularized segmented image or real segmentation label generated by the generator after passing through the encoder. With decoder The output is then used to determine whether the image was generated by the generator.

4. A system for regularizing the extraction and segmentation results of buildings from remote sensing images, characterized in that, include: Image acquisition unit: acquires remote sensing images, wherein the panchromatic resolution of the remote sensing images is greater than 1m; Preprocessing unit: Crops the image to 512*512 pixels as input data; Extraction Unit: An improved FCN based on DeepLabV3+ is used to extract building probability maps from image images; Result regularization unit: Regularizes the probability graph using a trained generative adversarial network, wherein the trained generative adversarial network includes a generator and a discriminator, and the generator includes an encoder and a decoder; The generator uses FCN to perform rule-making. ;in Irregular segmentation results The result of the rule-based segmentation. Images; The generator includes an encoder. With decoder encoder from Extract feature information and use the decoder The process generates regularized segmentation results, thus the entire process can be viewed as a combination of an encoder and a decoder: , , , i = 1, 2, 3... N, where N is the total number of images; Loss functions include: adversarial loss, reconstruction loss, Potts loss, and normalized cut loss; Combat losses: The formula for adversarial loss is as follows: ； The discriminator is used to distinguish whether the segmentation result was generated by the generator, and its loss function is as follows: ； in For real labels passed through the encoder With decoder The output after that, To regularize the original irregular segmented image; Reconstruction losses: The reconstruction loss uses cross-entropy loss; ； in, Here, M represents the weight, and M represents the number of categories. One-hot vectors can only take the values ​​0 and 1. A value of 1 is used if the category matches the sample's category, and 0 otherwise. To predict the probability that a sample belongs to c; The weights are given by the formula shown below, where N represents the total number of pixels. This represents the total number of pixels occupied by category C in the actual situation. ； Potts vs. normalized cut loss: The Potts loss formula is as follows: ； The formula for normalized cut loss is as follows: ； Where S = G(x, z) is a k-way softmax mask generated by the network, and k describes the vectorization of its k-th channel; and It is a pairwise discontinuous cost matrix, where each item describes the weight between two nodes or pixels, and T is a parameter value.

5. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the method according to any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-3.

Images