An Optimization Method for Building Extraction from High-resolution Remote Sensing Images Based on ConvCRF

By introducing Residual_Block and CBAM modules into the Unet neural network and combining ConvCRF, the problem of difficulty in balancing accuracy and time in building extraction with high score remote sensing images is solved, and fine extraction and serration of building edges are achieved, improving the extraction effect .

CN114943894BActive Publication Date: 2024-11-12TAIYUAN UNIVERSITY OF TECHNOLOGY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210513757.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-11
Publication Date
2024-11-12
Estimated Expiration
2042-05-11

AI Technical Summary

Technical Problem

The prior art is difficult to balance the extraction accuracy and time in the extraction of high-score remote sensing images, and often leads to sawtoothing at the edges of the segmentation results, making it impossible to accurately restore the building profile.

Method used

Using a method of combining full convolutional neural network with ConvCRF, the Residual_Block and CBAM modules are introduced into the Unet neural network, and the maximum pooling layer is replaced with a normal convolutional layer, and the conditionally limited FullCRF is introduced in the image segmentation post-processing to perform End-to-end model training to improve building edge extraction accuracy and reduce sawtoothing.

Benefits of technology

It improves the accuracy and segmentation effect of building extraction, eliminates the jagging of segmentation results, improves the average cross-parallel ratio and average pixel accuracy, and meets the extraction needs of the real outline of the building.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114943894B_ABST
    Figure CN114943894B_ABST
Patent Text Reader

Abstract

The invention provides a high-resolution remote sensing image building extraction optimization method based on ConvCRF, belonging to the technical field of image processing; the technical problem to be solved is: to provide an improvement of the high-resolution remote sensing image building extraction optimization method based on ConvCRF; the technical scheme adopted to solve the above technical problem is: to replace the ordinary convolution operation in the Unet network by a residual-block residual convolution structure, and to introduce a CBAM (Convolution Block Attention Module) convolution attention module in the Unet low receptive field encoding and decoding stage to improve the model's extraction accuracy for building edges; at the same time, the ConvCRF module is connected to perform separation model training to reduce the generation of edge jaggedness of segmentation results, eliminate noise, and fit the real contour of the building; the invention is applied to high-resolution remote sensing image building extraction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention provides a ConvCRF-based high-resolution remote sensing image building extraction optimization method, belonging to the technical field of image processing. Background Art

[0002] The advancement of remote sensing technology has closely integrated remote sensing applications with urban life, and urban planning cannot be separated from the support of remote sensing technology. High-resolution remote sensing images show more obvious features of buildings, and the clear and prominent shapes, structures, details and textures make it more effective to extract urban buildings. In order to ensure the accuracy and contour optimization of building extraction from high-resolution remote sensing images, manual interpretation is still the most common extraction method. Its time-consuming and labor-intensive weaknesses restrict the large-scale application of high-resolution remote sensing images.

[0003] In order to reduce the labor and time spent on building extraction, researchers have begun to try to combine deep learning with remote sensing images. The rapid development of deep learning technology in recent years has enabled researchers to use deep learning technology to build models, and continuously learn the data distribution characteristics of buildings in high-resolution remote sensing images during model training, thereby distinguishing buildings from their background environment.

[0004] With the expansion of the urban territory, the number of buildings has also increased, and people's demand for building extraction has also increased. However, there are still few methods that can better balance the extraction accuracy and interpretation time of buildings. In the problem of ground object segmentation, in order to improve the accuracy of building extraction, the method of deepening the neural network encoding structure is often used to extract high-level abstract features with low resolution, thereby losing the spatial information and edge information of the ground object; at the same time, the bilinear interpolation and deconvolution operations in the upsampling process of the full convolutional neural network make it impossible to accurately insert the edge pixels of the object in the process of restoring the abstract feature size of the ground object, resulting in more jagged edges on the edge of the ground object segmentation result, and the ground object contour cannot be accurately restored. At present, when people use deep learning technology to solve the problem of building extraction, they often use a neural network to segment the object once and manually correct the segmentation result to meet the city's demand for building extraction. In order to minimize the labor injected by manual secondary annotation, the present invention proposes a method combining a full convolutional neural network with ConvCRF to improve the accuracy of building extraction from high-resolution remote sensing images, eliminate the jagged edges of the extraction results, and meet the true contour of the building as much as possible. Summary of the invention

[0005] In order to overcome the deficiencies in the prior art, the present invention aims to solve the technical problem of providing an improved method for extracting buildings from high-resolution remote sensing images based on ConvCRF.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: a high-resolution remote sensing image building extraction optimization method based on ConvCRF, comprising the following steps:

[0007] S1: Construct a full convolutional neural network basic segmentation model, train the basic model for multiple rounds on high-resolution remote sensing image data, obtain a segmentation model that fits the feature distribution of high-resolution remote sensing image data, replace the ordinary convolution in the basic Unet with Residual_Block, replace the MaxPooling layer in the basic Unet with an ordinary convolution layer, and embed the CBAM module into the Residual_Block in the first two layers of encoding and the last two layers of decoding to obtain the improved Unet neural network model;

[0008] S2: Constructing image segmentation post-processing model: Introducing ConvCRF end-to-end connection model, performing secondary correction optimization on the segmentation results of the basic segmentation model, using ConvCRF improved based on FullCRF as image segmentation post-processing model, introducing conditional restrictions on the basis of FullCRF, and setting a given kernel-size so that the calculation range between pixels is limited to the kernel-size range centered on it;

[0009] S3: training the neural network model and image segmentation post-processing model;

[0010] S4: The average intersection-over-union (MIOU) and average pixel MPA are used to evaluate the segmentation effect of the model on the dataset.

[0011] Before model training, the remote sensing image dataset is obtained and preprocessed, and the dataset is divided into training set, validation set, and test set in proportion; the Inria_Aerial_Image_Labeling_Dataset remote sensing image urban building detection dataset is used. Each image in the dataset has a resolution of 5000*5000 pixels, and the labels are divided into buildings and non-buildings;

[0012] During the neural network training process, the large remote sensing image with a resolution of 5000*5000 pixels is cropped, and a 10% coverage rate is generated between the chunk images during the cropping process.

[0013] The training of the neural network model and the image segmentation post-processing model in step S3 specifically includes:

[0014] The torchvision.transforms in pytorch is used to normalize the input data, and the multi-classification cross entropy loss function is selected as the loss function for model training. Based on the adaptive gradient descent algorithm Adam, the StepLR interval adjustment learning rate decay strategy is used to adjust the learning rate. Based on the basic segmentation model, the ConvCRF algorithm is trained iteratively in a separated manner.

[0015] The learning rate is adjusted using the StepLR interval adjustment learning rate decay strategy.

[0016] The average intersection-over-union (MIOU) and average pixel MPA are used to evaluate the segmentation effect of the model on the dataset.

[0017] Assume there are n+1 categories, p ij It represents the number of pixels whose actual category is i but predicted to be j. The specific calculation formulas of MIOU and MPA are as follows:

[0018]

[0019]

[0020] The data in the training set are enhanced at a data enhancement ratio of 1:4. The enhancement methods include: image flipping, symmetrically flipping the original image with the midline as the axis; image rotation, rotating the original image 90° clockwise and counterclockwise; data noise addition, randomly adding noise to the original image at a pixel ratio of 50%.

[0021] Through the pytorch framework, all parameters in ConvCRF are placed in the GPU for training with gradients.

[0022] The beneficial effects of the present invention compared to the prior art are as follows: the present invention provides a high-resolution remote sensing image building extraction optimization method based on ConvCRF, which replaces the ordinary convolution operation in the Unet network with a residual-block residual convolution structure, and introduces the CBAM (Convolution Block Attention Module) convolution attention module in the Unet low receptive field encoding and decoding stage to improve the model's extraction accuracy for building edges. At the same time, the ConvCRF module is connected to perform separation model training to reduce the generation of edge jaggedness in the segmentation result, eliminate noise, and fit the true contour of the building. Experimental results show that the improved Unet neural network proposed in the present invention is superior to the classic semantic segmentation algorithm in segmentation effect and accuracy, with an increase of 2.67% and 3.17% in average accuracy and average intersection-over-union ratio; the ConvCRF separation model can effectively eliminate noise and reduce the generation of edge jaggedness. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The present invention will be further described below in conjunction with the accompanying drawings:

[0024] Figure 1 It is a schematic diagram of the structure of the traditional Unet neural network model;

[0025] Figure 2 It is a schematic diagram of the structure of the CBAM algorithm;

[0026] Figure 3 This is a schematic diagram of the structure of the CRF algorithm;

[0027] Figure 4 It is a structural diagram of the FullCRF algorithm;

[0028] Figure 5 It is a schematic diagram of the structure of the improved Unet neural network model of the present invention;

[0029] Figure 6 It is a structural diagram of the ConvCRF algorithm improved based on FullCRF in the present invention;

[0030] Figure 7 This is an example diagram of data samples used in the present invention;

[0031] Figure 8 A diagram of a data set after data enhancement for the present invention;

[0032] Fig. 9 Training accuracy loss plots for Unet, PSP, Improved Unet, and Separate ConvCRF models;

[0033] Fig.10 This is a comparison chart of the verification results of each model on the test set;

[0034] Fig.11 The following are the prediction results of each model. DETAILED DESCRIPTION

[0035] like Figure 1-11 As shown, given that the current full convolutional neural network is more inclined to extract the main part of the model for identifying objects, it ignores the feature extraction part of the low receptive field area, resulting in the model's lack of detection of the edge of the object. In order to improve this defect, the present invention proposes a neural network model that combines an improved Unet with ConvCRF. The residual-block residual convolution structure replaces the ordinary convolution operation in the Unet network, and the CBAM (Convolution Block Attention Module) convolution attention module is introduced in the Unet low receptive field encoding and decoding stage to improve the model's extraction accuracy for the edge of the building. At the same time, the ConvCRF module is connected to perform separation model training to reduce the generation of edge jaggedness in the segmentation result, eliminate noise, and fit the real outline of the building. The experimental results show that the improved Unet neural network proposed in the present invention is superior to the classic semantic segmentation algorithm in segmentation effect and accuracy, and the average accuracy and average intersection-over-union ratio are improved by 2.67% and 3.17%; the ConvCRF separation model can effectively eliminate noise and reduce the generation of edge jaggedness.

[0036] The following is an introduction to the basic model structures of Unet, CBAM, CRF, and FullCRF.

[0037] Unet: Figure 1 As shown in the figure, the basic Unet architecture has 5 layers, and the overall network has a symmetrical structure, namely the Encoder-Decoder structure. The left side is called the contraction path, also known as the encoding stage, which uses the convolutional neural network to extract the abstract feature information of the image; the right side is called the expansion path, also known as the decoding stage, which restores the image size of high-level abstract features through the bilinear interpolation algorithm. The left side is built with reference to VGG16, using a typical convolutional neural network structure, with 4 downsampling and 5 groups of convolutions. The input data size is 572*572*1, and each group of convolution operations consists of two convolution operations, with a convolution kernel size of 3*3, followed by a convolution operation, with a sampling kernel of 2*2 and a step size of 2. The result is that the size of the feature map continues to shrink, the number of feature map channels continues to increase, and the output data size on the left is 28*28*1024. The right side first performs upsampling, with a convolution kernel size of 2*2, and then superimposes it with the feature map of the corresponding layer on the contraction path on the left, and then performs a convolution operation.

[0038] During this period, both convolution and upsampling require activation functions. The Relu activation function is generally used in the Unet network layer, which can improve the gradient disappearance and speed up the training. At the end of the network, a 1*1 convolution is used to classify the output results. The activation function is generally converted into a probability map using Sigmoid or Softmax functions.

[0039] CBAM: Figure 2 As shown in the figure, CBAM (Convolutional Block Attention Module) consists of a channel attention mechanism and a spatial attention mechanism. Its main significance is to extract the main features in the feature map by obtaining the maximum and average values ​​of the feature map in different dimensions.

[0040] In the channel attention mechanism, the input feature map input is subjected to maximum pooling and average pooling respectively, and then connected with 1*1 convolution operation to obtain the main features. Finally, the sigmoid activation function is introduced to introduce the nonlinear relationship of the features.

[0041] Similar to the channel attention mechanism, in the spatial attention mechanism, the maximum and average values ​​of the features are obtained in the channel dimension respectively, and the convolution operation is connected with the sigmoid activation function.

[0042] CRF: In 2001, John et al. proposed conditional random fields, such as Figure 3 As shown, the conditional random field CRF (Conditional Random Field) consists of an observation sequence and a state sequence. The state sequence constitutes a Markov random field, which is expressed by formula (1) (2):

[0043] P(Y i-1 , Y i+1 |Y i )=P(Y i-1 |Y i )P(Y i+1 |Y i ) (1);

[0044] P(Y i-1 |Y j ,y i+1 )=P(Y i-1 |Y i ) (2);

[0045] A single observation point Xi uniquely determines a state point Yi, that is, P(Yi|Xi). In the conditional random field, each state point is only related to its adjacent state points and corresponding observation points, as shown in formulas (3) and (4):

[0046]

[0047]

[0048] Z(X) is the normalization factor, μ and s are the characteristic functions, and λ and u are the corresponding weights of the characteristic functions.

[0049] FullCRF: Philipp et al. proposed FullCRF (Fully Connected CRFs) in 2011. Based on the principle of CRF, FullCRF is similar to CRF. In FullCRF, a single pixel Xi is used as an observation point, and each pixel's corresponding category Yi is used as a state point. Unlike CRF, each state point is related to all state points in the image, such as Figure 4 shown.

[0050] The corresponding principle is shown in formula (5), (6), and (7), where I is each pixel point and observation sequence in the image, and X is the predicted label and state sequence corresponding to the observation point. In formula (5), E(X|I) is a potential energy function, which consists of a univariate potential energy and a binary potential energy. The univariate potential energy function ψu(Xi) is the probability that each pixel i corresponds to the relative category Xi. In the neural network, it can be calculated by the last layer of softmax activation function. The binary potential energy function ψp(Xi,Yj) calculates the probability that two pixels correspond to the same category. In formula (7), u(xi,yj) is a label compatibility term. In order to reduce the amount of FullCRF calculation, the operation between pixels corresponding to non-same categories is reduced. KG is a feature function that calculates the similarity probability of two pixels through the color difference and relative position difference between the two pixels, and w is the weight corresponding to the feature function.

[0051]

[0052]

[0053]

[0054] In FullCRF, since the label compatibility term limits the similarity probability operation between some pixels, the binary potential energy function is generally calculated through multiple inferences to complete the prediction of each pixel.

[0055] Based on the understanding of the basic model structures of Unet, CBAM, CRF, and FullCRF, Figure 5As shown, the symmetrical Encoder-Decoder structure adopted by Unet enables the model to fuse feature maps of the same size during upsampling, and the abstract feature information obtained is conducive to the segmentation and prediction of building contours. However, the VGG architecture structure adopted by traditional Unet is relatively shallow, and it is unable to obtain rich abstract features, resulting in the phenomenon of missed recognition and misdetection of building segmentation by Unet. Therefore, the present invention introduces the residual structure in ResNet (Residual Network), deepens the Unet neural network structure, and reduces the occurrence of gradient vanishing through the residual structure.

[0056] In order to fit the real outline of the building to the greatest extent and improve the model's segmentation accuracy for the building, the present invention introduces the CBAM convolutional attention module in the first two layers of encoding and the last two layers of decoding in Unet. The main features are extracted when the neural network obtains the feature map with a lower receptive field, which not only reduces the cases of missed recognition and misdetection, but also makes the model's prediction of the building edge more precise.

[0057] The present invention replaces the ordinary convolution in Unet with Residual_Block, deepens the neural network structure, and enriches the abstract feature extraction. At the same time, since the pooling layer loses feature spatial information and expands the feature receptive field, the neural network is relatively rough for edge detection. Therefore, the MaxPooling layer in the traditional Unet is replaced with an ordinary convolution layer to reduce the loss of spatial information, making the neural network more precise in detecting the edge of the building.

[0058] Figure 5 In the representation, the CBAM module is embedded into the Residual_Block of the first two encoding layers and the last two decoding layers to obtain the main features in the low receptive field feature stage, thereby improving the model's detailed segmentation of the entire building including the building outline.

[0059] FullCRF has always been one of the classic algorithms for refining segmentation results, and has also been used in traditional deeplabV1 and V2 neural networks. However, due to the large amount of calculation required for fully connected conditional random fields, even if the calculation of binary potential energy between some pixels is restricted by label compatibility terms, the calculation area is still all pixels in the entire image. After multiple inference calculations, as the predictions of the labels corresponding to some pixels change, it will still take a lot of computing time, resulting in the model's inability to predict images in real time.

[0060] The present invention introduces the ConvCRF improved based on FullCRF as the image segmentation post-processing model. Figure 6As shown, it is different from FullCRF in that it borrows the kernel-size in standard convolution and pooling, introduces conditional restrictions on the basis of FullCRF, and sets a given kernel-size so that the calculation range between pixels is limited to the kernel-size range centered on it, which greatly reduces the amount of calculation and time. At the same time, the present invention uses the pytorch framework to place all parameters in ConvCRF in the GPU for gradient training, thereby achieving the effect of real-time detection and segmentation without losing too much performance.

[0061] The present invention will be further described below by means of specific examples.

[0062] Experimental data set and hyperparameter selection of the present invention: The present invention uses the Inria_Aerial_Image_Labeling_Dataset remote sensing image urban building detection data set released by the French Institute in 2017. Each image in the data set has a resolution of 5000*5000 pixels, and the labels are divided into buildings and non-buildings, and the corresponding pixel values ​​are set to 255 and 0, such as Figure 7 shown.

[0063] During the neural network training process, due to the limitation of memory and video memory, it is impossible to directly input a whole remote sensing image with too large a resolution. Therefore, the present invention cuts a large remote sensing image with a resolution of 5000*5000 pixels into 441 chunks with a resolution of 256*256 pixels. Since the chunks obtained by cutting will have gaps during the splicing process, which affects the visibility of the interpretation results, the present invention proposes to generate a 10% coverage rate between the chunks during the cutting process to eliminate the gaps between the chunks when the interpretation results are spliced.

[0064] The present invention selects 2 remote sensing images as the training set of the model, 1 remote sensing image as the validation set, and 1 remote sensing image as the test set. During the data preparation process, the training data set is enhanced by 1:4, so the ratio of the number of training sets to validation sets and test sets in this embodiment is 8:1:1.

[0065] like Figure 8 As shown, the present invention uses traditional data enhancement (data rotation, data flipping, data noise addition) methods to enhance the training data set (such as Figure 8 As shown in the figure, the training data set is enhanced with a data enhancement ratio of 1:4. The enhancement methods include: image flipping, symmetrically flipping the original image with the midline as the axis; image rotation, rotating the original image 90° clockwise and counterclockwise; data noise addition, randomly adding noise to the original image at a pixel ratio of 50%.

[0066] The hyperparameter settings for the neural network model training experiment in this embodiment are shown in Table 1. The present invention uses torchvision.transforms in pytorch to normalize the input data. Normalize_Mean represents the mean of each channel, and Normalize_Std represents the standard deviation of each channel. In the present invention, the mean and standard deviation are set to the mean and standard deviation obtained by sampling in the imageNet training set;

[0067] Paramter Batch_Size Base_Lr Normalize_Mean Normalize_Std Value 8 0.01 (0.485,0.456,0.426) (0.229,0.224,0.225)

[0068] Table 1. Hyperparameter selection strategy for neural network model training.

[0069] Due to the influence of the equipment, the Batch_Size value in the present invention is set to 8; Epoch represents the number of iterations of the entire training set training, and the present invention sets Epoch to 200 times; the loss function plays a vital role in the model training process. During the training process, the model calculates the loss function for back propagation, and uses the gradient to find the optimal solution of each parameter of the model to reduce the value of the loss function, thereby fitting the characteristic distribution function of the data. The present invention uses a multi-classification cross entropy loss function. The cross entropy represents the chaos and uncertainty of the distribution of two information. In the calculation process of the cross entropy, the larger the value of a certain point, the larger its gradient at that point. Therefore, it can quickly achieve gradient descent as a loss function for model training. Its mathematical form is shown in formula (8):

[0070]

[0071] Based on the adaptive gradient descent algorithm Adam, the present invention adopts the StepLR interval adjustment learning rate attenuation strategy to adjust the learning rate, wherein the Step Size is set to 20 and the Gamma is set to 0.1, that is, the learning rate is adjusted by 0.1 times every 20 epochs.

[0072] Based on the improved Unet training model, the present invention performs a separate 100 epoch model training on the ConvCRF algorithm, and its hyperparameter settings are shown in Table 2:

[0073] Paramter Filter_Size Blur Inference Value 7 4 5

[0074] Table 2 ConvCRF algorithm model training hyperparameters.

[0075] In order to ensure the prediction accuracy of ConvCRF for images and reduce the operation time of ConvCRF, the present invention sets Filter_Size, i.e., Kernel_Size, to 7, Blur coefficient to 4, and Inference number to 5 in ConvCRF model training. This ensures the segmentation accuracy of the model while meeting the requirements of real-time segmentation.

[0076] The present invention uses Mean Intersection over Union (MIOU) and Mean Pixels Accuracy (MPA) to evaluate the segmentation effect of the model on the data set. Assume that there are n+1 categories, p ij It represents the number of pixels whose actual category is i but predicted to be j. The specific calculation formulas of MIOU and MPA are as follows:

[0077]

[0078]

[0079] From the formula of average intersection-union ratio (9), we can see that the calculation method of average intersection-union ratio is equivalent to the calculation method of set intersection-union ratio in mathematics. Its meaning is the intersection of building prediction results and corresponding labels divided by the union. When the prediction results completely correspond to the corresponding labels, the intersection is equal to the union, and its intersection-union ratio value becomes 1. In model evaluation, the larger the value of intersection-union ratio, the higher the accuracy of the model in building recognition.

[0080] From the formula of average pixel accuracy - formula (10), we can see that the average pixel accuracy is calculated by dividing the number of accurately identified pixels by the number of overall identified pixels. This means that the number of pixels accurately predicted for a building is divided by the total number of pixels predicted for that building. When the building prediction result is completely accurate, the number of accurate pixels is equal to the total number of pixels, that is, the calculated value of the pixel accuracy is 1. In model evaluation, the larger the value of the pixel accuracy, the higher the accuracy of the model in identifying buildings.

[0081] Model Training

[0082] like Fig. 9As shown, the training accuracy loss diagrams of Unet, PSP, improved Unet and separated ConvCRF models are respectively shown. Given the same hyperparameter selection strategy, the present invention performs model training for Unet, PSP and improved Unet neural networks for 200 epochs. Based on the improved Unet model, ConvCRF is trained for 100 epochs alone. The accuracy loss diagram shows that each neural network starts fitting at about 30 epochs, among which the improved Unet has the most stable fitting degree on the validation set, and the average intersection-over-union ratio and average pixel accuracy in the training set reach 0.945 and 0.987 respectively, and the average intersection-over-union ratio and average pixel accuracy in the validation set reach 0.883 and 0.939 respectively, with the highest accuracy among all models.

[0083] Since the calculation of ConvCRF includes a univariate potential function, that is, the probability of a pixel corresponding to a pixel category, its input must be the predicted probability of the segmentation model for the image and the original segmented image, so the invention adopts model separation training, that is, the ConvCRF model is trained on the basis of the improved Unet training model. It can be seen from the ConvCRF model accuracy loss graph that ConvCRF begins to fit at about the 7th epoch, and the stable accuracy after fitting approaches the model accuracy of the improved Unet. The average intersection-over-union ratio and average accuracy of ConvCRF in the training set are 0.947 and 0.987, and the average intersection-over-union ratio and average pixel accuracy in the validation set are 0.886 and 0.941 respectively.

[0084] In order to verify the effectiveness of the accuracy improvement of the improved network based on Unet, the present invention performs model training on traditional Unet, PSP and other neural networks on the Inria Aeria BuildingDataset data set, and compares the verification accuracy of each model on the test set. It can be seen from the model accuracy comparison table in Table 3 that the improved Unet neural network proposed in the present invention has the highest accuracy for building recognition, with an average intersection-over-union ratio of 0.8874, a pixel accuracy of 0.9483, and an F1 score of 0.9064. For single-class intersection-over-union of building recognition, the building single-class F1 score reaches 0.8071, the building single-class F1 score reaches 0.8337, and the building single-class pixel accuracy reaches 0.9115. The verification accuracy of the separated ConvCRF model on the test set does not change much, similar to the improved Unet, and its main purpose is to refine the edge detection of the improved Unet segmentation result and reduce jagged edges.

[0085]

[0086] Table 3 Comparison of the accuracy of each model.

[0087] like Fig.10As shown, the verification results of each model on the test set are shown. Fig.10 In the figure, the Unet and PSP models have a rough segmentation of the building edges, and Unet and PSP have missed recognition and misdetection of small buildings, as shown in the red box. Compared with the traditional Unet and PSP neural networks, ResUnet with the CBAM mechanism has a more detailed segmentation of the building edges, and can more accurately detect small buildings with remote locations and small areas. ConvCRF further refines the edges of buildings based on the improved Unet. As shown in the blue box, ConvCRF has not shown a significant improvement in verification accuracy, but it can achieve the refinement of building edge detection in the verification effect in the test set, while reducing the existence of jagged edges on the buildings.

[0088] like Fig.11 As shown in the figure, the model prediction result diagram shows the prediction results of each model for the entire building remote sensing image test set. Limited by the large resolution of the remote sensing image, insufficient GPU memory, and the inability to send the entire image to the model for prediction, the neural network model cannot obtain the global information of the remote sensing image. The cropped chunk image has the problem of not being able to accommodate the complete building, and the model cannot obtain complete semantic information for the recognition of some large categories of buildings. At the same time, in the remote sensing image, some buildings are blocked by other types of objects such as trees and bridges, which makes the model's prediction results for buildings have many gaps. However, by comparing the prediction results of each model on the test set, it can be seen that the model obtained by training the ResUnet neural network with the introduction of CBAM proposed in the present invention has more precise building edge recognition in the test set prediction. Even if there is a problem of occlusion by other objects, for the unoccluded part, the points on the building outline are sparser, and the recognition of the building boundary line is more delicate. On this basis, the introduction of the ConvCRF model further refines the segmentation effect of building edge detection, eliminates some noise points, and reduces the occurrence of missed detection and misrecognition of buildings by the model. This can further meet the application needs of building segmentation results and reduce the secondary labeling labor required for revising building detection results.

[0089] As shown in Table 4, it shows the time taken by each model to interpret a single 256*256 pixel chunk image. Among them, the traditional Unet neural network takes the shortest time due to its shallow network structure, and the interpretation time for a single chunk image reaches 0.02343 seconds per picture; PSP50 introduces a residual neural network and introduces a pyramid pooling layer at the end of the encoding stage. Therefore, the PSP neural network is higher than the traditional Unet neural network, and it takes 0.02809 seconds per picture to interpret a single chunk image3; the CBAM_ResUnet neural network improved based on Unet introduces the CBAM attention mechanism on the basis of replacing the residual structure, so its parameter volume is larger than PSP50, and the interpretation time for a single chunk image reaches 0.02944 seconds per picture; the improved Unet of the concatenated separated ConvCRF model takes 0.03407 seconds per picture to interpret a single chunk image, which takes 0.00463 seconds per picture more than the improved Unet, that is, the interpretation time for a single chunk image is 15% more. Meet the needs for real-time segmentation of buildings.

[0090]

[0091] Table 4: The time taken by each model to interpret a single 256*256 pixel chunk image.

[0092] According to the defects of the traditional Unet neural network, the present invention proposes a CBAM-ResUnet neural network based on Unet improvement. The purpose of the improvement of the network is mainly to improve the model accuracy and extract the main features of the low receptive field area by deepening the network structure to achieve fine segmentation of the building edge. The introduction of the Residual Block allows the improved network to avoid the gradient disappearance while deepening the structure; the introduction of the CBAM module allows the improved network to extract the main abstract features of the building at the stage of large spatial size of the feature map, thereby retaining more spatial information and realizing the feature extraction of the building edge, meeting the demand for fine segmentation of the building edge. The ConvCRF separation model improved based on FullCRF greatly improves the interpretation speed without excessive loss of accuracy, thereby realizing real-time segmentation of the model and reducing the generation of noise and edge jaggedness. Experiments have shown that the improved Unet network is more precise for the edge segmentation of the building, and reduces the occurrence of missed recognition and misdetection; the introduction of the ConvCRF separation model further refines the segmentation effect of the building edge without increasing a large number of parameters and interpretation time.

[0093] What needs to be explained about the specific structure of the present invention is that the connection relationship between the various component modules adopted in the present invention is definite and feasible. Except for the special instructions in the embodiments, the specific connection relationship can bring about the corresponding technical effects, and solve the technical problems raised by the present invention without relying on the execution of the corresponding software program. Except for the specific instructions, the models and connection methods of the components, modules and specific components appearing in the present invention belong to the existing technologies such as published patents, published journal articles, or common knowledge that can be obtained by technical personnel in this field before the application date, and there is no need to elaborate, so that the technical solution provided in this case is clear, complete and feasible, and the corresponding physical products can be reproduced or obtained according to the technical means.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A ConvCRF-based high-resolution remote sensing image building extraction optimization method, characterized by: The steps include: S1: Construct a full convolutional neural network basic segmentation model, train the basic model for multiple rounds on high-resolution remote sensing image data, obtain a segmentation model that fits the feature distribution of high-resolution remote sensing image data, replace the ordinary convolution in the basic Unet with Residual_Block, replace the MaxPooling layer in the basic Unet with an ordinary convolution layer, and embed the CBAM module into the Residual_Block in the first two layers of encoding and the last two layers of decoding to obtain the improved Unet neural network model; S2: Constructing image segmentation post-processing model: Introducing ConvCRF end-to-end connection model, performing secondary correction optimization on the segmentation results of the basic segmentation model, using ConvCRF improved based on FullCRF as image segmentation post-processing model, introducing conditional restrictions on the basis of FullCRF, and setting a given kernel-size so that the calculation range between pixels is limited to the kernel-size range centered on it; S3: training the neural network model and image segmentation post-processing model; S4: The average intersection-over-union (MIOU) and average pixel MPA are used to evaluate the segmentation effect of the model on the dataset.

2. The method for extracting buildings from high-resolution remote sensing images based on ConvCRF according to claim 1, characterized in that: Before model training, the remote sensing image dataset is obtained and preprocessed, and the dataset is divided into training set, validation set, and test set in proportion; the Inria_Aerial_Image_Labeling_Dataset remote sensing image urban building detection dataset is used. Each image in the dataset has a resolution of 5000*5000 pixels, and the labels are divided into buildings and non-buildings; During the neural network training process, the large remote sensing image with a resolution of 5000*5000 pixels is cropped, and a 10% coverage rate is generated between the chunk images during the cropping process.

3. The method for extracting and optimizing buildings from high-resolution remote sensing images based on ConvCRF according to claim 1, characterized in that: The training of the neural network model and the image segmentation post-processing model in step S3 specifically includes: The torchvision.transforms in pytorch is used to normalize the input data, and the multi-classification cross entropy loss function is selected as the loss function for model training. Based on the adaptive gradient descent algorithm Adam, the StepLR interval adjustment learning rate decay strategy is used to adjust the learning rate. Based on the basic segmentation model, the ConvCRF algorithm is trained iteratively in a separated manner.

4. The method for extracting and optimizing buildings from high-resolution remote sensing images based on ConvCRF according to claim 1, characterized in that: The learning rate is adjusted using the StepLR interval adjustment learning rate decay strategy.

5. The method for extracting buildings from high-resolution remote sensing images based on ConvCRF according to claim 1, characterized in that: The average intersection-over-union (MIOU) and average pixel MPA are used to evaluate the segmentation effect of the model on the dataset. Assume there are n+1 categories, p ij It represents the number of pixels whose actual category is i but predicted to be j. The specific calculation formulas of MIOU and MPA are as follows:

6. The method for extracting and optimizing buildings from high-resolution remote sensing images based on ConvCRF according to claim 2, characterized in that: The data in the training set are enhanced at a data enhancement ratio of 1:

4. The enhancement methods include: image flipping, symmetrically flipping the original image with the midline as the axis; image rotation, rotating the original image 90° clockwise and counterclockwise; data noise addition, randomly adding noise to the original image at a pixel ratio of 50%.

7. The method for extracting and optimizing buildings from high-resolution remote sensing images based on ConvCRF according to claim 1, characterized in that: Through the pytorch framework, all parameters in ConvCRF are placed in the GPU for training with gradients.