Building vector extraction model and extraction method based on deep learning

Abstract

The application discloses a kind of building vector extraction model and extraction method based on deep learning, comprising: S1.receiving remote sensing image, carries out multiple stages of feature extraction to remote sensing image, obtains multiple scale extraction features;S2.respectively to each scale extraction feature is optimized, obtains multiple scale optimization features;S3.multiple scale optimization features are fused, and obtain fusion features;S4.to fusion features are carried out feature recovery and class judgment and obtain building preliminary extraction result;S5.to building preliminary extraction result is post-processed and obtains the final building vector result.This scheme fuses the feature extraction advantage under multiple scales, effectively improves feature extraction capability, enhances the information extraction capability of network to original image, and is extracted to fusion features, post-processing is carried out to extraction result, can directly obtain relatively regular building vector result, greatly facilitates subsequent editing and application.

Description

Technical Field

[0001] The present invention belongs to the field of building extraction, and in particular relates to a building vector extraction model based on deep learning and an extraction method thereof. Background Art

[0002] Building extraction based on remote sensing images is a technique that analyzes the spectral and textural features of remote sensing images and determines the pixel-level semantic categories of the images. Common traditional classification methods mainly include methods based on the overall characteristics of buildings, methods based on auxiliary information, and object-oriented building extraction methods. Methods based on the overall characteristics of buildings include corner detection, line grouping, building index, and other methods that extract buildings based on the structural characteristics of the buildings themselves and combine empirical knowledge. Methods based on auxiliary information use auxiliary information such as digital surface models (DSMs) and building shadows to extract buildings. The most widely used is the object-oriented building extraction method, which segments remote sensing images into multiple clusters, calculates the spectral, textural, and morphological characteristics of the clusters, and uses classification methods such as Bayesian discriminant methods, decision trees, and support vector machines to extract buildings.

[0003] These traditional methods are subject to significant subjective factors, requiring manual adjustments to both the segmentation scale and feature selection based on individual images. This underutilizes the image data's unique characteristics, resulting in poor classification robustness. Compared to traditional algorithms, deep learning models utilize multiple layers of convolution and activation functions to mimic the neural architecture of the human brain, automatically learning the geometric structure, spectral, and texture characteristics of ground objects from large amounts of data. This reduces the influence of subjective factors on the results. Therefore, the use of deep learning technology in building extraction has become a new trend. While deep learning-based remote sensing image classification has been proposed, current methods are suboptimal in extracting and restoring shallow features, and further improvement is needed. Furthermore, convolutional neural networks for building extraction only produce raster extraction results, which cannot be subsequently edited, hindering the subsequent application of the extracted results. Summary of the Invention

[0004] The purpose of the present invention is to provide a building vector extraction model and extraction method based on deep learning to address the above problems.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A building vector extraction method based on deep learning, comprising the following steps:

[0007] S1. Receive remote sensing images, perform feature extraction on the remote sensing images at multiple stages, and obtain extracted features at multiple scales;

[0008] S2. Optimize the extracted features at each scale to obtain optimized features at multiple scales;

[0009] S3. Fusing the optimized features at multiple scales to obtain fused features;

[0010] S4. Perform feature recovery and category judgment on the fused features to obtain preliminary building extraction results;

[0011] S5. Post-process the preliminary building extraction results to obtain the final building vector results.

[0012] In the above-mentioned building vector extraction method based on deep learning, in step S1, the remote sensing image is a pre-processed high-resolution remote sensing image;

[0013] In step S1, at least three stages of feature extraction are performed on the remote sensing image to obtain extracted features at at least three scales;

[0014] In step S1, an encoder module is used to perform multiple stages of feature extraction on the remote sensing image, and the encoder module includes a dense convolution module and a transition convolution module, and feature extraction at each stage is performed by any one or both of the dense convolution module and the transition convolution module.

[0015] In the above-mentioned building vector extraction method based on deep learning, in step S1, four stages of feature extraction are performed on the remote sensing image to obtain extracted features at four scales;

[0016] In step S1, the first three stages are all performed by the dense convolution module and the transition convolution module for feature extraction, and the transition convolution module is executed once in each stage. The dense convolution module is repeated 6 times, 12 times, and 24 times in the first to third stages, respectively. The fourth stage is repeated 16 times by the dense convolution module for feature extraction.

[0017] In the above-mentioned deep learning-based building vector extraction method, in step S2, any one of the spatial pyramid pooling module and the self-attention module is used to optimize the extracted features of each stage.

[0018] In the above-mentioned deep learning-based building vector extraction method, in step S2, the spatial pyramid pooling module is used to optimize the features extracted in the fourth stage, and the self-attention module is used to optimize the features extracted in the first to third stages respectively;

[0019] In step S3, the optimized features of the fourth stage and the third stage are first fused, and then fused with the optimized features of the second stage and the first stage in sequence to obtain the final fused features.

[0020] In the above-mentioned building vector extraction method based on deep learning, step S4 specifically includes:

[0021] Use two depth-separated convolution modules, two dense upsampling modules and one convolution module to restore the final fusion features;

[0022] Use the softmax function to classify the restored features and obtain the preliminary building extraction results;

[0023] In step S5, the post-processing steps include four post-processing steps in sequence: removing small cluster areas, morphological operations, boundary point extraction, and Douglas Peucker thinning.

[0024] A building vector extraction model based on deep learning includes an encoder module, an optimization module, a decoder module and a post-processing module. The encoder module is used to perform multi-stage feature extraction on an input remote sensing image. The optimization module is used to optimize the extracted features of each scale output by the encoder module. The decoder module is used to fuse the optimized optimized features of each scale and perform feature recovery and category judgment based on the fused features to output a preliminary building extraction result. The post-processing module is used to post-process the preliminary building extraction result to obtain a final building vector result.

[0025] In the above-mentioned deep learning-based building vector extraction model, the encoder module includes a dense convolution module and a transition convolution module; the dense convolution module includes two convolution modules, and the dense convolution module is used to process the input through the two convolution modules in sequence, and then splice the input with the output band of the convolution module and output it;

[0026] The transition convolution module includes a convolution module and an average pooling layer. The transition convolution module is used to take the output of the dense convolution module as input, perform convolution and pooling on the input, and then output it.

[0027] In the above-mentioned deep learning-based building vector extraction model, the optimization module includes a self-attention module and a spatial pyramid pooling module, and the output of each stage is connected to a self-attention module or a spatial pyramid pooling module;

[0028] The decoder module includes multiple decoding convolution modules. The output of each self-attention module is connected to a decoding convolution module. The decoding convolution module is used to perform decoding convolution processing on the output of the next stage decoding convolution module / spatial pyramid pooling module, and then fuse and superimpose the output with the output of the corresponding self-attention module to output to the decoding convolution module corresponding to the previous stage. The final fusion feature is output by the decoding convolution module corresponding to the first stage.

[0029] This model also includes a branch structure for network training, and the branch structure includes multiple branch modules, and each decoding convolution module is connected to a branch module.

[0030] In the above-mentioned deep learning-based building vector extraction model, the encoder module is used to perform four-stage feature extraction on the input remote sensing image. The first three stages are performed by the dense convolution module and the transition convolution module, and the fourth stage is performed by the dense convolution module.

[0031] The optimization module includes three self-attention modules and a spatial pyramid pooling module. The inputs of the three self-attention modules are the extracted features of the first to third stages respectively, and the outputs are the inputs of the three decoding convolution modules respectively. The input of the spatial pyramid pooling module is the extracted features of the fourth stage;

[0032] The spatial pyramid pooling module performs four convolution operations and one global pooling on the features extracted in the fourth stage, concatenates the five features, and fuses them using a 1*1 convolution for output.

[0033] The decoding convolution module in the third stage performs decoding convolution processing on the output of the spatial pyramid pooling module, and then fuses and superimposes the optimized features output by the self-attention module in the third stage and outputs them to the corresponding decoding convolution module in the second stage;

[0034] The decoding convolution module of the second stage performs decoding convolution processing on the output of the decoding convolution module of the third stage, fuses and superimposes it with the optimized features output by the self-attention module of the second stage, and outputs it to the decoding convolution module of the first stage;

[0035] The decoding convolution module in the first stage performs decoding convolution processing on the output of the decoding convolution module in the second stage, and then fuses and superimposes the optimized features output by the self-attention module in the first stage to output the final fused features;

[0036] The decoding convolution process includes two depth separation convolution operations and one dense upsampling operation;

[0037] The post-processing module is used to perform four post-processing steps on the preliminary building extraction results, including removing small cluster areas, morphological operations, boundary point extraction and Douglas Peucker thinning, to obtain the final building vector results.

[0038] The advantages of the present invention are:

[0039] 1. This solution first performs multi-stage feature extraction on remote sensing images to obtain extracted features at multiple scales. It then optimizes the features at each scale and then performs feature fusion. This can integrate the advantages of feature extraction at multiple scales, effectively improve feature extraction capabilities, ensure the effectiveness of fused features, and improve subsequent feature recovery capabilities.

[0040] 2. Dense convolutional neural networks use self-attention modules and spatial pyramid pooling modules to optimize features and enhance the network's ability to extract information from original images;

[0041] 3. Finally, this solution extracts the results based on the fusion features and performs post-processing on the extracted results. This allows for direct acquisition of relatively regular building vector results, greatly facilitating subsequent editing and application.

[0042] 4. This proposal proposes a new dense convolutional neural network based on a dense convolutional module. This network is suitable for image classification tasks and has strong feature extraction capabilities, and can automatically extract building feature categories.

[0043] 5. Setting up a branch structure for auxiliary training for the network structure can improve the early training effect of the network, thereby improving the network's feature extraction ability;

[0044] 6. In the encoder module, this solution extracts features at each scale by superimposing dense convolution modules and transition convolution modules, optimizes the features extracted at each scale, and then fuses the optimized features through a series of convolution operations and superimposes them. The fused features are subjected to preliminary extraction of results including dense upsampling, thus ensuring the network's ability to recover image information from the overall process. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a model architecture diagram of the building vector extraction model of the present invention;

[0046] Figure 2 Schematic diagram of the structure of the dense convolution module in the building vector extraction model of the present invention;

[0047] Figure 3 Schematic diagram of the structure of the transition convolution module in the building vector extraction model of the present invention;

[0048] Figure 4 Schematic diagram of the structure of the self-attention module in the building vector extraction model of the present invention;

[0049] Figure 5 Schematic diagram of the structure of the spatial pyramid pooling module in the building vector extraction model of the present invention;

[0050] Figure 6 Schematic diagram of the structure of the depth separation convolution module in the building vector extraction model of the present invention;

[0051] Figure 7 Schematic diagram of the structure of the dense upsampling module in the building vector extraction model of the present invention;

[0052] Figure 8 Schematic diagram of the structure of the decoding convolution module corresponding to the second stage of this embodiment in the building vector extraction model of the present invention;

[0053] Figure 9 It is a structural diagram of the post-processing module in the building vector extraction model of the present invention. DETAILED DESCRIPTION

[0054] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] like Figure 1 As shown, this embodiment discloses a new building vector extraction model based on a dense convolutional neural network, including an encoder module 2, an optimization module 3, a decoder module 4 and a post-processing module 5. The method for extracting building vectors based on this model includes:

[0056] S1. The preprocessing module 1 performs preprocessing on the high-resolution remote sensing image, including remote sensing image cropping, data enhancement, and normalization. The encoder module 2 receives the preprocessed remote sensing image and performs feature extraction on the remote sensing image at multiple stages to obtain extracted features at multiple scales, outputting extracted features at each scale at each stage.

[0057] S2. The optimization module 3 performs feature optimization on the extracted features of each scale to obtain optimized features of multiple scales;

[0058] S3. The decoder module 4 performs feature fusion on the optimized features of multiple scales to obtain fused features;

[0059] S4. The decoder module 4 continues to perform feature recovery and category judgment on the fusion feature to obtain the preliminary extraction result of the building - output 1;

[0060] S5. The post-processing module 5 performs post-processing on the preliminary building extraction results to obtain the final building vector results.

[0061] Specifically, in step S1, feature extraction is performed on the remote sensing image at least three stages to obtain extracted features at at least three scales. In this embodiment, feature extraction is preferably performed on the remote sensing image at four stages to obtain extracted features at four scales.

[0062] This solution first extracts features from remote sensing images at multiple stages to obtain features at multiple scales. Features at each scale are then optimized and fused. This approach leverages the advantages of feature extraction at multiple scales, effectively improving feature extraction capabilities while ensuring the validity of the fused features and enhancing subsequent feature recovery. Finally, post-processing the fused feature extraction results directly yields relatively regular building vectors, greatly facilitating subsequent editing and application.

[0063] Furthermore, the building vector extraction model requires network training before it is put into use. This model further uses a branch structure 41 to assist the main structure in training to improve the training effect of the model. Specifically, the branch structure 41 includes N-1 branch modules, where N represents the number of stages. Specifically, in this embodiment, there are four stages, so there are 3 branch modules. The first three stages correspond to a branch module respectively. Each branch module includes a depth separation convolution module, a convolution module and a dense upsampling module. The output result size is 1 / 2 (output 2), 1 / 4 (output 3) and 1 / 8 (output 4) of the original input image size.

[0064] Specifically, the encoder module 2 includes a dense convolution module and a transition convolution module, such as Figure 2 As shown in , the dense convolution module includes two convolution modules, with convolution kernel sizes of 1*1 and 3*3 respectively, and a step size of 1. The input and output bands of the convolution module are spliced ​​and then output. Figure 3 As shown, the transition convolution module includes 1 convolution module and an average pooling layer, the convolution kernel size is 1*1, the step size is 1, the average pooling layer window size is 2*2, and the output feature space size is 1 / 2 of the input feature. Feature extraction at each stage is performed by any one or two of the dense convolution module and the transition convolution module. Specifically in this embodiment, each of the first three stages is composed of a dense convolution module and a transition convolution module, wherein the dense convolution module is repeated 6, 12, and 24 times respectively, the transition convolution module is only used once in each stage, and the last stage is repeated 16 times by the dense convolution module, and finally four scales of extracted features are obtained, and the feature map sizes output by the third and fourth stages are the same, so the encoder actually outputs three scales of extracted features.

[0065] Specifically, optimization module 3 includes a self-attention module and a spatial pyramid pooling module, and the output of each stage is connected to a self-attention module or a spatial pyramid pooling module, so as to optimize the extracted features of the corresponding stage with a self-attention module or a spatial pyramid pooling module respectively. In the present embodiment, optimization module 3 includes three self-attention modules and a spatial pyramid pooling module, and the inputs of the three attention modules are respectively the extracted features of the first-third stage, and the input of the spatial pyramid pooling module is the extracted features of the fourth stage.

[0066] like Figure 4 As shown in , the self-attention module adopts the space-reduced attention module, which can capture the internal correlation of data or features and reduce model resource consumption. Figure 5 As shown in the figure, the spatial pyramid pooling module performs four convolution operations and one global pooling on the output of the fourth stage, the final stage of encoder module 2. The non-dilated convolution kernel size is 1*1 and the stride is 1; the dilated convolution kernel size is 3*3 and the stride is 1, with three dilation rates: 6, 12, and 18. The five features are concatenated and fused using a 1*1 convolution before being output.

[0067] Specifically, the decoder module 4 includes multiple decoding convolution modules and basic component modules for feature recovery and category judgment of fused features. The basic component modules include a depth separation convolution module and a dense upsampling module. The upsampling process uses a dense upsampling module. The downsampling process in various convolutional neural networks will inevitably cause the loss of detail information, and the dense upsampling module compensates for the loss in length and width through the channel dimension to restore image information. Figure 6 As shown in , the depth separation convolution module consists of a (depth separation) convolution layer, a normalization layer (Batch Normalization, BN) and an activation layer (Rectified Linear Unit, ReLU). Figure 7 As shown, the dense upsampling module consists of a convolution module and a periodic filtering layer, where the convolution kernel size in the convolution module is 1*1 and the step size is 1. The periodic filtering layer reorders the features, reducing the number of feature bands and increasing the spatial size (length and width) of the features. Accordingly, step S4 specifically includes: using two depth-wise separation convolution modules, two dense upsampling modules, and one convolution module to restore the fused features to the size of the original input image; using the softmax function to classify the restored features to obtain the preliminary building extraction results.

[0068] like Figure 8As shown in the figure, the decoding convolution module consists of two depthwise separable convolution modules, one dense upsampling module, and one stacking layer. The main input is the output of the decoding convolution module / spatial pyramid pooling module in the next stage. The depthwise separable convolution is a channel-by-channel convolution with a 3*3 kernel and a stride of 1.

[0069] Furthermore, the output of each self-attention module is connected to a decoding convolution module, which is used to perform two depth-separated convolution operations and one dense upsampling operation on the output of the next stage decoding convolution module / spatial pyramid pooling module, and then fuse and superimpose the optimized features output by the self-attention module to the decoding convolution module of the previous stage. The final fusion feature is output by the decoding convolution module corresponding to the first stage.

[0070] Since this embodiment adopts four-stage feature extraction, in step S2, the spatial pyramid pooling module is used to optimize the features extracted in the fourth stage, and the self-attention module is used to optimize the features extracted in the first to third stages respectively; in step S3, the optimized features of the fourth stage and the third stage are first fused, and then fused with the optimized features of the second stage and the first stage in sequence to obtain the final fused features.

[0071] Specifically, the spatial pyramid pooling module performs four convolution operations and one global pooling on the features extracted in the fourth stage, concatenates the five features, and fuses them using a 1*1 convolution for output.

[0072] The decoding convolution module in the third stage performs two depth-separated convolution operations and one dense upsampling operation on the optimized features output by the self-attention module in the third stage, and then fuses and superimposes the output with the output of the spatial pyramid pooling module and outputs it to the corresponding decoding convolution module in the second stage;

[0073] The decoding convolution module of the second stage performs two depth separation convolution operations and one dense upsampling operation on the optimized features output by the self-attention module of the second stage, and then fuses and superimposes the output with the output of the decoding convolution module of the third stage and outputs it to the decoding convolution module of the first stage;

[0074] The decoding convolution module of the first stage performs two depth separation convolution operations and one dense upsampling operation on the optimized features output by the self-attention module of the first stage, and then fuses and superimposes the output of the decoding convolution module of the second stage to output the final fused features.

[0075] Furthermore, if Figure 9As shown, the post-processing module 5 is used to perform four post-processing steps on the preliminary extraction results of the building, including removing small cluster areas, morphological operations, boundary point extraction, and Douglas Peuker thinning, to obtain the final building vector result. The extraction result of the convolutional neural network is a raster image, and the building boundary has a certain difference from the true value. This solution uses the post-processing module to optimize the extraction results. The specific steps are to remove cluster areas with an area less than the threshold, which can reduce the obvious classification error in the result. The opening operation can remove isolated small points, burrs and small bridges, and the closing operation can fill small lakes (i.e., small holes) and bridge small cracks. Both morphological operations keep the overall position and shape unchanged. The contour extraction algorithm is used to convert the raster result into a vector result. There are many boundary points in this result, and the overall shape of the building is irregular. The Douglas Peuker thinning algorithm is then used to simplify the boundary points, so that the vector result is more consistent with the characteristics of the building, and finally a more regular building vector result is obtained.

[0076] The specific embodiments described herein are merely illustrative of the spirit of the present invention. Persons skilled in the art may make various modifications, additions, or substitutions to the described specific embodiments without departing from the spirit of the present invention or exceeding the scope of the appended claims.

[0077] Although this document frequently uses terms such as preprocessing module, encoder module, optimization module, decoder module, branch structure, post-processing module, depthwise separable convolution module, dense upsampling module, dense convolution module, transition convolution module, self-attention module, spatial pyramid pooling module, decoding convolution module, and post-processing module, the use of other terms is not excluded. These terms are used solely to more conveniently describe and explain the essence of the present invention; interpreting them as any additional limitations would be contrary to the spirit of the present invention.

Claims

1. A method for constructing a building vector extraction model based on deep learning, characterized in that: The invention comprises an encoder module (2), an optimization module (3), a decoder module (4), a post-processing module (5) and a branch structure (41) for network training, wherein the encoder module (2) is used to perform multi-stage feature extraction on an input remote sensing image, the optimization module (3) is used to optimize the extracted features of each scale output by the encoder module (2), the decoder module (4) is used to fuse the optimized optimized features of each scale and perform feature recovery and category judgment based on the fused features to output a preliminary building extraction result, and the post-processing module (5) is used to post-process the preliminary building extraction result to obtain a final building vector result; The encoder module (2) includes a dense convolution module and a transition convolution module, which are used to perform four-stage feature extraction on the input remote sensing image, wherein the dense convolution module and the transition convolution module perform feature extraction in the first three stages, and the dense convolution module performs feature extraction in the fourth stage; The decoder module (4) includes a plurality of decoding convolution modules; The optimization module (3) includes three self-attention modules and a spatial pyramid pooling module, and the output of each self-attention module is connected to a decoding convolution module; The inputs of the three self-attention modules are the extracted features from the first to third stages, and the outputs are the inputs of the three decoding convolution modules. The input of the spatial pyramid pooling module is the extracted features from the fourth stage. The decoding convolution module in the third stage performs decoding convolution processing on the output of the spatial pyramid pooling module, and then fuses and superimposes the optimized features output by the self-attention module in the third stage and outputs them to the corresponding decoding convolution module in the second stage; The decoding convolution module of the second stage performs decoding convolution processing on the output of the decoding convolution module of the third stage, fuses and superimposes it with the optimized features output by the self-attention module of the second stage, and outputs it to the decoding convolution module of the first stage; The decoding convolution module in the first stage performs decoding convolution processing on the output of the decoding convolution module in the second stage, and then fuses and superimposes the optimized features output by the self-attention module in the first stage to output the final fused features; The branch structure (41) includes a plurality of branch modules, and each decoding convolution module is connected to a branch module; The post-processing module (5) is used to perform post-processing steps including removing small cluster areas, opening operation, closing operation, boundary point extraction and Douglas Peucker thinning on the preliminary building extraction results to obtain the final building vector results.

2. The method for constructing a building vector extraction model based on deep learning according to claim 1, characterized in that: The dense convolution module includes two convolution modules, which are used to process the input through the two convolution modules in sequence, and then splice the input with the output band of the convolution module and output it; The transition convolution module includes a convolution module and an average pooling layer. The transition convolution module is used to take the output of the dense convolution module as input, perform convolution and pooling processing on the input, and then output it.

3. The method for constructing a building vector extraction model based on deep learning according to claim 2, wherein: The branch structure (41) includes N-1 branch modules, where N represents the number of stages; Each branch module consists of a depth-wise separable convolution module, a convolution module, and a dense upsampling module, and the output size is 1 / 2, 1 / 4, and 1 / 8 of the original input image size.

4. The method for constructing a building vector extraction model based on deep learning according to claim 3, characterized in that: The decoding convolution process includes two depth-separated convolution operations and one dense upsampling operation.

5. The method for constructing a building vector extraction model based on deep learning according to claim 3, characterized in that: The spatial pyramid pooling module is used to perform four convolution operations and one global pooling on the features extracted in the fourth stage, and then concatenate the five features and fuse them using a 1*1 convolution for output.

Images