Building contour line extraction method based on deep learning and joint optimization

Abstract

The invention discloses a building contour line extraction method based on deep learning and joint optimization, and the method comprises the steps: collecting a multi-view picture of an urban building, and constructing a DOM and a DSM; making a normalized digital surface model nDSM and a slope map, and marking a building contour line and a polygon; constructing a fishing net segmentation grid and a vector graph layer, establishing training sample images and labels, and substituting the training sample images and labels into a line feature extraction network MFLine-Net and a semantic segmentation network Res-UNet for respective training; extracting a line feature by using a line feature extraction network MFLine-Net; building roof semantic information is extracted by using a Res-UNet network; generating a cell cavity complex by using a motion space subdivision algorithm; correlating the semantic information of the cell units and the roof by adopting a graph cut optimization algorithm; and combining the cell units with the same roof semantic information to obtain a building contour map. According to the method, the city building contour map can be rapidly constructed, and the requirement for photogrammetry building drawing based on the unmanned aerial vehicle is met.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and remote sensing science and technology, and in particular to a building contour extraction method based on deep learning and joint optimization. Background Art

[0002] With the rapid development of drone technology, drone photogrammetry is becoming more and more popular due to its low price. As a new type of surveying and mapping technology, drone oblique photogrammetry is equipped with camera sensors to collect images from multiple perspectives simultaneously. Through data processing and analysis, it can generate a real-life three-dimensional model with high sense of reality. Drone oblique photogrammetry has the advantages of high resolution, high precision, large data volume and low cost. The surface model generated by photogrammetric three-dimensional reconstruction can be regarded as a continuous TIN triangulation network with a high-resolution image on the surface. In GIS management and application, this continuous surface model cannot meet the needs of object query and measurement. It can only be browsed as a base map like an image, which limits its further in-depth application. Therefore, it is usually necessary to identify and monomerize building objects.

[0003] In existing production tasks based on drone images, two-dimensional architectural plans are usually obtained by manual sketching, and then three-dimensional architectural models are constructed, which is inefficient. Although deep learning methods have been able to identify building instances with high accuracy, the extracted edges usually need to be regularized, so the identified building outline polygons are far from their true boundaries and cannot meet the needs of accurate architectural mapping. At present, the powerful feature description capabilities of deep neural networks can effectively extract line features of architectural scene images, which can ensure high geometric accuracy and integrity. Summary of the invention

[0004] The purpose of the present invention is to provide a building contour extraction method based on deep learning and joint optimization to meet the needs of building contour extraction based on drone images, improve the work efficiency of building model construction, and make the constructed building model have higher geometric accuracy and integrity.

[0005] To achieve the above functions, the present invention designs a building contour extraction method based on deep learning and joint optimization. For urban buildings, the following steps S1 to S3 are performed to complete the construction of the urban building contour map:

[0006] Step S1: Collect images of urban buildings, construct DOM and DSM of urban buildings; use DSM to make normalized digital surface model nDSM, and generate slope map at the same time; mark the contour lines and polygons of urban buildings based on DOM and slope map; build a fishing net for image cutting, segment raster and vector layers through the fishing net, and establish training sample images and annotations;

[0007] Step S2: pre-process the training sample images and annotations to generate distance and angle field training sample files; substitute the training sample files into the building contour extraction model for training, the building contour extraction model includes the line feature extraction network MFLine-Net and the semantic segmentation network Res-UNet, and obtain a trained building contour extraction model;

[0008] Step S3: Use the line feature extraction network MFLine-Net to extract the line features of urban buildings; then use the semantic segmentation network Res-UNet to extract the roof semantic information of urban buildings; use the motion space subdivision algorithm to generate a cell complex from the extracted line features; use the graph cut optimization algorithm to associate the cell units and roof semantic information in the cell complex; merge the cell units with the same semantic category to obtain the outline polygon of the urban buildings, and complete the construction of the urban building outline map.

[0009] Beneficial effects: Compared with the prior art, the advantages of the present invention include:

[0010] The present invention proposes a building contour extraction method based on deep learning and joint optimization. Compared with the method of simply extracting building contours through semantic segmentation and regularization, the present invention can extract building contour polygons with higher geometric accuracy and completeness, which is of great significance for the reconstruction of building three-dimensional models and can meet the needs of building contour extraction based on drone images. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a flowchart of a building contour extraction method based on deep learning and joint optimization provided according to an embodiment of the present invention;

[0012] Figure 2 is a schematic diagram of the structure of a line feature extraction network MFLine-Net provided according to an embodiment of the present invention;

[0013] Figure 3 is a schematic diagram of line features extracted from drone images provided by an embodiment of the present invention;

[0014] Figure 4 is a schematic diagram of semantic segmentation of building roofs using drone images provided by an embodiment of the present invention;

[0015] Figure 5 is a schematic diagram of obtaining a cell complex by performing motion space segmentation on a line feature according to an embodiment of the present invention;

[0016] Figure 6 The building outline is obtained by jointly optimizing the cell complex and the building roof semantic information provided by the embodiment of the present invention. DETAILED DESCRIPTION

[0017] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0018] A method for extracting building contours based on deep learning and joint optimization is provided in an embodiment of the present invention. Figure 1 , for urban buildings, execute the following steps S1 to S3 to complete the construction of the urban building outline map:

[0019] Step S1: In the training sample preparation stage, collect images of urban buildings, construct DOM (drone orthophoto) and DSM (digital surface model) of urban buildings; use DSM to make normalized digital surface model nDSM, and generate slope map at the same time; mark the contour lines and polygons of urban buildings based on DOM and slope map; construct a fishing net for image cutting, segment raster and vector layers through the fishing net, and establish training sample images and annotations;

[0020] The specific steps of step S1 are as follows:

[0021] Step S1.1: Use drones to collect multi-view photos of urban buildings, and construct the DOM, DSM and point cloud of urban buildings respectively through aerial triangulation, dense matching and texture mapping;

[0022] Step S1.2: Classify the point cloud into ground points, generate DEM by inverse distance weighted interpolation, use map algebra tools to perform DSM-DEM raster subtraction to obtain the digital surface model nDSM, normalize the pixel values ​​of the digital surface model nDSM to [0,255], and use DSM to generate a slope map with a pixel value range of [0,90];

[0023] Step S1.3: marking the outline polygons and building contours of urban buildings based on the DOM and slope map;

[0024] Step S1.4: constructing a fishing net for image cutting, wherein the fishing net is a square grid of a specific size for image cutting. In this embodiment, the fishing net is composed of a square grid of a size of 32m*32m;

[0025] Step S1.5: Segment the raster and vector layers through the fishnet to create training sample images and annotations, including slicing the DOM, nDSM, and slope map into training images and vector line and surface annotation files in JSON format.

[0026] Step S2: In the model training stage, the training sample images and annotations are preprocessed to generate distance and angle field training sample files; the training sample files are substituted into the building contour extraction model for training, and the building contour extraction model includes the line feature extraction network MFLine-Net and the semantic segmentation network Res-UNet, so as to obtain a trained building contour extraction model;

[0027] The specific steps of step S2 are as follows:

[0028] Step S2.1: pre-process the training sample images and annotation files, and generate distance field and angle field training sample files according to the building contours annotated by vector lines;

[0029] Step S2.2: Construct the line feature extraction network MFLine-Net, refer to Figure 2 , including a backbone network and two branch networks; the backbone network is based on the U-Net network, with a residual processing unit added to extract deep image features;

[0030] The improved line feature extraction network MFLine-Net with added residual processing units has better performance: 1. The residual unit will simplify network training; 2. The jump connections within the residual unit and between the low and high layers of the network will promote information propagation without causing performance degradation, forming a neural network with much fewer parameters, which can achieve better performance in image feature expression.

[0031] The structure of the network is constructed through multiple residual units. The backbone network includes an encoding path and a decoding path. The encoding path and the decoding path each have three residual units. Each residual unit consists of two 3×3 convolution blocks and an identity map; each convolution block includes a batch normalization BN layer, a ReLU activation layer and a convolution layer; the identity map connects the input and output of the residual unit; in each residual unit, a convolution operation with a stride of 2 is applied to the first convolution block to reduce the feature map size by half; the input of each residual unit is connected to the feature map from the corresponding encoding path by upsampling the low-level feature map.

[0032] The inputs of the two branch networks are connected to the outputs of the backbone network respectively. The two branch networks are in parallel and calculate the distance field and angle field respectively. Each branch network contains 3 convolutional layers. Except for the last layer, each convolutional layer is activated by ReLU and batch normalized. The activation function of the branch network that outputs the distance field is the ReLU function, and the activation function of the branch network that outputs the angle field is the Sigmoid function, without batch normalization. The loss function of the two branch networks is the sum of the losses of the normalized distance field branch and the angle field branch.

[0033] For each pixel in the image, the distance field D gives the distance from the current pixel to the point on the nearest line segment, and the line angle field A returns the direction of the nearest line segment; the distance field and angle field calculations performed by the two branch networks are as follows:

[0034]

[0035] Where D is the distance field, A is the angle field, (x, y) is the two-dimensional offset field pointing to the nearest point of the line segment, and (x, y)∈R H×W ×R H×W , H and W are the height and width of the training sample image respectively.

[0036] Step S2.3: Substitute the training sample image into the line feature extraction network MFLine-Net, set the number of training rounds, the number of batches, and the learning rate parameters, and obtain the trained line feature extraction network MFLine-Net;

[0037] Step S2.4: Substitute the training sample image into the semantic segmentation network Res-UNet for training to obtain the trained semantic segmentation network Res-UNet.

[0038] Step S3: In the model prediction and building mapping stage, the line feature extraction network MFLine-Net is used to extract the line features of urban buildings; then the semantic segmentation network Res-UNet is used to extract the roof semantic information of urban buildings; the extracted line features are used to generate a cell complex (Cell Complex) using the motion space subdivision algorithm; the graph cut optimization algorithm (Graph-cut) is used to associate the cell units in the cell complex and the roof semantic information; the cell units with the same semantic category are merged to obtain the outline polygon of the urban buildings, and the construction of the urban building outline map is completed.

[0039] The specific steps of step S3 are as follows:

[0040] Step S3.1: Use drones to collect images of buildings in the urban area to be tested, construct DOM, DSM, nDSM and slope images of the buildings in the urban area to be tested, input the DOM, nDSM and slope images of the buildings in the urban area to be tested into the line feature extraction network MFLine-Net for prediction, and extract the line feature set L. The extracted line features are as follows: Figure 3 As shown;

[0041] Step S3.2: Input the drone image and DSM of the urban building to be tested into the Res-UNet network, such as Figure 4 As shown, the roof semantic information of the urban building to be tested is extracted;

[0042] Step S3.3: Figure 5As shown, the extracted line features are used to generate the cell complex using the motion space partitioning algorithm;

[0043] Step S3.4: Use the graph cut optimization algorithm to associate the cell units and roof semantic information in the cell complex; the cell complex adopts a graph G with source points s and sink points t =<V,E> To express it, the node V corresponds to the polygonal unit of the cell complex, the undirected edge E corresponds to the adjacency relationship of the cell unit, and each cell unit is marked with two labels: free and non-free; the undirected edge E is defined as:

[0044] E=W(n)+w(t)

[0045] Among them, W( t ) is the weight between the source point s and the sink point t, defined as the probability that the source point s belongs to the indoor space, and W(n) is the weight between the source points s, as follows:

[0046]

[0047] Among them, p and q are adjacent cellular units, w<p,q> is the weight between p and q, cp is the value of the cell p; w<p,q> is the penalty function between cells p and q; if p and q cells are more similar, then w<p,q> The larger the value, the greater the difference between p and q.<p,q> Smaller and closer to 0;

[0048] Step S3.5: Figure 6 As shown, the cell units with the same roof semantic information are merged to obtain the building outline map of the urban area to be tested.

[0049] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A building contour extraction method based on deep learning and joint optimization, characterized in that: For urban buildings, the following steps S1 to S3 are performed to complete the construction of the urban building outline map: Step S1: Collect images of urban buildings, construct DOM and DSM of urban buildings; use DSM to make normalized digital surface model nDSM, and generate slope map at the same time; mark the contour lines and polygons of urban buildings based on DOM and slope map; build a fishing net for image cutting, segment raster and vector layers through the fishing net, and establish training sample images and annotations; Step S2: pre-process the training sample images and annotations to generate distance and angle field training sample files; Substitute the training sample file into the building contour extraction model for training. The building contour extraction model includes the line feature extraction network MFLine-Net and the semantic segmentation network Res-UNet to obtain a trained building contour extraction model; Step S3: Use the line feature extraction network MFLine-Net to extract the line features of urban buildings; then use the semantic segmentation network Res-UNet to extract the roof semantic information of urban buildings; use the motion space subdivision algorithm to generate a cell complex from the extracted line features; use the graph cut optimization algorithm to associate the cell units and roof semantic information in the cell complex; merge the cell units with the same semantic category to obtain the outline polygon of the urban buildings, and complete the construction of the urban building outline map.

2. A building contour extraction method based on deep learning and joint optimization according to claim 1, characterized in that: The specific steps of step S1 are as follows: Step S1.1: Use drones to collect multi-view photos of urban buildings, and construct the DOM, DSM and point cloud of urban buildings respectively through aerial triangulation, dense matching and texture mapping; Step S1.2: Classify the point cloud into ground points, generate DEM by inverse distance weighted interpolation, use map algebra tools to perform DSM-DEM raster subtraction to obtain the digital surface model nDSM, normalize the pixel values ​​of the digital surface model nDSM to [0,255], and use DSM to generate a slope map with a pixel value range of [0,90]; Step S1.3: marking the outline polygons and building contours of urban buildings based on the DOM and slope map; Step S1.4: construct a fishing net for image cutting, the fishing net consists of a square grid with a size of 32m*32m; Step S1.5: Segment the raster and vector layers through the fishnet to create training sample images and annotations, including slicing the DOM, nDSM, and slope map into training images and vector line and surface annotation files in JSON format.

3. The building contour extraction method based on deep learning and joint optimization according to claim 1 is characterized in that: The specific steps of step S2 are as follows: Step S2.1: pre-process the training sample images and annotation files, and generate distance field and angle field training sample files according to the building contours annotated by vector lines; Step S2.2: construct a line feature extraction network MFLine-Net, which includes a backbone network and two branch networks; the backbone network is based on the U-Net network, and a residual processing unit is added to perform image depth feature extraction; The inputs of the two branch networks are connected to the outputs of the backbone network respectively. The two branch networks are in parallel and calculate the distance field and angle field respectively. Each branch network contains 3 convolutional layers. Except for the last layer, each convolutional layer is activated by ReLU and batch normalized. The activation function of the branch network that outputs the distance field is the ReLU function, and the activation function of the branch network that outputs the angle field is the Sigmoid function, without batch normalization. The loss function of the two branch networks is the sum of the losses of the normalized distance field branch and the angle field branch. Step S2.3: Substitute the training sample image into the line feature extraction network MFLine-Net, set the number of training rounds, the number of batches, and the learning rate parameters, and obtain the trained line feature extraction network MFLine-Net; Step S2.4: Substitute the training sample image into the semantic segmentation network Res-UNet for training to obtain the trained semantic segmentation network Res-UNet.

4. The building contour extraction method based on deep learning and joint optimization according to claim 3 is characterized in that: The backbone network includes an encoding path and a decoding path. Each of the encoding path and the decoding path has three residual units. Each residual unit consists of two 3×3 convolution blocks and an identity map. Each convolution block includes a batch normalization BN layer, a ReLU activation layer and a convolution layer. The identity map connects the input and output of the residual unit. In each residual unit, a convolution operation with a stride of 2 is applied to the first convolution block to reduce the feature map size by half. The input of each residual unit is connected to the feature map from the corresponding encoding path by upsampling the low-level feature map.

5. The building contour extraction method based on deep learning and joint optimization according to claim 3 is characterized in that: The distance field and angle field calculations performed by the two branch networks are as follows: Where D is the distance field, A is the angle field, (x, y) is the two-dimensional offset field pointing to the nearest point of the line segment, and (x, y)∈R H ×W ×R H×W , H and W are the height and width of the training sample image respectively.

6. The building contour extraction method based on deep learning and joint optimization according to claim 1 is characterized in that: The specific steps of step S3 are as follows: Step S3.1: Use drones to collect images of the buildings in the urban area to be tested, construct DOM, DSM, nDSM and slope images of the buildings in the urban area to be tested, input the DOM, nDSM and slope images of the buildings in the urban area to be tested into the line feature extraction network MFLine-Net for prediction, and extract the line feature set L; Step S3.2: input the drone image and DSM of the urban building to be tested into the Res-UNet network to extract the roof semantic information of the urban building to be tested; Step S3.3: using the motion space partitioning algorithm to generate a cell complex based on the extracted line features; Step S3.4: Use the graph cut optimization algorithm to associate the cell units and roof semantic information in the cell complex; the cell complex adopts a graph G with source points s and sink points t =<V,E> To express it, the node V corresponds to the polygonal unit of the cell complex, the undirected edge E corresponds to the adjacency relationship of the cell unit, and each cell unit is marked with two labels: free and non-free; the undirected edge E is defined as: E=W (n) +W (t) Among them, W (t) is the weight between the source point s and the sink point t, defined as the probability that the source point s belongs to the indoor space, W (n) is the weight between source points s, as follows: Among them, p and q are adjacent cellular units, w <p,q> is the weight between p and q, c p is the value of the cell unit p; w <p,q> is the penalty function between the cell units p and q; Step S3.5: Merge the cell units with the same roof semantic information to obtain the building outline map of the urban area to be tested.

Images