Remote sensing image building extraction method based on deep learning and contour regularization

Abstract

The invention relates to a remote sensing image building extraction method based on deep learning and contour regularization, belongs to the technical field of remote sensing image data mining based on deep learning and a building post-processing algorithm, and aims to solve the problem that a building contour extracted by deep learning automation is greatly different from an actual contour. Compared with the prior art that the building contour is directly extracted from a deep learning model, the building contour extraction method has the advantages that subsequent post-processing operation is added, and the technical problem that the extracted building contour is greatly different from the actual contour is solved; according to the method, building distribution vectors with regular outlines and high applicability can be obtained under different image scales, and the method has the advantages of high efficiency, accuracy, high robustness and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image data mining based on deep learning and building post-processing algorithms. Background Art

[0002] At present, building extraction from high-resolution remote sensing images has been applied in urban planning and land use management, illegal building monitoring and map updating, three-dimensional modeling and digital cities, disaster risk assessment and emergency response, etc.

[0003] With the acceleration of urbanization, the number of buildings is increasing. Manual extraction of building information often has errors and loopholes and cannot meet actual needs. Deep learning, as a powerful tool, has shown significant advantages in automated building extraction. By constructing a deep learning sample set and training a deep learning model based on the constructed sample set, the building contour extraction is realized through the deep learning model. Constructing the sample set includes image preprocessing of the original remote sensing image buildings, and then annotating the selected image area. Figure 1 Examples of positive samples that meet the criteria for deep learning model input.

[0004] Existing deep learning automated building extraction methods are often limited to the extraction of local features and lack an understanding of the global information of the image. In some downstream applications in certain industries, such as three-dimensional reconstruction of buildings, the extracted building contours are often quite different from the actual contours. At the same time, the building contours directly extracted from the deep learning model have jagged or uneven boundaries, resulting in over-segmentation or under-segmentation. In addition, buildings are densely distributed in cities, and a large number of redundant points will exist after the large-scale building extraction results are directly converted into vector layers, which will not only cause the vector file to be too large but also affect subsequent further analysis. Therefore, it is necessary to post-process the direct extraction results of the building contours of the deep learning model. Summary of the invention

[0005] In order to solve the technical problem that there is a large difference between the building outline extracted by deep learning automatic building extraction and the actual outline, and the building outline has jagged or uneven boundaries, the present invention proposes a "remote sensing image building extraction method based on deep learning and outline regularization".

[0006] The remote sensing image building extraction method based on deep learning and contour regularization includes the following steps: Figure 2 As shown,

[0007] Step 1: construct a sample set of instance segmentation for building extraction from remote sensing images;

[0008] Step 2: training the instance segmentation model based on the instance segmentation sample set constructed in step 1;

[0009] Step 3: Automatic extraction of rough outlines of building tops from remote sensing images;

[0010] Step 4: Post-process the rough outline of the building top in the remote sensing image obtained in step 3, including simplification of the rough outline of the building top; calculate the intersection over union (IoU) of the minimum circumscribed rectangle of the rough outline of each building top with the initial outline to realize building type classification. The calculation formula of the intersection over union (IoU) is as follows:

[0011]

[0012] A is the minimum circumscribed rectangular pixel set of the rough outline of the building top, and B is the internal pixel set of the building's initial outline. The complex building outline is analyzed and simplified. A vector file is generated based on the obtained building outline results to complete the remote sensing image building extraction.

[0013] Technical effects:

[0014] In the prior art, after preliminary contour extraction of a building, a large amount of point data will be generated in the vector boundary. Too much point data will not only increase the computational burden, but may also contain redundant information. In addition, due to interference factors such as image noise, irregularities such as burrs and depressions will appear on the local boundary of the building. The present invention performs preliminary contour processing by using preprocessing such as point simplification, threshold screening, and hole filling to further improve the accuracy and regularity of building contour extraction. Figure 10-11 As shown, the problem of jagged or uneven boundaries of building contours extracted directly from the deep learning model and the phenomenon of over-segmentation or under-segmentation are solved.

[0015] The complex building contours are refined through vertex screening, vertex clustering, boundary fitting and other algorithms, and finally a fine contour boundary of the building top is obtained, which solves the technical problem that there is a large difference between the extracted building contour and the actual contour. The present invention can obtain building distribution vectors with regular contours and strong applicability at different image scales, and has the advantages of high efficiency, accuracy and strong robustness, providing technical support for urban planning and land use management, illegal building monitoring and map updating, three-dimensional modeling and digital cities, disaster risk assessment and emergency response, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Schematic diagram of positive samples according to the deep learning model input standard.

[0017] Figure 2 It is a schematic diagram of the process of the present invention.

[0018] Figure 3 This is a schematic diagram of the effect of extracting the original vertices of a simple building in step 4.

[0019] Figure 4 This is a diagram showing the effect of drawing the convex hull of a simple building in step 4.

[0020] Figure 5 This is a schematic diagram of the effect of extracting the approximate circumscribed rectangle of a simple building in step 4.

[0021] Figure 6 This is a schematic diagram of the effect of extracting the original vertices of the complex building in step 4.

[0022] Figure 7 This is a schematic diagram of the complex building vertex clustering effect in step 4.

[0023] Figure 8 This is a schematic diagram of the complex building boundary fitting effect in step 4.

[0024] Fig. 9 This is a schematic diagram of the final vertex extraction effect of the complex building in step 4.

[0025] Fig.10 This is a schematic diagram of the original remote sensing image building.

[0026] Fig.11 This is a schematic diagram of the final extraction result of the original remote sensing image building after the present invention. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by ordinary technicians in the field without creative work by adopting the embodiments of the present invention are within the scope of protection of the present invention.

[0028] In step 1, the original remote sensing image with a larger image size and the annotation vector are cropped with overlapping ratios using a sliding window. The window size is 512*512, the overlap size is 100, and 512*512 size sample pairs are output, which are finally made into a remote sensing image building extraction instance segmentation sample set.

[0029] The instance segmentation model in step 2 is the Mask2former model. The overall structure of Mask2former consists of three parts: Backbone extracts multi-scale features from the image; Pixel decoder performs multi-scale feature fusion; TransformerDecoder uses the query method to detect targets. First, the N*256 query feature is initialized, and then the query feature is iteratively updated using features of three resolutions: 1 / 32, 1 / 16, and 1 / 8, and the final class and mask prediction is performed.

[0030] In this embodiment, the software environment is: the operating system is Window Server 2019; the deep learning framework is MMdetection; the programming environment is Python 3.7;

[0031] Hardware environment: CPU model is Intel(R)Xeon(R)Gold 6254; GPU model is GTX3090;

[0032] The parameter settings for model training include: batchsize: 8; initial learning rate: 0.0025; epoch: 300.

[0033] Compared with the prior art, the present invention adds step 4 to extract the building outline directly from the deep learning model, that is, after obtaining the rough outline of the top of the building in the remote sensing image in step 3, a subsequent post-processing operation is performed. The specific process is as follows:

[0034] 1. Simplify the rough outline of the building top. Use the Ramer–Douglas–Peucker algorithm to simplify the points, set the threshold filter to 1.5m, and perform a 10m 2 Fill the following holes.

[0035] 2. Classify the building types. Calculate the intersection over union (IoU) of the minimum circumscribed rectangle of each building outline and the initial outline, set the classification threshold to 0.9, and define buildings with an IoU greater than 0.9 as simple buildings, and buildings with an IoU less than 0.9 as complex buildings. The IoU calculation formula is as follows:

[0036] A is the pixel set of the minimum circumscribed rectangle of the building, and B is the pixel set inside the initial outline of the building.

[0037] 3. If it is a simple building, the minimum enclosing rectangle outline can be used instead. The specific steps of minimum enclosing rectangle outline extraction are vertex extraction, convex hull drawing, and approximate enclosing rectangle extraction. The schematic diagram of each step is as follows Figure 3-5 shown.

[0038] If it is a complex building, analyze and simplify the complex building outline: Figure 6-9 As shown, firstly, some noise points are eliminated by vertex screening, then the vertex azimuth is calculated, vertices with similar azimuths are clustered, and the clustering results are used to classify and mark the boundary contour vertices. Finally, vertices of the same type are extracted in turn and the least squares boundary fitting is performed.

[0039] The vertex screening is specifically as follows:

[0040] Traverse all points in the contour point set, for any point p i , calculate point p i-1p i p i+1 The angle θ formed by the point p is (0-90°). If θ<15° or θ>165°, delete the point p. i ,Through the preliminary screening of vertex angles, some error points and redundant points can be eliminated, the jagged effect of the boundary can be reduced, thus reducing the error for subsequent calculations.

[0041] The vertex clustering is specifically as follows:

[0042] In order to further extract building boundary information from a large number of contour points and simplify subsequent processing steps, the azimuth clustering method is used to group the vertices, such as Figure 7 As shown, different outline colors represent different vertex clustering grouping results. The azimuth angle of each vertex is defined below:

[0043] Vertex p i and the next vertex p i+1 The angle (-90°~90°) between the straight line and the true north direction is the azimuth of point p. The deviation angle α of the two azimuths θ1 and θ2 is defined as:

[0044] α=min((180°-|θ1-θ2|),|θ1-θ2|). Based on the above definition, cluster the azimuths of the building outline vertices. Starting from the starting point of the longest edge, the azimuth of the point θ1 is used as the azimuth center θ of the first cluster. center1 , traverse all vertices in turn, if the azimuth angle of the next vertex is θ i The azimuth center θ of the current cluster k centerk If the deviation angle is less than 30°, the point belongs to cluster k, otherwise a new cluster k+1 is set with the azimuth center as θ i .

[0045] The boundary fitting is specifically:

[0046] The vertices belonging to the same cluster are extracted and fitted using the least squares method to obtain the boundary fitting curve. The vertices in the cluster are replaced by the fitting curve to achieve the purpose of vertex simplification.

[0047] 4. Generate a vector file based on the obtained building outline results to complete the remote sensing image building extraction.

[0048] In order to further improve the regional consistency of the building extraction results, the overall building direction can be fine-tuned. According to a fixed width, 500m is used in this embodiment, the entire area is gridded, and the endpoints p and q of the longest side of each building are found. Then, the azimuth θ of the main direction of each building is calculated, and the buildings in the same grid are clustered according to the main direction azimuth to ensure that the deviation angle between the azimuth of each element in the class and the azimuth center is less than 5°. Each building is rotated so that the deviation angle between the main direction azimuth of the building and the azimuth center of the class to which it belongs is 0, to ensure that the main direction of the building has regional consistency.

[0049] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. Remote sensing image building extraction method based on deep learning and contour regularization, It is characterized in that The steps include: Step 1: construct a sample set of instance segmentation for building extraction from remote sensing images; Step 2: training the instance segmentation model based on the instance segmentation sample set constructed in step 1; Step 3: Automatic extraction of rough outlines of building tops from remote sensing images; Step 4: Post-process the rough outline of the building top in the remote sensing image obtained in step 3, including simplification of the rough outline of the building top; calculate the intersection over union (IoU) of the minimum circumscribed rectangle of the rough outline of each building top with the initial outline to realize building type classification. The calculation formula of the intersection over union (IoU) is as follows: A is the minimum circumscribed rectangular pixel set of the rough outline of the building top, and B is the internal pixel set of the building's initial outline. The complex building outline is analyzed and simplified. A vector file is generated based on the obtained building outline results to complete the remote sensing image building extraction.

2. The remote sensing image building extraction method based on deep learning and contour regularization according to claim 1, It is characterized in that The rough outline simplification of the building top uses the Ramer-Douglas-Peucker algorithm for point simplification, with the threshold screening set to 1.5m and the 10m 2 Fill the following holes.

3. The remote sensing image building extraction method based on deep learning and contour regularization according to claim 1, It is characterized in that Building type classification is to calculate the intersection and union ratio of the minimum enclosing rectangle of each building outline and the initial outline, and set the classification threshold to 0.

9. Buildings with an intersection and union ratio greater than 0.9 are simple buildings, and buildings with an intersection and union ratio less than 0.9 are complex buildings. Simple buildings are replaced by the minimum enclosing rectangle outline, and complex buildings are analyzed and simplified.

4. The remote sensing image building extraction method based on deep learning and contour regularization according to claim 3, It is characterized in that The minimum circumscribed rectangle contour extraction specifically includes vertex extraction, convex hull drawing and approximate circumscribed rectangle extraction.

5. The remote sensing image building extraction method based on deep learning and contour regularization according to claim 3, It is characterized in that The complex building outline analysis and simplification specifically includes removing some noise points through vertex screening, calculating vertex azimuths, clustering vertices with similar azimuths, classifying and marking boundary outline vertices using the clustering results, and finally extracting similar vertices in turn to perform least square boundary fitting.

6. The remote sensing image building extraction method based on deep learning and contour regularization according to claim 5, It is characterized in that The vertex screening is specifically as follows: traverse all points in the contour point set, and for any point p i , calculate point p i-1 p i p i+1 The angle θ formed by the point p is (0-90°). If θ<15° or θ>165°, delete the point p. i The vertex clustering is specifically: taking the azimuth clustering method to group the vertices, vertex p i and the next vertex p i+1 The angle between the straight line and the north direction (-90°~90°) is the azimuth of point p. 1 ,θ 2 The deviation angle α is defined as: α=min((180 ° -|θ 1 -θ 2 |),|θ 1 -θ 2 |), Based on the deviation angle α, the azimuth of the complex building outline vertices is clustered, starting from the starting point of the longest side, and the azimuth of the point is clustered. 1 As the first type of clustering cluster azimuth center θ center1 , traverse all vertices in turn, if the azimuth angle of the next vertex is θ i The azimuth center θ of the current cluster k centerk If the deviation angle is less than 30°, the point belongs to cluster k, otherwise a new cluster k+1 is set with the azimuth center as θ i The boundary fitting is specifically as follows: extracting vertices belonging to the same cluster, fitting them using the least square method to obtain a boundary fitting curve, and replacing the vertices in the cluster with the fitting curve.

Images