Building instantiation segmentation method and device, terminal and storage medium

Abstract

The invention provides a building instantiation segmentation method and device, a terminal and a storage medium, and relates to the technical field of three-dimensional point cloud processing, and the method comprises the steps: carrying out the plane segmentation of a building point cloud of a target building, and extracting a potential vertical plane point cluster from a building plane point cluster obtained through the segmentation; mapping all three-dimensional point clouds in the potential vertical plane point cluster to a two-dimensional feature space to obtain a two-dimensional point set, and processing the two-dimensional point set by using a preset dual-threshold judgment mechanism to extract a building vertical structure point cluster; generating a building distance map based on the building vertical structure point clusters, and generating instance anchor points on the distance map; extracting a multi-level contour in the distance map based on the instance anchor points, constructing a tree-shaped hierarchical relationship corresponding to the multi-level contour, and constructing a mark matrix based on the tree-shaped hierarchical relationship; and building two-dimensional instantiation segmentation is carried out on the distance map based on the mark matrix. Through the unsupervised hierarchical segmentation framework, the dependence on the annotated data can be eliminated, and the segmentation precision is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional point cloud processing, and particularly to a method, device, terminal and storage medium for building instance segmentation. Background Art

[0002] At present, the instance-level segmentation of three-dimensional models of urban buildings has become a core requirement in fields such as smart city management, real estate registration, and BIM (Building Information Modeling) modeling.

[0003] Current mainstream lidar and oblique photogrammetry technologies can quickly obtain large-scale urban scene point cloud data. However, how to achieve accurate segmentation of building individual instances still faces major technical bottlenecks. However, the traditional manual annotation method is inefficient and limited by the subjective judgment of operators, making it difficult to meet the accuracy and timeliness requirements of urban-level three-dimensional modeling.

[0004] Moreover, existing instance segmentation methods based on deep learning are highly dependent on data and require large-scale labeled datasets. However, the labeling cost of urban scenes is high, and the generalization ability of this method is poor. The accuracy of the model drops significantly when migrating across scenes, and it has high hardware requirements and is difficult to be deployed to mobile devices. In addition, the clustering method based on local geometric features uses RANSAC (RANdom SAmple Consensus) plane fitting and combines Euclidean clustering to cluster building instantiation objects. However, this method only clusters based on local normal vector similarity, which may cause bridge-type buildings to be mismerged. Moreover, this method has poor anti-noise ability. When the two-dimensional point cloud density is uneven, the failure rate of facade structure extraction is relatively high, and the clustering radius threshold needs to be adjusted manually repeatedly.

[0005] In summary, the existing technology has technical defects such as strong dependence on labeled data, poor generalization ability, lack of global features, and projection distortion, and needs to be improved and developed. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a method, device, terminal and storage medium for building instance segmentation, which can achieve accurate segmentation of building individual instances, improve segmentation accuracy, reduce the misjudgment rate of vertical structures, and eliminate the dependence on labeled data, aiming at the above-mentioned defects of the existing technology.

[0007] The technical solution adopted by the present invention to solve the technical problem is as follows: A method for building instance segmentation, wherein the method includes: Construct a building point cloud of a target building, perform plane segmentation on the building point cloud, and extract potential vertical plane point clusters from the segmented building plane point clusters; Map all the three-dimensional point clouds in the point cloud set corresponding to the potential vertical plane point clusters to a two-dimensional feature space to obtain a two-dimensional point set, and use a preset double-threshold determination mechanism to extract the two-dimensional point clouds of the building vertical structure based on the two-dimensional point set, so as to obtain the building vertical structure point clusters; Generate a building distance map based on the building vertical structure point clusters, and generate building instance anchor points representing the positions of potential building instances on the building distance map; Extract multi-level contours in the building distance map based on the building instance anchor points, construct a tree-like hierarchical relationship corresponding to the multi-level contours, and construct a corresponding marking matrix based on the tree-like hierarchical relationship; Perform two-dimensional instantiation segmentation on the building distance map based on the marking matrix to obtain a two-dimensional instantiation object of the building; wherein, the two-dimensional pixels of the two-dimensional instantiation object of the building carry building instantiation labels.

[0008] In one implementation, the plane segmentation of the building point cloud and the extraction of potential vertical plane point clusters from the segmented building plane point clusters include: Use the plane region growing algorithm to perform plane segmentation on the building point cloud to obtain building plane point clusters; Use the least squares method to perform plane fitting on the building plane point clusters to obtain the optimal plane equation; Extract potential vertical plane point clusters based on the optimal plane equation and the size of the building plane point clusters.

[0009] In one implementation, the mapping of all the three-dimensional point clouds in the point cloud set corresponding to the potential vertical plane point clusters to a two-dimensional feature space to obtain a two-dimensional point set includes: Construct a ground point cloud and generate a ground horizontal reference plane based on the fitting of the ground point cloud; On the ground horizontal reference plane, establish a local coordinate system based on the centroid point of the ground point cloud; Project all the three-dimensional point clouds in the point cloud set corresponding to the potential vertical plane point clusters along the normal vector direction of the ground point cloud onto the local coordinate system to obtain a two-dimensional point set; Wherein, the use of the preset double-threshold determination mechanism and the extraction of the two-dimensional point clouds of the building vertical structure based on the two-dimensional point set to obtain the building vertical structure point clusters includes: Establish a two-dimensional grid of the two-dimensional point set on the ground horizontal reference plane; Count the number of two-dimensional point clouds in each grid cell of the two-dimensional grid to obtain the vertical projection density features corresponding to each grid cell; Determine the height of the highest point and the height of the lowest point of each grid cell in the two-dimensional grid, and calculate the difference between the height of the highest point and the height of the lowest point to obtain the vertical height difference feature corresponding to each grid cell; Determine a density threshold based on the vertical projection density feature, and determine a depth threshold based on the vertical height difference feature; Using a preset double-threshold determination mechanism, compare the vertical projection density feature corresponding to each grid cell with the density threshold, and compare the vertical height difference feature with the depth threshold; When the vertical projection density feature is greater than the density threshold and the vertical height difference feature is greater than the depth threshold, extract the two-dimensional point cloud in the grid cell as the two-dimensional point cloud of the building vertical structure to obtain a point cluster of the building vertical structure.

[0010] In one implementation, the determining a density threshold based on the vertical projection density feature and determining a depth threshold based on the vertical height difference feature includes: Based on the vertical projection density feature, determine the mean and variance of the two-dimensional point cloud density in the grid cell, and use a preset density threshold calculation formula to process the mean and the variance to obtain a density threshold; Use a preset depth threshold calculation formula to process the vertical height difference feature to obtain a depth threshold; Wherein, the preset density threshold calculation formula is: ; The preset depth threshold calculation formula is: ; ; And, represents the density threshold, represents the mean of the two-dimensional point cloud density in the grid cell, represents the variance of the two-dimensional point cloud density in the grid cell, represents the depth threshold, represents the vertical height difference feature, represents the height of the highest point in the grid cell, represents the height of the lowest point in the grid cell, represents the threshold control coefficient.

[0011] In one implementation, the generating a building distance map based on the point cluster of the building vertical structure and generating a building instance anchor point representing the position of a potential building instance on the building distance map includes: Project all the building vertical structure 2D point clouds in the building vertical structure point cluster onto the ground horizontal reference plane to obtain a set of building projection points; Determine the maximum and minimum longitudinal coordinates and the maximum and minimum transverse coordinates of the building projection points in the set of projection points; Based on a preset image resolution, the maximum and minimum longitudinal coordinates, and the maximum and minimum transverse coordinates, calculate the image width and image height respectively; Create an initial binary image according to the image width and the image height; Translate all the building projection points in the set of building projection points into the image coordinate system constructed based on the initial binary image to obtain a target binary image; Calculate the Euclidean distance between each pixel in the target binary image and the background pixel to obtain the distance value corresponding to each pixel; Linearly map the distance value corresponding to each pixel into the standard range of gray values to obtain a building distance map; Traverse the building distance map in the way of a sliding window, and sequentially calculate the average value of the distance values corresponding to all pixels in the window area to obtain an adaptive threshold corresponding to the window area; Compare the distance value corresponding to each pixel in the window area with the adaptive threshold, and determine the pixel area with the distance value higher than the adaptive threshold as the foreground to obtain building instance anchor points representing the positions of potential building instances on the building distance map.

[0012] In one implementation, before extracting multi-level contours from the building distance map based on the building instance anchor points, it further includes: Perform a morphological erosion operation on the building instance anchor points on the building distance map.

[0013] In one implementation, after performing building two-dimensional instantiation segmentation on the building distance map based on the marking matrix to obtain a building two-dimensional instantiation object, it further includes: Perform a morphological dilation operation on the building two-dimensional instantiation object, and extract the maximum contour of the dilated building two-dimensional instantiation object; Simplify the maximum contour to obtain a simplified maximum contour, and determine the two-dimensional pixels within the simplified maximum contour as target building instantiation pixels; Based on the pre-constructed mapping relationship between pixel coordinates and three-dimensional space coordinates, perform three-dimensional space mapping processing on the target building instantiation pixels to obtain a building three-dimensional instantiation object.

[0014] The present invention also discloses a building instantiation segmentation device, wherein the device includes: A building point cloud construction module for constructing a building point cloud of a target building; A planar point cluster extraction module for performing planar segmentation on the building point cloud and extracting potential vertical plane point clusters from the segmented building planar point clusters; A building vertical structure extraction module for mapping all three-dimensional point clouds in the point cloud set corresponding to the potential vertical plane point clusters to a two-dimensional feature space to obtain a two-dimensional point set, using a preset double-threshold determination mechanism, and extracting two-dimensional point clouds of the building vertical structure based on the two-dimensional point set to obtain a building vertical structure point cluster; An instance anchor generation module for generating a building distance map based on the building vertical structure point cluster and generating building instance anchors representing the positions of potential building instances on the building distance map; A tree-like hierarchical relationship construction module for extracting multi-level contours in the building distance map based on the building instance anchors and constructing a tree-like hierarchical relationship corresponding to the multi-level contours; A marking matrix construction module for constructing a corresponding marking matrix based on the tree-like hierarchical relationship; An instantiation segmentation module for performing two-dimensional building instantiation segmentation on the building distance map based on the marking matrix to obtain a two-dimensional building instantiation object; wherein, the two-dimensional pixels of the two-dimensional building instantiation object carry building instantiation labels.

[0015] The present invention also discloses a terminal, which includes: a memory, a processor, and a building instantiation segmentation program stored on the memory and executable on the processor. When the building instantiation segmentation program is executed by the processor, the steps of the building instantiation segmentation method described above are implemented.

[0016] The present invention also discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program that can be executed to implement the steps of the building instantiation segmentation method described above.

[0017] A method, device, terminal and storage medium for instantiating and segmenting buildings provided by the present invention, the method for instantiating and segmenting buildings includes: constructing a building point cloud of a target building, performing plane segmentation on the building point cloud, and extracting potential vertical plane point clusters from the segmented building plane point clusters; mapping all three-dimensional point clouds in the point cloud set corresponding to the potential vertical plane point clusters to a two-dimensional feature space to obtain a two-dimensional point set, using a preset double-threshold determination mechanism, and extracting two-dimensional building vertical structure point clouds based on the two-dimensional point set to obtain building vertical structure point clusters; generating a building distance map based on the building vertical structure point clusters, and generating building instance anchor points representing the positions of potential building instances on the building distance map; extracting multi-level contours in the building distance map based on the building instance anchor points, constructing a tree-like hierarchical relationship corresponding to the multi-level contours, and constructing a corresponding marking matrix based on the tree-like hierarchical relationship; performing two-dimensional instantiation segmentation on the building distance map based on the marking matrix to obtain a two-dimensional instantiation object of the building; wherein, the two-dimensional pixels of the two-dimensional instantiation object of the building carry building instantiation labels. It can be seen that the present invention realizes the accurate recognition of vertical structures through a preset double-threshold determination mechanism, reduces the misjudgment rate of vertical structures, and realizes the accurate division of complex building instances by mapping three-dimensional point clouds to a two-dimensional feature space, improving the segmentation accuracy. It also realizes an unsupervised hierarchical segmentation framework through a three-level optimization architecture of three-dimensional point clouds, plane point clusters and instance anchor points, and a contour tree structure, which can eliminate the dependence on labeled data in the process of segmenting building single instances, realize the rapid processing of urban-level point clouds, significantly improve the processing ability of complex buildings such as special-shaped buildings and adhesive structures, and improve the segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a schematic flowchart of a preferred embodiment of the method for instantiating and segmenting buildings in the present invention; Figure 2 is a schematic flowchart of a specific process for extracting building vertical structure point clusters disclosed in the present invention; Figure 3 is a schematic flowchart of a specific process for generating adaptive building anchor points disclosed in the present invention; Figure 4 is a schematic flowchart of a specific process for morphological-based building instantiation segmentation disclosed in the present invention; Figure 5 is a functional principle block diagram of a preferred embodiment of the device for instantiating and segmenting buildings in the present invention; Figure 6 is a functional principle block diagram of a preferred embodiment of the terminal in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] To make the objectives, technical solutions and advantages of the present invention more clear and definite, the following further describes the present invention in detail with reference to the accompanying drawings and by way of examples. It should be understood that the specific examples described herein are only for explaining the present invention and are not used to limit the present invention.

[0020] Please refer to Figure 1 , Figure 1 which is a flowchart of the method for instantiating and segmenting buildings in the present invention. As Figure 1 shown, the method for instantiating and segmenting buildings according to the embodiments of the present invention includes: Step S11: Construct a building point cloud of a target building, perform plane segmentation on the building point cloud, and extract a potential vertical plane point cluster from the building plane point clusters obtained by the segmentation.

[0021] In this embodiment, for the instance-level segmentation of a building three-dimensional model, first, a building point cloud of a target building is constructed, then the building point cloud is segmented into building plane point clusters, and a potential vertical plane point cluster is extracted from the building plane point clusters obtained by the segmentation.

[0022] Specifically, the plane region growing algorithm is used to perform plane segmentation on the building point cloud to obtain building plane point clusters, and then the least squares method is used to perform plane fitting on the building plane point clusters to obtain an optimal plane equation. Furthermore, a potential vertical plane point cluster is extracted based on the optimal plane equation and the size of the building plane point clusters. Among them, the optimal plane equation can describe the overall geometric structure of the point cluster, such as the wall or roof plane of a building.

[0023] For example, when using the plane region growing algorithm to implement the plane segmentation of the building point cloud, the minimum plane size can be set to 10 points, the maximum acceptable angle is , and the maximum point-to-plane distance is 0.5 m. Then, the least squares method is used to perform plane fitting on the building plane point clusters obtained by the segmentation to obtain the optimal plane equation ax + by + cz + d = 0. Let the size of the building plane point cluster be s, and c and s are jointly constrained, and a potential vertical plane point cluster is extracted based on the following formula, that is: ; where is the size of the largest plane point cluster.

[0024] Step S12: Map all three-dimensional point clouds in the point cloud set corresponding to the potential vertical plane point cluster to a two-dimensional feature space to obtain a two-dimensional point set, use a preset double-threshold determination mechanism, and extract two-dimensional point clouds of the building vertical structure based on the two-dimensional point set to obtain a building vertical structure point cluster.

[0025] In this embodiment, all three-dimensional point clouds in the point cloud set corresponding to the potential vertical plane points are mapped to a two-dimensional feature space to obtain a two-dimensional point set. Specifically, a ground point cloud is constructed, and a ground horizontal reference plane is generated by fitting based on the ground point cloud; on the ground horizontal reference plane, a local coordinate system is established based on the centroid point of the ground point cloud; all three-dimensional point clouds in the point cloud set corresponding to the potential vertical plane points are projected onto the local coordinate system along the normal vector direction of the ground point cloud to obtain a two-dimensional point set.

[0026] For example, the point cloud set corresponding to the potential vertical plane points can be , and then all points in the point cloud set are projected along the normal vector direction of the ground point cloud onto the ground horizontal reference plane L , and a local coordinate system is established with the centroid point of the ground point cloud , that is: ; Among them, the coordinates of the j-th three-dimensional point in the point cloud set are: , ; And, represents the x coordinate of the j-th three-dimensional point cloud in the point cloud set corresponding to the potential vertical plane points, represents the y coordinate of the j-th three-dimensional point cloud in the point cloud set corresponding to the potential vertical plane points, represents the z coordinate of the j-th three-dimensional point cloud in the point cloud set corresponding to the potential vertical plane points.

[0027] Then, all point clouds in the point cloud set are projected onto the local coordinate system along the normal vector direction of the ground point cloud to obtain a two-dimensional point set , and the two-dimensional coordinates of the i-th point in the two-dimensional point set are , .

[0028] Among them, ; And, represents the x coordinate of the i-th two-dimensional point cloud in the two-dimensional point set, represents the y coordinate of the i-th two-dimensional point cloud in the two-dimensional point set, represents the i-th three-dimensional point cloud in the point cloud set corresponding to the potential vertical plane points, and .

[0029] ​In this embodiment, a preset dual-threshold determination mechanism is utilized, and based on a two-dimensional point set, two-dimensional point clouds of building vertical structures are extracted to obtain clusters of building vertical structure points. That is, the preset dual-threshold determination mechanism is used to process the two-dimensional point set to extract clusters of building vertical structure points. Among them, the dual thresholds in the preset dual-threshold determination mechanism refer to the dual thresholds determined based on the two-dimensional point set, namely the density threshold and the depth threshold, rather than fixed thresholds set in advance. Then, the dual thresholds are used for determination to extract clusters of building vertical structure points. Specifically, a two-dimensional grid of the two-dimensional point set is established on the ground horizontal reference plane; the number of two-dimensional point clouds in each grid cell of the two-dimensional grid is counted to obtain the vertical projection density feature corresponding to each grid cell; the highest point height and the lowest point height of each grid cell in the two-dimensional grid are determined, and the difference between the highest point height and the lowest point height is calculated to obtain the vertical height difference feature corresponding to each grid cell; based on the vertical projection density feature corresponding to each grid cell, the density threshold of the two-dimensional grid is determined, and based on the vertical height difference feature corresponding to the grid cell, the corresponding depth threshold is determined; using the preset dual-threshold determination mechanism, the vertical projection density feature corresponding to each grid cell is compared with the density threshold, and the vertical height difference feature is compared with the depth threshold; when the vertical projection density feature corresponding to the grid cell is greater than the density threshold corresponding to the two-dimensional grid and the vertical height difference feature corresponding to the grid cell is greater than the corresponding depth threshold, the two-dimensional point cloud in the grid cell is extracted as the two-dimensional point cloud of the building vertical structure to obtain clusters of building vertical structure points.

[0030] In this embodiment, the calculation method of the dual thresholds is specifically as follows: based on the vertical projection density feature, the mean value and the variability of the two-dimensional point cloud density in the grid cell are determined, and the mean value and the variability are processed using a preset density threshold calculation formula to obtain the density threshold; the vertical height difference feature is processed using a preset depth threshold calculation formula to obtain the depth threshold; Among them, the preset density threshold calculation formula is: ; The preset depth threshold calculation formula is: ; ; And, represents the density threshold corresponding to the two-dimensional grid, represents the mean value of the two-dimensional point cloud density in the grid cell, represents the variability of the two-dimensional point cloud density in the grid cell, represents the depth threshold corresponding to the grid cell, represents the vertical height difference feature, represents the highest point height in the grid cell, represents the lowest point height in the grid cell, represents a threshold control coefficient, and the threshold control coefficient can be set to , at this time, the depth threshold can be .

[0031] For example, on the ground point horizontal reference plane L a two-dimensional grid covering all the two-dimensional point clouds in the two-dimensional point set is established, and the number of two-dimensional point clouds in each grid cell is counted to obtain the vertical projection density feature , expressing the spatial resolution as a two-dimensional density field to quantify the spatial distribution characteristics of the point cloud. At the same time, the height difference between the highest point and the lowest point in each grid cell of the two-dimensional grid is calculated to obtain the vertical height difference feature of each grid cell to quantify and express the spatial heterogeneity in the vertical direction of the point cloud data, that is: ; wherein, represents the vertical projection density feature of the m-th grid cell, represents the m-th grid cell, 1 is an indicator function, which takes the value of 1 when the condition is satisfied, otherwise 0. That is, the indicator function is a function that maps from the universal set to {0, 1}. In image processing, the indicator function can be used to mark whether a certain pixel belongs to the target area, represents the x coordinate of the i-th two-dimensional point cloud in the grid cell , n represents the number of two-dimensional point clouds in the grid cell , represents the vertical height difference feature of the m-th grid cell, represents the height of the highest point in the m-th grid cell, represents the height of the lowest point in the m-th grid cell.

[0032] For another example, the mean and variance of the two-dimensional point cloud density in the grid cell are determined through statistical analysis methods, that is: ; where M is the total number of grid cells in the two-dimensional grid.

[0033] Furthermore, based on the mean and variance, the density threshold corresponding to the two-dimensional grid is set, and based on the vertical height difference feature, the relative depth threshold constraint for the column height is set to remove the interfering point clouds below the set threshold, thereby enhancing the selectivity for relevant vertical features.

[0034] Finally, the final building vertical structure is extracted by combining the double thresholds, that is: .

[0035] For another example, see Figure 2As shown in the figure, the extraction process of the building vertical structure is as follows: The plane region growing algorithm is used to perform plane segmentation on the building point cloud to obtain building plane point clusters, realizing multi-scale point cluster segmentation. Then, the three-dimensional features of the point clusters are calculated. That is, the least squares method is used to perform plane fitting on the building plane point clusters to obtain the optimal plane equation ax + by + cz + d = 0. Assuming the size of the building plane point cluster is s, a joint constraint is imposed on c and s to extract potential vertical plane point clusters. At the same time, based on the ground point cloud, the least squares method is used to fit and generate the ground horizontal reference plane. L , and the normal vector N of the horizontal reference plane and the centroid point of the ground point cloud are used as the subsequent projection reference plane. On the ground horizontal reference plane, a local coordinate system is established based on the centroid point of the ground point cloud. All three-dimensional point clouds in the point cloud set corresponding to the potential vertical plane point clusters are projected along the normal vector direction of the ground point cloud to the local coordinate system to obtain a two-dimensional point set. Then, the vertical projection density feature and the vertical height difference feature are calculated. Based on the vertical projection density feature and the vertical height difference feature, a density threshold and a height threshold are set. Finally, the two thresholds are jointly used for the extraction of the final building vertical structure. That is, through the two-dimensional projection density-depth double-threshold determination mechanism, on the ground horizontal reference plane as the reference plane, a local coordinate system is established to accurately identify the vertical structure, which can reduce the misjudgment rate of the vertical structure.

[0036] It should be noted that the vertical structure of the building is the main basis for distinguishing different building instances. By constraining the projection to map the three-dimensional point cloud to the two-dimensional feature space, breaking through the limitations of traditional single-dimensional feature analysis, it can solve the problem of segmenting special-shaped buildings. It can achieve accurate division of complex building instances during the building instantiation segmentation process, and reach 92.3% of the instance segmentation IoU (Intersection over Union) on the standard test set. IoU is the core evaluation index of the instance segmentation task, measuring the degree of overlap between the prediction and the real area. 92.3% of the IoU indicates that the model has achieved extremely high segmentation accuracy on the standard test set, especially performing excellently in object boundary and detail processing.

[0037] Step S13: Generate a building distance map based on the building vertical structure point clusters, and generate building instance anchor points representing the positions of potential building instances on the building distance map.

[0038] In this embodiment, the two-dimensional point set corresponding to the potential vertical plane point cluster is processed by using a preset double-threshold determination mechanism. After extracting the building vertical structure point cluster, a building distance map is generated based on the building vertical structure point cluster. The building distance map is a grayscale map obtained by mapping the Euclidean distance from each pixel in the binary image generated based on the building vertical structure point cluster to the background pixel to the standard range of grayscale values. Each pixel in the grayscale map has a unique corresponding distance value. Then, a building instance anchor point representing the position of the potential building instance on the building distance map is generated, that is, the building instance anchor point on the building distance map is adaptively generated. The building anchor point refers to the position of the potential building instance on the building distance map.

[0039] Specifically, project all the building vertical structure two-dimensional point clouds in the building vertical structure point cluster onto the ground horizontal reference plane to obtain a set of building projection points; determine the maximum and minimum longitudinal coordinates and the maximum and minimum transverse coordinates of the building projection points in the set of projection points; calculate the image width and image height respectively based on the preset image resolution, the maximum and minimum longitudinal coordinates, and the maximum and minimum transverse coordinates; create an initial binary image according to the image width and image height; translate all the building projection points in the set of building projection points into the image coordinate system constructed based on the initial binary image to obtain a target binary image; calculate the Euclidean distance between each pixel in the target binary image and the background pixel to obtain the distance value corresponding to each pixel; linearly map the distance value corresponding to each pixel to the standard range of grayscale values to obtain the building distance map; traverse the building distance map in the way of a sliding window, and calculate the average value of the distance values corresponding to all the pixels in the window area in turn to obtain an adaptive threshold corresponding to the window area; compare the distance value corresponding to each pixel in the window area with the adaptive threshold, and determine the pixel area with a distance value higher than the adaptive threshold as the foreground to obtain the building instance anchor point representing the position of the potential building instance on the building distance map.

[0040] For example, as shown in Figure 3 project all the original point clouds in the building vertical structure point cluster onto the ground horizontal reference plane to obtain a set of building projection points, and determine the maximum and minimum longitudinal coordinates and the maximum and minimum transverse coordinates of the building projection points in the set of projection points, that is , , , The preset image resolution can be 0.08, calculate the image width width and the image height height, that is: ; ; Then, create an initial binary image Img according to the image width width and the image height height, and translate all the building projection points in the building projection point set into the image coordinate system constructed based on the initial binary image to obtain the target binary image, that is: ; ; where the preset offset , , height represents the image height, represents the abscissa of the building projection point, represents the ordinate of the building projection point.

[0041] Furthermore, calculate the Euclidean distance from each pixel in the target binary image to the background pixel (black pixel), linearly map the distance value to the standard range of gray values, that is, the range of [0, 255], retain the relative relationship with the distance value, and at the same time adapt to the image display format to obtain the building distance map in the grayscale image display format. Then, traverse the building distance map through a sliding window and calculate the average value of the distance values corresponding to all pixels in the window area , set as the adaptive threshold, and judge whether the distance value corresponding to each pixel in the window area is greater than the adaptive threshold , and set the pixel area with a distance value higher than the adaptive threshold as the foreground (white), and the rest as the background (black). Different areas in the foreground are the building instance anchor points.

[0042] Step S14: Extract multi-level contours from the building distance map based on the building instance anchor points, construct a tree-like hierarchical relationship corresponding to the multi-level contours, and construct a corresponding marking matrix based on the tree-like hierarchical relationship.

[0043] In this embodiment, after generating the building instance anchor points in the building distance map, building instance segmentation can be realized based on morphology, that is, based on the morphological characteristics of the building instance anchor points on the building distance map, two-dimensional building instance segmentation is performed. First, extract multi-level contours from the building distance map based on the building instance anchor points, construct a tree-like hierarchical relationship corresponding to the multi-level contours, and construct a corresponding marking matrix based on the tree-like hierarchical relationship. Then, perform the two-dimensional building instance segmentation in step S15. And before extracting the multi-level contours from the building distance map based on the building instance anchor points, it may specifically further include: performing a morphological erosion operation on the building instance anchor points on the building distance map. Among them, the erosion kernel size can be set to 2x2.

[0044] Among them, in the process of multi-level contour extraction and hierarchical relationship modeling, first, multi-level contours in the building distance map are extracted, and then the corresponding tree-like hierarchical relationship can be constructed using the RETR_TREE mode. Among them, the RETR_TREE mode is one of the retrieval modes of the contour detection function, which is used to completely reconstruct the hierarchical relationship (tree structure) of the contours, index and associate the parent and child nodes of each contour. Finally, the contours are classified as root nodes, intermediate nodes, and leaf nodes according to the hierarchical relationship, and visualization is achieved through color marking, such as marking the root node in red, the intermediate node in the default color, and the leaf node in green. And, in the process of constructing the marking matrix, first initialize the marking matrix markers, traverse all contours. If the number of valid pixels in the contour is zero and the area is greater than the average value of the contour areas, it is marked as the background (Scalar(255), the gray value of the background is 255), filter the contours with the number of valid pixels less than 5, and only assign a unique index value to the contours without child nodes or root nodes, and the rest are marked as the intermediate area. And the marking matrix can also be optimized, that is, background anchor points are added to the four corners of the marking matrix to enhance the stability of edge segmentation. Among them, the marking matrix is a two-dimensional matrix used to initialize region marking in image segmentation algorithms (such as the watershed algorithm). Its essence is an integer matrix with the same size as the original image. Each pixel value represents the initial region category to which the position belongs. The labels of key regions can be predefined to guide the segmentation process to more accurately distinguish the target from the background.

[0045] Step S15, perform two-dimensional instantiation segmentation of the building on the building distance map based on the marking matrix to obtain a two-dimensional instantiation object of the building; wherein, the two-dimensional pixels of the two-dimensional instantiation object of the building carry building instantiation labels.

[0046] In this embodiment, a marking matrix corresponding to multi-level contours in the distance map is constructed, and two-dimensional instantiation segmentation of the building can be performed on the building distance map based on the marking matrix. Specifically, the watershed algorithm can be used, and two-dimensional instantiation segmentation of the building is realized based on the marking matrix. For example, the marking matrix is input into the watershed algorithm, and region flooding segmentation is performed in combination with image gradient information to obtain a two-dimensional instantiation object of the building, and building instantiation labels are assigned to the two-dimensional pixels, that is, the two-dimensional pixels of the two-dimensional instantiation object of the building obtained by segmentation carry building instantiation labels. Among them, the image gradient information reflects the local change of pixel values and is often used to detect edges. Regions with large gradient intensities usually correspond to the boundaries of objects.

[0047] For example, see Figure 4As shown in the figure, based on the morphological features of the building anchor points on the building distance map, two-dimensional instantiation segmentation of the building is performed. The specific process is as follows: perform morphological erosion operation on the instantiated building anchor points in the building distance map, extract the multi-level contours in the building distance map, use the RETR_TREE mode to construct the tree-like hierarchical relationship corresponding to the multi-level contours, and then construct the corresponding marking matrix based on the tree-like hierarchical relationship. Finally, input the marking matrix into the watershed algorithm, and perform regional flooding segmentation in combination with the image gradient information to obtain the two-dimensional instantiation object of the building, and assign the building instantiation label to the two-dimensional pixels. Among them, combining the contour tree structure hierarchical organization strategy can improve the processing efficiency of large-scale difference scenarios.

[0048] In this embodiment, after performing two-dimensional instantiation segmentation of the building on the building distance map based on the marking matrix to obtain the two-dimensional instantiation object of the building, it may specifically further include: performing a morphological dilation operation on the two-dimensional instantiation object of the building, and extracting the maximum contour of the dilated two-dimensional instantiation object of the building; simplifying the maximum contour to obtain the simplified maximum contour, and determining the two-dimensional pixels within the simplified maximum contour as the target building instantiation pixels; based on the pre-constructed mapping relationship between the pixel coordinates and the three-dimensional space coordinates, performing three-dimensional space mapping processing on the target building instantiation pixels to obtain the three-dimensional instantiation object of the building.

[0049] For example, perform a morphological dilation operation on each two-dimensional instantiation object of the building to expand its boundary range, extract the maximum contour after dilation, use the Douglas-Peucker algorithm to simplify the maximum contour, and use the pixels within the simplified maximum contour as the target building instantiation pixels. Based on the mapping relationship between the pixel coordinates and the three-dimensional space coordinates, perform three-dimensional space mapping processing on the target building instantiation pixels to obtain the three-dimensional instantiation object of the building, that is: ; Among them, is the height acquisition function, which records the elevation value of the three-dimensional point corresponding to the two-dimensional point. represents the two-dimensional space coordinates of the two-dimensional point. represents the three-dimensional space coordinates of the three-dimensional point. represents the minimum x value in the three-dimensional space. represents the minimum y value in the three-dimensional space.

[0050] It can be seen that in the embodiments of the present invention, the accurate recognition of the vertical structure is achieved through a preset dual-threshold determination mechanism, reducing the misjudgment rate of the vertical structure. By mapping the three-dimensional point cloud to a two-dimensional feature space, the accurate division of complex building instances is realized, improving the segmentation accuracy. Moreover, through a three-level optimization architecture of supervoxels, planar point clusters, and instance anchor points, as well as a contour tree structure, an unsupervised hierarchical segmentation framework is implemented, which can eliminate the dependence on labeled data in the process of building single-instance segmentation, realize the rapid processing of urban-level point clouds, significantly improve the processing ability for complex buildings such as irregular buildings and adhesive structures, and enhance the segmentation accuracy.

[0051] That is, the above technical solution of the present application establishes an unsupervised segmentation mechanism without labeled data, breaks through the data dependence, realizes the collaborative optimization of three-dimensional geometric constraints and two-dimensional morphological features, thereby enhancing the feature expression. Through the extraction of adaptive spatial anchor points, the single-scene processing time is improved, and then the computing efficiency is enhanced. Also, through the strengthening of vertical structure features, the segmentation IoU of complex scenes is improved, the segmentation accuracy is improved, and the consumption of hardware resources is reduced. It can be widely applied in fields such as smart city modeling and three-dimensional registration of real estate.

[0052] In one embodiment, as Figure 5 shown, based on the above building instantiation segmentation method, the present invention also correspondingly provides a building instantiation segmentation device, including: A building point cloud construction module 11, configured to construct a building point cloud of a target building; A planar point cluster extraction module 12, configured to perform planar segmentation on the building point cloud and extract potential vertical planar point clusters from the segmented building planar point clusters; A building vertical structure extraction module 13, configured to map all three-dimensional point clouds in the point cloud set corresponding to the potential vertical planar point clusters to a two-dimensional feature space to obtain a two-dimensional point set, use a preset dual-threshold determination mechanism, and extract two-dimensional point clouds of the building vertical structure based on the two-dimensional point set to obtain a building vertical structure point cluster; An instance anchor generation module 14, configured to generate a building distance map based on the building vertical structure point cluster and generate building instance anchors representing the positions of potential building instances on the building distance map; A tree-like hierarchical relationship construction module 15, configured to extract multi-level contours in the building distance map based on the building instance anchors and construct a tree-like hierarchical relationship corresponding to the multi-level contours; A marking matrix construction module 16, configured to construct a corresponding marking matrix based on the tree-like hierarchical relationship; Instantiate the segmentation module 17 for performing two-dimensional instantiation segmentation of the building on the building distance map based on the marker matrix to obtain a two-dimensional instantiation object of the building; wherein, the two-dimensional pixels of the two-dimensional instantiation object of the building carry building instantiation labels.

[0053] Figure 6 The following is a schematic structural diagram of the terminal provided by the embodiments of the present application. The terminal may include: A memory 501, a processor 502, and a computer program stored on the memory 501 and executable on the processor 502.

[0054] When the processor 502 executes the program, it implements the building instantiation segmentation method provided in the above embodiments.

[0055] Further, the terminal further includes: A communication interface 503 for communication between the memory 501 and the processor 502.

[0056] The memory 501 is used to store a computer program executable on the processor 502.

[0057] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory.

[0058] If the memory 501, the processor 502, and the communication interface 503 are implemented independently, the communication interface 503, the memory 501, and the processor 502 may be interconnected through a bus and communicate with each other. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity, only one line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

[0059] Optionally, in a specific implementation, if the memory 501, the processor 502, and the communication interface 503 are integrated on a chip, the memory 501, the processor 502, and the communication interface 503 may communicate with each other through an internal interface.

[0060] The processor 502 may be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0061] This embodiment also provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the building instantiation segmentation method as described above is implemented.

[0062] Those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention, which follow the general principles of the present invention and include common general knowledge or conventional technical means in the technical field not disclosed in the present application. The specification and embodiments are only to be regarded as exemplary, and the true scope and spirit of the present invention are pointed out by the claims.

[0063] In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, without conflict, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples.

[0064] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, which can be specifically implemented in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, a system including a processor, or other systems that can read instructions from the instruction execution system, apparatus, or device and execute the instructions.

[0065] It should be understood that each part of the present application can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, any one of the following techniques known in the art or a combination thereof can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having suitable combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0066] It should be understood that the application of the present invention is not limited to the above examples. For those of ordinary skill in the art, improvements or transformations can be made according to the above description, and all such improvements and transformations should fall within the protection scope of the appended claims of the present invention.

Claims

1. A method for instantiating and segmenting a building, characterized in that, The method includes: Constructing a building point cloud of a target building, performing plane segmentation on the building point cloud, and extracting a potential vertical plane point cluster from the segmented building plane point clusters; Mapping all three-dimensional point clouds in the point cloud set corresponding to the potential vertical plane point cluster to a two-dimensional feature space to obtain a two-dimensional point set, using a preset double-threshold determination mechanism, and extracting a two-dimensional building vertical structure point cloud based on the two-dimensional point set to obtain a building vertical structure point cluster; Generating a building distance map based on the building vertical structure point cluster, and generating a building instance anchor point representing the position of a potential building instance on the building distance map; Extracting multi-level contours in the building distance map based on the building instance anchor point, constructing a tree-like hierarchical relationship corresponding to the multi-level contours, and constructing a corresponding marking matrix based on the tree-like hierarchical relationship; Performing two-dimensional instantiation segmentation on the building distance map based on the marking matrix to obtain a two-dimensional building instantiation object; wherein, the two-dimensional pixels of the two-dimensional building instantiation object carry building instantiation labels.

2. The building instantiation segmentation method according to claim 1, characterized in that The performing plane segmentation on the building point cloud and extracting a potential vertical plane point cluster from the segmented building plane point clusters includes: Performing plane segmentation on the building point cloud by using a plane region growing algorithm to obtain building plane point clusters; Performing plane fitting on the building plane point clusters by using the least squares method to obtain an optimal plane equation; Extracting a potential vertical plane point cluster based on the optimal plane equation and the size of the building plane point clusters.

3. The building instantiation segmentation method according to claim 2, wherein The mapping all three-dimensional point clouds in the point cloud set corresponding to the potential vertical plane point cluster to a two-dimensional feature space to obtain a two-dimensional point set includes: Constructing a ground point cloud and generating a ground horizontal reference plane based on the ground point cloud fitting; Establishing a local coordinate system on the ground horizontal reference plane based on the centroid point of the ground point cloud; Projecting all three-dimensional point clouds in the point cloud set corresponding to the potential vertical plane point cluster along the normal vector direction of the ground point cloud to the local coordinate system to obtain a two-dimensional point set; Wherein, the using a preset double-threshold determination mechanism and extracting a two-dimensional building vertical structure point cloud based on the two-dimensional point set to obtain a building vertical structure point cluster includes: Establishing a two-dimensional grid of the two-dimensional point set on the ground horizontal reference plane; Counting the number of two-dimensional point clouds in each grid cell of the two-dimensional grid to obtain a vertical projection density feature corresponding to each grid cell; Determining the highest point height and the lowest point height of each grid cell in the two-dimensional grid, and calculating the difference between the highest point height and the lowest point height to obtain a vertical height difference feature corresponding to each grid cell; Determining a density threshold based on the vertical projection density feature and determining a depth threshold based on the vertical height difference feature; Using a preset double-threshold determination mechanism to compare the vertical projection density feature corresponding to each grid cell with the density threshold, and comparing the vertical height difference feature with the depth threshold; When the vertical projection density feature is greater than the density threshold and the vertical height difference feature is greater than the depth threshold, the two-dimensional point cloud in the grid cell is extracted as the two-dimensional point cloud of the building vertical structure, and a point cluster of the building vertical structure is obtained.

4. The building instantiation segmentation method according to claim 3, wherein Determining the density threshold based on the vertical projection density feature and determining the depth threshold based on the vertical height difference feature includes: Based on the vertical projection density feature, determining the mean and variance of the two-dimensional point cloud density in the grid cell, and processing the mean and the variance using a preset density threshold calculation formula to obtain the density threshold; Processing the vertical height difference feature using a preset depth threshold calculation formula to obtain the depth threshold; Among them, the preset density threshold calculation formula is: ； The preset depth threshold calculation formula is: ； ； And, represents the density threshold, represents the mean value of the two-dimensional point cloud density within the grid cell, represents the variability of the two-dimensional point cloud density within the grid cell, represents the depth threshold, represents the vertical height difference feature, represents the height of the highest point within the grid cell, represents the height of the lowest point within the grid cell, represents the threshold control coefficient.

5. The building instantiation segmentation method according to claim 4, characterized in that Generating a building distance map based on the point cluster of the building vertical structure and generating a building instance anchor point representing the position of the potential building instance on the building distance map includes: Projecting all the two-dimensional point clouds of the building vertical structure in the point cluster of the building vertical structure onto the ground horizontal reference plane to obtain a set of building projection points; Determining the maximum and minimum longitudinal coordinates and the maximum and minimum transverse coordinates of the building projection points in the set of projection points; Based on the preset image resolution, the maximum and minimum longitudinal coordinates, and the maximum and minimum transverse coordinates, calculating the image width and the image height respectively; Creating an initial binary image according to the image width and the image height; Translating all the building projection points in the set of building projection points into the image coordinate system constructed based on the initial binary image to obtain a target binary image; Calculating the Euclidean distance between each pixel in the target binary image and the background pixel to obtain the distance value corresponding to each pixel; Linearly mapping the distance value corresponding to each pixel into the standard range of gray values to obtain a building distance map; Traversing the building distance map in the way of a sliding window, and sequentially calculating the average value of the distance values corresponding to all the pixels in the window area to obtain an adaptive threshold corresponding to the window area; Comparing the distance value corresponding to each pixel in the window area with the adaptive threshold, and determining the pixel area with the distance value higher than the adaptive threshold as the foreground to obtain a building instance anchor point representing the position of the potential building instance on the building distance map.

6. The building instantiation segmentation method according to claim 1, characterized in that, Before extracting the multi-level contour in the building distance map based on the building instance anchor point, it further includes: Performing a morphological erosion operation on the building instance anchor point on the building distance map.

7. The building instantiation segmentation method according to any one of claims 1 to 6, characterized in that After performing a building two-dimensional instantiation segmentation on the building distance map based on the marking matrix to obtain a building two-dimensional instantiation object, it further includes: Performing a morphological dilation operation on the building two-dimensional instantiation object and extracting the maximum contour of the dilated building two-dimensional instantiation object; Simplify the maximum contour to obtain a simplified maximum contour, and determine the two-dimensional pixels within the simplified maximum contour as target building instantiated pixels; Based on the pre-constructed mapping relationship between pixel coordinates and three-dimensional space coordinates, perform three-dimensional space mapping processing on the target building instantiated pixels to obtain a building three-dimensional instantiated object.

8. A building instantiation segmentation device, characterized in that, The device includes: A building point cloud construction module for constructing a building point cloud of a target building; A plane point cluster extraction module for performing plane segmentation on the building point cloud and extracting potential vertical plane point clusters from the segmented building plane point clusters; A building vertical structure extraction module for mapping all three-dimensional point clouds in the point cloud set corresponding to the potential vertical plane point clusters to a two-dimensional feature space to obtain a two-dimensional point set, using a preset double-threshold determination mechanism, and extracting two-dimensional point clouds of the building vertical structure based on the two-dimensional point set to obtain a building vertical structure point cluster; An instance anchor generation module for generating a building distance map based on the building vertical structure point cluster and generating building instance anchors representing the positions of potential building instances on the building distance map; A tree-like hierarchical relationship construction module for extracting multi-level contours in the building distance map based on the building instance anchors and constructing a tree-like hierarchical relationship corresponding to the multi-level contours; A marking matrix construction module for constructing a corresponding marking matrix based on the tree-like hierarchical relationship; An instantiation segmentation module for performing building two-dimensional instantiation segmentation on the building distance map based on the marking matrix to obtain a building two-dimensional instantiated object; wherein, the two-dimensional pixels of the building two-dimensional instantiated object carry building instantiation labels.

9. A terminal, characterized in that, Includes: A memory, a processor, and a building instantiation segmentation program stored on the memory and executable on the processor. When the building instantiation segmentation program is executed by the processor, the steps of the building instantiation segmentation method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed to implement the steps of the building instantiation segmentation method according to any one of claims 1 to 7.

Images