Point cloud plane segmentation method and device based on super-point and deep learning

Abstract

The invention provides a point cloud plane segmentation method and device based on super-point and deep learning, and belongs to the technical field of three-dimensional point cloud processing, and the method comprises the steps: determining a rough super-point and a preliminary segmentation result based on an input point cloud; optimizing the boundary of the preliminary segmentation result, and determining unfitted points in the input point cloud; respectively refining the rough super points and the unfitted points to obtain refined super points; extracting depth features of each three-dimensional point in the input point cloud, and determining superpoint features of the refined superpoints based on the depth features; and inputting the super-point feature into a mask prediction model to obtain a super-point mask, and inputting the super-point mask into a segmentation model to obtain a target segmentation result. According to the method, the refined super points are used as calculation elements in the mask prediction model, so that the accuracy of a target segmentation result is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional point cloud processing, and particularly to a method and device for point cloud plane segmentation based on superpoints and deep learning. Background Art

[0002] In the field of 3D city modeling, the point cloud plane segmentation technology shows its core importance. By analyzing the point cloud data collected by a Light Detection and Ranging (LiDAR) system, this technology enables the high-precision extraction of building plane structures. This is of great application value for the three-dimensional reconstruction of buildings, the accuracy of urban planning, and the digital preservation of cultural heritage. In addition, the plane segmentation technology is also indispensable in the design and optimization of building roof photovoltaic systems. The key to the plane segmentation technology is how to accurately identify and segment planes.

[0003] Traditional plane segmentation algorithms, such as region-growing-based and model-fitting-based algorithms, usually rely on prior knowledge of point cloud data, which limits their adaptability and generalization ability in dealing with complex scenes. In practical applications, the limitations of such algorithms are significant, especially when facing the challenges of diverse building structures. In contrast, deep learning methods, with their powerful feature extraction and learning capabilities, have shown broad application prospects in point cloud segmentation tasks. The existing deep learning models segment point clouds by extracting features from each three-dimensional point data of the input point cloud to obtain the segmentation result. This method will significantly increase the computational complexity and resource consumption. To solve this technical problem, it is proposed to first determine the superpoints of the input point cloud and use the superpoints as the computational primitives of the deep learning model to reduce the computational complexity and resource consumption of plane segmentation. However, it has the following technical problems: 1. When the superpoint size is too large, the number of superpoints used to represent the input point cloud is small, which in turn leads to a small amount of feature quantities for deep learning, resulting in a decrease in the accuracy of the segmentation result. 2. The three-dimensional points at the boundaries in the input point cloud are not considered, resulting in blurred boundaries in the plane segmentation result, that is, the segmentation accuracy is low.

[0004] Therefore, there is an urgent need to provide a method and device for point cloud plane segmentation based on superpoints and deep learning to improve the segmentation accuracy of point cloud planes. Summary of the Invention

[0005] In view of this, it is necessary to provide a method and device for point cloud plane segmentation based on superpoints and deep learning to solve the technical problem in the prior art that when using superpoints with a large size for plane segmentation and not considering the three-dimensional points at the boundaries, the plane segmentation accuracy is low.

[0006] In a first aspect, the present invention provides a method for point cloud plane segmentation based on superpoints and deep learning, including: Determine rough superpoints and a preliminary segmentation result based on the input point cloud; Optimize the boundary of the preliminary segmentation result to determine the unfitted points in the input point cloud; Refine the rough superpoints and the unfitted points respectively to obtain refined superpoints; Extract the depth features of each three-dimensional point in the input point cloud, and determine the superpoint features of the refined superpoints based on the depth features; Input the superpoint features into a mask prediction model to obtain a superpoint mask, and input the superpoint mask into a segmentation model to obtain a target segmentation result.

[0007] In some possible implementation manners, the refining the rough superpoints and the unfitted points respectively to obtain refined superpoints includes: Determine a first expected number of clusters of the rough superpoints and a second expected number of clusters of the unfitted points; Perform K-means clustering on the rough superpoints and the unfitted points respectively based on the first expected number of clusters and the second expected number of clusters to obtain the refined superpoints.

[0008] In some possible implementation manners, the second expected number of clusters is 2 times the first expected number of clusters.

[0009] In some possible implementation manners, when the size of the rough superpoints is greater than the size of each cluster after the expected K-means clustering, the first expected number of clusters is:

[0010] When the size of the rough superpoints is less than or equal to the size of each cluster after the expected K-means clustering, the first expected number of clusters is:

[0011] wherein, is the number of three-dimensional points of the input point cloud; is the number of rough superpoints; is the size of each cluster after the expected K-means clustering; is the ceiling symbol.

[0012] In some possible implementation manners, the extracting the depth features of each three-dimensional point in the input point cloud includes: Extract the geometric features and position information of each three-dimensional point, where the geometric features include linearity, flatness, scattering degree, perpendicularity, and plane contour features; Input the input point cloud, the geometric features, and the position information into a feature extraction network for feature extraction to obtain the depth features.

[0013] In some possible implementation manners, determining the superpoint features of the refined superpoints based on the depth features includes: Determine a plurality of target 3D points corresponding to the refined superpoints based on the attribution relationship between the refined superpoints and the 3D points; Perform average pooling operation on the depth features of the plurality of target 3D points to obtain the superpoint features.

[0014] In some possible implementation manners, the mask prediction model includes an instance branch module, a mask branch module, and a prediction head; The instance branch module is used to perform cross-attention learning on the superpoint features to obtain instance features; The mask branch module is used to extract mask features from the superpoint features to obtain mask-aware features; The prediction head is used to multiply the instance features and the mask-aware features to obtain multiplied features, and perform Sigmoid activation processing on the multiplied features to obtain the superpoint masks.

[0015] In some possible implementation manners, the mask branch module uses Fourier transform to capture the non-linear relationship of the superpoint features.

[0016] In some possible implementation manners, the segmentation model is a model based on bipartite graph matching.

[0017] In a second aspect, the present invention further provides a point cloud plane segmentation device based on superpoints and deep learning, including: A rough superpoint and preliminary segmentation unit, configured to determine rough superpoints and a preliminary segmentation result based on an input point cloud; A boundary optimization unit, configured to optimize the boundary of the preliminary segmentation result to determine the unfitted points in the input point cloud; A superpoint refinement unit, configured to refine the rough superpoints and the unfitted points respectively to obtain refined superpoints; A superpoint feature determination unit, configured to extract the depth features of each 3D point in the input point cloud, and determine the superpoint features of the refined superpoints based on the depth features; A plane segmentation unit, configured to input the superpoint features into a mask prediction model to obtain superpoint masks, and input the superpoint masks into a segmentation model to obtain a target segmentation result.

[0018] The beneficial effects of adopting the above embodiments are as follows: The method for point cloud plane segmentation based on superpoints and deep learning provided by the present invention refines the rough superpoints after determining them to obtain refined superpoints, and then determines the superpoint features of the refined superpoints, that is, using the refined superpoints as the computational primitives in the mask prediction model, which increases the number of features of the superpoint features used in the mask prediction model, thereby improving the accuracy of the target segmentation result determined based on the superpoint features.

[0019] Furthermore, the present invention also optimizes the boundaries of the preliminary segmentation result, determines the unfitted points in the input point cloud, and refines the unfitted points. The unfitted points are three-dimensional points with blurred boundaries, that is, the present invention realizes the refinement of the blurred boundaries, thereby improving the boundary accuracy in the segmentation result. In other words, it further improves the accuracy of the target segmentation result. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those skilled in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0021] Figure 1 It is a schematic flowchart of an embodiment of the method for point cloud plane segmentation based on superpoints and deep learning provided by the present invention; Figure 2 For the present invention Figure 2 It is a schematic flowchart of an embodiment of step S103 in the present invention; Figure 3 It is a schematic diagram of the effect of an embodiment of the refined superpoints provided by the present invention; Figure 4 It is a schematic flowchart of an embodiment of extracting depth features in step S104 of the present invention; Figure 5 It is a schematic flowchart of an embodiment of determining the superpoint features of the refined superpoints in step S104 of the present invention; Figure 6 It is a schematic structural diagram of an embodiment of the mask prediction model provided by the present invention; Figure 7 It is a schematic structural diagram of an embodiment of the device for point cloud plane segmentation based on superpoints and deep learning provided by the present invention; Figure 8 It is a schematic structural diagram of an embodiment of the urban modeling device provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0023] It should be understood that the schematic drawings are not drawn to scale. The flowcharts used in the present invention illustrate the operations implemented according to some embodiments of the present invention. It should be understood that the operations in the flowchart may not be implemented in sequence, and steps without logical context relationships may be reversed or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart or remove one or more operations from the flowchart under the guidance of the content of the present invention. Some of the block diagrams shown in the drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor systems and / or microcontroller systems.

[0024] The descriptions such as "first" and "second" involved in the embodiments of the present invention are only for descriptive purposes, and cannot be understood as indicating or implying their relative importance or implicitly indicating the quantity of the indicated technical features. Therefore, the technical features defined with "first" and "second" may explicitly or implicitly include at least one such feature.

[0025] Referring to "embodiments" herein means that the specific features, structures, or characteristics described in connection with the embodiments may be included in at least one embodiment of the present invention. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0026] The present invention provides a point cloud plane segmentation method and device based on superpoints and deep learning, which will be described separately below.

[0027] Figure 1 It is a schematic flowchart of an embodiment of the point cloud plane segmentation method based on superpoints and deep learning proposed for the embodiments of the present invention. As Figure 1 shown, the point cloud plane segmentation method based on superpoints and deep learning includes: S101. Determine rough superpoints and a preliminary segmentation result based on the input point cloud.

[0028] Among them, the methods for determining rough superpoints based on the input point cloud include, but are not limited to, clustering based on point cloud texture features to obtain rough superpoints and obtaining rough superpoints based on point cloud energy information, etc. These are all existing mature methods and will not be specifically limited here.

[0029] Among them, the method for determining the preliminary segmentation result based on the input point cloud is: fitting and obtaining it using an octree-based segmentation method. And, to improve the accuracy of the preliminary segmentation result, after obtaining the preliminary segmentation result by fitting using the octree-based segmentation method, the preliminary segmentation result can also be optimized based on the region growing algorithm.

[0030] Among them, the method of segmenting the input point cloud using the octree-based segmentation method is an existing mature method and will not be elaborated here.

[0031] It should be understood that: the input point cloud in the embodiments of the present invention is a building point cloud to achieve planar segmentation of a building, and it can also be other types such as equipment point cloud, road point cloud, etc., which will not be elaborated one by one here.

[0032] It should also be understood that: the result of the segmentation in the embodiments of the present invention is a plane, for example: roof, wall, etc., to achieve the recognition of the roof plane and the wall.

[0033] S102. Optimize the boundary of the preliminary segmentation result to determine the unfitted points in the input point cloud.

[0034] In a specific embodiment of the present invention, optimizing the boundary of the preliminary segmentation result specifically means: using a local boundary optimization algorithm based on energy optimization to optimize the boundary of the preliminary segmentation result. The local boundary optimization algorithm based on energy optimization is a local optimization algorithm that only optimizes the attribution of points at the intersection of different planes, with high efficiency and can effectively prevent points on different planes from being divided into the same superpoint.

[0035] Among them, specifically, this algorithm transforms the boundary optimization problem into an energy maximization problem, and the energy function is composed of a distance term and a boundary term. The energy function is optimized through a boundary relabeling method, that is, local adjustment is performed in the boundary region to relabel the boundary points to the adjacent optimal plane.

[0036] Among them, unfitted points refer to points whose attribution to a certain plane cannot be determined, that is: three-dimensional points that cannot be fitted to any plane.

[0037] S103. Refine the rough superpoints and the unfitted points respectively to obtain refined superpoints.

[0038] Among them, the refined superpoints are the union of the refined rough superpoints and the refined unfitted points respectively, that is: the refined superpoints include the superpoints refined from the rough superpoints and the superpoints refined from the unfitted points.

[0039] It should be noted that: in the embodiments of the present invention, all the un-fitted points are refined as a large rough superpoint.

[0040] Specifically: refining the superpoint is:

[0041] In the formula, is the superpoint refined from the i-th rough superpoint; is the superpoint after refining the un-fitted points; is the union operator; is the union of the superpoints refined from multiple rough superpoints.

[0042] S104. Extract the depth features of each three-dimensional point in the input point cloud, and determine the superpoint features of the refined superpoints based on the depth features.

[0043] Among them, the difference between the depth features and the superpoint features is that: the superpoint features are features adapted to the mask prediction model, and the depth features are the statistical features of each three-dimensional point.

[0044] S105. Input the superpoint features into the mask prediction model to obtain a superpoint mask, and input the superpoint mask into the segmentation model to obtain the target segmentation result.

[0045] It should be understood that: the point cloud plane segmentation method based on superpoints and deep learning in the embodiments of the present invention can be implemented in any device based on the point cloud plane segmentation based on superpoints and deep learning, such as: urban modeling devices, etc. Specifically, the point cloud plane segmentation method based on superpoints and deep learning is stored in the above device in the form of a prepared program. When the device is started, the program is called, and the point cloud plane segmentation method based on superpoints and deep learning is implemented.

[0046] Compared with the prior art, the point cloud plane segmentation method based on superpoints and deep learning provided by the embodiments of the present invention refines the rough superpoints after determining the rough superpoints to obtain refined superpoints, and then determines the superpoint features of the refined superpoints, that is: using the refined superpoints as the calculation basis in the mask prediction model, increasing the number of features of the superpoint features used in the mask prediction model, and thus improving the accuracy of the target segmentation result determined based on the superpoint features.

[0047] Furthermore, the embodiments of the present invention also optimize the boundary of the preliminary segmentation result, determine the un-fitted points in the input point cloud, and refine the un-fitted points. The un-fitted points are the three-dimensional points with blurred boundaries, that is: the embodiments of the present invention realize the refinement of the blurred boundaries, and thus improve the boundary accuracy in the segmentation result. In other words, the accuracy of the target segmentation result is further improved.

[0048] In some embodiments of the present invention, as Figure 2 shown, step S103 includes: S201. Determine the first expected number of clusters for rough superpoints and the second expected number of clusters for unfitted points.

[0049] Among them, to make the size of the refined superpoints more reasonable, the number of the first expected number of clusters and the second expected number of clusters is not a fixed value, and it changes dynamically to adapt to rough superpoints and unfitted points with different sizes.

[0050] S202. Perform K-means clustering on the rough superpoints and the unfitted points respectively based on the first expected number of clusters and the second expected number of clusters to obtain refined superpoints.

[0051] Among them, the specific process of K-means clustering is as follows: In the feature space, randomly select multiple rough superpoints as the initial cluster centers, and calculate the distances between the spatial coordinates of all three-dimensional points included in the feature space and the spatial coordinates of each initial cluster center. Assign this point to the cluster represented by the nearest cluster center. For each cluster, recalculate its cluster center. The update method of the cluster center is usually to take the mean value of the three-dimensional point coordinates of all superpoints in the cluster. Repeat the above steps until the change in the cluster center is less than a preset threshold or reaches a preset number of iterations, thus completing the clustering process.

[0052] In the embodiments of the present invention, by performing K-means clustering on the rough superpoints and the unfitted points, the size of each cluster is restricted to be similar during K-means clustering. Therefore, the sizes of the clusters generated by K-means clustering are similar, that is: the sizes of the generated refined superpoints are relatively consistent; and the feature used during K-means clustering is the spatial coordinates of three-dimensional points, and the shapes of the finally clustered clusters will also be similar, that is: the shapes of the generated refined superpoints are similar. The refined superpoints with consistent sizes and similar shapes meet the superpoint generation criteria in the mask prediction model, further improving the accuracy of the target segmentation result.

[0053] Since the unfitted points may contain a large amount of noise, in order to ensure that each refined superpoint is mainly composed of points or noise on the same plane, during the refinement process of the unfitted points, a strategy of reducing the scale of the superpoints is adopted. That is: try to set a relatively large second expected number of clusters. To achieve the refinement of the unfitted points and avoid the refined superpoints generated from the unfitted points being on different planes, thereby further improving the accuracy of the target plane segmentation result.

[0054] In the specific embodiments of the present invention, the second expected number of clusters is greater than the first expected number of clusters.

[0055] By setting the second expected number of clusters to be greater than the first expected number of clusters, the embodiment of the present invention can achieve finer-grained clustering of unfitted points, thereby increasing the possibility that the refined super points are located in the same plane, that is, the segmentation accuracy can be further improved.

[0056] In a specific embodiment of the present invention, the second expected number of clusters is twice the first expected number of clusters.

[0057] In a specific embodiment of the present invention, when the size of the rough superpoint is larger than the expected size of each cluster after K-means clustering, the first expected number of clusters is for:

[0058] When the size of the rough superpoint is less than or equal to the expected size of each cluster after K-means clustering, the first expected number of clusters for:

[0059] In the formula, is the number of 3D points in the input point cloud; is the number of rough super points; is the expected size of each cluster after K-means clustering; The symbol for rounding up.

[0060] The embodiment of the present invention sets the first expected number of clusters to be dynamically variable, thereby enabling adjustment of the size of the refined super-points, accurately obtaining a suitable super-point scale, and improving the segmentation speed while ensuring segmentation accuracy.

[0061] In a specific embodiment of the present invention, Figure 3 It can be seen that the size of the refined super point is smaller than that of the coarse super point, that is, the refinement of the coarse super point is achieved.

[0062] In order to achieve comprehensive extraction of deep features, in some embodiments of the present invention, Figure 4 As shown, the step S104 of extracting the depth features of each 3D point in the input point cloud includes: S401, extracting geometric features and position information of each three-dimensional point, where the geometric features include linearity, flatness, scattering, verticality, and plane contour features.

[0063] The position information is a property of the input point cloud, that is, when the input point cloud is obtained, the position information of each three-dimensional point is also obtained.

[0064] Among them, the region with high linearity usually corresponds to edge or straight-line features. The region with high flatness indicates a possible planar structure, which can lock the flat surface during the segmentation process. The scattering degree is used to evaluate the density of the point distribution in each region of the point cloud data. Through this feature, dense regions (such as walls and roofs) and relatively scattered regions (which may be noise or railings) can be effectively identified. The perpendicularity can more accurately identify vertical planes and effectively distinguish vertical surfaces from horizontal planes.

[0065] It should be noted that: the planar contour feature refers to the planar contour feature of the part where the normal vector changes significantly.

[0066] In the embodiment of the present invention, by setting the geometric features including 5 geometric features: linearity, flatness, scattering degree, perpendicularity, and planar contour feature, the geometric features of three-dimensional points can be comprehensively evaluated to ensure the comprehensiveness of the depth features.

[0067] Furthermore, in the embodiment of the present invention, by selecting the planar contour feature of the part where the normal vector changes significantly, the number of features can be reduced while ensuring the comprehensiveness of the features, thereby improving the speed of plane segmentation.

[0068] S402: Input the input point cloud, geometric features, and position information into the feature extraction network for feature extraction to obtain depth features.

[0069] Among them, the purpose of the feature extraction network is to integrate and transform the geometric features and position information of each three-dimensional point in the input point cloud to generate depth features adapted to the mask prediction model.

[0070] In a specific embodiment of the present invention, the feature extraction network is a U-Net network.

[0071] To improve the extraction speed and extraction accuracy of the feature extraction network, in some embodiments of the present invention, before inputting the input point cloud into the feature extraction network, the input point cloud needs to be voxelized. Through voxelization, the irregular point cloud data is converted into a regular voxel grid, thereby providing a standardized input format for subsequent network processing to improve the extraction speed and extraction accuracy of the feature extraction network.

[0072] As can be seen from the foregoing description: the depth feature is the per-point feature of the input point cloud, rather than the superpoint feature of the refined superpoint. And in the embodiment of the present invention, the superpoint is used as the calculation primitive. Therefore, in some embodiments of the present invention, as Figure 5 shown, determining the superpoint feature of the refined superpoint based on the depth feature in step S104 includes: S501: Determine a plurality of target three-dimensional points corresponding to the refined superpoint based on the attribution relationship between the refined superpoint and the three-dimensional points.

[0073] Among them, the attribution relationship can be determined when refining the rough superpoints, that is: determine the three-dimensional points corresponding to each refined superpoint.

[0074] S502. Perform average pooling operation on the depth features of multiple target three-dimensional points to obtain superpoint features.

[0075] Among them, to make the superpoint features more convenient to input into the mask prediction model, after obtaining the superpoint features, the superpoint features can also be projected into a new feature space based on linear projection.

[0076] In a specific embodiment of the present invention, as Figure 6 shown, the mask prediction model includes an instance branch module, a mask branch module, and a prediction head; The instance branch module is used to perform cross-attention learning on the superpoint features to obtain instance features; The mask branch module is used to extract mask features from the superpoint features to obtain mask-aware features; The prediction head is used to multiply the instance features and the mask-aware features to obtain a multiplied feature, and perform Sigmoid activation processing on the multiplied feature to obtain a superpoint mask.

[0077] In a specific embodiment of the present invention, the instance branch module is a Transfoemer structure, and the mask branch module is a KAN (Kolmogorov-Arnold Networks) structure.

[0078] The Transformer structure is used to handle the disorder and quantity uncertainty of superpoints, enabling it to effectively process variable-length inputs. Decode the learnable query vector through the superpoint cross-attention mechanism. Assume that the feature input from the superpoint pooling layer is L query vectors. We can pre-define the feature of the query vector of the Transformer decoder layer as:

[0079] In the formula, D is the number of embedding layers, and i is the index of the transformer layer. Capture context information through the superpoint cross-attention mechanism The formula can be expressed as:

[0080] Among them, A is the superpoint attention mask, Q is the query vector, K represents the importance of the input vector, and V is the superpoint feature with different linear projections.

[0081] In the embodiments of the present invention, by setting the mask branch module to the KAN structure, since the KAN structure can use a more parameter - efficient form to approximate non - linear functions, it has fewer parameters under the same complexity and stronger expressive power under the same number of parameters. Therefore, the accuracy and precision of plane segmentation can be further improved.

[0082] To further improve the performance of plane segmentation, in some embodiments of the present invention, the mask branch module uses Fourier transform to capture the non - linear relationship of super - point features. That is: the mask branch module is of the FourierKAN structure.

[0083] Among them, the core implementation of FourierKAN is based on Fourier transform, that is: the feature transformation is completed through the linear combination of cosine and sine functions.

[0084] In the embodiments of the present invention, FourierKAN is used to replace the combination of traditional linear layers and non - linear activation functions. FourierKAN can effectively capture the periodic patterns in the input data by using Fourier coefficients for feature transformation, especially suitable for dealing with problems with complex non - linear relationships and periodic features. By constraining the high - frequency Fourier coefficients through a regularization term, the smoothness of the function is ensured, and the performance of the mask prediction model is further improved, that is: the accuracy of plane segmentation is further improved.

[0085] More specifically, the embodiments of the present invention use one - dimensional Fourier coefficients to replace the B - spline coefficients in KAN. And the Fourier coefficient g is set to 5, replacing two - layer multi - layer perceptrons. Let the network learn g groups of Fourier coefficients for weighting the transformed features. The weighted features are passed through the inverse Fourier transform to obtain the final output. The formula of FourierKAN is as follows:

[0086] In the formula, and are the weights to be learned in the mask branch module.

[0087] In some embodiments of the present invention, the segmentation model is a model based on bipartite graph matching.

[0088] It should be noted that: the segmentation model needs to be trained with samples to be labeled before use.

[0089] In the embodiments of the present invention, by converting the real - label assignment problem into an optimal assignment problem through a bipartite - graph - matching model, an end - to - end training framework is realized, making the iteration speed of the segmentation model faster, the overall learning ability stronger, and the ability to handle scenarios improving faster.

[0090] In summary, the point cloud plane segmentation method based on superpoints and deep learning proposed in the embodiments of the present invention uses refined superpoints as the basic processing unit, significantly reducing the computational complexity of the Transformer. In addition, two characteristics that high-quality superpoints for the Transformer should possess are proposed, and a corresponding multi-stage superpoint generation process is proposed, making the generated superpoints not only have precise boundaries but also consistent geometric sizes, which are very beneficial to the feature learning of the boutonniere superpoint Transformer. Moreover, in order to make up for the deficiency of deep learning features under a limited training set, geometric features in multiple dimensions are introduced into the model, further improving the plane segmentation performance. Further, the embodiments of the present invention also construct a mask prediction model combining the Kolmogorov-Arnold Network and the Transformer module, effectively optimizing the instance prediction and mask extraction processes and enhancing the overall segmentation performance. Finally, the real label assignment problem is formulated as an optimal assignment problem, and an end-to-end training framework is achieved through bipartite graph matching based on the superpoint mask. In summary, the point cloud plane segmentation method based on superpoints and deep learning proposed in the embodiments of the present invention can accurately segment planes, and its performance is significantly better than other traditional methods and existing deep learning-based models.

[0091] To better implement the point cloud plane segmentation method based on superpoints and deep learning in the embodiments of the present invention, correspondingly, the embodiments of the present invention also provide a point cloud plane segmentation device based on superpoints and deep learning, as Figure 7 shown. The point cloud plane segmentation device 700 based on superpoints and deep learning includes: A rough superpoint and preliminary segmentation unit 701, configured to determine rough superpoints and a preliminary segmentation result based on the input point cloud; A boundary optimization unit 702, configured to optimize the boundaries of the preliminary segmentation result to determine the unfitted points in the input point cloud; A superpoint refinement unit 703, configured to refine the rough superpoints and the unfitted points respectively to obtain refined superpoints; A superpoint feature determination unit 704, configured to extract the depth features of each three-dimensional point in the input point cloud and determine the superpoint features of the refined superpoints based on the depth features; A plane segmentation unit 705, configured to input the superpoint features into the mask prediction model to obtain a superpoint mask, and input the superpoint mask into the segmentation model to obtain a target segmentation result.

[0092] The point cloud plane segmentation device 700 based on superpoints and deep learning provided in the above embodiments can implement the technical solutions described in the above embodiments of the point cloud plane segmentation method based on superpoints and deep learning. The specific implementation principles of the above modules or units can be referred to the corresponding content in the above embodiments of the point cloud plane segmentation method based on superpoints and deep learning, and will not be elaborated here.

[0093] As Figure 8 shown, the present invention also correspondingly provides an urban modeling device 800. The urban modeling device 800 includes a processor 801, a memory 802, and a display 803. Figure 8 Only some components of the urban modeling device 800 are shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented.

[0094] The memory 802 can be an internal storage unit of the urban modeling device 800 in some embodiments, such as the hard disk or memory of the urban modeling device 800. The memory 802 can also be an external storage device of the urban modeling device 800 in other embodiments, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the urban modeling device 800.

[0095] The processor 801 can be a Central Processing Unit (CPU), a microprocessor, or other data processing chips in some embodiments, and is used to run the program code stored in the memory 802 or process data, such as the point cloud plane segmentation method based on superpoints and deep learning in the present invention.

[0096] The display 803 can be an LED display, a liquid crystal display, a touch liquid crystal display, and an OLED (Organic Light-Emitting Diode) toucher, etc. in some embodiments. The display 803 is used to display the information of the urban modeling device 800 and to display a visual user interface. The components 801 - 803 of the urban modeling device 800 communicate with each other through a system bus.

[0097] In some embodiments of the present invention, when the processor 801 executes the point cloud plane segmentation program in the memory 802, the following steps can be implemented: Determine rough superpoints and a preliminary segmentation result based on the input point cloud; Optimize the boundary of the preliminary segmentation result to determine the unfitted points in the input point cloud; Refine the rough superpoints and the unfitted points respectively to obtain refined superpoints; Extract the depth features of each three-dimensional point in the input point cloud, and determine the superpoint features of the refined superpoints based on the depth features; Input the superpoint features into a mask prediction model to obtain a superpoint mask, and input the superpoint mask into a segmentation model to obtain a target segmentation result.

[0098] It should be understood that when the processor 801 executes the point cloud plane segmentation program based on superpoints and deep learning in the memory 802, in addition to the above functions, other functions can also be implemented. For specific details, please refer to the description of the relevant method embodiments above.

[0099] On the other hand, an embodiment of the present invention further provides a computer-readable storage medium, which is used to store computer-readable programs or instructions. When the programs or instructions are executed by a processor, the steps or functions in the point cloud plane segmentation method based on superpoints and deep learning provided by the above method embodiments can be implemented.

[0100] Those skilled in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a computer-readable storage medium. The computer-readable storage medium is a disk, an optical disk, a read-only memory or a random access memory, etc.

[0101] The above has introduced in detail the point cloud plane segmentation method and device based on superpoints and deep learning provided by the present invention. Specific examples are used in this article to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A method for point cloud plane segmentation based on superpoints and deep learning, characterized in that, Including: Determine rough superpoints and a preliminary segmentation result based on the input point cloud; Optimize the boundary of the preliminary segmentation result to determine the unfitted points in the input point cloud; Refine the rough superpoints and the unfitted points respectively to obtain refined superpoints; Extract the depth features of each three-dimensional point in the input point cloud, and determine the superpoint features of the refined superpoints based on the depth features; Input the superpoint features into a mask prediction model to obtain a superpoint mask, and input the superpoint mask into a segmentation model to obtain a target segmentation result.

2. The method for segmenting point cloud planes based on superpoints and deep learning according to claim 1, wherein The refining the rough superpoints and the unfitted points respectively to obtain refined superpoints includes: Determine the first expected number of clusters of the rough superpoints and the second expected number of clusters of the unfitted points; Perform K-means clustering on the rough superpoints and the unfitted points respectively based on the first expected number of clusters and the second expected number of clusters to obtain the refined superpoints.

3. The method for segmenting point cloud planes based on superpoints and deep learning according to claim 2, characterized in that, The second expected number of clusters is twice the first expected number of clusters.

4. The method for segmenting a point cloud plane based on superpoints and deep learning according to claim 2, characterized in that When the size of the rough superpoints is greater than the size of each cluster after the desired K-means clustering, the first desired number of clusters is as follows: When the size of the rough superpoints is less than or equal to the size of each cluster after the desired K-means clustering, the first desired number of clusters is: In the formula, is the number of three-dimensional points of the input point cloud; is the number of rough superpoints; is the size of each cluster after the expected K-means clustering; is the ceiling symbol.

5. The method for segmenting point cloud planes based on superpoints and deep learning according to claim 1, wherein The extracting the depth features of each three-dimensional point in the input point cloud includes: Extract the geometric features and position information of each three-dimensional point, where the geometric features include linearity, flatness, scattering degree, perpendicularity, and plane contour features; Input the input point cloud, the geometric features, and the position information into a feature extraction network for feature extraction to obtain the depth features.

6. The method for segmenting point cloud planes based on superpoints and deep learning according to claim 1, wherein Determining the superpoint features of the refined superpoints based on the depth features includes: Determine multiple target three-dimensional points corresponding to the refined superpoints based on the attribution relationship between the refined superpoints and the three-dimensional points; Perform average pooling operation on the depth features of the multiple target three-dimensional points to obtain the superpoint features.

7. The method for segmenting a point cloud plane based on super points and deep learning according to claim 1, characterized in that, The mask prediction model includes an instance branch module, a mask branch module, and a prediction head; The instance branch module is used to perform cross-attention learning on the superpoint features to obtain instance features; The mask branch module is used to extract mask features from the superpoint features to obtain mask-aware features; The prediction head is used to multiply the instance features and the mask-aware features to obtain a multiplied feature, and perform Sigmoid activation processing on the multiplied feature to obtain the superpoint mask.

8. The method for segmenting point cloud planes based on superpoints and deep learning according to claim 7, characterized in that, The mask branch module uses Fourier transform to capture the non-linear relationship of the superpoint features.

9. The method for segmenting a point cloud plane based on superpoints and deep learning according to claim 1, characterized in that, The segmentation model is a model based on bipartite graph matching.

10. A point cloud plane segmentation device based on superpoints and deep learning, characterized in that, Including: A rough superpoint and preliminary segmentation unit for determining rough superpoints and a preliminary segmentation result based on the input point cloud; A boundary optimization unit for optimizing the boundary of the preliminary segmentation result to determine the unfitted points in the input point cloud; A superpoint refinement unit for refining the rough superpoints and the unfitted points respectively to obtain refined superpoints; A superpoint feature determination unit for extracting the depth features of each three-dimensional point in the input point cloud and determining the superpoint features of the refined superpoints based on the depth features; A plane segmentation unit for inputting the superpoint features into a mask prediction model to obtain a superpoint mask, and inputting the superpoint mask into a segmentation model to obtain a target segmentation result.

Images