A point cloud plane segmentation method and device based on superpoint and deep learning

Abstract

The present invention provides a method and device for point cloud plane segmentation based on superpoints and deep learning, belonging to the technical field of three-dimensional point cloud processing. The method includes: determining rough superpoints and a preliminary segmentation result based on an input point cloud; optimizing the boundaries of the preliminary segmentation result to determine unfitted points in the input point cloud; respectively refining the rough superpoints and the unfitted points to obtain refined superpoints; extracting depth features of each three-dimensional point in the input point cloud and determining superpoint features of the refined superpoints based on the depth features; inputting the superpoint features into a mask prediction model to obtain a superpoint mask, and then inputting the superpoint mask into a segmentation model to obtain a target segmentation result. The present invention uses refined superpoints as computational primitives in the mask prediction model, thereby improving the accuracy of the target segmentation result.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional point cloud processing, and specifically to a point cloud plane segmentation method and device based on super points and deep learning. Background Art

[0002] In the field of 3D urban modeling, point cloud plane segmentation technology has demonstrated its central importance. By analyzing point cloud data collected by LiDAR systems, this technology enables high-precision extraction of building planar structures. This has significant applications in 3D reconstruction of buildings, accurate urban planning, and the digital preservation of cultural heritage. Furthermore, plane segmentation technology is indispensable in the design and optimization of rooftop photovoltaic systems. The key to plane segmentation technology lies in accurately identifying and segmenting planes.

[0003] Traditional plane segmentation algorithms, such as those based on region growing and model fitting, typically rely on prior knowledge of point cloud data, limiting their adaptability and generalization capabilities when dealing with complex scenes. In practical applications, these algorithms exhibit significant limitations, particularly when addressing the challenges of diverse architectural structures. In contrast, deep learning methods, with their powerful feature extraction and learning capabilities, have shown broad application prospects in point cloud segmentation tasks. Existing deep learning models perform plane segmentation on point clouds by extracting features from each 3D point in the input point cloud to generate a segmentation result. This approach significantly increases computational complexity and resource consumption. To address this technical issue, a method has been proposed to first identify superpoints in the input point cloud and use them as computational primitives in the deep learning model, thereby reducing the computational complexity and resource consumption of plane segmentation. However, this method suffers from the following technical issues: 1. When the superpoint size is too large, the number of superpoints used to represent the input point cloud is reduced, resulting in a smaller number of features for deep learning and reduced segmentation accuracy. 2. It fails to consider 3D points at the boundary of the input point cloud, resulting in blurred boundaries in the plane segmentation results, i.e., low segmentation accuracy.

[0004] Therefore, there is an urgent need to provide a point cloud plane segmentation method and device based on super points and deep learning to improve the segmentation accuracy of point cloud planes. Summary of the Invention

[0005] In view of this, it is necessary to provide a point cloud plane segmentation method and device based on superpoints and deep learning to solve the technical problem in the existing technology that large superpoints are used for plane segmentation and the three-dimensional points at the boundary are not considered, resulting in low plane segmentation accuracy.

[0006] In a first aspect, the present invention provides a point cloud plane segmentation method based on super points and deep learning, comprising:

[0007] Determine rough superpoints and preliminary segmentation results based on the input point cloud;

[0008] Optimizing the boundaries of the preliminary segmentation result to determine unfitted points in the input point cloud;

[0009] Refining the rough super points and the unfitted points respectively to obtain refined super points;

[0010] Extracting depth features of each three-dimensional point in the input point cloud, and determining super-point features of the refined super-point based on the depth features;

[0011] The superpoint features are input into a mask prediction model to obtain a superpoint mask, and the superpoint mask is input into a segmentation model to obtain a target segmentation result.

[0012] In some possible implementations, respectively refining the rough super point and the unfitted point to obtain a refined super point includes:

[0013] Determining a first expected number of clusters of the rough superpoints and a second expected number of clusters of the unfitted points;

[0014] K-means clustering is performed on the rough super points and the unfitted points based on the first expected number of clusters and the second expected number of clusters to obtain the refined super points.

[0015] In some possible implementations, the second expected number of clusters is twice the first expected number of clusters.

[0016] In some possible implementations, when the size of the rough superpoint is larger than the expected size of each cluster after K-means clustering, the first expected number of clusters is for:

[0017]

[0018] When the size of the rough superpoint is less than or equal to the expected size of each cluster after K-means clustering, the first expected number of clusters is for:

[0019]

[0020] Where, is the number of 3D points in the input point cloud; is the number of rough super points; The size of each cluster after the desired K-means clustering; The symbol for rounding up.

[0021] In some possible implementations, extracting a depth feature of each three-dimensional point in the input point cloud includes:

[0022] Extracting geometric features and position information of each of the three-dimensional points, wherein the geometric features include linearity, flatness, scattering, perpendicularity, and plane profile features;

[0023] The input point cloud, the geometric features and the position information are input into a feature extraction network for feature extraction to obtain the depth features.

[0024] In some possible implementations, determining the super-point feature of the refined super-point based on the depth feature includes:

[0025] Determining a plurality of target three-dimensional points corresponding to the refined super-points based on the attribution relationship between the refined super-points and the three-dimensional points;

[0026] An average pooling operation is performed on the depth features of the multiple target three-dimensional points to obtain the super-point features.

[0027] In some possible implementations, the mask prediction model includes an instance branch module, a mask branch module, and a prediction head;

[0028] The instance branch module is used to perform cross-attention learning on the super-point features to obtain instance features;

[0029] The mask branch module is used to perform mask feature extraction on the super point feature to obtain mask perception features;

[0030] The prediction head is used to multiply the instance feature and the mask perception feature to obtain a multiplied feature, and perform Sigmoid activation processing on the multiplied feature to obtain the super point mask.

[0031] In some possible implementations, the mask branch module uses Fourier transform to capture the nonlinear relationship of the super-point features.

[0032] In some possible implementations, the segmentation model is a model based on bipartite graph matching.

[0033] In a second aspect, the present invention further provides a point cloud plane segmentation device based on super points and deep learning, comprising:

[0034] A rough superpoint and preliminary segmentation unit, configured to determine a rough superpoint and preliminary segmentation result based on an input point cloud;

[0035] a boundary optimization unit, configured to optimize the boundary of the preliminary segmentation result and determine unfitted points in the input point cloud;

[0036] a superpoint refinement unit, configured to refine the rough superpoints and the unfitted points respectively to obtain refined superpoints;

[0037] a super-point feature determination unit, configured to extract depth features of each three-dimensional point in the input point cloud, and determine super-point features of the refined super-point based on the depth features;

[0038] The plane segmentation unit is used to input the superpoint feature into the mask prediction model to obtain the superpoint mask, and input the superpoint mask into the segmentation model to obtain the target segmentation result.

[0039] The beneficial effect of adopting the above embodiment is: the point cloud plane segmentation method based on superpoints and deep learning provided by the present invention, after determining the rough superpoints, refines the rough superpoints to obtain refined superpoints, and then determines the superpoint features of the refined superpoints, that is: the refined superpoints are used as calculation primitives in the mask prediction model, which increases the number of features of the superpoint features used in the mask prediction model, thereby improving the accuracy of the target segmentation results determined based on the superpoint features.

[0040] Furthermore, the present invention also optimizes the boundaries of the preliminary segmentation results, determines the unfitted points in the input point cloud, and refines the unfitted points. The unfitted points are three-dimensional points with blurred boundaries. That is, the present invention realizes the refinement of the fuzzy boundaries, thereby improving the boundary accuracy in the segmentation results. In other words, the accuracy of the target segmentation results is further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 A schematic flow chart of an embodiment of a point cloud plane segmentation method based on super points and deep learning provided by the present invention;

[0043] Figure 2 For the present invention Figure 2 A schematic flow chart of an embodiment of step S103;

[0044] Figure 3 A schematic diagram showing the effect of an embodiment of the refinement of super points provided by the present invention;

[0045] Figure 4 This is a schematic diagram of an embodiment of the process of extracting depth features in step S104 of the present invention;

[0046] Figure 5This is a flow chart of an embodiment of determining super-point features of a refined super-point in step S104 of the present invention;

[0047] Figure 6 A schematic structural diagram of an embodiment of the mask prediction model provided by the present invention;

[0048] Figure 7 A schematic structural diagram of an embodiment of a point cloud plane segmentation device based on super points and deep learning provided by the present invention;

[0049] Figure 8 This is a schematic structural diagram of an embodiment of the city modeling device provided by the present invention. DETAILED DESCRIPTION

[0050] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0051] It should be understood that the schematic drawings are not drawn to scale. The flowcharts used in the present invention illustrate operations implemented according to some embodiments of the present invention. It should be understood that the operations of the flowcharts can be implemented out of sequence, and steps that have no logical contextual relationship can be reversed in order or implemented simultaneously. In addition, those skilled in the art, guided by the content of the present invention, can add one or more other operations to the flowcharts or remove one or more operations from the flowcharts. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and / or processor systems and / or microcontroller systems.

[0052] The terms "first" and "second" in the embodiments of the present invention are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature designated as "first" or "second" may explicitly or implicitly include at least one such feature.

[0053] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0054] The present invention provides a point cloud plane segmentation method and device based on super points and deep learning, which are described below.

[0055] Figure 1 This is a flow chart of an embodiment of the point cloud plane segmentation method based on super points and deep learning proposed in an embodiment of the present invention. Figure 1 As shown in Figure 2, point cloud plane segmentation methods based on super points and deep learning include:

[0056] S101 : Determine rough superpoints and preliminary segmentation results based on the input point cloud.

[0057] Among them, the methods of determining rough super points based on the input point cloud include but are not limited to obtaining rough super points by clustering based on point cloud texture features and obtaining rough super points based on point cloud energy information, etc., which are all existing mature methods and are not specifically limited here.

[0058] The initial segmentation result is determined based on the input point cloud by fitting it using an octree-based segmentation method. Furthermore, to improve the accuracy of the initial segmentation result, after obtaining the initial segmentation result using the octree-based segmentation method, the initial segmentation result can be optimized using a region growing algorithm.

[0059] Among them, the octree-based segmentation method for segmenting the input point cloud is an existing mature method and will not be described in detail here.

[0060] It should be understood that the input point cloud in the embodiment of the present invention is a building point cloud to achieve plane segmentation of the building. It can also be other point clouds such as equipment point clouds, road point clouds, etc., which are not described in detail here.

[0061] It should also be understood that the segmentation result in the embodiment of the present invention is a plane, such as a roof, a wall, etc., so as to realize the recognition of the roof plane and the wall surface.

[0062] S102: Optimize the boundary of the preliminary segmentation result to determine unfitted points in the input point cloud.

[0063] In a specific embodiment of the present invention, optimizing the boundaries of the preliminary segmentation result involves employing a local boundary optimization algorithm based on energy optimization. This local boundary optimization algorithm is a local optimization algorithm that optimizes only the attribution of points at the intersection of different planes. This algorithm is highly efficient and effectively prevents points from different planes from being grouped together as a single superpoint.

[0064] Specifically, the algorithm formulates the boundary optimization problem as an energy maximization problem, where the energy function consists of a distance term and a boundary term. The energy function is optimized using a boundary relabeling method, which performs local adjustments in the boundary region and relabels boundary points to the adjacent optimal plane.

[0065] Among them, unfitted points refer to points that cannot be determined to which plane they belong, that is, three-dimensional points that cannot be fitted to any plane.

[0066] S103 , respectively refine the rough super points and the unfitted points to obtain refined super points.

[0067] The refined superpoints are the union of the rough superpoints and the unfitted points after being refined, that is, the refined superpoints include the superpoints obtained by refining the rough superpoints and the superpoints obtained by refining the unfitted points.

[0068] It should be noted that the embodiment of the present invention treats all unfitted points as a large rough point for refinement.

[0069] Specifically: Refine the super point for:

[0070]

[0071] Where, is the super point refined from the i-th rough super point; is the super point after the unfitted point is refined; is the union operator; It is the union of superpoints refined from multiple rough superpoints.

[0072] S104: Extracting depth features of each 3D point in the input point cloud, and determining super-point features of the refined super-point based on the depth features.

[0073] The difference between deep features and super-point features is that super-point features are features adapted to the mask prediction model, while deep features are statistical features of each three-dimensional point.

[0074] S105: Input the superpoint feature into the mask prediction model to obtain a superpoint mask, and input the superpoint mask into the segmentation model to obtain a target segmentation result.

[0075] It should be understood that the super-point and deep learning-based point cloud plane segmentation method in the embodiments of the present invention can be implemented in any device that uses super-point and deep learning-based point cloud plane segmentation, such as a city modeling device. Specifically, the super-point and deep learning-based point cloud plane segmentation method is stored in the aforementioned device as a pre-programmed program. When the device is powered on, the program is invoked and the super-point and deep learning-based point cloud plane segmentation method is implemented.

[0076] Compared with the prior art, the point cloud plane segmentation method based on superpoints and deep learning provided by an embodiment of the present invention, after determining the rough superpoints, refines the rough superpoints to obtain refined superpoints, and then determines the superpoint features of the refined superpoints, that is: the refined superpoints are used as calculation primitives in the mask prediction model, which increases the number of features of the superpoint features used in the mask prediction model, thereby improving the accuracy of the target segmentation results determined based on the superpoint features.

[0077] Furthermore, the embodiment of the present invention also optimizes the boundaries of the preliminary segmentation results, determines the unfitted points in the input point cloud, and refines the unfitted points. The unfitted points are three-dimensional points with blurred boundaries. That is, the embodiment of the present invention realizes the refinement of the fuzzy boundaries, thereby improving the boundary accuracy in the segmentation results. In other words, the accuracy of the target segmentation results is further improved.

[0078] In some embodiments of the present invention, Figure 2 As shown, step S103 includes:

[0079] S201: Determine a first expected number of clusters of rough super points and a second expected number of clusters of unfitted points.

[0080] In order to make the size of the refined super-point more reasonable, the first expected number of clusters and the second expected number of clusters are not fixed values, but are dynamically changed to adapt to the rough super-points and unfitted points of different sizes.

[0081] S202 : Perform K-means clustering on the rough super points and the unfitted points based on the first expected number of clusters and the second expected number of clusters to obtain refined super points.

[0082] The specific process of K-means clustering is as follows: within the feature space, multiple rough superpoints are randomly selected as initial cluster centers. The distance between the spatial coordinates of all 3D points contained in the feature space and each initial cluster center is calculated. The point is assigned to the cluster represented by the cluster center closest to it. For each cluster, its cluster center is recalculated. The cluster center is usually updated by taking the mean of the 3D coordinates of all superpoints in the cluster. These steps are repeated until the change in cluster center is less than a preset threshold or the preset number of iterations is reached, thus completing the clustering process.

[0083] The present invention performs K-means clustering on rough superpoints and unfitted points. K-means clustering constrains the size of each cluster to be similar. Therefore, the sizes of the clusters generated by K-means clustering are similar, meaning the sizes of the resulting refined superpoints are relatively consistent. Furthermore, because the features used in K-means clustering are the spatial coordinates of three-dimensional points, the shapes of the resulting clusters are also similar, meaning the shapes of the resulting refined superpoints are similar. Refined superpoints of consistent size and similar shape meet the superpoint generation criteria in the mask prediction model, further improving the accuracy of the target segmentation results.

[0084] Because unfitted points may contain a significant amount of noise, a strategy of reducing the size of superpoints is employed during the unfitted point refinement process to ensure that each refined superpoint is primarily composed of points or noise on the same plane. Specifically, a larger second desired cluster number is set as possible. This allows for refinement of unfitted points and prevents refined superpoints generated from unfitted points from being located on different planes, further improving the accuracy of the target plane segmentation results.

[0085] In a specific embodiment of the present invention, the second expected number of clusters is greater than the first expected number of clusters.

[0086] By setting the second expected number of clusters to be greater than the first expected number of clusters, the embodiment of the present invention can achieve finer-grained clustering of unfitted points, thereby increasing the possibility that the refined super points are located in the same plane, that is, further improving the segmentation accuracy.

[0087] In a specific embodiment of the present invention, the second expected number of clusters is twice the first expected number of clusters.

[0088] In a specific embodiment of the present invention, when the size of the rough superpoint is larger than the expected size of each cluster after K-means clustering, the first expected number of clusters is for:

[0089]

[0090] When the size of the rough superpoint is less than or equal to the expected size of each cluster after K-means clustering, the first expected number of clusters for:

[0091]

[0092] Where, is the number of 3D points in the input point cloud; is the number of rough super points; The size of each cluster after the desired K-means clustering; The symbol for rounding up.

[0093] The embodiment of the present invention sets the first expected number of clusters to be dynamically variable, thereby enabling adjustment of the size of the refined super-points, accurately obtaining a suitable super-point scale, and improving the segmentation speed while ensuring segmentation accuracy.

[0094] In a specific embodiment of the present invention, Figure 3 It can be seen from the figure that the size of the refined superpoint is smaller than that of the coarse superpoint, that is, the refinement of the coarse superpoint is achieved.

[0095] In order to achieve comprehensive extraction of depth features, in some embodiments of the present invention, such as Figure 4 As shown, the step S104 of extracting the depth features of each 3D point in the input point cloud includes:

[0096] S401 , extracting geometric features and position information of each three-dimensional point, where the geometric features include linearity, flatness, scattering, perpendicularity, and plane profile features.

[0097] The position information is an inherent attribute of the input point cloud, that is, when the input point cloud is obtained, the position information of each 3D point is also obtained.

[0098] Regions with high linearity typically correspond to edges or straight line features. Regions with high flatness indicate possible planar structures, which helps identify flat surfaces during segmentation. Scattering measures the density of point distribution within each region of point cloud data. This feature can effectively distinguish dense areas (such as walls and roofs) from more dispersed areas (such as noise or railings). Perpendicularity more accurately identifies vertical planes and effectively distinguishes vertical surfaces from horizontal ones.

[0099] It should be noted that the plane contour feature refers to the plane contour feature of the part where the normal vector changes significantly.

[0100] The embodiment of the present invention sets five geometric features including linearity, flatness, scattering, perpendicularity and plane contour features, so as to comprehensively evaluate the geometric features of three-dimensional points and ensure the comprehensiveness of depth features.

[0101] Furthermore, the embodiment of the present invention selects the plane contour features of the portion where the normal vector changes significantly, which can reduce the number of features while ensuring the comprehensiveness of the features, thereby increasing the speed of plane segmentation.

[0102] S402: Input the input point cloud, geometric features and position information into the feature extraction network to extract features and obtain deep features.

[0103] Among them, the purpose of the feature extraction network is to integrate and transform the geometric features and position information of each 3D point in the input point cloud to generate deep features that are suitable for the mask prediction model.

[0104] In a specific embodiment of the present invention, the feature extraction network is a U-Net network.

[0105] To improve the extraction speed and accuracy of the feature extraction network, some embodiments of the present invention require voxelization of the input point cloud before feeding it into the feature extraction network. Voxelization converts irregular point cloud data into a regular voxel grid, providing a standardized input format for subsequent network processing, thereby improving the extraction speed and accuracy of the feature extraction network.

[0106] From the above description, it can be seen that the deep feature is the point-by-point feature of the input point cloud, rather than the super-point feature of the refined super-point. However, the embodiment of the present invention uses the super-point as the calculation primitive. Therefore, in some embodiments of the present invention, such as Figure 5 As shown, the step S104 of determining the super-point features of the refined super-point based on the depth features includes:

[0107] S501 : Determine a plurality of target three-dimensional points corresponding to the refined super-points based on the attribution relationship between the refined super-points and the three-dimensional points.

[0108] The attribution relationship can be determined when the rough superpoints are refined, that is, the three-dimensional point corresponding to each refined superpoint is determined.

[0109] S502: Perform an average pooling operation on the depth features of multiple target three-dimensional points to obtain super-point features.

[0110] In order to make the super-point feature more convenient for inputting into the mask prediction model, after obtaining the super-point feature, the super-point feature can be projected into a new feature space based on linear projection.

[0111] In a specific embodiment of the present invention, Figure 6 As shown, the mask prediction model includes an instance branch module, a mask branch module and a prediction head;

[0112] The instance branch module is used to perform cross-attention learning on super-point features to obtain instance features;

[0113] The mask branch module is used to extract mask features from super-point features to obtain mask-aware features;

[0114] The prediction head is used to multiply the instance feature and the mask-aware feature to obtain the multiplied feature, and then perform Sigmoid activation on the multiplied feature to obtain the super-point mask.

[0115] In a specific embodiment of the present invention, the instance branch module is a Transfoemer structure, and the mask branch module is a KAN (Kolmogorov-Arnold Networks) structure.

[0116] The Transformer structure is used to handle the disorder and uncertainty of the number of super points, making it able to effectively handle variable-length input. The learnable query vector is decoded through the super point cross attention mechanism. Assume that the feature input from the super point pooling layer is L query vector, we can predefine the features of the query vector of the Transformer decoder layer as:

[0117]

[0118] Where D is the number of embedding layers and i is the index of the transformer layer. The context information is captured by the super-point cross attention mechanism The formula can be expressed as:

[0119]

[0120] Where A is the super-point attention mask, Q is the query vector, K represents the importance of the input vector, and V is the super-point feature with different linear projections.

[0121] In this embodiment of the present invention, the mask branch module is configured as a KAN structure. Because the KAN structure can express and approximate nonlinear functions in a more parameter-efficient form, it has a smaller number of parameters when the complexity is comparable, and has stronger expressive power when the number of parameters is comparable. Therefore, the precision and accuracy of plane segmentation can be further improved.

[0122] To further improve the performance of plane segmentation, in some embodiments of the present invention, the mask branch module uses Fourier transform to capture the nonlinear relationship of super-point features. That is, the mask branch module is a FourierKAN structure.

[0123] Among them, the core implementation of FourierKAN is based on Fourier transform, that is, feature conversion is completed through the linear combination of cosine and sine functions.

[0124] In this embodiment of the present invention, FourierKAN replaces the traditional combination of linear layers and nonlinear activation functions. By utilizing Fourier coefficients for feature transformation, FourierKAN effectively captures periodic patterns in input data and is particularly suitable for processing problems with complex nonlinear relationships and periodic features. By constraining the high-frequency Fourier coefficients through regularization terms, the smoothness of the function is ensured, further improving the performance of the mask prediction model and, consequently, the accuracy of plane segmentation.

[0125] More specifically, this embodiment of the present invention replaces the B-spline coefficients in KAN with one-dimensional Fourier coefficients. The Fourier coefficient g is set to 5, replacing the two-layer multilayer perceptron. The network learns g sets of Fourier coefficients to weight the transformed features. The weighted features are then inverse Fourier transformed to obtain the final output. The FourierKAN formula is as follows:

[0126]

[0127] Where, and are the weights to be learned in the mask branch module.

[0128] In some embodiments of the present invention, the segmentation model is a model based on bipartite graph matching.

[0129] It should be noted that the segmentation model needs to be trained with samples to be labeled before use.

[0130] The embodiment of the present invention realizes an end-to-end training framework by converting the real label assignment problem into an optimal assignment problem through a bipartite graph matching model, which makes the iteration speed of the segmentation model faster, the overall learning ability stronger, and the ability to cope with scenarios improves faster.

[0131] In summary, the superpoint and deep learning-based point cloud plane segmentation method proposed in this embodiment of the present invention utilizes refined superpoints as the basic processing unit, significantly reducing the computational complexity of the Transformer. Furthermore, two features that high-quality superpoints for the Transformer should possess are proposed, along with a corresponding multi-stage superpoint generation process. This ensures that the generated superpoints have precise boundaries and consistent geometric dimensions, all of which are highly beneficial for feature learning in the Transformer for corsage superpoints. Furthermore, to address the limitations of deep learning features with limited training sets, multi-dimensional geometric features are introduced into the model, further improving plane segmentation performance. Furthermore, this embodiment of the present invention constructs a mask prediction model that combines the Kolmogorov-Arnold Network and the Transformer module, effectively optimizing the instance prediction and mask extraction processes and improving overall segmentation performance. Finally, the ground truth label assignment problem is formulated as an optimal assignment problem, and an end-to-end training framework is implemented through bipartite graph matching based on superpoint masks. In summary, the point cloud plane segmentation method based on superpoints and deep learning proposed in the embodiments of the present invention can accurately segment planes, and its performance is significantly better than other traditional methods and existing deep learning-based models.

[0132] In order to better implement the point cloud plane segmentation method based on super point and deep learning in the embodiment of the present invention, on the basis of the point cloud plane segmentation method based on super point and deep learning, the embodiment of the present invention also provides a point cloud plane segmentation device based on super point and deep learning, such as Figure 7 As shown, the point cloud plane segmentation device 700 based on super points and deep learning includes:

[0133] A rough superpoint and preliminary segmentation unit 701 is used to determine rough superpoints and preliminary segmentation results based on the input point cloud;

[0134] A boundary optimization unit 702 is used to optimize the boundary of the preliminary segmentation result and determine the unfitted points in the input point cloud;

[0135] A superpoint refinement unit 703 is used to refine the rough superpoints and the unfitted points to obtain refined superpoints;

[0136] A super-point feature determination unit 704 is configured to extract depth features of each 3D point in the input point cloud and determine super-point features of the refined super-point based on the depth features;

[0137] The plane segmentation unit 705 is used to input the superpoint feature into the mask prediction model to obtain the superpoint mask, and input the superpoint mask into the segmentation model to obtain the target segmentation result.

[0138] The point cloud plane segmentation device 700 based on super points and deep learning provided in the above embodiment can implement the technical solution described in the above embodiment of the point cloud plane segmentation method based on super points and deep learning. The specific implementation principles of the above modules or units can refer to the corresponding contents in the above embodiment of the point cloud plane segmentation method based on super points and deep learning, which will not be repeated here.

[0139] like Figure 8 As shown, the present invention also provides a city modeling device 800. The city modeling device 800 includes a processor 801, a memory 802 and a display 803. Figure 8 Only some of the components of the city modeling apparatus 800 are shown, but it should be understood that implementing all of the shown components is not a requirement, and more or fewer components may alternatively be implemented.

[0140] In some embodiments, the memory 802 may be an internal storage unit of the city modeling device 800, such as a hard disk or memory of the city modeling device 800. In other embodiments, the memory 802 may be an external storage device of the city modeling device 800, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the city modeling device 800.

[0141] In some embodiments, the processor 801 can be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run the program code stored in the memory 802 or process data, such as the point cloud plane segmentation method based on super points and deep learning in the present invention.

[0142] In some embodiments, display 803 can be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 803 is used to display information about city modeling device 800 and to present a visual user interface. Components 801-803 of city modeling device 800 communicate with each other via a system bus.

[0143] In some embodiments of the present invention, when the processor 801 executes the point cloud plane segmentation program based on super points and deep learning in the memory 802, the following steps may be implemented:

[0144] Determine rough superpoints and preliminary segmentation results based on the input point cloud;

[0145] Optimize the boundaries of the preliminary segmentation results and determine the unfitted points in the input point cloud;

[0146] Refine the rough super points and unfitted points respectively to obtain refined super points;

[0147] Extract the depth features of each 3D point in the input point cloud, and determine the super-point features of the refined super-point based on the depth features;

[0148] The superpoint features are input into the mask prediction model to obtain the superpoint mask, and the superpoint mask is input into the segmentation model to obtain the target segmentation result.

[0149] It should be understood that when the processor 801 executes the point cloud plane segmentation program based on super points and deep learning in the memory 802, in addition to the above functions, it can also implement other functions. For details, please refer to the description of the previous related method embodiments.

[0150] On the other hand, an embodiment of the present invention also provides a computer-readable storage medium, which is used to store computer-readable programs or instructions. When the program or instructions are executed by a processor, it can implement the steps or functions of the point cloud plane segmentation method based on super points and deep learning provided in the above-mentioned method embodiments.

[0151] Those skilled in the art will appreciate that all or part of the process flow of the above-described method embodiment can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program, and the computer program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0152] The above is a detailed introduction to the point cloud plane segmentation method and device based on super points and deep learning provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A point cloud plane segmentation method based on superpoints and deep learning, characterized in that: include: Determine rough superpoints and preliminary segmentation results based on the input point cloud; Optimizing the boundaries of the preliminary segmentation result to determine unfitted points in the input point cloud; Refining the rough super points and the unfitted points respectively to obtain refined super points; Extracting depth features of each three-dimensional point in the input point cloud, and determining super-point features of the refined super-point based on the depth features; The superpoint features are input into a mask prediction model to obtain a superpoint mask, and the superpoint mask is input into a segmentation model to obtain a target segmentation result.

2. The point cloud plane segmentation method based on super points and deep learning according to claim 1, characterized in that Refining the rough super points and the unfitted points respectively to obtain refined super points includes: Determining a first expected number of clusters of the rough superpoints and a second expected number of clusters of the unfitted points; K-means clustering is performed on the rough super points and the unfitted points based on the first expected number of clusters and the second expected number of clusters to obtain the refined super points.

3. The point cloud plane segmentation method based on super points and deep learning according to claim 2, characterized in that The second expected number of clusters is twice the first expected number of clusters.

4. The point cloud plane segmentation method based on super points and deep learning according to claim 2, characterized in that When the size of the rough superpoint is larger than the expected size of each cluster after K-means clustering, the first expected number of clusters is for: When the size of the rough superpoint is less than or equal to the expected size of each cluster after K-means clustering, the first expected number of clusters is for: Where, is the number of 3D points in the input point cloud; is the number of rough super points; The size of each cluster after the desired K-means clustering; The symbol for rounding up.

5. The point cloud plane segmentation method based on super points and deep learning according to claim 1, characterized in that The extracting the depth feature of each three-dimensional point in the input point cloud includes: Extracting geometric features and position information of each of the three-dimensional points, wherein the geometric features include linearity, flatness, scattering, perpendicularity, and plane profile features; The input point cloud, the geometric features and the position information are input into a feature extraction network for feature extraction to obtain the depth features.

6. The point cloud plane segmentation method based on super points and deep learning according to claim 1, characterized in that Determining the super-point feature of the refined super-point based on the depth feature includes: Determining a plurality of target three-dimensional points corresponding to the refined super-points based on the attribution relationship between the refined super-points and the three-dimensional points; An average pooling operation is performed on the depth features of the multiple target three-dimensional points to obtain the super-point features.

7. The point cloud plane segmentation method based on super points and deep learning according to claim 1, characterized in that The mask prediction model includes an instance branch module, a mask branch module and a prediction head; The instance branch module is used to perform cross-attention learning on the super-point features to obtain instance features; The mask branch module is used to perform mask feature extraction on the super point feature to obtain mask perception features; The prediction head is used to multiply the instance feature and the mask perception feature to obtain a multiplied feature, and perform Sigmoid activation processing on the multiplied feature to obtain the super point mask.

8. The point cloud plane segmentation method based on super points and deep learning according to claim 7, characterized in that: The mask branch module uses Fourier transform to capture the nonlinear relationship of the super-point features.

9. The point cloud plane segmentation method based on super points and deep learning according to claim 1, characterized in that The segmentation model is a model based on bipartite graph matching.

10. A point cloud plane segmentation device based on super points and deep learning, characterized in that: include: A rough superpoint and preliminary segmentation unit, for determining a rough superpoint and preliminary segmentation result based on an input point cloud; a boundary optimization unit, configured to optimize the boundary of the preliminary segmentation result and determine unfitted points in the input point cloud; a superpoint refinement unit, configured to refine the rough superpoints and the unfitted points respectively to obtain refined superpoints; a super-point feature determination unit, configured to extract depth features of each three-dimensional point in the input point cloud, and determine super-point features of the refined super-point based on the depth features; The plane segmentation unit is used to input the superpoint feature into the mask prediction model to obtain the superpoint mask, and input the superpoint mask into the segmentation model to obtain the target segmentation result.

Images