Neural network point cloud segmentation algorithm based on hyperpoint

Abstract

The invention discloses a point cloud segmentation algorithm named CityFormer. Compared with a previous point cloud segmentation algorithm, the innovation point of the algorithm mainly lies in optimization of a super-point processing part: by providing a new super-point algorithm, dependence on mesh or point cloud normal vectors contained in a data set is eliminated; the application range of super-point processing and the adaptive capability and processing efficiency of the algorithm to a super-large scale aviation point cloud data set are improved. According to the CityFormer point cloud segmentation algorithm, a good result is obtained on a large-scale urban aviation point cloud data set through a large number of experiments.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and machine learning, and in particular to semantic segmentation, instance segmentation, and panoptic segmentation technologies for three-dimensional point cloud data. Background Art

[0002] With the acceleration of urbanization and the rapid development of remote sensing technology, obtaining three-dimensional information about large urban areas is becoming increasingly common. The widespread use of technologies such as drones, satellites, and commercial aircraft has made it possible to obtain high-precision aerial point cloud data of cities. This data captures the physical form of cities in detail, including building heights, road layouts, and vegetation, providing a foundation for the construction of digital twins of cities. However, while acquiring this data has become easier, effectively processing it and extracting valuable information from it remains a significant challenge.

[0003] When processing ultra-large-scale urban aerial point cloud data, traditional point cloud processing technologies often face the challenges of massive data volumes, high computational complexity, and significant memory consumption. Existing processing methods, such as raster-based image processing techniques, are often unable to adapt to the demands of these large-scale datasets. Point cloud data is sparse and irregular, and effective data analysis and segmentation becomes a challenge, especially in the absence of structured information (such as meshes and normal vectors). Traditional point cloud segmentation methods typically rely on data preprocessing, such as denoising and filtering, but these steps consume significant computational resources and can result in a loss of data accuracy.

[0004] Current mainstream point cloud segmentation algorithms, including those based on convolutional neural networks (CNNs) and Transformers, have demonstrated promising performance in tasks such as semantic segmentation and instance segmentation. However, these algorithms still face the following challenges when processing extremely large point cloud data: Firstly, they consume significant computing resources, resulting in slow processing speeds and even potential memory overflows. Secondly, the sparsity and irregular distribution of point cloud data create bottlenecks in the efficiency and accuracy of existing processing methods.

[0005] For example, algorithms based on convolutional neural networks (CNNs) capture local features in point cloud data by designing special convolution kernels and network structures, effectively processing data with clear local structures. However, when processing very large-scale data, the computational complexity and memory requirements of these methods increase rapidly, making them unable to meet the processing needs of large-scale datasets. Transformer-based methods use self-attention mechanisms to capture global features in point cloud data, but they also suffer from high computational resource consumption and slow processing speeds. Especially when faced with complex scenarios, the efficiency of these methods still needs to be further improved.

[0006] Furthermore, while instance segmentation and panoptic segmentation algorithms have made progress in separating fine-grained objects and differentiating multiple objects, their performance is still constrained by computational resources and algorithm design when processing complex, dense, and extremely large-scale point cloud data. Superpoint processing algorithms, as an important means of preprocessing point cloud data, can reduce data volume by dividing point clouds into superpoints with certain geometric characteristics. However, most existing algorithms rely on the normal vectors or mesh structure of the point cloud. In the absence of this information, the effectiveness of superpoint generation and the performance of the algorithm are often limited.

[0007] To address these challenges, emerging algorithms such as OneFormer3D have been proposed in recent years to optimize point cloud segmentation and processing using deep learning techniques. However, these methods still face significant computational bottlenecks when faced with ultra-large-scale urban aerial point cloud data. Therefore, an algorithm that can effectively process extremely large-scale point cloud data without mesh and normal vector information is urgently needed to meet the needs of smart city construction and management.

[0008] To this end, we propose the CityFormer algorithm. Its innovation lies in its unique super-pointing technology for spatially distributed features, enabling direct processing of urban aerial point cloud data without mesh or normal information. This algorithm significantly simplifies the data processing workflow, improves the efficiency and accuracy of point cloud data processing, and demonstrates excellent performance in applications involving ultra-large-scale urban aerial point cloud data. Summary of the Invention

[0009] The technical problem to be solved by this invention is to provide an algorithm that can efficiently process very large-scale, sparse, and irregular urban aerial point cloud data, and to address the high computational complexity, high memory consumption, slow processing speed, and poor data segmentation of existing point cloud processing methods when dealing with data without mesh and normal vector information. Specifically, this invention aims to simplify the point cloud data processing process through an innovative super-pointing algorithm, improving the segmentation accuracy and processing efficiency of large-scale point cloud data to meet the application needs of smart city construction, autonomous driving, environmental monitoring, and other fields.

[0010] The present invention solves its technical problems by adopting the following technical solutions.

[0011] The CityFormer algorithm provided by this invention comprises four modules. The first module is the super-pointing module, which groups similar points in the point cloud data through clustering and merges each group of points into a super-point. The second module is the backbone network module, which uses a sparse convolutional neural network structure (Sparse 3D U-Net) to extract global and local features from the super-pointed point cloud data. The third module is the pooling layer module, which uses an adaptive pooling strategy to automatically adjust the size and step size of the pooling window based on the density and distribution of the point cloud data, extract key features, and simplify feature data. The fourth module is the query decoder module, which includes an instance branch and a mask branch. It uses the self-attention mechanism of the Transformer to process the input query information and generate the final segmentation result.

[0012] Specifically, the superpoint processing module spatially partitions the input point cloud data. We employ a spatial grid-based partitioning method, dividing the entire 3D space into several equally sized small spatial regions. This partitioning method evenly covers the entire point cloud data space, ensuring that points within each region are properly processed. The grid size selection needs to be adjusted based on the density and distribution of the point cloud data. The requirement is that the grid size should be small enough to ensure that each grid contains enough points, but not too small to result in an excessive number of grids and increase the computational burden. By setting an appropriate grid size, we ensure uniform coverage of the entire 3D space and avoid losing detailed information in certain areas due to overly large grids. For each spatial region, we calculate the adjacency relationship of points within the region. By constructing relationships between adjacent points and calculating the degree of similarity between adjacent points using the dot-wise similarity of their spatial geometric distance and RGB color values, we quantitatively reflect the clustering and distribution of points within the region. Using adjacency relationships and similarity measures, we then divide the points within the region into different superpoints. The superpoint labels of each region are then merged to obtain superpoint information for all input points.

[0013] In the backbone network module, the initial features of the point cloud data are extracted. CityFormer uses an efficient convolutional neural network structure (Sparse 3D U-Net) based on SPFormer. This module takes N input point clouds ( ) voxelization. The U-Net-like architecture implemented through sparse 3D convolution can quickly capture the local feature information of the point cloud. This module gradually converts the original point cloud data into a feature representation with higher-level semantic information through multi-layer convolution and pooling operations ( ) where C represents the number of feature channels. This multi-layer convolution and pooling operation not only extracts local features from the point cloud but also restores detailed information through upsampling and skip connections, ensuring the integrity and hierarchy of the feature representation. This module utilizes sparse convolution to process 3D data, avoiding ineffective computations in void areas and significantly improving computational efficiency. The multi-layer design of the U-Net architecture simultaneously captures both local details and global features of point cloud data. The skip connection mechanism enables the network to better preserve detailed information, improving segmentation accuracy and adapting to point cloud data of varying densities and distributions.

[0014] In the pooling layer module, features extracted by the backbone network are further aggregated and simplified. CityFormer employs an adaptive pooling strategy that automatically adjusts pooling parameters based on the distribution of point cloud data, thereby better preserving important feature information while reducing data volume and improving the efficiency of subsequent processing. Specifically, the core of the adaptive pooling strategy lies in its ability to dynamically adjust the size and step size of the pooling window to ensure that key features can be effectively extracted from point cloud data of varying densities and distributions. This not only reduces the amount of computation but also enhances the algorithm's adaptability to varying data distributions.

[0015] In the query decoder module, a set of query points is predefined, which are considered potential segmentation region candidates. The initial positions and feature information of these query points are carefully designed to ensure that they cover the entire point cloud data and are representative. This initialization strategy improves the convergence speed and stability of the algorithm when processing large-scale datasets. In the Transformer decoder, the self-attention mechanism enables each query point to pay attention to the information of other related points, thereby better capturing global context. This approach is particularly effective when processing complex scenes because it can account for long-range dependencies in point cloud data. The self-attention mechanism not only improves segmentation accuracy but also enhances the model's adaptability to diverse scenarios. The matching strategy is used to determine the segmentation region corresponding to each query point. By dynamically adjusting the matching threshold and strategy, the algorithm can more accurately identify different instances and generate corresponding segmentation masks for each instance. This strategy significantly improves segmentation accuracy and reliability when processing multiple objects in complex scenes.

[0016] A set of query points is predefined, which can be considered as potential segmentation region candidates. The initial positions and feature information of these query points are carefully designed to ensure that they cover the entire point cloud data and are representative. This initialization strategy can improve the convergence speed and stability of the algorithm when processing large-scale datasets. In the Transformer decoder, the self-attention mechanism enables each query point to pay attention to the information of other related points, thereby better capturing global context. This approach is particularly effective when processing complex scenes because it can account for long-range dependencies in point cloud data. The self-attention mechanism not only improves segmentation accuracy but also enhances the model's adaptability to different scenarios. The matching strategy is used to determine the segmentation region corresponding to each query point. By dynamically adjusting the matching threshold and strategy, the algorithm can more accurately identify different instances and generate a corresponding segmentation mask for each instance. This strategy can significantly improve segmentation accuracy and reliability when processing multiple objects in complex scenes.

[0017] This paper proposes a point cloud segmentation algorithm, named CityFormer, designed to process ultra-large-scale urban aerial point cloud data. The algorithm focuses on optimizing superpoint processing. By proposing a new superpointing algorithm, it eliminates the need for datasets containing meshes or point cloud normals, broadening the application scope of superpointing and improving the algorithm's adaptability and processing efficiency for ultra-large-scale aerial point cloud datasets.

[0018] Compared with the prior art, the present invention has the following main advantages.

[0019] Expanded Application Scope: The CityFormer algorithm eliminates the need for datasets containing meshes or normals, meaning it can process a wide range of point cloud datasets. CityFormer can effectively process datasets containing rich structural information, or datasets that are pure point clouds without meshes or normals. This feature significantly broadens the algorithm's application scenarios. For example, in urban aerial monitoring projects that only acquire raw point cloud data without additional structural information, the CityFormer algorithm can still effectively process and analyze the data. This gives CityFormer broad application potential in fields such as smart cities, autonomous driving, and environmental monitoring. In urban aerial monitoring projects, due to limitations in data acquisition methods and technologies, in many cases only raw point cloud data is available without mesh or normal information. Even in these situations, the CityFormer algorithm can still play a vital role, ensuring the accuracy and effectiveness of data processing.

[0020] Improved processing efficiency: The new super-pointing algorithm is based on the spatial distribution characteristics of point clouds, and its calculation process is relatively simple and efficient. Compared with traditional super-point processing algorithms that rely on normal vectors or mesh structures, CityFormer does not require complex normal vector calculations or mesh structure analysis, thereby reducing the amount of calculation and processing time. CityFormer's overall architecture has been optimized to facilitate information transfer between modules, further improving processing efficiency. For example, through the design of an efficient backbone network, adaptive pooling layers, and query decoder modules, CityFormer is able to maintain high speeds when processing extremely large-scale datasets. When processing extremely large-scale urban aerial point cloud data, CityFormer can obtain processing results more quickly, meeting application scenarios with high real-time requirements. This is particularly important for scenarios requiring rapid decision-making, such as autonomous driving and drone navigation.

[0021] Strong Adaptability: The CityFormer algorithm has significantly enhanced its adaptability to extremely large-scale urban aerial point cloud datasets. It automatically adjusts its superpoint segmentation and processing methods based on the characteristics of each dataset, better adapting to the diversity and complexity of the data. CityFormer effectively processes urban aerial point cloud datasets of enormous volumes, as well as datasets with diverse semantic and geometric features, extracting valuable information. This adaptability enables the CityFormer algorithm to perform well in a variety of complex real-world application scenarios. For example, in urban reconstruction projects, datasets may contain a variety of structures, such as high-rise buildings, bridges, and trees. CityFormer effectively processes this diverse data and produces high-quality segmentation results. The CityFormer algorithm's flexibility is also reflected in its ability to adapt to point cloud data of varying density and distribution. Through adaptive pooling strategies and dynamically adjusted matching strategies, the algorithm automatically adapts to the distribution characteristics of point cloud data in different scenarios, thereby better preserving key features. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is the overall structure diagram of the algorithm of the present invention.

[0023] Figure 2 It is a flow chart of the super-pointing part of the algorithm of the present invention.

[0024] Figure 3 This is a structural diagram of the sparse three-dimensional convolutional neural network (Sparse 3D U-Net) in the backbone network module of the algorithm of the present invention. DETAILED DESCRIPTION

[0025] This invention relates to a point cloud segmentation algorithm for CityFormer, designed to process very large-scale urban aerial point cloud data. It focuses on optimizing superpoint processing. By proposing a new superpointing algorithm, it eliminates the need for datasets containing meshes or point cloud normals. This improves the application scope of superpointing, the algorithm's adaptability to very large-scale aerial point cloud datasets, and its processing efficiency, resulting in high practical value.

[0026] The present invention will be further described below with reference to examples and drawings, but the present invention is not limited thereto.

[0027] The CityFormer point cloud segmentation algorithm, designed to process ultra-large-scale urban aerial point cloud data, comprises four modules. The first module is the super-pointing module, which groups similar points in the point cloud data through clustering and merges each group of points into a super-point. The second module is the backbone network module, which uses a sparse convolutional neural network (Sparse 3D U-Net) to extract global and local features from the super-pointed point cloud data. The third module is the pooling layer module, which uses an adaptive pooling strategy to automatically adjust the size and step size of the pooling window based on the density and distribution of the point cloud data, extract key features, and simplify the feature data. The fourth module is the query decoder module, which includes an instance branch and a mask branch. It uses the self-attention mechanism of the Transformer to process the input query information and generate the final segmentation result.

[0028] The environment required for running the algorithm can be selected as CPU with 25 vCPU AMD EPYC 7T83 64-CoreProcessor, GPU with NVIDIA L40 (48GB), equipped with Ubuntu 20.04 system, Cuda version 11.6, and Python 3.8.

[0029] The CityFormer point cloud segmentation algorithm provided by this invention has the following specific workflow: Figure 1 shown.

[0030] Perform initial environment configuration.

[0031] Perform super-point processing, such as Figure 2 As shown, input point cloud data P, Point cloud data, N is the number of point clouds, 6 is the feature dimension of the point cloud, the specific features include: xyzrgb, through the specified number of cuts (Preset to 2) Divide the point cloud into multiple regions according to coordinates. For each region, calculate the n adjacent points of each point, n is preset to 16, and obtain the edge information of the point cloud in the region , and then calculate the similarity represented by each edge through the xyzrgb information of the two points connected by each edge , then delete the edges whose similarity is less than the threshold thr, which is preset to 0.01, and calculate the number of super points connected at this time ,like Greater than threshold , the preset value is 5000000, then the super point information is output, otherwise the similarity threshold thr is dynamically adjusted, and the edge deletion operation is repeated until the super point label information is output .

[0032] Step 3. Preprocess the input data and randomly sample the input point cloud to reduce the number of point clouds. ,in The default value is 180,000. The sampled point cloud is randomly flipped along the x-axis or y-axis, the xyz coordinates are globally scaled by 0.9-1.1 times, and the RGB information is normalized.

[0033] Step 4. Perform voxelization. Use the given voxel size Voxel_size (preset to 0.8) to perform spatial alignment of the point cloud. Convert the point cloud to voxels, that is, average the features of all points in the same voxel. The processed features are where voxels The number of

[0034] Step 5. Through the backbone network module, the five-layer sparse convolutional neural network structure (Sparse 3D U-Net) performs a three-dimensional convolution on the input voxels with a convolution kernel size of 3, increasing the number of feature channels from 6 to 64. The new feature is .like Figure 3 As shown in the figure, in the backbone neural network UNet, the number of feature channels is gradually increased from 64 to 128, 192, 256, and 320 through 3D convolution with a convolution kernel size of 3. Then, the number of features is restored through the reverse 3D convolution with a convolution kernel size of 2, from 320 to 256, 192, 128, and 64. The final output feature is The features are passed through a one-dimensional batch normalization layer and a ReLU layer, and the feature size remains unchanged.

[0035] Step 6. Next, enter the features Entering the decoder module for processing, the input features first pass through the input projection layer (input_proj), reducing the number of channels to 256, generating a query tensor of size (Q, 256). Each query tensor then passes through a cross-attention layer, interacting with all input feature points. This updates the internal representation of the query tensor, while maintaining the shape of (Q, 256). A self-attention layer (self_attn_layers) interacts with the query tensor, updating the internal representation of the query tensor, while maintaining the shape of (Q, 256). A feedforward neural network (FFN) further enhances the query tensor's representation, while preserving its shape. The prediction head generates classification predictions, score predictions, and mask predictions. The classification prediction is Q×(C+1), where C is the number of categories; the score prediction is Q×1; and the mask prediction is Q×Nvoxel.

[0036] Step 7. Feature Mapping and Superpoint Generation: Nvoxel features are mapped back to a 180,000-point cloud, where each point has the same features as the voxel it resides in. Points are then mapped to superpoints, significantly reducing the number of points that need to be processed.

[0037] Step 8. Instance segmentation label generation: Calculate the instance score and perform a softmax operation on the classification prediction to obtain the probability that each query belongs to each category. Multiply the classification probability by the score prediction to obtain the final score for each query. Select the top 450 highest-scoring instances, take the mask predictions for these selected instances, and convert them into probability values using a sigmoid operation. Apply a threshold of 0.5 to convert the probability values into binary masks. Based on the binary values of the mask, generate the instance label for each point. The instance label of each point is the index of the instance it belongs to. Semantic segmentation label generation: Calculate the semantic mask: For each category, take its mask prediction and convert it into a probability value using a sigmoid operation. Apply a threshold of 0.5 to convert the probability values into binary masks. Generate semantic labels: For each superpoint, determine its semantic label based on its position in each category mask. The semantic label of each superpoint is the index of its category. Panoramic segmentation label generation generates a panoramic mask, combining the instance mask and the semantic mask to ensure that each point has a unique identifier. Panoramic labels are generated: for each point, its panoramic label is generated based on its position in the panoramic mask. The panoramic label is a combined identifier composed of the semantic label and the instance label.

[0038] Step 9. Repeat the above training steps 512 times to obtain the trained CityFormer model parameters that can be used to process ultra-large-scale urban aerial point cloud data.

[0039] The CityFormer point cloud segmentation algorithm provided by the present invention is designed to process ultra-large-scale urban aerial point cloud data and has the following characteristics.

[0040] A super-pointing algorithm based on the spatial distribution characteristics of point clouds is proposed. This algorithm eliminates the need for datasets containing meshes or point cloud normals, resulting in a more versatile and adaptable super-pointing algorithm. It can handle pure point cloud datasets without meshes or normals, and can adaptively adjust the super-pointing division based on the actual distribution of the point cloud data, improving the accuracy and efficiency of super-pointing. This algorithm's increased versatility, adaptability, and robustness provide strong support for the CityFormer algorithm in processing very large-scale urban aerial point cloud data.

[0041] Although the present invention has been particularly shown and described in conjunction with preferred embodiments, it will be understood by those skilled in the art that various changes may be made to the present invention in form and detail without departing from the spirit and scope of the invention as defined in the appended claims, and all such changes are within the scope of protection of the present invention.

Claims

1. An algorithm for segmenting ultra-large-scale urban aerial point cloud data, called CityFormer, is characterized by: A new super-pointing algorithm is used to improve the processing efficiency and segmentation accuracy of point cloud data. The algorithm consists of four modules. The first module is the super-pointing module, which groups similar points in the point cloud data through clustering and merges each group of points into a super-point. The second module is the backbone network module, which uses a sparse convolutional neural network structure (Sparse 3D U-Net) to extract global and local features from the super-pointed point cloud data. The third module is the pooling layer module, which adopts an adaptive pooling strategy to automatically adjust the size and step size of the pooling window according to the density and distribution of the point cloud data, extract key features, and simplify feature data. The fourth module is the query decoder module, which includes an instance branch and a mask branch. It uses the self-attention mechanism of the Transformer to process the input query information and generate the final segmentation result.

2. The CityFormer algorithm for ultra-large-scale urban aerial point cloud data segmentation according to claim 1 is characterized in that: The super-pointing algorithm uses an adaptive clustering strategy based on density or distance to dynamically adjust the size of super-points according to the local characteristics and density distribution of the point cloud, generating features that represent the area where each super-point is located. This can reduce the amount of point cloud data to be processed and the amount of computation required when processing large-scale urban aerial point cloud data, while maintaining high segmentation accuracy and enhancing the algorithm's adaptability to complex urban environments.

3. The CityFormer algorithm for ultra-large-scale urban aerial point cloud data segmentation according to claim 1 is characterized in that: The sparse convolutional neural network structure (Sparse 3D U-Net) in the backbone network module uses sparse convolution to avoid invalid calculations in void areas, improving computational efficiency. It also constructs high-level semantic information through multi-layer convolution and deconvolution operations, while retaining detailed information through jump connections, thereby enhancing segmentation accuracy.

4. The CityFormer algorithm for ultra-large-scale urban aerial point cloud data segmentation according to claim 1 is characterized in that: The pooling layer module dynamically adjusts the pooling parameters according to the distribution of the point cloud, can automatically adapt to different density distributions in different scenarios, and effectively extract key features, thereby improving processing efficiency and reducing computational burden.

5. The CityFormer algorithm for ultra-large-scale urban aerial point cloud data segmentation according to claim 1 is characterized in that: The query decoder module enhances the representation ability of query points through the self-attention mechanism and feedforward neural network (FFN), updates the features of query points through cross-attention layer and self-attention layer, and finally generates instance segmentation labels, semantic segmentation labels and panoptic segmentation labels through mask branch and instance branch respectively.

Images