A point cloud semantic segmentation method based on sampling enhancement and multi-view

Abstract

The application discloses a point cloud semantic segmentation method based on sampling enhancement and multi-view, first, airborne laser radar point cloud data is collected, a triangular facet model is used to generate virtual samples to enhance data diversity, and a dataset is obtained after preprocessing; then, the dataset is sampled, a boundary point analysis module is used to optimize the sampling strategy, and boundary features of ground objects are mainly captured; then, a point cloud semantic segmentation model is constructed, three-dimensional geometric features are extracted by using a kernel point convolution network, two-dimensional image features generated by multi-view side projection are combined, multi-dimensional feature complementation is realized, the model is trained and optimized according to a loss function; finally, the point cloud semantic segmentation model is used for semantic segmentation of new airborne laser radar point cloud data. The application can improve the recognition accuracy of ground object boundaries in complex urban scenes and the classification performance of small-scale targets that are easy to confuse, effectively reduces the burden of relevant staff, and improves production efficiency.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and 3D point cloud processing technology, specifically to a point cloud semantic segmentation method based on sampling enhancement and multi-viewpoint, which is particularly suitable for automated semantic analysis of large-scale airborne lidar point cloud data at the city level. Background Technology

[0002] With the rapid development of smart city construction, airborne LiDAR technology has become an important means of acquiring urban 3D spatial data. Point cloud semantic segmentation, as a core task of 3D image scene understanding, aims to assign accurate semantic labels to each point, segmenting point cloud data into different semantic categories to achieve precise spatial analysis. It has significant application value in fields such as basic surveying and mapping management and digital city construction. Traditional point cloud semantic segmentation methods mainly rely on manually designed geometric features, which suffer from limited feature representation capabilities and poor adaptability to complex scenes. In recent years, although deep learning-based methods have made significant progress, they still face many challenges when processing large-scale urban point cloud data, such as extremely uneven distribution of ground feature categories, uneven sparse density distribution of point clouds, and difficulties in extracting features from boundary regions.

[0003] Therefore, there is an urgent need for an automatic and accurate point cloud semantic segmentation method to effectively reduce the burden on relevant personnel, save costs, and improve production efficiency. Summary of the Invention

[0004] The purpose of this invention is to address the problems in the background technology by proposing a point cloud semantic segmentation method based on sampling enhancement and multi-view projection. This method, based on deep learning technology, first optimizes the sampling strategy through virtual sample enhancement and boundary awareness. Then, it extracts three-dimensional geometric features using a convolutional network and combines them with two-dimensional features from multi-view projection. An adaptive fusion mechanism achieves feature complementarity, and finally, a cascaded segmentation network completes the refined semantic segmentation of city-level point cloud data.

[0005] The technical solution adopted in this invention is as follows:

[0006] In a first aspect, the present invention provides a point cloud semantic segmentation method based on sampling enhancement and multi-viewpoint, comprising the following steps:

[0007] Collect large-scale airborne lidar point cloud data at the city level and perform data augmentation. Preprocess the augmented data to construct a dataset, which is then divided into a training set and a test set.

[0008] Sampling of the dataset;

[0009] A point cloud semantic segmentation model is constructed by inputting sampled point cloud data into the model. The model employs an encoder-decoder architecture. The encoder extracts features from the input point cloud data, including multi-level residual blocks and residual bottleneck blocks. Each residual block and bottleneck block includes a 3D feature extraction branch based on kernel convolution and a 2D feature extraction branch based on multi-view side projection. The 2D feature extraction branch extracts local structural features from different perspectives through 2D convolution, normalization, and pooling operations, and fuses these features with the 3D feature extraction branch at various layers to obtain a multimodal feature representation of the point cloud. The decoder upsamples and classifies the fused features point by point, outputting a semantic category label for each point.

[0010] The point cloud semantic segmentation model was trained and tested using the sampled training and test sets, respectively.

[0011] The trained and tested point cloud semantic segmentation model is used to perform semantic segmentation on the new airborne lidar point cloud data.

[0012] Furthermore, the airborne lidar point cloud data includes xyz coordinates, intensity, and, for the training set, includes manually labeled real classification class information.

[0013] Furthermore, the data augmentation utilizes a mesh model to simulate easily confused categories with a small number of samples, and simulates different angles and occlusion situations during airborne lidar sampling. The mesh model is converted into virtual samples of point cloud data, and reasonable insertion positions are selected based on the road topology of the original point cloud data and the distribution of existing samples of this category. The reasonable orientation of the virtual sample when placing it at the position is analyzed based on SIFT features. The virtual sample is then inserted into the dataset to achieve data augmentation.

[0014] Furthermore, the preprocessing includes: firstly, eliminating spatial offsets from different acquisition scenarios through coordinate standardization, and reducing data density while maintaining geometric features using a voxel grid downsampling algorithm; then, constructing a KDTree spatial index structure to optimize neighborhood query efficiency, and establishing a multi-resolution pyramid hierarchy to support multi-scale feature learning; finally, intelligently dividing the training set and validation set based on scene content, and integrating online data augmentation transformations such as random rotation and scaling to form an efficient training data pipeline suitable for deep learning models.

[0015] Furthermore, during sampling, the local curvature and normal vector changes of the point cloud are calculated based on the k-nearest neighbor algorithm for boundary points, and boundary point sampling enhancement is performed.

[0016] Furthermore, the encoder part comprises multiple simple blocks and residual bottleneck blocks (resnetb), where the downsampled residual blocks (resnetb_strided) progressively expand the receptive field and reduce the point cloud resolution. Each convolutional layer adaptively learns the local geometry through deformable kernel points, and its core operation is defined as:

[0017]

[0018] Where K is the number of kernel points, and Φ is the linear influence function. The coordinates of the deformable core point. These are the convolution weights. The decoder gradually restores the resolution through nearest-neighbor upsampled blocks and unary blocks, ultimately outputting the 3D semantic features of each point.

[0019] Furthermore, during the extraction of three-dimensional features, branches are extracted and fused in combination with two-dimensional features;

[0020] The two-dimensional feature extraction branch performs the following steps: First, each sampling collects points within a specified sampling radius around the sampling center. At this time, a copy of the point cloud data obtained with the original sampling radius during this sampling is pre-stored, as well as a copy of the point cloud data obtained with twice the sampling radius during this sampling (to more fully learn neighborhood information). Multi-view side-view projection mapping is then performed, mapping the three-dimensional spatial coordinates to the two-dimensional image plane using a rotation matrix. That is, for a given point cloud... The rotation operation around the Z-axis can be represented as:

[0021]

[0022] in Taking seven specific angles, one every 30° within the range of 0-180°; the rotated point cloud The projection onto the XZ plane is normalized to a fixed-resolution (256×256 pixels) image grid through a linear transformation, where near and far information (Y coordinate) is represented by a color mapping function. Encode the color using the Viridis color space to achieve continuous gradients, thus preserving the three-dimensional spatial relationships in a two-dimensional projection;

[0023] A total of 14 two-dimensional projection images with 4 channels RGBA were used. Feature extraction was performed using a convolutional neural network architecture, with the input tensor set as follows: The feature extraction process can be formalized as follows:

[0024]

[0025] in It is a convolutional encoder that extracts local features step by step through three layers of convolution and pooling operations; This is a feature compression layer that uses 1×1 convolutions and global pooling to reduce the feature dimension to 64; ⊕ indicates a feature concatenation operation; To achieve network fusion, a fully connected layer is used to aggregate features from multiple perspectives.

[0026] Furthermore, the feature fusion includes:

[0027] First, the 64-dimensional two-dimensional features are aligned to the three-dimensional feature dimension through linear projection. Then, the two-dimensional features are concatenated with the three-dimensional features of each point through a broadcast mechanism. The fusion process is implemented using the following formula:

[0028]

[0029] The MLP consists of linear layers, batch normalization layers, and a LeakyReLU activation function (negative slope 0.1). In the residual bottleneck block, the fused features are further added to the original features via shortcut connections to enhance feature propagation efficiency.

[0030]

[0031] This design ensures effective complementarity between the geometric structure and the multi-view texture features.

[0032] Secondly, the present invention provides a point cloud semantic segmentation system for implementing the above method, comprising the following modules:

[0033] The data preprocessing module is used to collect point cloud data and perform preprocessing.

[0034] The point cloud semantic segmentation module performs semantic segmentation on preprocessed point cloud data using a trained point cloud semantic segmentation model.

[0035] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the above-described method.

[0036] Fourthly, the present invention provides a computing device, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the above-described method.

[0037] Compared with the prior art, the beneficial effects of the present invention are:

[0038] The point cloud semantic segmentation method proposed in this invention, based on sampling enhancement and multi-view, achieves automated and refined processing of large-scale city-level point cloud data through deep learning technology, significantly reducing the time cost and subjective error of traditional manual interpretation, and greatly improving the efficiency of 3D geographic information production.

[0039] This invention innovatively combines virtual sample enhancement with boundary-aware sampling strategies, effectively solving the problems of class imbalance and boundary feature loss in point cloud data. It can improve the model's ability to recognize sparse classes and complex structures without requiring a large number of abnormal samples.

[0040] This invention fully leverages the complementary advantages of two-dimensional visual features and three-dimensional geometric features through an adaptive fusion mechanism of multi-view projection and three-dimensional features.

[0041] This invention has a wide range of applications, achieving higher boundary accuracy and small target recognition rate in point cloud semantic segmentation tasks in urban scenarios, and has important application value in fields such as smart cities and autonomous driving. Attached Figure Description

[0042] Figure 1 This is a flowchart of the present invention.

[0043] Figure 2 This is a network model structure diagram of the present invention.

[0044] Figure 3 This is an example image of a virtual sample generated by the present invention.

[0045] Figure 4 This is an example diagram illustrating the selection of virtual sample insertion positions according to the present invention.

[0046] Figure 5 This is an example diagram of the enhanced position in the boundary point sampling enhancement logic of this invention.

[0047] Figure 6 This is an example diagram of the multi-view side projection generated by the present invention for point cloud samples.

[0048] Figure 7 This is a semantic segmentation effect diagram of the present invention. The first column is the input point cloud sample (the sample is three-dimensional data, and a part of the view is captured in the visualization software). The second column is a top-down mapping effect diagram of the complete segmentation result after semantic segmentation (each category corresponds to a color). The third column is a single target correctness diagram with vehicles as an example (blue indicates correct reasoning, red indicates that the model misclassifies it as a vehicle, and green indicates that the model misses the classification as a vehicle). Detailed Implementation

[0049] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0050] like Figure 1 As shown, the implementation process of the point cloud semantic segmentation method based on sampling enhancement and multi-viewpoints provided by this invention is as follows:

[0051] Step 1: Raw Data Collection and Virtual Sample Augmentation – Collect city-level point cloud data from UAV aerial photography, including XYZ coordinates, intensity information, and manually labeled real classification labels. For categories with few samples in the dataset (such as vehicles), generate virtual samples using a triangular patch model (e.g., Figure 3 (As shown).

[0052] In this invention, the point cloud data processed can reach the level of millions or tens of millions, enabling the processing of large-scale point cloud data at the city level.

[0053] In practical implementation, taking vehicles as an example, firstly, reasonable virtual sample placement locations are selected based on road topology and existing ground feature distribution (e.g., Figure 4 As shown in the figure, the placement is ensured by combining distance constraints, terrain constraints and ground flatness constraints; then, the road texture direction is analyzed based on SIFT features to determine the reasonable orientation of the virtual sample; finally, the triangular patch model is non-uniformly scaled and morphologically edited using the trimesh library to generate diverse virtual samples, which are then converted into point cloud data by simulated LiDAR scanning and inserted into the original scene after density resampling.

[0054] The detailed implementation steps are as follows:

[0055] ①Based on the road topology and existing vehicle distribution of point cloud data, combined with distance constraints (maintaining minimum spacing (usually 2.5-3.0 meters) to avoid vehicle overlap), terrain constraints (excluding unsuitable areas such as building exterior walls and roof obstructions), and ground flatness constraints (checking that the ground height change does not exceed 0.5 meters), reasonable candidate locations for placing virtual samples are selected.

[0056] ② For each reasonable candidate location extracted in ①, based on the road texture of SIFT features, analyze the reasonable orientation of placing a virtual sample vehicle at that location. That is, firstly, extract the ground point cloud within a radius of 5 meters centered on the target location; then convert the 3D point cloud into a 2D height map, using a grid precision of 10cm×10cm, and calculate the average elevation of each grid through double histogram statistics, mapping the height values ​​to a grayscale range of 0-255; then use OpenCV's SIFT algorithm to detect feature points. When SIFT detects feature points, further use the Canny edge detection algorithm to extract significant edge information in the image; finally, based on the edge detection results, use Hough transform technology to analyze the directional distribution of straight line features, determine the most significant principal direction angle through statistical peak analysis, and perform coordinate system transformation to convert the angle values ​​of the image coordinate system into the vehicle orientation angle in the actual scene.

[0057] ③ Select a suitable triangular facet vehicle model, deform the model using the trimesh library in Python (including non-uniform scaling and radian editing), generate a specified number of vehicle models of different shapes (e.g., 100), and simulate LiDAR scanning at different angles using the OpenCV library to convert the triangular facet model into PLY point cloud data as virtual samples to be inserted.

[0058] ④ Resample the density of the sample to be inserted to make it consistent with the point cloud density distribution of the scene obtained in step 1);

[0059] ⑤ Insert the virtual samples obtained in ④ into the original scene according to the positions and directions obtained in ① and ②, and delete the relevant point clouds by simulating laser occlusion. For example, the occluded ground point clouds need to be deleted after the vehicle is placed. Finally, the complete point cloud data after virtual sample enhancement is obtained.

[0060] Step 2: Data Preprocessing and Boundary Point Sampling Enhancement—The enhanced point cloud data undergoes systematic preprocessing, including coordinate standardization, voxel grid downsampling to reduce data density, and the construction of a KDTree spatial index to optimize query efficiency. Subsequently, the local curvature change rate and normal vector angle of each point are calculated using the boundary point analysis module to identify feature boundary regions. High-curvature boundary points (such as...) are then analyzed. Figure 5 (As shown) Set the sampling frequency 2-3 times higher than that of flat areas, and introduce a minimum distance constraint to ensure that the sampling points are evenly distributed and effectively preserve the detailed structural features of the scene; in other cases, sample according to potential, that is, prioritize areas that are sampled less often, to ensure that each area is fully sampled.

[0061] Specifically, conventional sampling methods are based on potential sampling, a point cloud data sampling strategy based on spatial probability distribution, which initially assigns a small random value (e.g., 1×10⁻⁶) to each point. -3 The potential value of a point within a sphere is determined by the sphere's center point c and a radius r. During the sampling process, a spherical region is defined with the center point c as the center and a radius r as the radius. The potential values ​​of points within this region are then updated.

[0062]

[0063] Here, Tukey weights reflect the probability of each point being selected, and d is the distance from the point to the center of the sphere. If d > r, the weights are reset to zero; otherwise, the weights are accumulated to the potential value of the corresponding point. In each round of sampling, the point with the smallest global potential value is selected as the center of the sphere to ensure that the model continuously focuses on the region with the poorest current prediction performance.

[0064] Then, based on the conventional sampling method, a strategy for enhanced sampling of boundary points is added for each point in the point cloud. and its k-nearest neighbor set Calculate the local covariance matrix, i.e.:

[0065]

[0066] Among them The neighborhood centroid is then obtained through eigenvalue decomposition. And calculate the local curvature:

[0067]

[0068] The sampling frequency is dynamically adjusted based on the curvature value, and a sampling density 2-3 times higher than that in flat areas is set for high curvature boundary points, while distance constraints are introduced. This ensures the uniform spatial distribution of sampling points.

[0069] Step 3: Fusion of 3D and Multi-view Feature Extraction

[0070] ① Extracting 3D geometric features: A convolutional network is used to extract the 3D geometric features of the point cloud. The network adopts an encoder-decoder architecture. The encoder gradually expands the receptive field through multiple layers of deformable convolution, and the decoder restores the resolution through nearest neighbor upsampling.

[0071] ② Extracting 2D multi-view features: Simultaneously, multi-view side-view projection is performed on the point cloud data obtained from the original sampling radius and twice the sampling radius, rotating once around the Z-axis every 30°, generating 7 2D projection images for each, for a total of 14 images (e.g., ...). Figure 6As shown), near and far information is encoded into image pixels using a Viridis color map. A two-dimensional convolutional neural network is used to extract image features from each viewpoint, which are then compressed using global pooling to form a 64-dimensional multi-view feature vector. Specifically:

[0072] First, each sampling involves collecting points within a specified sampling radius around the sampling center. Therefore, it's necessary to pre-store both the original sampling radius (for point cloud data acquired during that sampling) and a second sampling radius (for point cloud data acquired during that sampling) (to more fully learn neighborhood information). Then, multi-view side-view projection mapping is performed, mapping the 3D spatial coordinates to the 2D image plane using a rotation matrix. That is, for a given point cloud... The rotation operation around the Z-axis can be represented as:

[0073]

[0074] in Taking seven specific angles, one every 30° within the range of 0-180°; the rotated point cloud The projection onto the XZ plane is normalized to a fixed-resolution (256×256 pixels) image grid through a linear transformation, where near and far information (Y coordinate) is represented by a color mapping function. Encode the color using the Viridis color space to achieve continuous gradients, thus preserving the three-dimensional spatial relationships in a two-dimensional projection;

[0075] A total of 14 two-dimensional projection images with 4 channels RGBA were used. Feature extraction was performed using a convolutional neural network architecture, with the input tensor set as follows: The feature extraction process can be formalized as follows:

[0076]

[0077] in It is a convolutional encoder that extracts local features step by step through three layers of convolution and pooling operations; This is a feature compression layer that uses 1×1 convolutions and global pooling to reduce the feature dimension to 64; ⊕ indicates a feature concatenation operation; To achieve network fusion, a fully connected layer is used to aggregate features from multiple perspectives.

[0078] ③ Feature fusion: The 64-dimensional multi-view features are aligned to the three-dimensional feature dimension through linear projection, and then stitched with the three-dimensional geometric features of each point through a broadcast mechanism. The result is input into a multilayer perceptron (including batch normalization and LeakyReLU activation) to achieve adaptive feature fusion. In the residual bottleneck block, the fused features are added to the original features through shortcut connections to enhance information transmission efficiency.

[0079] Step 4: Model Training – The training process employs a multi-task loss function, including cross-entropy loss and deformable convolutional regularization loss. The total loss function is defined as:

[0080]

[0081] in For cross-entropy loss, For deformable convolution regularization loss, The weights are used for the deformation regularization loss, which includes the fitting loss and the repulsion loss, and is expressed as:

[0082]

[0083] in This represents the distance from the kernel point to the nearest input point. The distance between core points This is the exclusion range threshold. Training uses the Adam optimizer with an initial learning rate of 10. -3 The batch size is 8, and the training lasts for 200 epochs, with the learning rate decreasing by 0.5 times every 50 epochs.

[0084] Step 5: Input real-world examples into the model for testing. Specific implementation examples:

[0086] This invention adopts Figure 7 The first column is the input point cloud sample (the sample is 3D data, and a portion of the view is captured in the visualization software). The second column is the top-down mapping effect of the complete segmentation result after semantic segmentation (each category corresponds to a color). The third column is the single-target correctness graph with vehicles as an example (blue indicates correct reasoning, red indicates that the model misclassifies it as a vehicle, and green indicates that the model misses the classification as a vehicle).

[0087] The experimental results show that this invention achieves efficient and relatively accurate semantic segmentation, effectively reducing human resource costs and improving production efficiency. Furthermore, this method can be transferred to other datasets for training, demonstrating significant potential application value.

[0088] The above specific embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A point cloud semantic segmentation method based on sampling enhancement and multi-viewpoint, characterized in that, The method includes the following steps: Collect airborne lidar point cloud data and perform data augmentation. Preprocess the augmented data to construct a dataset, which is then divided into a training set and a test set. The dataset is sampled; during sampling, the local curvature and normal vector changes of the point cloud are calculated based on the k-nearest neighbor algorithm for boundary points, and boundary point sampling enhancement is performed. A point cloud semantic segmentation model is constructed. This model employs an encoder-decoder architecture. The encoder extracts features from the input point cloud data, including multi-level residual blocks and residual bottleneck blocks. Each residual block and bottleneck block includes a 3D feature extraction branch based on kernel convolution and a 2D feature extraction branch based on multi-view side projection. The 2D feature extraction branch extracts local structural features from different perspectives through 2D convolution, normalization, and pooling operations, and fuses these features with the 3D feature extraction branch at each layer. The decoder upsamples and classifies the fused features point by point, outputting a semantic category label for each point. The 2D feature extraction branch performs the following steps: selecting a sampling center, collecting points within a specified sampling radius around the sampling center, pre-storing point cloud data with the original sampling radius and point cloud data with twice the sampling radius, and performing multi-view side projection mapping. A rotation matrix maps the 3D spatial coordinates to a 2D image plane. That is, for a given point cloud... The rotation operation around the Z-axis can be represented as: in Taking seven specific angles, one every 30° within the range of 0-180°; the rotated point cloud The projection onto the XZ plane is normalized to a fixed-resolution image grid through a linear transformation, where the Y coordinate is determined by a color mapping function. Encode the color using the Viridis color space to achieve continuous gradients, thus preserving the three-dimensional spatial relationships in a two-dimensional projection; A total of 14 two-dimensional projection images with 4 channels RGBA were used. Feature extraction employed a convolutional neural network architecture, with the input tensor set as follows: The feature extraction process can be formalized as follows: in It is a convolutional encoder that extracts local features step by step through three layers of convolution and pooling operations; This is a feature compression layer that uses 1×1 convolutions and global pooling to reduce the feature dimension to 64; ⊕ indicates a feature concatenation operation; To achieve network fusion, a fully connected layer is used to aggregate features from multiple perspectives. The point cloud semantic segmentation model was trained and tested using the sampled training and test sets, respectively. The trained and tested point cloud semantic segmentation model is used to perform semantic segmentation on the new airborne lidar point cloud data.

2. The point cloud semantic segmentation method according to claim 1, characterized in that, The airborne lidar point cloud data includes xyz coordinates, intensity, and manually labeled real classification class information.

3. The point cloud semantic segmentation method according to claim 1, characterized in that, The data augmentation uses a triangular facet model to simulate different angles and occlusions during airborne lidar sampling. The triangular facet model is then converted into virtual point cloud data samples and inserted into the dataset to achieve data augmentation.

4. The point cloud semantic segmentation method according to claim 1, characterized in that, The feature fusion includes: Two-dimensional features are aligned to the three-dimensional feature dimension through linear projection, and then the two-dimensional features are spliced ​​with the three-dimensional features of each point through a broadcast mechanism; in the residual bottleneck block, the fused features are further added to the original features through shortcut connections.

5. The point cloud semantic segmentation method according to claim 1, characterized in that, The loss function used during training employs a multi-task loss function, including cross-entropy loss and deformable convolutional regularization loss, defined as: in For cross-entropy loss, For deformable convolution regularization loss, The weights are used for the deformation regularization loss, which includes the fitting loss and the repulsion loss, and is expressed as: in This represents the distance from the kernel point to the nearest input point. The distance between core points The threshold for the exclusion range, , For weights.

6. A point cloud semantic segmentation system implementing the method as described in any one of claims 1-5, characterized in that, Includes the following modules: The data preprocessing module is used to collect point cloud data and perform preprocessing. The point cloud semantic segmentation module performs semantic segmentation on preprocessed point cloud data using a trained and tested point cloud semantic segmentation model.

7. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed in a computer, causes the computer to perform the method as described in any one of claims 1-5.

8. A computing device, characterized in that, The method includes a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method as described in any one of claims 1-5.

Images