A three-dimensional city natural landscape point cloud online processing system based on deep learning

Abstract

The application discloses a ground three-dimensional laser scanning point cloud collection system and relates to the field of three-dimensional point cloud data processing of the ground; the size of data input is 10 square meters; a KPConv algorithm is improved; the characteristics of each input point are expanded; Max Pooling is used to aggregate the relative position and the Euclidean distance of the field points and the center point; Average Pooling is used to aggregate the field characteristics of each point extracted through the multi-layer KPConv into global characteristics; and a set of point cloud visualization websites is deployed by using WebGL.

Description

Technical Field

[0001] This invention relates to the field of 3D point cloud data processing, specifically to an online processing system for 3D urban natural landscape point clouds based on deep learning; an online service system for intelligent processing, editing, and browsing of urban or rural landscape point cloud data. It can assist personnel involved in urban and rural planning, scenic area planning, heritage protection, and related fields in efficiently analyzing and browsing point cloud data. The website's point cloud computing is deployed on a cloud server, allowing users to quickly and intelligently edit, statistically analyze, and browse point cloud scenes using only a browser. Background Technology

[0002] An analysis of the current state of research both domestically and internationally reveals that extensive methodological discussions and practical applications have been conducted in the digital preservation of traditional settlements and their architecture, making significant contributions to my country's cultural heritage protection efforts. However, most current research focuses on basic theories, historical background, and basic layouts, lacking in-depth spatial feature identification and exploration for disappearing ancient villages. Without causing damage to traditional settlements, combining new surveying and mapping technologies, virtual reality, cloud computing, and big data technologies to acquire complete and detailed geometric and appearance data, and then digitally modeling, storing, and managing this data—including obtaining point cloud semantic segmentation—can significantly reduce or mitigate losses caused by natural disasters, climate change, and human negligence.

[0003] Point cloud semantic segmentation is an indispensable part of 3D scene understanding. Due to the high complexity of sampling methods and the computationally intensive preprocessing steps, most existing models can only input the raw point cloud data into small-scale point cloud blocks for training. The RandLA-Net model proposes a novel input method, replacing the high-complexity sampling method with random sampling, enabling it to input the entire point cloud data without cutting it into small point cloud blocks. However, the good performance of RandLA-Net does not necessarily indicate the effectiveness of its input method, as the network's framework itself is quite unique.

[0004] In summary, the current process of digitizing traditional settlements and their buildings faces the following technical challenges: point cloud data is obtained through scanning devices, and the buildings are then digitized from this data. For point clouds with unordered characteristics, many deep learning algorithms cannot achieve similar results; therefore, point clouds are typically converted into voxels or image sets beforehand. PointNet was the first algorithm to directly feed point clouds into a deep learning framework and achieve good results, but most of these algorithms are limited to extremely small 3D point clouds. The reason for this limitation is that commonly used downsampling methods are either computationally intensive or consume a lot of memory. For outdoor scenes, the volume of building labels that need to be predicted in most datasets is not comparable to the volume of labels for tables and chairs in indoor scenes. If the original point cloud is cut into 1m×1m small point clouds and fed into the network for learning, the shape and structure of large objects such as buildings will inevitably be severely damaged, and spatial geometric information may even be almost completely lost.

[0005] In indoor scenes, such as the Stanford University dataset S3DIS, the semantic labels for objects like tables and chairs are relatively small. Using 1m×1m block-sized inputs, as chosen by most models, is perfectly acceptable and has limited impact on object shape. However, for outdoor scenes, typical labels like houses, trees, roads, and bridges are several times larger than indoor objects. A block size that is too small will obviously result in input points lacking geometric shape information, forcing the network to make predictions based solely on height and basic color information. Summary of the Invention

[0006] To address the aforementioned problems in existing technologies, this invention provides an online processing system for three-dimensional urban natural landscape point clouds based on deep learning.

[0007] To achieve the above-mentioned technical objectives and effects, the present invention is implemented through the following technical solution:

[0008] Step 1: Input data size is 10 square meters; test the impact of the shape and size of the input data on the results. Input methods include block input and sphere input. Input sizes range from 1 square meter to 25 square meters; an input size of approximately 10 square meters yields optimal segmentation accuracy and training efficiency.

[0009] Step 2: Improve the KPConv algorithm to obtain KP-SG (improved point cloud core point convolution); under the same input mode, test the model performance using PointNet, PointNet++, Randla-Net, KPconv, and DGCNN. KPConv is the optimal model choice.

[0010] Step 3: Expand the features of each input point and use Max Pooling to aggregate the relative positions and Euclidean distances between neighborhood points and the center point. This module is the FE module.

[0011] Step 4: Use Average Pooling to aggregate the domain features extracted from each point through multiple KPConv layers into global features, and distribute them to each point. This module is called Global Feature; this improves each point's perception of the global context and improves classification accuracy.

[0012] Step 5: We deployed a point cloud visualization website using WebGL. This website integrates six pre-trained scene segmentation models to intelligently segment scenes, including our improved KP-SG algorithm, and supports browsing, manual point cloud segmentation, and statistical functions.

[0013] The beneficial effects of this invention are:

[0014] A deep learning-based online processing system for 3D urban natural landscape point clouds, along with an online service system for editing and browsing, is presented. This system can assist professionals in urban and rural planning, scenic area planning, and heritage conservation to efficiently analyze and browse point cloud data. The website's point cloud computing is deployed on a cloud server, allowing users to quickly and intelligently edit, analyze, and browse point cloud scenes using only a browser.

[0015] A deep learning-based online point cloud processing system for 3D urban natural landscapes achieves results similar to those of existing block inputs by directly inputting large-scale point cloud data. It also selects different input modes based on label size and distribution. In outdoor scene datasets, it increases the volume of the input blocks to retain the spatial structure features of large labels. Experiment 2 shows that using fixed-point input in the input stage before model training yields higher prediction results than fixed-volume input.

[0016] The improved algorithm KP-SG achieves a two-percentage-point improvement in accuracy compared to the original kpconv.

[0017] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is an overview of the Taoping Qiang Village dataset described in the embodiments of the present invention;

[0020] Figure 2 This refers to the label distribution of the Taoping Qiang Village dataset described in this embodiment of the invention.

[0021] Figure 3 This refers to the time required to complete one epoch for different block sizes as described in the embodiments of the present invention.

[0022] Figure 4 To visualize the raw data and label prediction results as described in this embodiment of the invention, different labels are represented by different colors, and to avoid repetition, only one tile is visualized;

[0023] Figure 5 This is to visualize real input data with different bloc sizes as described in the embodiments of the present invention;

[0024] Figure 6 The red box shows the prediction results of a tile in the second group of the visualization experiment 2 described in this embodiment of the invention, indicating the areas with significant differences.

[0025] Figure 7 The matrix is ​​a relative position encoding of point position information fused by the KP-SG algorithm described in this embodiment of the invention;

[0026] Figure 8 The KP-SG algorithm described in this embodiment of the invention combines the global feature matrix of the entire input sphere;

[0027] Figure 9 This is the overall network structure of the KP-SG algorithm described in this embodiment of the invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Example 1

[0030] Experiment 1: Verify the impact of block size on segmentation results. With other variables remaining the same, increase the block size and compare the model's prediction results.

[0031] Agreement

[0032] 1. The method of cutting the original point cloud is called constant volume, and the input method of RandLA-Net is called constant number.

[0033] 2. The length of the cut edge is called the block size.

[0034] 3. For ease of explanation, the number of input points mentioned below refers to the number of points input within a single batch.

[0035] 4. The mIoU index is used to evaluate the prediction results of the model, and the formula is shown below.

[0036]

[0037]

[0038] Where N is the total number of labels, and n is the nth label. TP, FP, and FN represent the number of true positives, false positives, and false negatives, respectively, as well as IoU (Intersection over Union) and mIoU (Mean Intersection over Union).

[0039] 5. The number of steps required to complete one epoch is denoted as epoch steps.

[0040] 6. In grid sampling, the grid side length is denoted as grid size.

[0041] like Figure 1 As shown:

[0042] Point cloud data was obtained from Taoping Qiang Village, a scenic spot. It contains 200 million point cloud data points covering nearly 3,000 square meters. Each point cloud data point consists of three-dimensional coordinates and color information, and is tagged with eleven categories: building, vegetation, traffic road, ground, meadow, rock wall, rock road, car, others, bridge, and water.

[0043] like Figure 2 As shown:

[0044] The dataset contains 20 tiles, each with an area of ​​50×50 square meters. Sixteen tiles were selected as the training set, and four tiles as the test set (referencing the partitioning of the Sensat Urban and S3DIS datasets, and considering the uneven distribution of labels in outdoor scene datasets). Figure 2 The number of points for each type of label was counted in both the training and test sets; these included: Sensat Urban (the overall graph of the dataset, including a continuous area of ​​York, UK, extending to 3 square kilometers), Sematic labels, Number of points, Training set, and Test set.

[0045] For fixed-point input method:

[0046] Step 1: Randomly assign a "probability" to each point;

[0047] Step 2: Before inputting, select the tile containing the point with the lowest probability, and use that point as the center point to search for the nearest set of a specified number of points within this tile, starting from the center point.

[0048] Step 3: Assign weights to each point in the selected point set based on its distance from the center point, thereby increasing the probability of each point in the point set accordingly;

[0049] Step 4: Repeat step 2. This input method results in an irregular point cloud with varying shapes each time.

[0050] Step 5: The experiment was conducted on the RandLA-Net model;

[0051] Because the Taoping Qiang Village dataset has uneven outdoor scene distribution, and RandLA-Net uses a weighted cross-entropy loss function with the number of labels in each category as the weight, it solves the problem caused by uneven outdoor scene distribution. Therefore, in the source code, the weights are changed to the number of labels in the dataset used. In the experiment: weights = [72495175, 127473936, 4898801, 10988151, 15958409, 9233404, 3324402, 430049, 9459596, 965145, 3419016]. To ensure time comparability, a single variable is controlled and the experiment is run on a consistent GPU. The experiment was tested on an RTX 2080Ti×1 GPU.

[0052] Step 6: Based on the scene type of the dataset, set the block size to 5m×5m for datasets with large buildings and 2m×2m for datasets with small labels. After building the model, improve the model performance by selecting the appropriate input method.

[0053] The block size was set to 1m×1m, 2m×2m, 3m×3m, 4m×4m, and 5m×5m, respectively. The number of input points was increased proportionally to the area, fixed at 1024, 1024×4, 1024×9, 1024×16, and 1024×25, respectively, while maintaining the network downsampling method and basic network architecture. Due to the sparse density of the Taoping Qiang Village dataset, 1024 input points were used for the 1m×1m block. The IoU for each of the 11 categories and the overall mIoU were calculated, and the time required for the model to complete one epoch of training was also recorded. Experimental results are shown in Table 1. Figure 3 and Figure 4 As shown; Figure 5 As shown, the inputs of different block sizes are visualized, allowing a direct view of the characteristics of the input points.

[0054] Table 1. Performance of RandLA-Net on the Taoping Qiangzhai dataset with different block sizes

[0055]

[0056] As shown in Table 1, the overall prediction performance of the model improves with increasing block size. The mIoU increases by 5.3% as the block size increases from 1m×1m to 5m×5m. Specifically, the mIoU of buildings and bridges, which have large spatial geometric features and are relatively large, increases by 7.92% and 21.14%, respectively. Conversely, the prediction performance of cars, which have smaller volumes, decreases. Expanding the range that a point can perceive preserves more geometric information. RandLA-Net aggregates the coordinates of neighboring points and the relative distances between points in the feature information encoding part of the point, thus amplifying the spatial structural features of the point. Therefore, for labels with large spatial volumes, the larger the input volume, the more geometric information can be obtained, resulting in better performance. However, for labels with small volumes, the input volume can no longer obtain more effective information beyond a certain point.

[0057] like Figure 3As shown, due to the high efficiency of random sampling, the overall time difference is not significant. The time required to complete one epoch initially decreases from 81s / epoch to 56s / epoch, then gradually increases to 65s / epoch. This is because initially, increasing the number of sampling points has less impact on time than reducing the number of epoch steps; as the block size increases, the number of epoch steps decreases accordingly. However, as the block size and the number of input points increase, the impact of sampling on time exceeds the impact of epoch steps, leading to a gradual increase in time. In practice, choosing random sampling with low time complexity and reasonably increasing the block size and the number of input points can achieve an effective balance between computational efficiency and performance.

[0058] like Figure 4 As shown, a fixed volume input results in inconsistencies because the same object is divided into different blocks and input separately, leading to different segmentation results. However, the inconsistencies gradually decrease as the block size is increased.

[0059] Example 2

[0060] Experiment 2:

[0061] Compare two different input methods and observe the different effects of fixed volume and fixed number of points; determine the input point coverage area, select the appropriate block size and number of input points, and compare the differences between fixed volume and fixed point input methods; conduct the experiment in two groups.

[0062] Step 1: For both methods, grid downsampling is performed during the preprocessing stage, with a grid size of 0.06m, reducing the number of point clouds to about 1 / 10 of the original number of point clouds; based on the conclusion of Experiment 1, a large coverage area is obtained.

[0063] Step 2: Design two sets of experiments: one with a fixed block size of 4m×4m and a fixed number of points of 4096, and the other with a block size of 10m×10m and a fixed number of points of 4096×6. For each set of experiments, select the same number of points to ensure comparisons are made within similar coverage areas.

[0064] The prediction results are shown in Table 2; the prediction results for one tile in the second group of experiments are visualized as follows. Figure 6 As shown;

[0065] Table 2. Evaluation of prediction results on RandLA-Net for two experimental groups using the Taoping Qiang Village dataset with a grid size of 0.06m.

[0066]

[0067]

[0068] As shown in Table 2, the segmentation effect of fixed-point input is better than that of fixed-volume input, with mIoU being 4.22% and 3.77% higher, respectively. According to the visualization results, fixed-point prediction is more reasonable than fixed-volume prediction. The fixed-volume input method also exhibits the inconsistency phenomenon described in Experiment 1 when the block size is at its maximum of 10m×10m, but the fixed-point input method does not have this problem. Therefore, the fixed-point input method should be used in the input stage before model training.

[0069] In the two sets of experiments, we compared the same input method with different numbers of input points. By increasing the number of input points and fixing the volume of the fixed points, the conclusions were the same as in Experiment 1, namely, the more input points there are, the better the model learning effect.

[0070] like Figure 6 As shown, in Experiments 1 and 2, the IoU of cars and bridges with relatively few labels is very low, while the prediction results of buildings, traffic roads, and vegetation with relatively many labels are similar to the actual results. It can be seen that the uneven distribution of the number of labels affects the prediction results, resulting in a large difference between the labels with fewer labels and the actual results.

[0071] The above findings provide preliminary verification of the feasibility of the algorithm improvement.

[0072] Example 3

[0073] The main improvements of the KP-SG (Improved Point Cloud Core Point Convolution) algorithm are as follows:

[0074] 1. A matrix that integrates point location information and relative position encoding, such as... Figure 7 The FE module (feature-rich module) enhances kpconv's ability to understand point cloud shapes; (point set: point set; K: nearest neighbor; M: max pooling; +: concatenation).

[0075] 2. Before performing softmax (normalized exponential function) classification prediction on each point, the global feature matrix of the entire input sphere is combined, such as... Figure 8 The global feature in this context refers to the average pooling or shared multilayer perceptron (MLP) technology.

[0076] 3. The overall network structure is as follows: Figure 9As shown, the FE module and Global Feature (point: point cloud; FEmodule: finite element module; simple block: simple block; resnet block: residual module; resnet stridcd: deep residual network; nearest_upsample: nearest upsampled; lconv: vector convolution operation, convolutional layer; globalfeature: global feature; skip connection: skip connection;)

[0077] Table 3: Semantic segmentation results (%) of different methods

[0078] PointNet PointNet++ DGCNN RandLA-Net KP-FCNN Ours mIoU 33.88 21.24 39.06 42.53 50.50 53.03 OA 73.28 67.03 69.03 76.90 84.30 84.61 build 58.40 37.30 63.68 64.84 79.49 81.02 veg 79.20 72.00 65.08 77.72 82.77 83.25 tra.road 55.50 58.30 77.25 87.05 92.01 93.02 ground 31.20 15.40 44.41 53.96 53.87 59.41 mead 25.00 3.00 23.30 29.75 24.77 30.60 wall 10.20 0.90 19.02 37.44 31.84 33.80 rock 52.50 23.30 62.01 56.64 73.50 72.31 cars 7.20 0.00 4.21 5.54 12.24 18.97 others 7.70 2.90 2.69 8.76 13.24 14.04 bridge 2.90 0.00 5.42 0.91 27.42 25.26 water 42.90 20.50 62.62 45.27 64.30 71.62

[0079] The above three points are the core of the improvement. Comparative experiments have shown that the improved algorithm has a two percentage point higher accuracy than the original kpconv (as shown in Table 4).

[0080] Table 4: Comparison of KP-SG and kpconv

[0081]

[0082]

[0083] In summary, the present invention provides an online processing system for 3D urban natural landscape point clouds based on deep learning. The improved algorithm KP-SG achieves a two percentage point higher accuracy than the original kpconv.

[0084] This service supports the following functions:

[0085] 1. Supports various point cloud data sources: terrestrial laser scanning, aerial laser scanning, tilt model reconstruction point cloud, mobile lidar scanning, and other data sources.

[0086] 2. Point cloud preprocessing: Remove outliers and support downsampling at a set ratio (farthest point downsampling, grid downsampling, random downsampling).

[0087] 3. Based on the deep learning algorithm model, scene segmentation is automatically performed, and the results are visualized. The server deploys four pre-trained scene segmentation models (PointNet, PointNet++, Randla-Net, and KPconv), capable of recognizing 11 semantic labels: plants, buildings, roads, grass, ground, bridges, water surfaces, street facilities such as trash cans and utility poles, walls, stones, and cars. These four models are trained on commonly used open-source urban or rural point cloud datasets, achieving an average accuracy of over 70%, with semantic annotation accuracy for objects such as buildings and plants exceeding 90%, demonstrating strong generalization ability.

[0088] 4. Interactive Point Cloud Editing and Browsing: Users can select desired points on the screen. The algorithm intelligently helps users select points based on color growth regions, normal growth regions, or judgments from deep learning models regarding whether they belong to the same semantic label. It supports editing point colors, positions, manually setting labels, and deleting points.

[0089] 5. Supports automatic point cloud segmentation.

[0090] 6. Select the point cloud measurement: point distance measurement, area measurement, angle measurement, volume measurement.

[0091] 7. Saving of sub-point clouds.

[0092] 8. Based on WebGL point cloud rendering technology, the frame rate can reach 20 frames per second. High-quality scene browsing is possible, and users can set browsing routes, including flight browsing and ground browsing, and configure the browsing route and speed.

[0093] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0094] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A ground-based three-dimensional laser scanning point cloud acquisition system, characterized in that: Step 1: Input the size of the data: 10 square meters; Step 2: Expand the features of each input point in the KPConv algorithm, and use Max Pooling to aggregate the relative positions and Euclidean distances between neighborhood points and the center point. This module is the FE module. Step 3: Use Average Pooling to aggregate the domain features extracted by the KPConv algorithm for each point into global features, and distribute them to each point. This module is called Global Feature. Step 4: Improve the KPConv algorithm to obtain KP-SG; the main improvements to the KP-SG algorithm are as follows: The FE module is used to fuse the position information of points and the matrix of relative position encoding. It enhances KPConv's ability to understand the shape of point clouds through point sets, nearest neighbor points, max pooling and concatenation techniques. The Global Feature module uses the techniques of averaging and sharing multilayer perceptrons to combine the global feature matrix of the entire input sphere before performing softmax classification prediction on each point. The overall network structure of the KP-SG algorithm includes a point cloud input layer, an FE module, a simple block, a resnet block, a resnet strided layer, an lconv layer, a global feature layer, and a nearest_upsample module. The modules are connected via skip connections to achieve collaborative feature transfer. The FE module and the global feature module are responsible for expanding the input point features and aggregating the global features, respectively.

Images