Real-time semantic three-dimensional reconstruction method based on laser radar

Abstract

The invention provides a real-time semantic three-dimensional reconstruction method based on a laser radar, according to the characteristics of sparse point cloud data and non-uniform density in an outdoor and indoor large-scale scene environment, through cylindrical division and an asymmetric three-dimensional convolutional network, the influence of sparse point cloud characteristics on a segmentation result is reduced, and the precision of the segmentation result is ensured; secondly, the invention also provides a semantic three-dimensional grid map, and the semantic three-dimensional grid map is used as a tool to realize the fusion of semantic information and a three-dimensional reconstructed grid map; and finally, in order to ensure the fusion precision, the invention further provides a constraint relation between grids, and the influence caused by semantic segmentation errors is reduced by establishing the relation between the grids, so that a better effect is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot positioning and reconstruction, and in particular relates to a real-time semantic three-dimensional reconstruction method based on laser radar. Background Art

[0002] The process of point cloud mesh reconstruction generally includes data preprocessing, feature extraction, mesh generation and post-processing. Among them, feature extraction is a key step. By extracting the geometric features of the point cloud (such as normal vectors, curvature, etc.), the shape information of the object can be better reflected. The mesh generation algorithm connects the point cloud data into triangular facets based on the extracted features to form a continuous surface. For 3D reconstruction technology, there are several existing solutions: Poisson-based surface reconstruction method, Delaunay tetrahedronization (i.e. 3D Delaunay triangulation) and graph cut method (Delaunay-based).

[0003] The main task of point cloud semantic segmentation is to output a predefined label value for each 3D point based on the input 3D point cloud, which helps to obtain a rich understanding of the scene. Real-time fine-grained semantic prediction can accelerate the realization of full autonomy. For point cloud semantic segmentation technology, there are several existing solutions:

[0004] Solution 1: The PointNet algorithm was proposed in the literature (Qi CR, Su H, Mo K, et al. PointNet: Deep Learning on PointSets for 3D Classification and Segmentation: 30th IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR) [C]. New York: IEEE, 2017.). The contribution of the PointNet algorithm is that it directly extracts and classifies features from disordered point cloud data, breaking through the limitation of traditional deep learning methods that require regular data formats. However, the PointNet algorithm has limited local feature extraction capabilities and is sensitive to point cloud density, both of which will lead to reduced performance and accuracy when segmenting point cloud data.

[0005] Solution 2: The paper (Zhao H, Jiang L, Jia J, et al. Point Transformer: 17th IEEE / CVFInternational Conference on Computer Vision (ICCV) [C]. Piscataway: IEEE, 2021.) proposed the Point Transformer algorithm. The contribution of the Point Transformer algorithm is to introduce the self-attention mechanism into point cloud processing, effectively capturing the long-range dependencies between different points in the point cloud. Replacing the traditional MLP with the self-attention layer enables the model to better learn the global and local features of the point cloud. However, the Point Transformer algorithm is sensitive to hyperparameters and has a high computational complexity. Summary of the invention

[0006] In order to solve the problem that existing grid maps lack scene semantic information, the present invention provides a real-time semantic 3D reconstruction method based on laser radar, which uses laser radar as a sensing device to construct a real-time 3D reconstruction grid map with semantic information.

[0007] A real-time semantic 3D reconstruction method based on laser radar includes the following steps:

[0008] Cylindrical segmentation is performed on the original 3D lidar point cloud to obtain a plurality of cylindrical voxels;

[0009] Input each cylindrical voxel into the trained asymmetric 3D convolutional network to obtain the semantic label corresponding to each cylindrical voxel;

[0010] The semantic labels corresponding to the cylindrical voxels are respectively used as the semantic labels corresponding to the point clouds falling into the cylindrical voxels, so as to complete the mapping from the voxel semantic labels to the point cloud semantic labels;

[0011] Divide the original 3D laser radar point cloud into 3D grids to obtain multiple 3D grids;

[0012] The grid semantics corresponding to each three-dimensional grid are obtained respectively, and the grid semantics corresponding to any three-dimensional grid is obtained by counting the number of point clouds under different point cloud semantic labels in the three-dimensional grid, and taking the point cloud semantic label with the largest number of point clouds as the grid semantics corresponding to the three-dimensional grid;

[0013] The original 3D lidar point cloud is reconstructed in three dimensions, and each vertex is colored according to the vertex semantics corresponding to each vertex of the grid map obtained by the three-dimensional reconstruction, so as to complete the three-dimensional reconstruction with semantic information; wherein, the vertex semantics corresponding to any vertex is obtained by taking the three-dimensional grid in which the vertex falls as the wrapping grid, and judging whether there are more than 5 three-dimensional grids in the surrounding three-dimensional grids adjacent to the wrapping grid whose grid semantics are the same as the grid semantics corresponding to the wrapping grid; if so, the grid semantics corresponding to the wrapping grid is the vertex semantics corresponding to the vertex; if not, the vertex semantics corresponding to the vertex is the set default semantics.

[0014] Furthermore, the method of cylindrically segmenting the original 3D lidar point cloud to obtain multiple cylindrical voxels is as follows:

[0015] The Cartesian coordinates (x, y, z) of each point cloud are converted to cylindrical coordinates (ρ, θ, z), where the radius ρ is defined as the distance from the point cloud to the Cartesian origin on the xy plane, and the azimuth θ is defined as the angle of the point cloud from the x-axis to the y-axis;

[0016] In the cylindrical coordinate system, the point cloud is non-uniformly segmented according to the three dimensions of radius, azimuth and height to obtain multiple cylindrical voxels; wherein the cylindrical voxel with a larger distance from the origin of the cylindrical coordinate system has a larger size.

[0017] Furthermore, the method of performing three-dimensional reconstruction on the original three-dimensional lidar point cloud to obtain a grid map is as follows:

[0018] Transform each point cloud into world coordinates using an initial guess of the constant velocity model;

[0019] Allocate each point cloud to each irregular voxel unit in the world coordinate system;

[0020] The Gaussian process is used to reconstruct the local surface inside each irregular voxel unit to obtain vertices, and the vertices are connected to form triangular meshes;

[0021] The original 3D lidar point cloud is aligned with the constructed triangular mesh to obtain a mesh map.

[0022] Furthermore, when real-time semantic 3D reconstruction of the park is performed, the semantic labels include public facilities, roads, plants, and buildings.

[0023] Furthermore, when real-time semantic 3D reconstruction of the vehicle driving environment is performed, the semantic labels include buses, cars, motorcycles, bicycles, plants, and buildings.

[0024] Furthermore, when real-time semantic 3D reconstruction is performed on the underground garage, the semantic labels include buses, cars, motorcycles, bicycles, and buildings.

[0025] Furthermore, when each vertex is colored according to the vertex semantics corresponding to each vertex of the mesh graph obtained by three-dimensional reconstruction, different semantic labels correspond to different colors.

[0026] Beneficial effects:

[0027] 1. The present invention provides a real-time semantic 3D reconstruction method based on LiDAR. According to the characteristics of sparse and uneven density of point cloud data in large outdoor and indoor scene environments, cylindrical division and asymmetric 3D convolutional network are used to reduce the influence of sparse point cloud characteristics on segmentation results, thereby ensuring the accuracy of segmentation results. Secondly, the present invention also proposes a semantic 3D grid map, which is used as a tool to realize the fusion of semantic information and 3D reconstructed grid map. Finally, in order to ensure the fusion accuracy, the present invention also proposes a constraint relationship between grids, which reduces the influence of semantic segmentation errors by establishing connections between grids, thereby achieving better results.

[0028] 2. The present invention provides a real-time semantic 3D reconstruction method based on LiDAR, which accelerates point cloud processing through Gaussian regression process, realizes real-time 3D reconstruction relying only on CPU, allows positioning and mapping to benefit each other, and solves the problem of generally poor real-time performance of current grid maps. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is the overall design idea of ​​the present invention.

[0030] Figure 2 Flowchart of the semantic segmentation module of the present invention.

[0031] Figure 3 This is the result output by the semantic segmentation module of the present invention.

[0032] Figure 4 FIG. 4 is a flow chart of a three-dimensional reconstruction module of the present invention.

[0033] Figure 5 This is the construction process of the semantic grid map of the present invention.

[0034] Figure 6 This is a grid map with semantic information reconstructed by the present invention.

[0035] Figure 7 This is a schematic diagram of the asymmetric residual module provided by the present invention.

[0036] Figure 8 This is a schematic diagram of the downsampling module provided by the present invention.

[0037] Fig. 9 This is a schematic diagram of the upsampling module provided by the present invention.

[0038] Fig.10 A schematic diagram of a context modeling module for dimensional decomposition provided by the present invention. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0040] At present, most laser SLAM systems directly use point clouds to build maps, but when the point cloud map is enlarged, it is very sparse. With the increase in the complexity of robot tasks, this map can no longer meet the needs of robot navigation, positioning and obstacle avoidance, and a denser high-precision map is required. Due to the low memory cost, grid-based 3D reconstruction has become a competitive dense model mapping method. However, the existing grid maps only cover the geometric information of the modeling scene, do not cover the semantic information in the scene, and have poor real-time performance, which greatly limits the robot's ability to perform advanced and complex tasks. Therefore, the present invention proposes a real-time semantic 3D reconstruction method based on laser radar to solve the above problems.

[0041] A real-time semantic 3D reconstruction method based on laser radar includes the following steps:

[0042] S1: Cylindrical segmentation of the original 3D LiDAR point cloud to obtain multiple cylindrical voxels;

[0043] S2: Input each cylindrical voxel into the trained asymmetric 3D convolutional network to obtain the semantic label corresponding to each cylindrical voxel;

[0044] For example, when performing real-time semantic 3D reconstruction of a park, the semantic tags include public facilities, roads, plants, buildings, etc.; when performing real-time semantic 3D reconstruction of a vehicle driving environment, the semantic tags include buses, cars, motorcycles, bicycles, plants, buildings, etc.; when performing real-time semantic 3D reconstruction of an underground garage, the semantic tags include buses, cars, motorcycles, bicycles, buildings, etc.

[0045] S3: taking the semantic labels corresponding to the cylindrical voxels as the semantic labels corresponding to the point clouds falling into the cylindrical voxels, and completing the mapping from the voxel semantic labels to the point cloud semantic labels;

[0046] S4: dividing the original 3D laser radar point cloud into 3D grids to obtain multiple 3D grids;

[0047] S5: respectively obtaining grid semantics corresponding to each three-dimensional grid, and the grid semantics corresponding to any three-dimensional grid is obtained by counting the number of point clouds under different point cloud semantic labels in the three-dimensional grid, and taking the point cloud semantic label with the largest number of point clouds as the grid semantics corresponding to the three-dimensional grid;

[0048] S6: Perform three-dimensional reconstruction on the original three-dimensional lidar point cloud, and color each vertex according to the vertex semantics corresponding to each vertex of the grid diagram obtained by the three-dimensional reconstruction, so as to complete the three-dimensional reconstruction with semantic information, and different semantic labels correspond to different colors; wherein, the method for obtaining the vertex semantics corresponding to any vertex is: taking the three-dimensional grid in which the vertex falls as the wrapping grid, and judging whether there are more than 5 three-dimensional grids in the surrounding three-dimensional grids adjacent to the wrapping grid whose grid semantics are the same as the grid semantics corresponding to the wrapping grid; if so, the grid semantics corresponding to the wrapping grid is the vertex semantics corresponding to the vertex; if not, the vertex semantics corresponding to the vertex is the set default semantics.

[0049] It can be seen that the present invention adopts a modular design and is divided into three modules: semantic segmentation, semantic 3D grid map, and 3D reconstruction. Each module can be run independently, which solves the problem that the existing grid map has poor real-time performance and lacks semantic information. The LiDAR point cloud can be input and semantic segmentation can be performed on the LiDAR point cloud through the semantic segmentation network. According to the semantic information obtained by segmentation, the output is as follows: Figure 3 The result of semantic segmentation is input into the grid map module to obtain a semantic 3D grid map. The semantic 3D grid map is combined with the grid map obtained by the 3D reconstruction module to achieve the fusion of object-level semantic information and grid map, and generate real-time Figure 6 The semantic grid map shown.

[0050] It should be noted that steps S1 to S3 of the present invention are actually for semantic segmentation, and the specific implementation of semantic segmentation is described in detail below.

[0051] In order to solve the two difficult problems of sparsity and density variation of outdoor radar point clouds, the present invention first uses cylindrical segmentation to generate a more balanced point distribution (making it more robust to density variations), and then uses an asymmetric three-dimensional convolutional network to enhance the horizontal and vertical weights, thereby well matching the target point distribution and enhancing the robustness to sparse point clouds.

[0052] like Figure 2As shown in the figure, the framework consists of two parts: cylindrical segmentation and asymmetric 3D convolutional network. First, the lidar point cloud is cylindrically segmented, based on which the features extracted by the multi-layer perceptron (MLP) are reconfigured. Then, an asymmetric 3D convolutional network is applied to produce voxelized results. Finally, by introducing a point-by-point optimization module, the error caused by voxel unit label encoding can be effectively reduced, thereby improving the quality of the final output. This series of steps aims to improve the accuracy and efficiency of point cloud data processing.

[0053] In view of the above process, the present invention has the following design features:

[0054] 1. Use cylindrical segmentation. In outdoor environments, the point clouds scanned by LiDAR usually show uneven density, that is, the point cloud density near is much higher than that far away. If regular cells are used to segment these point clouds, it will lead to uneven distribution problems such as too high a proportion of empty cells. In contrast, the cylindrical coordinate system covers distant areas by gradually expanding the grid size, which can distribute the point cloud more evenly and provide a more balanced representation for areas of different densities. In addition, unlike projection-based methods that project point clouds onto a two-dimensional plane, cylindrical segmentation retains the representation of a three-dimensional grid, thereby better maintaining the geometric structure of the point cloud.

[0055] The specific workflow is as follows: First, convert the Cartesian coordinates (x, y, z) of each point into cylindrical coordinates (ρ, θ, z), where the radius ρ is defined as the distance from the point to the origin on the xy plane, and the azimuth θ is defined as the angle of the point from the x-axis to the y-axis. In the cylindrical coordinate system, segmentation is performed based on the three dimensions of radius, azimuth, and height. In this segmentation strategy, the farther the area is from the origin, the larger the corresponding cell size. The point features calculated using the multi-layer perceptron (MLP) are redistributed according to the previous results to form a cylindrical feature set. Finally, the cylinder is expanded from 0 degrees to obtain a three-dimensional cylindrical representation R∈H×W×L, where H represents the radius, W represents the azimuth, and L represents the height. The subsequent asymmetric three-dimensional convolutional network will also be processed based on this representation.

[0056] 2. Use an asymmetric three-dimensional convolutional network. Point cloud data in a driving environment usually contains objects of specific shapes, such as cars, buses, and motorcycles, which can often be represented by cubes. The present invention is based on this observation and improves the effect of standard three-dimensional convolutional networks in processing such data by modeling these objects as cubes. In addition, the center cross weight is very important in the square convolution kernel. In view of this specific distribution of point cloud data, the present invention uses asymmetric residual blocks to enhance the responsiveness in the horizontal and vertical directions to better match the point distribution of the target object. Based on the asymmetric residual blocks, asymmetric downsampling and upsampling blocks are further developed to perform the downsampling and upsampling processes.

[0057] Among them, the asymmetric residual block is as follows Figure 7 As shown in the figure. First, a convolutional layer with a kernel size of 3×1×3 is used, followed by a 1×3×3 convolutional layer. The receptive field is the same as that of a 3×3×3 three-dimensional convolution with a kernel size of 3×3×3. However, the computational cost of this method is 33% lower than that of a 3×3×3 convolution with the same number of output filters. The asymmetric residual block, which splits the 3×3×3 convolution into 1×3×3 and 3×1×3 parts, has two advantages: first, since the targets to be detected are close to rectangular blocks, this form of convolution is more conducive to extracting the features of rectangular styles; second, similar to deep convolution, it can greatly reduce the amount of calculation and improve efficiency.

[0058] In the proposed architecture, the asymmetric residual block (A) is the core of the downsampling (D) and upsampling (U) modules. Figure 8 and Fig. 9 As shown in the figure, the downsampling module (D) consists of an asymmetric residual block and a 3D convolution with a stride of 2 to achieve data downsampling. The upsampling module (U) integrates low-level features and optimizes the features through an asymmetric residual block (A). This design effectively improves the expressiveness of features and the performance of the network.

[0059] At the same time, the dimensional decomposition-based context modeling (DDCM) technology is introduced to improve the ability to handle high-order global context in the decomposition-aggregation strategy. The combination of these technologies provides strong support for the accurate processing of point cloud data. Specifically, Fig.10As shown in the figure, DDCM often requires a high-rank tensor to effectively encode the context differences between point clouds in three-dimensional space when dealing with them. This is because the distribution and relationship of point cloud data in space are complex and varied, and a single low-rank representation often cannot capture all the context information. However, due to its high-rank characteristics, it is expensive to construct and compute such a high-rank tensor in three-dimensional space. Inspired by the high-rank matrix decomposition theory, complex high-order context can be decomposed into simpler low-rank representations. In the point cloud segmentation task, this means that the high-order context can be decomposed along three main dimensions: height, width, and depth. The representations on each dimension are low-rank, but together they form a complete high-order context. This decomposition-aggregation strategy handles high-order complexity from different dimensions while simplifying computation based on low-order constraints. As shown in the figure below, by using three rank-1 kernels, low-rank encodings can be generated on each dimension. Then, the convolution results are modulated by the Sigmoid function to generate weights for each dimension. These rank-1 tensors based on different dimensions can mine co-occurrence context information. Finally, by aggregating the low-rank activations in all three dimensions, we obtain a sum of features that can represent the complete context.

[0060] 3. Use a point-by-point optimization module. Voxelization methods are indeed very effective in processing large-scale point cloud data, but they also have the problem of classifying points of different categories into the same voxel unit, which may lead to information loss. To solve this problem, the present invention can introduce a point-by-point refinement module to reduce the interference caused by inaccurate voxel label encoding.

[0061] The process includes the following steps: First, the cylindrical voxel features are projected back to the original point cloud according to the inverse point-voxel mapping table. This means that all points in the same voxel unit will be assigned the same features. Then, the point features before and after the 3D convolutional network processing are quantitatively fused, and the fused features are input into the point-by-point refinement module to refine the final output result (i.e., the category labels corresponding to all point clouds).

[0062] In this way, each point can get a more refined and accurate label, thereby improving the overall segmentation quality and mIoU. This point-by-point refinement method helps to retain more detailed information, especially when there are multiple categories inside the voxel unit. Such a strategy can reduce information loss and improve segmentation accuracy while maintaining the advantages of voxelization methods in exploring a large range of point clouds.

[0063] Furthermore, in order to merge the semantic information with the three-dimensionally reconstructed grid map, the present invention also proposes a semantic three-dimensional grid map obtained by steps S4 to S5, which is specifically implemented as follows: first, read and save all point cloud data of the current frame, and convert it to the world coordinate system using the updated transformation matrix of each frame for subsequent processing. Next, read the folder containing semantic labels, and each txt file in the folder stores the semantic labels of all 3D points of the corresponding frame, so as to prepare for the subsequent grid semanticization. A custom function is defined in the Octree class, which receives the point cloud set and the semantic label storage path in the world coordinate system, and reads the semantic information from the file. Then, traverse all points in the point cloud, and update their occupancy information and semantic information in the octree. Finally, classify the semantic information of the points in each grid into target class and non-target class, and store them separately. Using the majority voting mechanism, the semantic label with the largest number of points in the grid is the semantic label of the grid.

[0064] Furthermore, a mesh is a three-dimensional solid surface composed of faces, edges, and vertices. In the field of three-dimensional modeling, such as Figure 6 The triangular facets shown are the main representation form because they are simple and can approximate most complex three-dimensional structures. The present invention improves the existing method by using Gaussian process (GP) to process point cloud data, so that the system can run in real time only by CPU, and can achieve real-time update of grid map even when using sensors with denser point clouds such as 64-line laser radar.

[0065] The overall flow chart of the 3D reconstruction module is as follows: Figure 4 As shown. It mainly consists of three parts: meshing, registration and mesh management. First, each new radar scan is converted to the world coordinate system using the initial guess of the constant velocity model. Subsequent operations are performed in the world coordinate system. Then, the points are assigned to voxel units. GP reconstructs the local surface inside each unit to obtain vertices, which are connected to form a mesh. In the registration component, a point-to-mesh registration is designed to align the reconstructed current scan image with the constructed mesh map, and finally the mesh map is iteratively updated.

[0066] Building and updating the grid is very time-consuming. To solve this problem, the present invention adopts a reconstruction and connection strategy to facilitate subsequent steps and enable the entire system to run in real time. GP recovers local surfaces from the noise and sparse point clouds within the voxels, and these vertices are the interpolation results of the surface. Two coordinates of the three-dimensional vertex are uniformly distributed (called locations), and the other coordinate (called prediction) has a continuous value range. These locations are used as indexes to achieve fast queries in constant time. The predicted value range is continuous to avoid the loss of accuracy caused by discretization. The present invention uses a spatial Gaussian process to reconstruct local surfaces from noisy distance observations. In order to obtain more feasible data association and map updates, discrete samples are extracted from the recovered surface. This process is called regionalized Gaussian process map construction. In other words, it can be regarded as a kind of surface interpolation. This process includes two parts: regionalization and reconstruction.

[0067] First, in order to establish different functional relationships locally, the entire domain is divided into several uniformly distributed cubic cells in the world coordinate system {W}. This decomposition also speeds up the reconstruction process. The side length of each cell is a. The original point The subset located in the kth cell is denoted as Then we need to determine the functional relationship between the coordinates, x = f(y,z), or y = f(x,z), or z = f(x,y). Considering that a function can only express a 2.5-dimensional surface, for complex three-dimensional structures, it is generally assumed that there are three functions in a cell. Each function provides corresponding constraints along its direction. If the surface in a cell is perpendicular to a coordinate plane, the corresponding function in the cell can be omitted because it only provides constraints along its normal. This situation is judged based on principal component analysis (PCA). For example, when a set of raw data is extracted from a vertical wall, the function in the direction of z is ignored because the wall cannot provide vertical constraints.

[0068] After regionalization, GP map reconstruction is performed in each non-empty cell. Lidar measures noise samples of the environment. Its noise model can be derived from manufacturing data. The present invention simply assumes that each Lidar point follows an isotropic variance σ 2 In this case, GP regression is used, which produces the best linear unbiased predictions.

[0069] The GP regression problem is as follows: given n k training points D = {(f i ,l i ),i=1,…n k}, training position With the observed value f i The relationship between ∈R is expressed as f i =f(li )+ε i ,i=1,…,n k , where ε i is to follow The goal is to achieve n test Prediction f * At the preset test position Use f *j =f(l *j )+ε *j ,j=1,…,n test Definition and The predicted distribution f given f * for

[0071] The average value As f * In the test position * The point forecast of . Its variance is estimated as Here, k ** = k(l * ,l * ), k l* = k(l,l * ) T , and K ll is a k ×n k Matrix, K ll (i,j) = k(i,j), k(.,.) represents the kernel function. Select the commonly used exponential covariance function k(l i ,l j )=exp(-κ|l i -l j |), which has a preset length scale parameter κ.

[0072] In this case, the training points are the original points in the cell The coordinate used as the observation is called the direction, and the other two coordinates are used as the training position. Each test point is set evenly with an interval of r. Since the side length of the cell is a, let a be the integral times r, which means n test =(a / r) 2 These predictions with variance are samples drawn from the implicit surface, and each group of samples is named a layer. Predictions that are farther from the original data are less reliable. These samples are then used as the reconstruction results. After reconstruction, there are 0-3 layers in a cell. The cells are stored in a hash table data structure. A sample is represented by p i =(f i ,l i) is expressed as and test coordinates as indices. This allows for direct query of samples.

[0073] Based on the regional decomposition, the concept of local regression is used to further speed up the GP training process. The core idea is to predict the influence of observations whose training locations are closer to the predicted test location. Therefore, the training process can be accelerated by principled downsampling of the training points without causing much loss of accuracy. The specific approach is to choose to keep the original data, but only use all the nearest points of each test location in each GP map reconstruction. In this way, the number of filtered training points will be significantly reduced. One way to complete this filtering process is to use Kd-tree. However, in the case of n inputs, the initialization cost of this data structure is O(nlogn) and the average search cost is O(logn). Although Kd-tree is faster than brute force search, it is still time-consuming, especially when the number of points is large. Since the search target in this design application is uniformly distributed, an improved two-dimensional voxel filter is used to approximate this process. In a cell, the training locations are distributed in a two-dimensional domain, which is divided into smaller grids, and the center of the grid is the test location. The original voxel filter calculates the average of all the original data in each small grid. The modified approach is to keep a point if it is the closest to the test position among all the points in the same smaller grid. Therefore, the filtering process can be completed with a linear complexity cost.

[0074] Based on this, in the process of fusing semantic information and grid map provided in step S6, it is necessary to judge whether Figure 6 The mesh vertex colors shown. If only the grid semantics of the vertex are used to color the vertex, the accuracy will be completely dependent on the semantic segmentation accuracy. When the accuracy is not ideal, the fusion effect is also not ideal. Therefore, the present invention designs a grid constraint method. Take the judgment of the semantics of a car as an example: when each vertex determines the semantics, the semantics of the surrounding grids are judged. If the number of surrounding grids with the semantics of a car reaches 5 or more, the vertex is considered to have the semantics of a car, and then colored according to the predefined color relationship. This approach utilizes the constraint relationship between grids, effectively alleviates the impact of semantic segmentation errors, and improves the fusion quality.

[0075] Finally, the present invention uses the proposed method to test the complex environment, and the results are as follows Figure 6As shown. It can be seen that the real-time semantic three-dimensional reconstruction method based on laser radar of the present invention has achieved relatively impressive results. In general, the point cloud semantic segmentation module provides semantic information for the subsequent reconstruction module and the three-dimensional grid module, and can independently run to output a point cloud map with semantic information. As a means of connecting semantic information and three-dimensional reconstruction grid maps in series, the three-dimensional grid provides semantic information from the perspective of spatial coordinates by designing the constraint relationship between grids for use by the three-dimensional reconstruction module. The three-dimensional reconstruction module reconstructs the grid map, and judges the semantics of the grid vertices by the coordinates in the world coordinate system, and uses different colors to express the semantic information. The three-dimensional reconstruction module uses a Gaussian regression process to ensure the speed of reconstruction and achieve real-time operation relying only on the CPU.

[0076] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may certainly make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A real-time semantic 3D reconstruction method based on laser radar, characterized in that: The following steps are involved: Cylindrical segmentation is performed on the original 3D lidar point cloud to obtain multiple cylindrical voxels; Input each cylindrical voxel into the trained asymmetric 3D convolutional network to obtain the semantic label corresponding to each cylindrical voxel; The semantic labels corresponding to the cylindrical voxels are respectively used as the semantic labels corresponding to the point clouds falling into the cylindrical voxels, so as to complete the mapping from the voxel semantic labels to the point cloud semantic labels; Divide the original 3D laser radar point cloud into 3D grids to obtain multiple 3D grids; The grid semantics corresponding to each three-dimensional grid are obtained respectively, and the grid semantics corresponding to any three-dimensional grid is obtained by counting the number of point clouds under different point cloud semantic labels in the three-dimensional grid, and taking the point cloud semantic label with the largest number of point clouds as the grid semantics corresponding to the three-dimensional grid; The original 3D lidar point cloud is reconstructed in three dimensions, and each vertex is colored according to the vertex semantics corresponding to each vertex of the grid map obtained by the three-dimensional reconstruction, so as to complete the three-dimensional reconstruction with semantic information; wherein, the vertex semantics corresponding to any vertex is obtained by taking the three-dimensional grid in which the vertex falls as the wrapping grid, and judging whether there are more than 5 three-dimensional grids in the surrounding three-dimensional grids adjacent to the wrapping grid whose grid semantics are the same as the grid semantics corresponding to the wrapping grid; if so, the grid semantics corresponding to the wrapping grid is the vertex semantics corresponding to the vertex; if not, the vertex semantics corresponding to the vertex is the set default semantics.

2. The real-time semantic 3D reconstruction method based on laser radar according to claim 1, characterized in that: The method of cylindrical segmentation of the original 3D LiDAR point cloud to obtain multiple cylindrical voxels is as follows: The Cartesian coordinates (x, y, z) of each point cloud are converted to cylindrical coordinates (ρ, θ, z), where the radius ρ is defined as the distance from the point cloud to the Cartesian origin on the xy plane, and the azimuth θ is defined as the angle of the point cloud from the x-axis to the y-axis; In the cylindrical coordinate system, the point cloud is non-uniformly segmented according to the three dimensions of radius, azimuth and height to obtain multiple cylindrical voxels; wherein the cylindrical voxel with a larger distance from the origin of the cylindrical coordinate system has a larger size.

3. The real-time semantic 3D reconstruction method based on laser radar according to claim 1, characterized in that: The method for reconstructing the original 3D lidar point cloud to obtain a grid map is: Transform each point cloud into world coordinates using an initial guess of the constant velocity model; Allocate each point cloud to each irregular voxel unit in the world coordinate system; The Gaussian process is used to reconstruct the local surface inside each irregular voxel unit to obtain vertices, and the vertices are connected to form triangular meshes; The original 3D lidar point cloud is aligned with the constructed triangular mesh to obtain a mesh map.

4. The real-time semantic 3D reconstruction method based on laser radar according to claim 1, characterized in that: When performing real-time semantic 3D reconstruction of a park, semantic labels include public facilities, roads, plants, and buildings.

5. The real-time semantic 3D reconstruction method based on laser radar according to claim 1, characterized in that: When performing real-time semantic 3D reconstruction of the vehicle driving environment, the semantic labels include buses, cars, motorcycles, bicycles, plants, and buildings.

6. The real-time semantic 3D reconstruction method based on laser radar according to claim 1, characterized in that: When performing real-time semantic 3D reconstruction of an underground garage, the semantic labels include buses, cars, motorcycles, bicycles, and buildings.

7. A real-time semantic 3D reconstruction method based on laser radar as claimed in any one of claims 1 to 6, characterized in that: When each vertex is colored according to the vertex semantics corresponding to each vertex of the mesh graph obtained by three-dimensional reconstruction, different semantic labels correspond to different colors.

Images