A real-time semantic 3D reconstruction method based on lidar

Abstract

The application provides a kind of real-time semantic three-dimensional reconstruction method based on laser radar, according to the characteristics of point cloud data sparsity and uneven density in outdoor and indoor large scene environment, through cylindrical division and asymmetric three-dimensional convolution network, the influence of sparse point cloud characteristics on segmentation result is reduced, and the accuracy of segmentation result is guaranteed;Secondly, the application also proposes a semantic three-dimensional grid map, which is used as a tool to realize the fusion of semantic information and three-dimensional reconstructed grid map;Finally, to ensure the fusion accuracy, the application also proposes a constraint relationship between grids, by establishing the relationship between grids, the influence of semantic segmentation error is reduced, to achieve better effect.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of robot positioning and reconstruction, and particularly relates to a real-time semantic three-dimensional reconstruction method based on a laser radar. BACKGROUND

[0002] The process of point cloud mesh reconstruction generally includes data preprocessing, feature extraction, mesh generation and post-processing steps. Among them, feature extraction is a key step, through which the geometric features (such as normal vector, curvature, etc.) of the point cloud can be extracted to better reflect the shape information of the object. The mesh generation algorithm connects the point cloud data into triangular facets according to the extracted features to form a continuous surface. For three-dimensional reconstruction technology, there are the following existing solutions: Poisson-based method, Delaunay tetrahedralization (i.e. 3D Delaunay triangulation) and graph cut method.

[0003] The main task of point cloud semantic segmentation is to output a predefined label value for each 3D point according to the input three-dimensional point cloud, which helps to obtain a rich understanding of the scene. Real-time fine-grained semantic prediction can accelerate the realization of full autonomy. For point cloud semantic segmentation technology, there are the following existing solutions:

[0004] Solution 1: The literature (Qi C R, Su H, Mo K, et al. PointNet: Deep Learning on PointSets for 3D Classification and Segmentation: 30th IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR) [C]. New York: IEEE, 2017.) proposes a PointNet algorithm. The contribution of the PointNet algorithm is that it directly extracts features and classifies unordered point cloud data, breaking through the limitation of traditional deep learning methods that require regular data formats. However, the PointNet algorithm has limited local feature extraction capability and is sensitive to point cloud density, both of which can lead to performance and accuracy degradation when segmenting point cloud data.

[0005] Scheme 2: The Point Transformer algorithm is proposed in the literature (Zhao H, Jiang L, Jia J, et al. Point Transformer: 17th IEEE / CVF International Conference on Computer Vision (ICCV) [C]. Piscataway: IEEE, 2021.). The contribution of the Point Transformer algorithm is to introduce the self-attention mechanism into the point cloud processing, effectively capturing the long-range dependency relationship between different points in the point cloud. By replacing the traditional MLP with a self-attention layer, the model can better learn the global and local features of the point cloud. However, the Point Transformer algorithm is sensitive to hyperparameters, and the computational complexity is also high. SUMMARY

[0006] To solve the problem that the existing grid map lacks scene semantic information, the present application provides a real-time semantic three-dimensional reconstruction method based on a laser radar, which uses a laser radar as a perception device to construct a real-time three-dimensional reconstruction grid map with semantic information.

[0007] A real-time semantic three-dimensional reconstruction method based on a laser radar, comprising the following steps:

[0008] The original three-dimensional laser radar point cloud is cylindrically segmented to obtain a plurality of cylindrical voxels;

[0009] Each cylindrical voxel is input into a trained asymmetric three-dimensional convolutional network to obtain the corresponding semantic label of each cylindrical voxel;

[0010] Each cylindrical voxel corresponding semantic label is used as the corresponding semantic label of the point cloud falling into each cylindrical voxel to complete the mapping of voxel semantic labels to point cloud semantic labels;

[0011] The original three-dimensional laser radar point cloud is three-dimensionally gridded to obtain a plurality of three-dimensional grids;

[0012] The corresponding grid semantics of each three-dimensional grid are obtained, and the method for obtaining the grid semantics corresponding to any three-dimensional grid is as follows: the number of point clouds under different point cloud semantic labels in the three-dimensional grid is counted, and the point cloud semantic label with the largest number of point clouds is taken as the grid semantics corresponding to the three-dimensional grid;

[0013] The original three-dimensional laser radar point cloud is reconstructed in three dimensions, and each vertex of the mesh graph obtained by the three-dimensional reconstruction is colored according to the vertex semantics corresponding to the vertex, so as to complete the three-dimensional reconstruction with semantic information; wherein, the method for obtaining the vertex semantics corresponding to any vertex is: taking the three-dimensional grid in which the vertex falls as a wrapped grid, and judging whether there are more than 5 three-dimensional grids in the surrounding three-dimensional grids adjacent to the wrapped grid whose grid semantics are the same as the grid semantics corresponding to the wrapped grid, if yes, the grid semantics corresponding to the wrapped grid is the vertex semantics corresponding to the vertex, if no, the vertex semantics corresponding to the vertex is the default semantics set.

[0014] Further, the method for performing cylindrical segmentation on the original three-dimensional laser radar point cloud to obtain a plurality of cylindrical voxels is as follows:

[0015] Each point cloud Cartesian coordinate (x, y, z) is converted into cylindrical coordinates (p, q, z), wherein the radius p is defined as the distance of the point cloud in the x-y plane to the Cartesian coordinate origin, and the azimuth angle q is defined as the angle of the point cloud from the x axis to the y axis.

[0016] In the cylindrical coordinate system, the point cloud is non-uniformly segmented according to the three dimensions of radius, azimuth angle and height to obtain a plurality of cylindrical voxels; wherein, the larger the distance from the cylindrical coordinate system origin, the larger the size of the cylindrical voxel.

[0017] Further, the method for performing three-dimensional reconstruction on the original three-dimensional laser radar point cloud to obtain a mesh graph is as follows:

[0018] Each point cloud is converted into a world coordinate system using an initial guess of a constant speed model;

[0019] Each point cloud is respectively distributed into each irregular voxel unit in the world coordinate system;

[0020] A local surface inside each irregular voxel unit is reconstructed using a Gaussian process to obtain vertices, and the vertices are connected to form each triangular mesh;

[0021] The original three-dimensional laser radar point cloud is aligned with the constructed triangular mesh to obtain a mesh graph.

[0022] Further, when performing real-time semantic three-dimensional reconstruction on a park, the semantic labels include public facilities, roads, plants and buildings.

[0023] Further, when performing real-time semantic three-dimensional reconstruction on a vehicle driving environment, the semantic labels include buses, cars, motorcycles, bicycles, plants and buildings.

[0024] Further, when the underground garage is reconstructed in real time, the semantic labels include buses, cars, motorcycles, bicycles and buildings.

[0025] Further, when the vertices of the grid map obtained according to the three-dimensional reconstruction are colored according to the vertex semantics corresponding to each vertex, the colors corresponding to different semantic labels are different.

[0026] Beneficial effects:

[0027] 1. The application provides a real-time semantic three-dimensional reconstruction method based on a laser radar, according to the characteristics of sparse and uneven density of point cloud data in outdoor and indoor large scene environments, the influence of sparse point cloud characteristics on the segmentation result is reduced through cylindrical division and asymmetric three-dimensional convolution network, and the accuracy of the segmentation result is ensured; secondly, the application also provides a semantic three-dimensional grid map, which is used as a tool to realize the fusion of semantic information and the grid map reconstructed by three-dimensional reconstruction; finally, in order to ensure the fusion accuracy, the application also provides a constraint relationship between grids, which reduces the influence of semantic segmentation error by establishing the relationship between grids, so as to achieve better results.

[0028] 2. The application provides a real-time semantic three-dimensional reconstruction method based on a laser radar, which accelerates point cloud processing through a Gaussian regression process, realizes real-time three-dimensional reconstruction only relying on CPU, and makes positioning and mapping benefit each other, solving the problem of generally poor real-time performance of the grid map. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 The overall design idea of the application.

[0030] Figure 2 The flowchart of the semantic segmentation module of the application.

[0031] Figure 3 The result output by the semantic segmentation module of the application.

[0032] Figure 4 The flowchart of the three-dimensional reconstruction module of the application.

[0033] Figure 5 The construction process of the semantic grid map of the application.

[0034] Figure 6 The grid map with semantic information reconstructed by the application.

[0035] Figure 7 The asymmetric residual module provided by the application is shown in the schematic diagram.

[0036] Figure 8 The down-sampling module provided by the application is shown in the schematic diagram.

[0037] Figure 9 An upsampling module provided by the present application.

[0038] Figure 10 A context modeling module provided by the present application. DETAILED DESCRIPTION

[0039] In order to enable persons skilled in the art to better understand the scheme of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application.

[0040] At present, most laser SLAM systems directly use point clouds to construct a map, but when the point cloud map is enlarged, it is very sparse. With the improvement of the complexity of the robot task, such a map cannot meet the needs of robot navigation, positioning and obstacle avoidance, and a more dense high-precision map is needed. Due to the lower memory cost, grid-based three-dimensional reconstruction has become a competitive dense model mapping method. However, the existing grid map only covers the geometric information of the modeled scene, does not cover the semantic information in the scene, and the real-time performance is poor, which greatly limits the ability of the robot to perform advanced complex tasks. Therefore, the present application proposes a real-time semantic three-dimensional reconstruction method based on a laser radar to solve the above problems.

[0041] A real-time semantic three-dimensional reconstruction method based on a laser radar, comprising the following steps:

[0042] S1: performing cylindrical segmentation on the original three-dimensional laser radar point cloud to obtain a plurality of cylindrical voxels;

[0043] S2: respectively inputting each cylindrical voxel into a trained asymmetric three-dimensional convolution network to obtain a semantic label corresponding to each cylindrical voxel;

[0044] For example, when performing real-time semantic three-dimensional reconstruction on a park, the semantic labels include public facilities, roads, plants, buildings, etc.; when performing real-time semantic three-dimensional reconstruction on a vehicle driving environment, the semantic labels include buses, cars, motorcycles, bicycles, plants, buildings, etc.; when performing real-time semantic three-dimensional reconstruction on an underground garage, the semantic labels include buses, cars, motorcycles, bicycles, buildings, etc.

[0045] S3: respectively taking the semantic labels corresponding to each cylindrical voxel as the semantic labels corresponding to the point cloud falling into each cylindrical voxel, to complete the mapping of voxel semantic labels to point cloud semantic labels;

[0046] S4: performing three-dimensional grid division on the original three-dimensional laser radar point cloud to obtain a plurality of three-dimensional grids;

[0047] S5: respectively acquire the grid semantics corresponding to each three-dimensional grid, and the acquisition method of the grid semantics corresponding to any one three-dimensional grid is: counting the number of point clouds under different point cloud semantic labels in the three-dimensional grid, and taking the point cloud semantic label with the largest number of point clouds as the grid semantics corresponding to the three-dimensional grid;

[0048] S6: three-dimensional reconstruction is performed on the original three-dimensional laser radar point cloud, and each vertex of the mesh graph obtained by three-dimensional reconstruction is colored according to the vertex semantics corresponding to the vertex, so that three-dimensional reconstruction with semantic information is completed, and the colors corresponding to different semantic labels are different; wherein the acquisition method of the vertex semantics corresponding to any one vertex is: taking the three-dimensional grid in which the vertex falls as a wrapped grid, and judging whether there are more than 5 three-dimensional grids in the surrounding three-dimensional grids adjacent to the wrapped grid whose grid semantics are the same as the grid semantics corresponding to the wrapped grid, if yes, the grid semantics corresponding to the wrapped grid is the vertex semantics corresponding to the vertex, if no, the vertex semantics corresponding to the vertex is the default semantic set.

[0049] It can be seen that the present application adopts modular design and is divided into three modules of semantic segmentation, semantic three-dimensional grid map and three-dimensional reconstruction, each module can be operated independently, and solves the problems of poor real-time performance and lack of semantic information of the existing grid map. By inputting the laser radar point cloud, the laser radar point cloud can be segmented by the semantic segmentation network, and according to the semantic information obtained by segmentation, the visual point cloud with object-level semantic information as shown in Figure 3 is output. The result of semantic segmentation is input into the grid map module to obtain a semantic three-dimensional grid map. The semantic three-dimensional grid map is combined with the mesh graph obtained by the three-dimensional reconstruction module to realize the fusion of object-level semantic information and the grid map, and a semantic grid map as shown in Figure 6 is generated in real time.

[0050] It should be noted that steps S1-S3 of the present application are actually for semantic segmentation, and the specific implementation of semantic segmentation will be described in detail below.

[0051] In order to solve the two difficult problems of sparsity and density variation of outdoor radar point cloud, the present application first adopts cylindrical segmentation to generate more balanced point distribution (to make it more robust to density variation), and then adopts an asymmetric three-dimensional convolution network to enhance horizontal and vertical weights, so as to well match the target point distribution and enhance the robustness to sparse point cloud.

[0052] As shown in Figure 2As shown, the framework consists of two parts: cylindrical partitioning and asymmetric 3D convolutional network. Firstly, the laser radar point cloud is partitioned cylindrically, and the features extracted by the multi-layer perceptron (MLP) are reconfigured based on this. Then, an asymmetric 3D convolutional network is applied to generate a voxelized result. Finally, by introducing a point-wise optimization module, the error caused by the label encoding of the voxel units can be effectively reduced, thereby improving the quality of the final output. This series of steps aims to improve the accuracy and efficiency of point cloud data processing.

[0053] For the above process, the present application has the following design features:

[0054] 1. Cylindrical partitioning is adopted. In outdoor environments, the point cloud scanned by the laser radar usually exhibits uneven density characteristics, i.e., the density of the point cloud near the scanner is much higher than that far away. If regular cells are used to partition these point clouds, it will lead to the problem of uneven distribution of the proportion of empty cells. In contrast, the cylindrical coordinate system covers the far area by gradually expanding the grid size, which can more evenly distribute the point cloud and provide a more balanced representation for areas of different densities. In addition, unlike projection-based methods that project point clouds onto a two-dimensional plane, cylindrical partitioning preserves the three-dimensional grid representation, thereby better maintaining the geometric structure of the point cloud.

[0055] The specific workflow is as follows: First, convert the Cartesian coordinates (x, y, z) of each point to cylindrical coordinates (ρ, θ, z), where the radius ρ is defined as the distance of the point in the x-y plane to the origin, and the azimuth angle θ is defined as the angle of the point from the x-axis to the y-axis. In the cylindrical coordinate system, partitioning is performed according to the three dimensions of radius, azimuth angle, and height. In this partitioning strategy, the farther the area is from the origin, the larger the corresponding cell size. The point features calculated using the multi-layer perceptron (MLP) are re-distributed according to the previous results to form a cylindrical-shaped feature set. Finally, the cylinder is unwrapped from 0 degrees to obtain a three-dimensional cylindrical representation R ∈ H × W × L, where H represents the radius, W represents the azimuth angle, and L represents the height. The subsequent asymmetric 3D convolutional network is also based on this representation for processing.

[0056] 2. Asymmetric 3D convolution network is adopted. Point cloud data in driving environment usually contains objects with specific shapes, such as cars, buses and motorcycles, etc., which can often be represented by cubes. Based on this observation, the invention improves the effect of standard 3D convolution network in processing such data by modeling these objects as cubes. In addition, the center cross weight is very important in the square convolution kernel. For this specific distribution of point cloud data, the invention enhances the response ability in horizontal and vertical directions by using asymmetric residual blocks to better match the point distribution of the target object. Based on the asymmetric residual block, asymmetric down-sampling and up-sampling blocks are further developed to perform the down-sampling and up-sampling processes.

[0057] wherein the asymmetric residual block is as shown in Figure 7 First, a convolution layer with a kernel size of 3x1x3 is used, followed by a 1x3x3 convolution layer, which has the same receptive field as a 3x3x3 three-dimensional convolution. However, the computational cost of this method is reduced by 33% compared to a 3x3x3 convolution with the same number of output filters. This method of splitting a 3x3x3 convolution into 1x3x3 and 3x1x3 has two advantages: first, since the target to be detected is close to a cuboid, this form of convolution is more conducive to extracting cuboid-style features; second, the method is similar to deep convolution, which can greatly reduce the amount of calculation and improve efficiency.

[0058] In the proposed architecture, the asymmetric residual block (A) is the core of the down-sampling (D) and up-sampling (U) modules. As shown in Figure 8 and Figure 9 The down-sampling module (D) is composed of an asymmetric residual block and a three-dimensional convolution with a step size of 2 to achieve data down-sampling processing. The up-sampling module (U) integrates low-level features and optimizes the features through the asymmetric residual block (A). This design effectively improves the expression ability of the features and the performance of the network.

[0059] At the same time, the dimension decomposition-based context modeling (DDCM) technology is introduced to improve the ability to handle high-order global context in the decomposition-aggregation strategy. The combination of these technologies provides strong support for accurate processing of point cloud data. Specifically, as shown in Figure 10As shown, DDCM often needs a high-rank tensor to effectively encode the contextual differences between the point clouds in 3D space when processing. This is because the distribution and relationship of point cloud data in space are complex and varied, and a single low-rank representation often cannot capture all the contextual information. However, due to its high-rank characteristics, constructing and calculating such a high-rank tensor is expensive in 3D space. Inspired by the high-rank matrix decomposition theory, the complex high-rank context can be decomposed into a simpler low-rank representation. In the point cloud segmentation task, this means that the high-rank context can be decomposed along the three main dimensions: height, width, and depth. The representation on each dimension is low-rank, but they collectively constitute a complete high-rank context. This decomposition-aggregation strategy handles the complexity of high-rank from different dimensions, while simplifying the calculation based on low-rank constraints. As shown in the following figure, by using three rank-1 kernels, low-rank encoding can be generated on each dimension. Then, the convolution results are modulated by the Sigmoid function to generate weights for each dimension, which can extract co-occurrence context information based on different dimension rank-1 tensors. Finally, by aggregating the low-rank activations on all three dimensions, a total sum of features that can represent the complete context is obtained.

[0060] 3. Adopting a point-wise optimization module. The voxelization method is indeed very effective in processing large-scale point cloud data, but it also has the problem of grouping different categories of points into the same voxel unit, which may cause information loss. To solve this problem, the invention can introduce a point-wise refinement module to reduce the interference caused by inaccurate voxel label encoding.

[0061] This process includes the following steps: first, according to the inverse point-voxel mapping table, project the cylindrical voxel features back to the original point cloud. This means that all points within the same voxel unit will be assigned the same feature. Then, the number of point features before and after processing by the three-dimensional convolutional network is fused, and the fused features are input into the point-wise refinement module to refine the final output result (i.e., the corresponding class label for all point clouds).

[0062] In this way, each point can obtain a more detailed and accurate label, thereby improving the overall segmentation quality and mIoU. This point-wise refinement method helps to preserve more detailed information, especially in the case of multiple categories within a voxel unit. Such a strategy can maintain the advantages of voxelization methods in exploring large-scale point clouds while reducing information loss and improving segmentation accuracy.

[0063] Further, in order to fuse the semantic information with the three-dimensional reconstructed grid map, the application also proposes a semantic three-dimensional grid map obtained by steps S4-S5, and the specific implementation manner is as follows: firstly, all point cloud data of the current frame is read and saved, and the updated transformation matrix of each frame is used to convert the point cloud data to the world coordinate system, so as to perform subsequent processing. Then, a folder containing semantic labels is read, and each txt file in the folder stores the semantic labels of all 3D points of the corresponding frame, so as to prepare for subsequent grid semanticization. A function is defined in the Octree class, which receives the point cloud set in the world coordinate system and the semantic label storage path, reads the semantic information from the file. Then, all points in the point cloud are traversed, and the occupation information and semantic information of the points in the octree are updated. Finally, the semantic information of the points in each grid is classified into target class and non-target class, and is stored respectively. A majority voting mechanism is adopted, and the semantic label with the largest number of points in the grid is the semantic label of the grid.

[0064] Further, the grid is a three-dimensional entity surface composed of faces, edges and vertices. In the field of three-dimensional modeling, triangular facets as shown in the drawings are the main representation form because they are simple and can approximate most complex three-dimensional structures. The application improves the existing method by processing point cloud data by using a Gaussian process (GP), so that the system can run in real time only by using a CPU, and even when a sensor with denser point cloud such as a 64-line laser radar is used, real-time updating of the grid map can be realized. Figure 6

[0065] The overall flowchart of the three-dimensional reconstruction module is shown in Figure 4 . It mainly consists of three parts: grid division, registration and grid management. Firstly, the initial guess of the constant speed model is used to convert each new radar scan to the world coordinate system. Subsequent operations are performed in the world coordinate system. Then, the points are distributed to the voxel units. The GP reconstructs the local surface inside each unit to obtain vertices, and the vertices are connected to form a grid. In the registration component, the registration of the points to the grid is designed, the reconstructed current scan image is aligned with the constructed grid map, and finally the grid map is iteratively updated.

[0066] ​Building and updating the mesh is very time-consuming. To address this issue, this invention employs a reconstruction and connection strategy to facilitate subsequent steps, enabling the entire system to operate in real time. GP recovers local surfaces from noise and sparse point clouds within voxels; these vertices are interpolated results of surfaces. Two coordinates of a 3D vertex are uniformly distributed (called locations), and the other coordinate (called prediction) has a continuous value domain. These locations are used as indexes to enable fast lookups in constant time. The prediction's value domain is continuous to avoid accuracy loss caused by discretization. This invention uses a spatial Gaussian process to reconstruct local surfaces from noisy distance observations. To obtain more feasible data associations and map updates, discrete samples are extracted from the recovered surface. This process is called regionalized Gaussian process map building. In other words, it can be viewed as a kind of surface interpolation. This process includes two parts: regionalization and reconstruction.

[0067] First, to establish different functional relationships locally, the entire domain is divided into several uniformly distributed cubic units in the world coordinate system {W}. This decomposition also accelerates the reconstruction process. Each cell has a side length of 'a'. (Original point) The subset located in the k-th cell is denoted as Next, the functional relationship between the coordinates needs to be determined: x = f(y,z), or y = f(x,z), or z = f(x,y). Considering that a function can only represent a 2.5-dimensional surface, for complex 3D structures, it is generally assumed that there are three functions in a cell. Each function provides a corresponding constraint along its direction. If the surface in a cell is perpendicular to a coordinate plane, since it only provides constraints along its normal, the corresponding function in that cell can be omitted. This situation is determined based on Principal Component Analysis (PCA). For example, when extracting a set of raw data from a vertical wall, since the wall cannot provide vertical constraints, the function in the z-direction is ignored.

[0068] After regionalization, GP map reconstruction is performed in each non-empty cell. The lidar measures environmental noise samples. Its noise model can be derived from manufacturing data. This invention simply assumes that each lidar point follows an isotropic variance σ. 2 The independent normal distribution. In this case, using GP regression can produce the best linear unbiased prediction.

[0069] The GP regression problem is as follows: given n k Training points D = {(f i ,l i ), i = 1, ..., n k Training position With the observed value f i The relationship between ∈R is represented by f i =f(li )+ ε i i = 1,..., n k where ε i is a noise term following a distribution. The goal is to achieve n test predictions f * at pre-set test locations denoted by f *j = f(l *j )+ ε *j j = 1,..., n test . Define and the predictive distribution f * given f

[0070]

[0071] where the average is taken over f * at the test locations l * . Its variance is estimated by Here, k ** = k(l * , l * ), k l* = k(l, l * ) T , and K ll is an n k x n k matrix, K ll (i, j) = k(i, j), k(.,.) denotes the kernel function. The commonly used exponential covariance function k(l i , l j ) = exp(-κ | l i - l j |) is chosen, which has a pre-set length scale parameter κ.

[0072] In this case, the training points are the original points in the cells as observed coordinates are called directions, and the other two coordinates are as training locations. Each test point is set uniformly with interval r. Since the side length of the cell is a, set a as the integral multiple of r, which means n test = (a / r) 2 . These predictions with variance are samples drawn from the implicit surface, and each set of samples is named as a level. Those predictions far away from the original data are less reliable. Then these samples are used as the reconstruction results. After reconstruction, there are 0-3 levels in a cell. The cells are stored in a hash table data structure. A sample is denoted by p i = (fi , i ) represents. Wherein the and test coordinates as an index. This can be directly queried samples.

[0073] On the basis of regional decomposition, the training process of GP is further accelerated by the concept of local regression. The core idea is to predict the influence of the observation data that is closer to the predicted test position. Therefore, the training process can be accelerated by principle down-sampling of training points without causing too much accuracy loss. The specific approach is: select to retain the original data, but only use all the nearest points of each test position in each GP map reconstruction. The number of filtered training points will be significantly reduced in this way. One way to complete this filtering process is to use Kd-tree. However, in the case of n inputs, the initialization cost of this data structure is O(nlogn), and the average search cost is O(logn). Although Kd-tree is faster than brute force search, it is still time-consuming, especially when the number of points is large. Since the search target in this design application is uniformly distributed, an improved two-dimensional voxel filter is used to approximate this process. In a unit, the training positions are distributed in a two-dimensional domain, and this two-dimensional domain is divided into smaller grids, and the center of the grid is the test position. The original voxel filter calculates the average value of all original data in each small grid. The modified method is to retain the point if it is the closest to the test position among all points in the same smaller grid. Therefore, the filtering process can be completed with linear complexity cost.

[0074] Based on this, the fusion of semantic information provided in step S6 and the grid map process needs to judge the grid vertex color as shown in Figure 6 If only the grid semantics of the vertex is used to color the vertex, the accuracy will completely depend on the semantic segmentation accuracy. When the accuracy is not ideal, the fusion effect is also not ideal. Therefore, the invention designs a grid constraint method. Taking the judgment of semantic as a car as an example: when determining the semantics of each vertex, judge the semantics of the surrounding grid. If the number of surrounding grid semantics as a car is more than 5, it is considered that the semantics of this vertex is a car, and then coloring is performed according to the pre-defined color relationship. This method uses the constraint relationship between grids to effectively alleviate the influence of semantic segmentation errors and improves the fusion quality.

[0075] Finally, the proposed method is used to test the complex environment, and the results are as shown in Figure 6As shown. It can be seen that the real-time semantic three-dimensional reconstruction method based on laser radar of the present application achieves considerable effect. In general, the point cloud semantic segmentation module provides semantic information for the subsequent reconstruction module and three-dimensional grid module, and can independently run to output point cloud graph with semantic information. The three-dimensional grid serves as a means of concatenating semantic information and three-dimensional reconstruction grid graph, and gives semantic information from the spatial coordinate angle by designing the constraint relationship between grids, so as to be used by the three-dimensional reconstruction module. The three-dimensional reconstruction module reconstructs the grid graph, and judges the semantics of the grid vertex through the coordinates in the world coordinate system, and uses different colors to express semantic information. The three-dimensional reconstruction module uses the Gaussian regression process to ensure the speed of reconstruction, and realizes real-time operation only relying on CPU.

[0076] Of course, the present application can have other various embodiments, and those skilled in the art can certainly make various corresponding changes and modifications according to the present application without departing from the spirit and essence of the present application. However, these corresponding changes and modifications should all belong to the protection scope of the claims attached to the present application.

Claims

1. A real-time semantic 3D reconstruction method based on lidar, characterized in that, The method comprises the following steps: cylindrical segmentation is performed on the original three-dimensional laser radar point cloud to obtain a plurality of cylindrical voxels; each cylindrical voxel is input into a trained asymmetric three-dimensional convolution network to obtain a corresponding semantic label of each cylindrical voxel; the semantic label corresponding to each cylindrical voxel is taken as the semantic label of the point cloud falling into each cylindrical voxel, so as to complete the mapping of the voxel semantic label to the point cloud semantic label; three-dimensional grid division is performed on the original three-dimensional laser radar point cloud to obtain a plurality of three-dimensional grids; the corresponding grid semantics of each three-dimensional grid are obtained, and the method for obtaining the grid semantics of any three-dimensional grid is as follows: the number of point clouds under different point cloud semantic labels in the three-dimensional grid is counted, and the point cloud semantic label with the largest number of point clouds is taken as the grid semantics corresponding to the three-dimensional grid; three-dimensional reconstruction is performed on the original three-dimensional laser radar point cloud, and each vertex of the mesh graph obtained by the three-dimensional reconstruction is colored according to the vertex semantics corresponding to each vertex, so as to complete the three-dimensional reconstruction with semantic information; wherein the method for obtaining the vertex semantics corresponding to any vertex is as follows: the three-dimensional grid in which the vertex falls is taken as a wrapped grid, and it is judged whether there are more than five three-dimensional grids in the surrounding three-dimensional grids adjacent to the wrapped grid whose grid semantics are the same as that of the wrapped grid; if yes, the grid semantics of the wrapped grid is the vertex semantics corresponding to the vertex; if no, the vertex semantics corresponding to the vertex is a default semantic set in advance; the method for performing cylindrical segmentation on the original three-dimensional laser radar point cloud to obtain a plurality of cylindrical voxels is as follows: Cartesian coordinates (x, y, z) of each point cloud are converted to cylindrical coordinates where the radius is defined as the distance of the point cloud from the origin of the Cartesian coordinates in the x-y plane, the azimuth is defined as the angle of the point cloud from the x-axis to the y-axis; in the cylindrical coordinate system, the point cloud is non-uniformly segmented according to three dimensions of radius, azimuth and height to obtain a plurality of cylindrical voxels; wherein the size of the cylindrical voxel is larger as the distance from the origin of the cylindrical coordinate system is larger; the method for performing three-dimensional reconstruction on the original three-dimensional laser radar point cloud to obtain a mesh graph is as follows: each point cloud is converted to the world coordinate system using an initial guess of a constant speed model; each point cloud is distributed to each irregular voxel unit in the world coordinate system; the local surface inside each irregular voxel unit is reconstructed by using a Gaussian process to obtain vertices, and the vertices are connected to form triangular meshes; the original three-dimensional laser radar point cloud is aligned with the constructed triangular meshes to obtain a mesh graph.

2. A real-time semantic 3D reconstruction method based on lidar as claimed in claim 1, wherein, When performing real-time semantic three-dimensional reconstruction on a park, the semantic labels include public facilities, roads, plants and buildings.

3. A real-time semantic 3D reconstruction method based on lidar as claimed in claim 1, wherein, When performing real-time semantic three-dimensional reconstruction on a vehicle driving environment, the semantic labels include buses, cars, motorcycles, bicycles, plants and buildings.

4. A real-time semantic 3D reconstruction method based on lidar as claimed in claim 1, wherein, When performing real-time semantic three-dimensional reconstruction on an underground garage, the semantic labels include buses, cars, motorcycles, bicycles and buildings.

5. A real-time semantic 3D reconstruction method based on laser radar according to any one of claims 1 to 4, characterized in that, When coloring each vertex according to the vertex semantics corresponding to each vertex of the mesh graph obtained by the three-dimensional reconstruction, the colors corresponding to different semantic labels are different.

Images