A method for harsh environment perception based on multi-lidar fusion

Abstract

This invention relates to a method for harsh environment perception based on multi-LiDAR fusion, comprising the following steps: S1: Obtaining ground point clouds and non-ground point clouds based on raw point cloud data; S2: Constructing a point cloud map table and performing PCA analysis: Constructing a map table for the ground and non-ground point clouds, simultaneously representing the point clouds as voxels, and then performing PCA analysis on the point clouds within each voxel; S3: Obtaining a fused global map table: Based on the results of the PCA analysis of the voxels, determining whether the grid corresponding to the voxel in the map table belongs to the dust category, and then performing a fusion operation on the analyzed map tables to obtain a fused global map table. This invention can improve the working efficiency of vehicles working on unstructured roads in harsh environments.

Description

Technical Field

[0001] This invention relates to the field of environmental perception for autonomous driving on unstructured roads, and specifically to a method for perceiving adverse environments based on the fusion of multiple lidar sensors. Background Technology

[0002] With the development and evolution of autonomous driving technology, autonomous driving technology supported by vehicle-mounted multi-LiDAR can achieve a high degree of automation in unstructured road scenarios, such as quarries and mining areas. However, unstructured roads are affected by harsh environments, such as dust, which poses a greater challenge to the environmental perception of autonomous driving in unstructured road conditions.

[0003] Existing dust analysis methods include discreteness analysis, multi-sensor fusion, and deep learning-based dust filtering algorithms. Conventional discreteness analysis, taking advantage of the low density of dust, allows for feature extraction in two- or three-dimensional space. However, in unstructured road scenarios, dust noise exhibits high density, risking the failure of conventional discreteness analysis methods. Multi-sensor fusion methods, such as those described in patent CN202010944090.5, fuse millimeter-wave radar and lidar detection results to filter out interference from rain, fog, and dust. When obstacles are inconsistent between sensors, millimeter-wave sensing results are used as output, utilizing the Doppler effect of millimeter waves—their long wavelength and strong penetration. However, multi-sensor fusion requires multiple sensors, placing higher demands on computing power. Deep learning-based methods can be used to represent point clouds as voxels and classify the voxel grids using CNN convolutional neural networks to remove dust. Alternatively, semantic segmentation networks can be used to perform semantic segmentation of dust point clouds, which is also a deep learning-based dust filtering method. However, deep learning-based dust recognition algorithms often face the problems of over-reliance on samples and slow computation speed.

[0004] In summary, unstructured road environments are complex and variable, posing challenges to the stable operation of multi-sensor systems. Furthermore, sensor variations introduce new problems. High dust density in mining areas makes discreteness analysis methods ineffective in describing characteristics. Deep learning methods rely heavily on data samples, resulting in long processing times. Autonomous vehicles typically use multiple LiDAR sensors to achieve full environmental coverage. To address the challenges of autonomous driving applications in unstructured scenarios, a method that can effectively identify dust and ensure the reliability of dust identification is urgently needed. Summary of the Invention

[0005] The technical problem to be solved:

[0006] To address the challenges of autonomous driving in harsh unstructured road environments, and to overcome the complexity of fusion between different sensors, as well as the limitations of effective dust identification due to high dust levels and discreteness in unstructured roads, this invention provides a dust perception method based on multiple lidar sensors. This method utilizes point cloud voxelization representation and PCA data analysis results, employing feature fusion from multiple lidar sensors to improve the working efficiency of vehicles operating on unstructured roads in challenging scenarios.

[0007] The technical solution adopted is as follows:

[0008] This invention utilizes voxel representation of point clouds based on multiple lidar sensors, PCA data penetration characteristic analysis, and map fusion algorithm technology of multiple lidar sensors to achieve effective identification of various harsh environments such as dust, rain, fog, and snow, ensuring the safe operation of unmanned driving in unstructured road environments.

[0009] The present invention provides a method for harsh environment perception based on multi-lidar fusion, which specifically includes the following steps:

[0010] S1: Obtain ground point cloud and non-ground point cloud based on raw point cloud data: Based on the raw point cloud data information of obstacles acquired by each lidar and the installation and configuration information of the lidar, the raw point cloud data information is separated into ground point cloud and non-ground point cloud through a ground filtering algorithm.

[0011] S2: Construct a point cloud map table and perform PCA analysis: Construct a map table for the ground point cloud and non-ground point cloud, and simultaneously perform voxelization characterization on the ground point cloud and non-ground point cloud obtained in step S1; then perform PCA analysis on all point clouds within the voxel.

[0012] S3: Obtain the fused global Map table: Based on the results of the PCA analysis of the voxels, determine whether the grid corresponding to the voxel in the map table is a dust category, and then perform a fusion operation on all the map tables that have undergone PCA analysis to obtain the fused global Map table.

[0013] Furthermore, the harsh environment includes a dusty environment.

[0014] Furthermore, the ground filtering algorithm in step S1 specifically includes:

[0015] S11: The ground filtering algorithm uses a pass-through filtering method to separate the original point cloud acquired by the lidar into ground point cloud and non-ground point cloud based on the installation parameters of each lidar installed at different locations and the ground installation height.

[0016] Furthermore, the construction of the point cloud map table in step S2 specifically includes:

[0017] S21: Construct a map table based on the ground point cloud and non-ground point cloud obtained in S11. The map table is a grid of size 0.3m*0.3m that is uniformly divided on the xy plane parallel to the ground.

[0018] S22: Since the ground filtering algorithm will cause some obstacle point cloud to be missing, it is necessary to perform non-ground point cloud completion operation. The specific operation of non-ground point cloud completion operation is as follows: according to the intersection of the grid information occupied by the ground point cloud and the grid information occupied by the non-ground point cloud in the map table constructed in step S21, the existing ground point and non-ground point grids are completed to add spatial information of obstacles.

[0019] Furthermore, step S2, specifically performing PCA analysis on the voxels, includes:

[0020] S23: Based on the map table after the completion operation in step S22, construct voxels in the vertical direction of the xy plane grid. The size of each voxel is a cube of 0.3m*0.3m*0.3m.

[0021] S24: Based on the grid information occupied by non-ground points in the map table, perform grid clustering. The clustering result is the obstacle represented by grid coordinates. For the coordinates of the obstacle in the map table, perform neighborhood expansion of 2, that is, add the voxel information of the ground point cloud obtained by neighborhood expansion to its corresponding obstacle information.

[0022] S25: Perform PCA analysis on all point clouds within each voxel to identify point clouds suspected of being dust.

[0023] Furthermore, in step S25, the PCA analysis of all n point cloud points of the voxel is performed as follows:

[0024] (1) For each voxel, obtain the representation matrix of n point cloud points inside the voxel, decenter the representation matrix, and solve its covariance matrix.

[0025] (2) Perform singular value decomposition on the covariance matrix. The eigenvalues ​​after decomposition are λ0, λ1, and λ2, where λ0 is the smallest eigenvalue. δ = λ0 / (λ0 + λ1 + λ2) is the discrete characteristic characterization value of the entire voxel. If the voxel exhibits planar characteristics and δ approaches 0, it indicates that the corresponding voxel is an obstacle point cloud. Conversely, if the point cloud points in the voxel exhibit discrete and penetrating characteristics and δ approaches 0.33, it indicates that the corresponding voxel is a suspected dust point cloud.

[0026] Furthermore, step S3 specifically includes the following methods:

[0027] S31: After completing the dust detection and analysis using the PCA analysis method, the voxels are mapped vertically on the map table to obtain the corresponding grids. The grids corresponding to the voxels are classified according to the proportion of obstacle categories and dust point categories of the voxels. They are then marked as dust grids or obstacle grids according to different categories. Then, the obstacle information is projected and dimensionality reduced to project its voxel information onto the map table to facilitate the fusion operation of multiple lidars.

[0028] S32: The multi-LiDAR fusion operation is as follows: For the point cloud map tables of multiple LiDARs, 0 represents that the grid has no point cloud, 1 represents that the grid's point cloud is an obstacle point cloud, and 2 represents that the grid's point cloud is a dust point cloud; when multiple map tables are fused, if the grid category at the same position contains 2, that is, when at least one LiDAR detects dust, the fused global map table will have a value of 2 at that grid position, which is a dust category grid; if the grid at the same position represents 0 or 1, the fused global map table will have a value of 1 at that grid position, which is an obstacle category grid;

[0029] S33: Based on the fusion operation described in step S32, assign the category of all grids to the values ​​corresponding to the dust category grid, obstacle category grid, and point cloud-free grid, i.e., the corresponding values ​​are 2, 1, or 0, and obtain the final fused global Map table to realize dust detection on unstructured roads.

[0030] Furthermore, the harsh environment also includes rain, snow, or fog. When sensing the different environments mentioned above, it is only necessary to replace the treatment of dust with rain, snow, or fog.

[0031] A mining unmanned driving system includes an unmanned vehicle and multiple lidars installed at different locations on the unmanned vehicle. The system uses the aforementioned harsh environment perception method based on multi-lidar fusion to perceive the harsh environment of the unmanned vehicle.

[0032] The beneficial effects of this invention are as follows:

[0033] 1. In this invention, voxels are constructed in the vertical direction of the grid of the point cloud map table. This form fully preserves the positional relationship between point clouds and improves the detection speed of the algorithm. The square voxels provide support for the normalization of PCA feature values, enabling the algorithm to better characterize dust characteristics. At the same time, the map table information is used to expand the neighborhood of obstacles to characterize the instability of the dust neighborhood and improve the detection accuracy of dust.

[0034] 2. One of the key innovations of this invention is the use of PCA data analysis to characterize the penetration and discrete characteristics of dust point clouds. Utilizing voxel representation, PCA data analysis effectively characterizes dust characteristics, providing a reliable guarantee for the identification of unstructured road dust. The PCA analysis method performs feature analysis on clustered point clouds to characterize the chaotic nature of dust. This represents a new form of discreteness representation and offers better identification capabilities for high-concentration dust.

[0035] 3. The harsh environment in mining areas prevents the full performance of lidar from being realized. This invention proposes a multi-feature fusion method. Based on voxel information, the PCA algorithm is used for dust analysis, and voxels are classified. Through vertical mapping, the sensing results from different lidars are stored in a map table. The map tables of different lidars are then fused to achieve reliable dust detection. The fusion algorithm increases the robustness of the algorithm to cope with the changing physical environment under unstructured roads. Attached Figure Description

[0036] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0037] Figure 1 This is an intentional representation of a point cloud projection map.

[0038] Figure 2 This is a schematic diagram of PCA discreteness analysis of point clouds within voxels.

[0039] Figure 3 This is a schematic diagram of the fusion of multiple map tables.

[0040] Figure 4 This is an overall flowchart of a method for perceiving harsh environments based on multi-LiDAR fusion. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] This invention specifically relates to an unmanned mining system, comprising an unmanned vehicle and multiple lidar sensors installed at different locations on the unmanned vehicle. The system employs the aforementioned adverse environment perception method based on multi-lidar fusion to perceive the adverse environment of the unmanned vehicle. The specific adverse environment perception method based on multi-lidar fusion is as follows:

[0043] S1: Input the raw point cloud data, and use the radar installation and configuration information to perform ground filtering of the point cloud to obtain ground points and non-ground point clouds. The specific ground filtering algorithm is as follows:

[0044] S11: Input the raw point cloud. For radars installed at different locations on the vehicle, the ground filtering algorithm uses a pass-through filter to separate ground points from non-ground points based on the radar installation parameters and ground installation height. The identification algorithm used before the fusion operation is consistent for different radars.

[0045] S2: Based on the processing results of step S1, construct a map for ground points and non-ground points. Simultaneously, perform voxelization representation on both the ground and non-ground point clouds, complete non-ground points, expand the neighborhood, and perform PCA analysis. The specific methods are as follows:

[0046] S21: Construct a map table based on the ground point cloud and non-ground point cloud obtained in step S11. The map table is a grid of 0.3m * 0.3m evenly divided on the xy plane parallel to the ground, such as... Figure 1 As shown.

[0047] S22: Since the ground filtering algorithm will cause some obstacle point clouds to be missing, a non-ground point completion operation is performed here. Specifically, based on the grid information occupied by the ground points and the grid information occupied by the non-ground points, the intersection is calculated, and the non-ground point operation is performed on the existing ground point and non-ground point network to add target spatial information.

[0048] S23: After the map table is completed in step S22, voxels are constructed in the vertical direction of the grid. The size of each voxel is a cube of 0.3m * 0.3m * 0.3m, as shown below. Figure 1 As shown.

[0049] Point cloud representation is a prerequisite for the execution of LiDAR methods. Point cloud representation methods are generally divided into three categories: point cloud representation based on raw points, point cloud representation based on voxels, and point cloud representation based on projection. Point-based representation methods have the advantage of rich spatial characteristic information, but the algorithm suffers from high resource consumption. Projection-based methods have low resource consumption and fast computation speed, but the spatial positional relationship of the point cloud is lost. Voxel-based point cloud representation methods combine the advantages of projection and point cloud representation, that is, they preserve spatial information while improving computation speed. They have many applications in the field of deep learning, such as VoxelNet, and point cloud downsampling scenarios.

[0050] S24: Based on the information occupied by non-ground points, grid clustering is performed using a map table. The clustering result yields obstacles represented by grid coordinates. For the map coordinates of the obstacles, a neighborhood expansion of 2 is performed. That is, the voxel information of the ground point cloud obtained by the neighborhood expansion is added to the corresponding obstacle information. This neighborhood expansion aims to utilize the obvious discrete characteristics of the neighborhood of small targets to increase the accuracy of dust detection. After adding neighborhood information, the characteristics of dust are enhanced, which is beneficial to dust filtration.

[0051] S25: Perform PCA analysis on the point cloud within each voxel to identify obstacles suspected of causing dust pollution.

[0052] Principal Component Analysis (PCA) is significant in two main ways: dimensionality reduction and hyperplane fitting. In statistics, PCA is a technique for simplifying datasets. Through linear transformation, it maps data to a new coordinate system. In this new coordinate system, based on the variance of the data projected onto the new axes, the axes are sequentially named the first, second, and third principal components. PCA reduces the dimensionality of the dataset while preserving the characteristic of maximizing variance contribution. Mathematically, PCA is also used for hyperplane fitting. The smallest eigenvalue corresponds to the eigenvector that is the normal vector of the fitted hyperplane, and the magnitude of the eigenvalues ​​can measure the spatial characteristics of the data.

[0053] In this invention, since point clouds are composed of isolated points, PCA is used to estimate the distribution characteristics of point cloud voxels; the analysis steps are as follows:

[0054] PCA analysis of n points on a voxel, such as Figure 2 As shown.

[0055] 1. For any voxel, obtain n point cloud points. Decenter the n discrete points and solve for their covariance matrix, as follows;

[0056] Let the representation matrix of n point cloud data within each voxel be:

[0057]

[0058] Find the mean of n point clouds in each coordinate direction:

[0059]

[0060] The result after decentralization is:

[0061]

[0062] Construct the covariance matrix C:

[0063]

[0064] 2. Perform singular value decomposition on the covariance matrix C, and its eigenvalues ​​are λ0, λ1, and λ2, where λ0 is the smallest eigenvalue; δ = λ0 / (λ0 + λ1 + λ2) is the discrete characteristic characterization value of the entire voxel. If the voxel exhibits planar characteristics and δ approaches 0, it indicates that the corresponding voxel is an obstacle point cloud. Conversely, if the point cloud points within the voxel exhibit discrete and penetrating characteristics and δ approaches 0.33, it indicates that the corresponding point cloud is a suspected dust point cloud.

[0065] S3: Based on the PCA analysis results, determine whether the voxels belong to the dust category, perform vertical mapping on the voxels, construct a map table, perform a fusion operation based on the results of multiple map tables, and update the global map table. The specific method is as follows:

[0066] S31: Because the voxels are cut vertically on the map, such as Figure 1 As shown, after detecting dust using the PCA algorithm, the corresponding meshes are analyzed based on the proportion of obstacle categories and dust categories in the vertical voxels of the map table. Meshes are labeled as obstacle meshes and dust meshes according to their categories. Here, obstacle information is projected and dimensionality reduced from three-dimensional space onto a two-dimensional map table to facilitate the fusion operation of multiple LiDAR systems.

[0067] S32: The multi-LiDAR fusion operation is as follows: Obtain the detection results of LiDAR 1, such as... Figure 3 The left-hand image shows the detection results obtained by LiDAR 2. Figure 3 The middle image shows the result. 0 indicates no point cloud at that location, 1 indicates the point cloud of that grid is an obstacle point cloud, and 2 indicates the point cloud of that grid is a dust point cloud. During map fusion, if grids at the same location contain a category of 2 (meaning at least one lidar detected dust), the final fused map (e.g., ...) will be considered. Figure 3 (As shown in the right-hand diagram) At this grid position, it is 2, which is the dust category grid; if the grid at the same position represents 0 or 1, then the merged global map table will have a grid position of 1, which is the obstacle category grid; as shown in the right-hand diagram. Figure 3 As shown overall.

[0068] S33: Based on the final merged Map table, assign an overall point category value to the point cloud points of the corresponding grid to realize dust detection of unstructured roads.

[0069] Furthermore, the sensing method of the present invention is also suitable for sensing various harsh environments such as rain, fog, and snow. It is only necessary to replace the dust processing with the corresponding parameter recognition for rain, fog, and snow when dealing with different harsh environments.

[0070] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A harsh environment perception method based on multi-lidar fusion, characterized in that, Specifically comprising the following steps: S1: obtaining ground point cloud and non-ground point cloud based on original point cloud data: based on the original point cloud data information of each laser radar obtained obstacle and the installation configuration information of the laser radar, the original point cloud data information is separated into ground point cloud and non-ground point cloud by ground filtering algorithm; S2: constructing point cloud map table and performing PCA analysis: constructing map table for the ground point cloud and non-ground point cloud, and voxelizing the ground point cloud and non-ground point cloud obtained in step S1; then performing PCA analysis operation on all point clouds in the voxel; S3: obtaining fused global map table: based on the results of PCA analysis of the voxel, determining whether the grid corresponding to the voxel in the map table is dust category, then performing fusion operation on all the map tables after PCA analysis to obtain the fused global map table; The harsh environment includes dust environment; The ground filtering algorithm in step S1 specifically comprises: S11: the ground filtering algorithm uses the pass-through filtering method to separate the original point cloud obtained by the laser radar into ground point cloud and non-ground point cloud according to the installation parameters of the laser radar installed at different positions and the ground installation height; The step S2 of constructing point cloud map table specifically comprises: S21: constructing map table according to the ground point cloud and non-ground point cloud obtained in step S11, the map table is a grid with size of 0.3m*0.3m uniformly separated on the xy plane parallel to the ground; S22: since the ground filtering algorithm will cause the loss of part of the obstacle point cloud, non-ground point cloud completion operation is needed, the specific operation of non-ground point cloud completion operation is: according to the intersection of the grid information occupied by the ground point cloud and the grid information occupied by the non-ground point cloud in the map table constructed in step S21, the grid of the ground point and the non-ground point is completed, and the spatial information of the obstacle is increased; The step S2 of performing PCA analysis on the voxel specifically comprises: S23: based on the map table after the completion operation in step S22, constructing voxel in the vertical direction of the xy plane grid, the size of the voxel is a cube with size of 0.3m*0.3m*0.3m; S24: according to the grid information occupied by the non-ground point in the map table, performing grid clustering, and the result of clustering is the obstacle represented by grid coordinates; for the coordinates of the obstacle in the map table, the neighborhood is expanded to 2, that is, the voxel information of the ground point cloud obtained by neighborhood expansion is added to the corresponding obstacle information; S25: performing PCA analysis on all point clouds in each voxel to find out the point cloud suspected of dust.

2. The method of claim 1, wherein, In the step S25, the PCA analysis steps of all n point cloud points of the voxel are as follows: (1) for each voxel, obtaining the representation matrix of n point cloud points in the voxel, decentering the representation matrix, and solving the covariance matrix thereof; (2) Singular value decomposition is performed on the covariance matrix, and the decomposed eigenvalues are λ0, λ1, λ2, wherein λ0 is the minimum eigenvalue; δ = λ0 / (λ0+λ1+λ2) is a discrete characteristic value of the entire voxel, if the voxel presents a plane characteristic δ tends to 0, indicating that the corresponding voxel is an obstacle point cloud, on the contrary, if the point cloud points in the voxel present a discrete and penetrating characteristic, δ tends to 0.33, indicating that the corresponding voxel is a suspected dust point cloud.

3. The method of claim 2, wherein, The step S3 specifically comprises the following method: S31: After the detection and analysis of dust using the PCA analysis method, the voxel is mapped in the vertical direction on the map table to obtain the corresponding grid, the grid corresponding to the voxel is analyzed according to the obstacle category and the proportion of dust point category of the voxel, and the grid is marked as a dust grid or an obstacle grid according to the different categories, then the information of the obstacle is projected and dimensionally reduced, and the voxel information is projected onto the map table, so as to facilitate the fusion operation of multiple laser radars; S32: The multi-laser radar fusion operation specifically comprises: the point cloud map table of multiple laser radars is represented by 0, 1 and 2, wherein 0 represents that the grid has no point cloud, 1 represents that the point cloud of the grid is an obstacle point cloud, and 2 represents that the point cloud of the grid is a dust point cloud; when multiple map tables are fused, if the grid category at the same position contains 2, that is, at least one laser radar detects dust, the global map table after fusion is 2 at the grid position, that is, a dust category grid; if the grid at the same position represents 0 or 1, the global map table after fusion is 1 at the grid position, that is, an obstacle category grid; S33: Based on the fusion operation of step S32, the categories of all grids are assigned to the values corresponding to the dust category grid, the obstacle category grid and the no-point cloud grid, that is, the corresponding values are 2, 1 or 0, and the final fused global map table is obtained, realizing dust detection on an unstructured road.

4. The method of claim 3, wherein, The harsh environment also includes rain, snow or fog environment, when perceiving different environments, only the processing of dust in the environment needs to be replaced by rain, snow or fog.

5. A mine unmanned system comprising an unmanned vehicle, and a plurality of laser radars installed at different positions of the unmanned vehicle, characterized in that, The system adopts the harsh environment perception method based on multi-laser radar fusion of any one of claims 1-4 to perceive the harsh environment of an unmanned vehicle.

Images