Unstructured road condition sensing method and device

Abstract

The invention discloses an unstructured road condition sensing method, and relates to the field of automatic driving and intelligent traffic systems, and the method comprises the steps: determining a ground point cloud and an obstacle point cloud according to the original point cloud data of an in-vehicle laser radar; determining singular points in the ground point cloud by using a ground singular point detection algorithm; performing point cloud pose matching according to the ground point cloud, the obstacle point cloud and the singular point, and determining a point cloud matching relationship; the point cloud matching relationship comprises a rotation matrix and a translation matrix; augmenting the sparse point cloud according to the matching relationship; according to the augmented point cloud, determining a two-dimensional grid map, and according to the two-dimensional grid map, determining a vehicle body surrounding grid map under a bird's-eye view angle; based on a deep neural network, obstacle and road surface drivable area detection is carried out on a target area in the grid map around the vehicle body, and different road condition driving areas are determined; the road condition driving area comprises a rigid obstacle area, a flexible obstacle area and a drivable area, so that the boundary of the unstructured road is clearer.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving and intelligent transportation systems, and in particular to a method for perceiving unstructured road conditions. Background Art

[0002] In order to improve the universality and robustness of autonomous driving technology, in addition to studying autonomous driving technology on well-paved structured road scenarios, it is also necessary to study autonomous driving technology on unstructured roads.

[0003] During the process of autonomous driving, autonomous vehicles need to perceive the surrounding environment, including obstacles and road boundaries. However, unstructured roads often have a large number of potholes and obstacles, some of which are even difficult to detect. The boundaries of unstructured roads are not clear enough, resulting in great uncertainty and safety hazards in the driving of autonomous vehicles. Summary of the invention

[0004] The purpose of this application is to provide a method for perceiving unstructured road conditions, so as to achieve clarity of unstructured road boundaries, so as to solve the problem that the boundaries of unstructured roads cannot be clearly identified, resulting in great uncertainty and safety hazards in the driving of autonomous driving vehicles.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides an unstructured road environment perception method, comprising:

[0007] The original point cloud data of the lidar is input, and the point cloud data is filtered according to the preset height threshold to obtain the ground point cloud and obstacle point cloud.

[0008] The computer's ground singular point detection algorithm is used to calculate the angle θ between each point of each Ring and the origin of the laser radar coordinate system relative to the positive direction of the vehicle head; the computer's angle singular point detection algorithm is used to calculate the maximum height difference Δz and the average height within each angle range; the horizontal horizontal distance and the maximum horizontal distance of all points within the current angle range are calculated; based on the values ​​of Δz, average height, average horizontal distance and maximum horizontal distance, it is determined whether there is a singular point in the angle θ of the Ring.

[0009] Point cloud pose matching is performed based on a computer using the original point cloud containing obstacle points, singular points and ground points.

[0010] According to Mask R-cnn (Mask Region-based Convolutional Neural Network), the target area is divided into rigid obstacle area, flexible obstacle area and drivable area according to its passability.

[0011] In a second aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-described steps of perceiving an unstructured road environment.

[0012] According to the specific embodiments provided in this application, this application has the following technical effects:

[0013] The present application provides a method for perceiving unstructured road conditions, which determines ground point clouds and obstacle point clouds based on original point cloud data of an in-vehicle laser radar; determines singular points in the ground point cloud using a ground singular point detection algorithm; performs point cloud pose matching based on the ground point cloud, obstacle point cloud and singular points to determine a point cloud matching relationship; the point cloud matching relationship includes a rotation matrix and a translation matrix; augments the sparse point cloud based on the matching relationship; determines a two-dimensional grid map based on the augmented point cloud, and determines a grid map around the vehicle body from a bird's-eye view based on the two-dimensional grid map; detects obstacles and drivable road areas in target areas in the grid map around the vehicle body based on a deep neural network to determine driving areas with different road conditions; the driving area with road conditions includes a rigid obstacle area, a flexible obstacle area and a drivable area, and different types of areas are obtained by dividing the road conditions, so that the boundaries of unstructured roads are clearer, so that different driving decisions can be made more accurately according to different areas, thereby improving the certainty and safety of identifying drivable areas on unstructured roads. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0015] Figure 1 A schematic diagram of a process flow of an embodiment of the present application;

[0016] Figure 2This is a schematic diagram of the comparison of point clouds of a flat road surface and a road surface with singular points in the embodiment of the present application; wherein: Figure 2 (a) is a schematic diagram of the flat road surface point cloud. Figure 2 (b) is a schematic diagram of the road surface point cloud with singular points;

[0017] Figure 3 A schematic diagram of point cloud augmentation provided in an embodiment of the present application;

[0018] Figure 4 (a) is a schematic diagram of the sparse point cloud before augmentation. Figure 4 (b) is a schematic diagram of the augmented point cloud comparison;

[0019] Figure 5 A schematic diagram of the point cloud dimensionality reduction effect provided in an embodiment of the present application;

[0020] Figure 6 (a) is a schematic diagram of the effect of mask on the raster map after dimensionality reduction. Figure 6 (b) is a schematic diagram of the effect of the class raster map after dimensionality reduction. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0022] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0023] like Figure 1 As shown, the embodiment of the present application provides a method for perceiving unstructured road conditions, which specifically includes:

[0024] Step 101: Determine the ground point cloud and obstacle point cloud based on the original point cloud data of the in-vehicle laser radar.

[0025] Step 102: using a ground singular point detection algorithm to determine singular points in the ground point cloud.

[0026] Step 103: Perform point cloud pose matching based on the ground point cloud, the obstacle point cloud and the singular point to determine a point cloud matching relationship, wherein the point cloud matching relationship includes a rotation matrix and a translation matrix. Augment the point clouds of several adjacent frames based on the point cloud matching relationship.

[0027] Step 104: determining a two-dimensional grid map according to the augmented point cloud, and determining a grid map around the vehicle body from a bird's-eye view according to the two-dimensional grid map.

[0028] Step 105: Detect obstacles and drivable road areas in the target area in the grid map around the vehicle body based on a deep neural network to determine driving areas with different road conditions; the driving areas with different road conditions include rigid obstacle areas, flexible obstacle areas and drivable areas.

[0029] In order to improve the universality and robustness of autonomous driving technology, in addition to studying autonomous driving technology in well-paved structured road scenarios, it is also necessary to study autonomous driving technology in unstructured roads. Unstructured roads refer to roads without obvious driving guidance factors such as lane lines, traffic signs, and signal lights, as well as undeveloped irregular driving areas. Such scenarios, due to their incomplete road paving and uncertain surrounding environment, pose great challenges to autonomous driving technology, especially the traversable area identification link in autonomous driving technology. Most of the existing traversable area identification algorithms are for structured roads. Compared with structured roads, unstructured roads cannot be described by high-precision maps. At the same time, the roads themselves are easily affected by human or natural factors and change. There is a lack of common reference objects, so it is difficult to formulate a general discrimination strategy for traversable area identification algorithms.

[0030] In the process of autonomous driving, autonomous vehicles first need to perceive the surrounding environment, including obstacles and road boundaries. However, this kind of unstructured road surface often has a large number of potholes and obstacles, some of which are even difficult to detect; the boundaries of unstructured roads are far less clear than those of structured roads. This brings great uncertainty and safety hazards to the driving of autonomous vehicles. Therefore, how to improve the reliability and accuracy of identifying drivable areas in unstructured road scenarios is crucial for autonomous driving technology on unstructured roads. Depending on the sensors used, unstructured road environment perception methods are divided into three categories. The first category is a detection method based on camera visual information; the second category is a detection method based on lidar three-dimensional point cloud information; and the third category is a detection method based on multi-sensor fusion.

[0031] The vision-based perception method was initially widely used in the perception task of structured roads. In terms of environmental perception of unstructured roads, P. Jansen et al. used a color-based Gaussian mixture model to classify objects in unstructured scenes; R. Manduchi et al. used color features combined with 3D stereo images constructed by laser ranging for classification. The test scenes of these traditional visual perception algorithms are relatively simple, and the effect of actual application may not be very robust. In order to improve the reliability of perception, some scholars introduced convolutional neural networks. C. Holder et al. performed transfer learning on a neural network pre-trained with urban road data to enable it to perform perception tasks in unstructured roads. In order to improve the reliability of visual recognition algorithms under different weather and light conditions, Q. Wang et al. improved the Lite-Mobilenetv2 feature extraction module and integrated the pyramid pooling module with the attention mechanism. At the same time, a defogging preprocessing module was introduced to achieve recognition and perception of drivable areas in foggy scenes. Despite this, the effect of visual perception will be greatly reduced in low light or even dark environments. However, unstructured roads often do not have complete lighting facilities, which makes visual recognition solutions unable to perform perception tasks under such conditions.

[0032] The laser radar emits lasers and calculates the relative relationship between itself and the surrounding environment through the laser flight time and reflection characteristics to obtain the three-dimensional point cloud of the surrounding environment. Compared with camera vision, laser radar is not easily affected by changes in ambient light and has better environmental adaptability. The work of laser radar-related traversable area recognition is currently mainly focused on structured roads. In the field of unstructured road recognition, T.Liu et al. used traditional feature extraction methods to extract radial and lateral features of point clouds for road segmentation. Z.Zhu et al. proposed a road boundary enhancement Point-Cylinder network that can directly use laser radar point cloud data to extract point cloud features and segment roads. Similar methods include RandLA-Net, KPConv, PointNet, etc. This type of method can use neural networks to directly extract the features of three-dimensional point clouds, but at the same time greatly increase the amount of calculation, resulting in limited computing speed and affecting the real-time performance of the algorithm. In the multi-sensor fusion perception scheme, most of the work maps three-dimensional point cloud information to two-dimensional image information, which makes up for the problem that image information does not have depth information and is difficult to estimate the target distance. For example, L.Caltagirone used this method to realize the perception of unstructured roads using a fully convolutional neural network. However, although the unstructured roads mentioned in the article do not have road markings, they are still town roads with clear boundaries. And this type of multi-sensor fusion algorithm cannot avoid the impact of light problems on camera vision.

[0033] In summary, the problems with the current perception algorithms for unstructured road environments are: algorithms involving images are not suitable for unstructured roads with similar road backgrounds, and are easily affected by external factors such as light; perception algorithms based on lidar 3D point clouds have good adaptability, but the amount of calculation required to directly extract 3D point cloud features is too large and the real-time performance is poor. In addition, existing research mainly considers obstacles and road boundaries, ignoring factors such as potholes and gullies that may affect vehicle driving.

[0034] In response to the above problems, this application uses Velodyne 16-line laser radar to identify passable areas on unstructured roads. An obstacle point cloud detection method based on the longitudinal distribution characteristics of the point cloud is used to pre-process the point cloud. A Ring-lateral singular point detection algorithm is proposed. In order to improve the information density of the point cloud and the accuracy of detection, a point cloud augmentation algorithm based on the normal distribution transformation point cloud matching algorithm is proposed to augment the sparse point cloud to make the point cloud features dense. In order to improve the calculation speed and real-time performance while retaining the necessary point cloud information as much as possible, this application proposes a three-dimensional point cloud dimensionality reduction method, which uses the RGB channel of the image to store obstacle and singular point information, and uses the Mask-Rcnn algorithm to perceive the surrounding point cloud after dimensionality reduction in real time. The overall algorithm is developed using the Robot Operating System (ROS), which can achieve real-time visualization and facilitate the access of subsequent autonomous driving algorithms.

[0035] Further, in an exemplary embodiment, step 101 may be replaced by the following steps.

[0036] Step 1011: filtering out point cloud data above a set height threshold in the original point cloud data, determining the filtered point cloud data, and obtaining first point cloud data; the original point cloud data includes vehicle body point cloud data.

[0037] Step 1012: Using an obstacle detection algorithm, filter out point clouds within a preset radius from the vehicle body point cloud data in the first point cloud data to obtain second point cloud data; the second point cloud data includes obstacle points and ground points.

[0038] Step 1013: Divide the XY plane circumference of the laser radar coordinate system according to the set degree, and differentiate the divided area to obtain multiple rays; the laser radar coordinate system is established with the vehicle's driving direction as the X-axis, the axis perpendicular to the vehicle's driving direction as the Y-axis, and the position of the laser radar as the origin.

[0039] Step 1014: Differentiate each ray to determine the original points in the original point cloud data distributed on the ray.

[0040] Step 1015: traverse each point in the second point cloud data to obtain the horizontal distance from each point to the origin of the laser radar coordinate system.

[0041] Step 1016: Calculate the distance between two adjacent points on each ray according to the horizontal distance.

[0042] Step 1017: Calculate the local height domain value and the global height threshold according to the distance and the local slope threshold.

[0043] Step 1018: Determine the ground point cloud and the obstacle point cloud according to the local height domain value and the global height threshold.

[0044] Step 1019: Obstacles are defined as objects that the vehicle cannot get too close to or collide with during driving. Such objects often protrude a certain height from the ground and have obvious obstacle point cloud features in the vertical direction. This application uses an obstacle point cloud detection method based on the longitudinal distribution characteristics of point clouds to segment obstacles above the ground and ground point clouds.

[0045] The computer's obstacle detection algorithm subscribes to the original point cloud ROS data of the lidar, and filters out the point cloud data above the threshold according to the preset height threshold, reducing the amount of point cloud data and improving the overall processing speed.

[0046] The computer's obstacle detection algorithm filters out point clouds within a certain radius around the lidar based on a preset threshold to prevent the vehicle's own point cloud from interfering with subsequent point cloud processing.

[0047] The computer's obstacle detection algorithm differentiates the circumference of the laser radar detection range in the XY plane of the laser radar coordinate system by 0.18°. Each equal part of the differentiation is approximately regarded as a ray, and each ray is differentiated again every 0.001 meter along the radial direction. Then, each point in the point cloud is traversed, and the horizontal distance r from each point to the origin of the laser radar coordinate system and the angle θ between the line connecting the point and the origin of the laser radar coordinate system relative to the positive direction of the vehicle head (the positive direction of the X-axis of the laser radar coordinate system) are calculated. The calculation method is as follows:

[0048]

[0049] Among them, x is the X-axis coordinate of the laser radar point in the laser radar coordinate system, and y is the Y-axis coordinate of the laser radar point in the laser radar coordinate system. According to the calculated horizontal distance and angle, all laser radar reflection points are classified and rearranged according to the angle and radial differentials.

[0050] The computer's obstacle detection algorithm calculates the distance d between the two adjacent laser radar reflection points before and after each ray in turn based on the rearranged laser radar reflection points, and calculates the local height domain value h through the preset local slope domain value local-max-slope. The global height domain value H is calculated through the global slope domain value general-max-slope and the horizontal distance r of the current laser radar point. The calculation method is as follows:

[0051] h = tan(local-max-slope).

[0052] H=tan(local-max-slope)×d.

[0053] d=r i -r i-1 .

[0054] Specifically, determining the ground point cloud and the obstacle point cloud according to the local height domain value and the global height threshold includes:

[0055] Judgement i-1 -h≤z i ≤z i-1 +h is established, and the first judgment result is obtained.

[0056] If the first judgment result is yes, determine whether the previous point is a ground point to obtain a second judgment result; wherein z i-1 is the height of the previous point adjacent to the current point i in the laser radar coordinate system; h is the local height threshold; z i It is the height of the current point in the laser radar coordinate system.

[0057] If the second judgment result is yes, the current point is determined to be a ground point.

[0058] If the second judgment result is no, judge -h sensor -H≤z i ≤-h sensor +H is established, and the third judgment result is obtained; wherein, h sensor is the installation height of the laser radar, and H is the global height threshold.

[0059] If the third judgment result is yes, the current point is determined to be a ground point.

[0060] If the third judgment result is no, the current point is determined to be an obstacle point.

[0061] If the first judgment result is no, determine whether the distance is less than the reclassification threshold and whether it satisfies -h sensor -H≤z i ≤-hsensor +H, and obtain the fourth judgment result.

[0062] If the fourth judgment result satisfies the condition that the value is less than the reclassification threshold and satisfies -h sensor -H≤z i ≤-h sensor +H, determine that the current point is ground fixed.

[0063] If the fourth judgment result is that it does not satisfy the value less than the reclassification threshold or does not satisfy -h sensor -H≤z i ≤-h sensor +H, the current point is an obstacle point.

[0064] The rules for judging obstacle points and ground points are as follows:

[0065] If z i-1 -h≤z i ≤z i-1 +h, where z i is the height of the current point on the Z axis of the laser radar coordinate system, i is the index of the current point, z i-1 The height of the previous point on the Z axis of the LiDAR coordinate system. If i = 0, then z i-1 Negative laser radar installation height -h sensor .

[0066] Further determine whether the previous point is a ground point. If the previous point is a ground point, then the current point is also a ground point. If the previous point is not a ground point, then further determination is required. sensor -H≤z i ≤-h sensor +H, the current point is a ground point; if this condition is not met, the current point is an obstacle point.

[0067] If z is not satisfied i-1 -h≤z i ≤z i-1 +h, then determine whether the distance d between the two points is less than the preset reclassification threshold and -h sensor -H≤z i ≤-h sensor +H, the current point is a ground point; if any of the two conditions is not met, the current point is an obstacle point.

[0068] The computer's obstacle detection algorithm divides all points in the point cloud into two categories: obstacle points and ground points according to the above rules.

[0069] In an exemplary embodiment, step 102 may be replaced by the following steps: calculating the angle between a line connecting any point in the second point cloud data and the origin of the laser radar coordinate system and the positive direction of the vehicle head.

[0070] According to the maximum height within the current angle range and the minimum height within the current angle range, the height difference within the angle range is calculated.

[0071] The average height of all points within the angle range is calculated based on the sum of the heights of all points in the second point cloud data within the current angle range and the total number of all points within the current angle range; all points include the ground point cloud.

[0072] Calculate the horizontal distance of all points relative to the origin of the lidar coordinate system and calculate the average horizontal distance of all points.

[0073] Based on the horizontal distances of all points relative to the origin of the LiDAR coordinate system, the horizontal distance difference of all points within the angle range is calculated according to the maximum horizontal distance within the current angle range and the minimum horizontal distance within the current angle range.

[0074] The adjacent angular distance difference of all points within the angle range is calculated based on the horizontal horizontal distance of all points in the previous angle range within the current angle range, the horizontal horizontal distance of all points in the current angle range, and the horizontal horizontal distance of all points in the next angle range of the current angle.

[0075] The singular points in the ground point cloud are determined according to the height difference, the average height, the horizontal distance difference and the adjacent angular distance difference.

[0076] In the invention, singular points are defined as points that do not conform to the imaging law of rotating laser radar on flat ground. During the rotating scanning process of rotating laser radar, each laser beam will form a cone structure. The projection of each cone on the flat ground should be approximately a smooth circle. In mathematical expression, the current point p i [x i ,y i , z i ] and the adjacent points on the uniform conic projection plane in the parameter and z i The difference needs to be kept very small. Figure 2 (a), road surface point cloud with singular points such as Figure 2 (b) as shown.

[0077] In an exemplary embodiment, step 102 may be replaced by the following steps: Is it true, get the fifth judgment result; where i Ring Is the index of the current Ring scan line.

[0078] judge Whether it is established or not, the sixth judgment result is obtained.

[0079] If the fifth judgment result is Not true and the sixth judgment result is If not, determine the singular points in the ground point cloud within the current angle range.

[0080] If the fifth judgment result is The sixth judgment result is established or It is established that there is no singular point in the ground point cloud within the current angle range.

[0081] judge Whether it is established or not, the seventh judgment result is obtained.

[0082] If the seventh judgment result is If not, determine the singular points in the ground point cloud within the current angle range.

[0083] If the seventh judgment result is It is established that there is no singular point in the ground point cloud within the current angle range.

[0084] The computer's ground singular point detection algorithm calculates the maximum height difference Δz in each angular range; the average height in each angular range Maximum horizontal distance difference Δr; Maximum distance difference between adjacent angles Δr adj The specific calculation method is as follows:

[0085] Δz=z max -z min .

[0086]

[0087] Δr=r max -r min .

[0088]

[0089] where z max is the maximum height within the current angle range; z min is the minimum height within the current angle range; ∑z is the sum of the heights of all points within the current angle range; n is the total number of all points within the current angle range; r max is the maximum horizontal distance within the current angle range; r min is the minimum horizontal distance within the current angle range; Represents the average horizontal distance of all points within the previous angle range of the current angle; Represents the average horizontal distance of all points within the previous angle range of the current angle; Represents the average horizontal distance of all points within the current angle range; Represents the horizontal horizontal distance of all points within the range of the next angle of the current angle. If the current angle is 0°, the previous angle is 359°; if the current angle is 359°, the next angle is 0°. The calculation method of the horizontal horizontal distance is:

[0090]

[0091] Among them, ∑r is the sum of the horizontal distances of all points within the current angle range.

[0092] The specific judgment conditions of the computer's ground singular point detection algorithm are as follows:

[0093] where i Ring Is the index of the current Ring.

[0094]

[0095] If the condition and Not satisfied or not satisfied Then there are singular points in the point cloud at this angle.

[0096] In an exemplary embodiment, step 103 may be replaced by the following steps, where the point cloud matching relationship is:

[0097]

[0098] in, represents the transformation matrix, R represents the rotation matrix, t represents the translation matrix, R 11 Represents the first row and first column of the rotation matrix, R 12 Represents the first row and second column of the rotation matrix, R 13 Represents the first row and third column of the rotation matrix, R 21 Represents the second row and first column of the rotation matrix, R 22 Represents the second row and second column of the rotation matrix, R 23 Represents the second row and third column of the rotation matrix, R 31 Represents the third row and first column of the rotation matrix, R 32 Represents the third row and second column of the rotation matrix, R 33 represents the third row and third column of the rotation matrix, t1 represents the first row of the translation matrix, t2 represents the second row of the translation matrix, and t3 represents the third row of the translation matrix.

[0099] The point cloud of 16-line laser is relatively sparse, and a single frame of point cloud may be difficult to fully describe the features of the object due to angle and occlusion problems. Therefore, it is necessary to use several frames of point cloud to describe the surrounding environment. This application uses the Normal Distribution Transform (NDT) matching algorithm to perform point cloud matching. The specific steps are as follows:

[0100] Divide the XY plane of the three-dimensional space of the laser radar into several grids, traverse each grid and filter out the grids with less than 3 laser points in the grid, and calculate the mean q and covariance matrix Σ of each grid:

[0101]

[0102] where p i Represents the i-th point in a grid, where there are n points in total. T represents the transpose of the matrix.

[0103] According to the mean and covariance matrix, we can see the normal distribution N(q,Σ), and the probability distribution density of the normal distribution is:

[0104]

[0105] Solve the point cloud matching relationship:

[0106] Point cloud matching relationship through transformation matrix Description, which consists of the rotation matrix R and the translation matrix t:

[0107]

[0108] The computer's point cloud matching algorithm will first initialize the point cloud matching relationship transformation matrix Initialize the adjacent frames to zero matrices and converge through iteration. When the rotation or translation transformation between the two frames of point clouds is large, it is necessary to use devices such as odometers and inertial navigation to estimate the transformation matrix. Map the new point cloud Φ to the target point cloud to get Φ′:

[0109]

[0110] Where p is a point in the point cloud Φ. c represents the cosine value, s represents the sin value; the subscripts x, y, and z represent the XYZ coordinate axes, such as s x is the sin value of the counterclockwise rotation angle of the x-axis, t x ,t y ,t z Represents the distance of translation along the XYZ axis.

[0111] The NDT registration score is obtained by superimposing the probability density of each grid:

[0112]

[0113] Define variable Q = p′ i -q i . Define each grid-score as s:

[0114]

[0115] Use the chain rule to calculate the gradient direction of s:

[0116]

[0117] Where Q is R i The partial derivative of is the Jacobian matrix J:

[0118]

[0119] in:

[0120]

[0121] The objective function s is optimized by Newton's method, that is, the transformation matrix is ​​found Make the value of s minimum. Construct s about the variable The second-order partial derivative H ij :

[0122]

[0123] in:

[0124]

[0125] After the solution is completed, the steps of solving the point cloud matching relationship are repeated until the convergence condition is met.

[0126] In this application, the computer uses the original point cloud containing obstacle points, singular points and ground points to perform point cloud pose matching. The original point cloud has more point cloud features, which is conducive to improving the accuracy of matching.

[0127] The price of LiDAR will increase significantly as the number of beams increases. In order to save hardware costs, this application uses a 16-line LiDAR and augments the sparse point cloud to make the point cloud features more obvious.

[0128] The original point cloud is used to perform NDT matching calculations to obtain the entire transformation matrix R of the current frame and the historical frame. The relative posture relationship between any two frames can be expressed by the transformation matrix between them as:

[0129]

[0130] in represent Relative to The posture, and Represents two pose matrices, such as Figure 3 As shown, Ф represents the point cloud of different frames, Represents the position transformation relationship between point clouds in different frames, and the subscript is the serial number. When augmenting, calculate the transformation matrix of the previous 1 to k frames (historical frames nk to n-1 frames) relative to the current frame (nth frame), such as Figure 2 As shown in the figure, the red circle is a 1.5cm book and the black circle is a 3cm deep box. This is used to illustrate the detection effect of singular points.

[0131] The historical frames are combined into the current frame using the transformation matrix to form an augmented point cloud. Figure 3 shown.

[0132] The computer will publish the augmented obstacle point cloud and singular point cloud ROS messages according to steps 101 to 104.

[0133] In an exemplary embodiment, step 104 may be replaced by the following steps:

[0134] Step 1041: Use a point cloud dimensionality reduction algorithm to intercept second point cloud data within a preset length on the front, rear, left and right sides of the vehicle, and determine the intercepted second point cloud data.

[0135] Step 1042: Determine a two-dimensional grid map using a cloud dimensionality reduction algorithm.

[0136] Step 1043: Project the intercepted second point cloud data into the two-dimensional grid map in proportion to form a grid map around the vehicle body from a bird's-eye view.

[0137] Step 1044: Compared with image recognition, although the laser radar point cloud has better performance in resisting environmental interference, the type of information contained in the point cloud has disadvantages in target recognition and classification. Therefore, this application proposes a method of reducing the dimension of the point cloud, projecting the three-dimensional point cloud containing singular points and obstacle information onto a 384×512 two-dimensional grid map. This is beneficial for the detection of road conditions and can greatly reduce the amount of calculation. The specific dimensionality reduction method is as follows:

[0138] The computer's point cloud dimensionality reduction algorithm will capture a point cloud of 12m by 16m in total, 12m in front of the vehicle, 4m in the rear, and 6m on each side.

[0139] The computer's point cloud dimensionality reduction algorithm creates a blank 384×512 pixel grid map. The value range of each pixel grid is 0-255.

[0140] Step 1045: The computer's point cloud dimensionality reduction algorithm traverses the point cloud containing singular points and obstacle points, and projects them proportionally into a 384×512 two-dimensional grid map to form a grid map of the bird's-eye view around the vehicle body. The B channel of the image is used to store the density information of the singular point cloud; the G channel is used to store the strength information of the singular point cloud. If the singular point strength is less than 80, a value of 3 times the point cloud strength is assigned; if it is less than 80, a value of 255 is directly assigned; the R channel is used to store the obstacle height information. If the point cloud is an obstacle point cloud, a value of 255×the obstacle height is assigned when its height is less than 1m; if it is greater than 1m, a value of 255 is directly assigned.

[0141] Step 1046: In order to make the discrete point cloud have continuity at the pixel points representing the same target and optimize the target recognition effect, the computer's point cloud dimensionality reduction algorithm will expand and corrode the reduced dimensionality raster image to filter out discrete points while enhancing dense continuous points.

[0142] In an exemplary embodiment, step 105 can be replaced by the following steps: using a mask region-based convolutional neural network algorithm based on Mask R-cnn (Mask Region-based Convolutional Neural Network) to detect obstacles and drivable areas of the grid map around the vehicle body from a bird's-eye view, and dividing the target area into a rigid obstacle area, a flexible obstacle area and a drivable area according to the passability; the drivable area includes a free driving area and a poor driving area.

[0143] Deep neural networks are well suited for target recognition in disordered point clouds. Among the many deep neural networks, the Mask R-cnn (Mask Region-based Convolutional Neural Network) mask region-based convolutional neural network algorithm can accurately outline the target while identifying the specified target. The Mask R-cnn (Mask Region-based Convolutional Neural Network) mask region-based convolutional neural network is an instance segmentation algorithm that can segment the target in the form of a mask. This application uses the Mask R-cnn (Mask Region-based Convolutional Neural Network) mask region-based convolutional neural network algorithm to detect obstacles and drivable areas on the grid map processed by steps 101 to 104. The specific implementation steps are as follows:

[0144] This application divides the target area into rigid obstacle areas, flexible obstacle areas and drivable areas according to the degree of passability. Rigid areas are areas that cannot be touched by pedestrians, vehicles, buildings, etc. Flexible obstacle areas are areas that can be entered when necessary, such as grass and shrubs; drivable areas are further divided into free driving areas and poor driving areas, where vehicles are not restricted in free driving areas, and there are potholes and other rough roads in poor driving areas, and the speed of vehicles needs to be limited according to the unevenness.

[0145] The roughness in the bad driving area is:

[0146]

[0147] Among them, i represents the row coordinate of the pixel point, j represents the column coordinate of the pixel point, m represents the total number of rows in the grid map, and n represents the total number of columns in the grid map. Represents the G channel value of the pixel, Represents the B channel value of the pixel.

[0148] Use the on-board industrial computer to record the lidar raw point cloud ROS message data packet (Rosbag) on ​​a road section that includes flat, uneven and irregular roads; and on both sides of the road, there are various shapes of walls, bushes, grass, vehicles, cluttered objects, pedestrians and other targets. And intercept several frames of point cloud in Rosbag and use the above steps for preprocessing.

[0149] Use the labelme tool to label the raster map after the step. Standard targets are divided into two categories: obs and obs_pass. Among them, obs represents inaccessible rigid obstacles; obs_pass represents non-rigid obstacle areas such as bushes and grass that can be driven into when necessary. After each training image is labeled, a json file will be generated to record all the annotation points and the corresponding category information. After the labeling is completed, use the json_to_dataset tool to convert the json file into an xml file that stores the target mask location and a yaml file that represents the target category (class), such as Figure 5 shown.

[0150] The labeled data file is put into the Mask R-cnn (Mask Region-based Convolutional Neural Network) mask-based region convolutional neural network for training. After the training is completed, the Mask R-cnn (Mask Region-based Convolutional Neural Network) mask-based region convolutional neural network will be reloaded into the on-board industrial computer for testing.

[0151] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0152] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for perceiving unstructured road conditions, characterized in that: The unstructured road condition perception method comprises: Determine the ground point cloud and obstacle point cloud based on the original point cloud data of the in-vehicle LiDAR; Determining singular points in the ground point cloud using a ground singular point detection algorithm; Performing point cloud pose matching according to the ground point cloud, the obstacle point cloud and the singular point to determine a point cloud matching relationship; the point cloud matching relationship includes a rotation matrix and a translation matrix; and augmenting the sparse point cloud according to the matching relationship; Determine a two-dimensional grid map according to the augmented point cloud, and determine a grid map around the vehicle body from a bird's eye view according to the two-dimensional grid map; Based on a deep neural network, obstacles and drivable road areas are detected in the target area in the grid map around the vehicle body to determine driving areas with different road conditions; the driving areas with different road conditions include rigid obstacle areas, flexible obstacle areas and drivable areas.

2. The unstructured road condition perception method according to claim 1, characterized in that: Determining the ground point cloud and the obstacle point cloud based on the original point cloud data of the in-vehicle laser radar specifically includes: Filtering out point cloud data above a set threshold in the original point cloud data, determining the filtered point cloud data, and obtaining first point cloud data; the original point cloud data includes vehicle body point cloud data; Using an obstacle detection algorithm, filtering out point clouds within a preset radius from the vehicle body point cloud data in the first point cloud data to obtain second point cloud data; the second point cloud data includes obstacle points and ground points; The XY plane circumference of the laser radar coordinate system is divided according to the set degree, and the divided area is differentiated to obtain multiple rays; the laser radar coordinate system is established with the vehicle driving direction as the X axis, the direction perpendicular to the vehicle driving direction as the Y axis, and the position of the laser radar as the origin; Differentiate each ray to determine the original point in the original point cloud data distributed on the ray; Traversing each point in the second point cloud data to obtain the horizontal distance from each point to the origin of the laser radar coordinate system; Calculate the distance between two adjacent points of each ray according to the horizontal distance; Calculate the local height domain value and the global height threshold according to the distance and the local slope threshold; A ground point cloud and an obstacle point cloud are determined according to the local height domain value and the global height threshold.

3. According to the unstructured road condition perception method of claim 2, the determining of the ground point cloud and the obstacle point cloud according to the local height domain value and the global height threshold specifically comprises: Judge z i-1 -h≤z i ≤z i-1 +h is established, and the first judgment result is obtained; If the first judgment result is z i-1 -h≤z i ≤z i-1 +h is established, it is determined whether the previous point is a ground point, and a second determination result is obtained; wherein z i-1 is the height of the previous point adjacent to the current point i in the laser radar coordinate system; h is the local height threshold; z i is the height of the current point in the laser radar coordinate system; If the second judgment result is that the previous point is a ground point, determining that the current point is a ground point; If the second judgment result is that the previous point is not a ground point, judge -h sensor -H≤z i ≤-h sensor +H is established, and the third judgment result is obtained; wherein, h sensor is the installation height of the laser radar, and H is the global height threshold; If the third judgment result is -h sensor -H≤z i ≤-h sensor +H is established, and the current point is determined to be a ground point; If the third judgment result is -h sensor -H≤z i ≤-h sensor +H is not true, and the current point is determined to be an obstacle point; If the first judgment result is z i-1 -h≤z i ≤z i-1 +h is not true, determine whether the distance is less than the reclassification threshold and whether it satisfies -h sensor -H≤z i ≤-h sensor +H, get the fourth judgment result; If the fourth judgment result satisfies the condition that the value is less than the reclassification threshold and satisfies -h sensor -H≤z i ≤-h sensor +H, determine that the current point is ground fixed; If the fourth judgment result is that it does not satisfy the value less than the reclassification threshold or does not satisfy -h sensor -H≤z i ≤-h sensor +H, the current point is an obstacle point.

4. The unstructured road condition perception method according to claim 2, characterized in that: The method of determining the singular points in the ground point cloud by using a ground singular point detection algorithm specifically includes: Calculate the angle between the line connecting any point in the second point cloud data and the origin of the laser radar coordinate system and the positive direction of the vehicle head; Calculate the height difference within the angle range according to the maximum height within the current angle range and the minimum height within the current angle range; Calculate the average height of all points within the angle range according to the sum of the heights of all points in the second point cloud data within the current angle range and the total number of all points within the current angle range; all points include the ground point cloud; Calculate the horizontal distance of all points relative to the origin of the LiDAR coordinate system and calculate the average horizontal distance of all points; Based on the horizontal distances of all points relative to the origin of the laser radar coordinate system, the horizontal distance difference of all points within the angle range is calculated according to the maximum horizontal distance within the current angle range and the minimum horizontal distance within the current angle range; Calculate the adjacent angular distance difference of all points within the angle range according to the horizontal horizontal distance of all points within the previous angle range, the horizontal horizontal distance of all points within the current angle range, and the horizontal horizontal distance of all points within the next angle range of the current angle; The singular points in the ground point cloud are determined according to the height difference, the average height, the horizontal distance difference and the adjacent angular distance difference.

5. The unstructured road condition perception method according to claim 1, wherein the step of determining the singular points in the ground point cloud by using a ground singular point detection algorithm specifically comprises: judge Is it true, get the fifth judgment result; where i Ring is the index of the current Ring scan line; judge Whether it is established, the sixth judgment result is obtained; If the fifth judgment result is Not true and the sixth judgment result is If it is not true, determine the singular point in the ground point cloud within the current angle range; If the fifth judgment result is The sixth judgment result is established or Established, it is determined that there is no singular point in the ground point cloud within the current angle range; judge Whether it is established, the seventh judgment result is obtained; If the seventh judgment result is If it is not true, determine the singular point in the ground point cloud within the current angle range; If the seventh judgment result is It is established that there is no singular point in the ground point cloud within the current angle range.

6. The unstructured road condition perception method according to claim 1, characterized in that: The point cloud matching relationship is: in, represents the transformation matrix, R represents the rotation matrix, t represents the translation matrix, R 11 Represents the first row and first column of the rotation matrix, R 12 Represents the first row and second column of the rotation matrix, R 13 Represents the first row and third column of the rotation matrix, R 21 Represents the second row and first column of the rotation matrix, R 22 Represents the second row and second column of the rotation matrix, R 23 Represents the second row and third column of the rotation matrix, R 31 Represents the third row and first column of the rotation matrix, R 32 Represents the third row and second column of the rotation matrix, R 33 represents the third row and third column of the rotation matrix, t1 represents the first row of the translation matrix, t2 represents the second row of the translation matrix, and t3 represents the third row of the translation matrix.

7. The unstructured road condition perception method according to claim 2, wherein determining a two-dimensional grid map according to the point cloud matching relationship, and determining a grid map around the vehicle body from a bird's eye view according to the two-dimensional grid map, specifically comprises: Using a point cloud dimensionality reduction algorithm to intercept second point cloud data within a preset length on the front, rear, left, and right sides of the vehicle, and determine the intercepted second point cloud data; Determine the two-dimensional grid map using the cloud dimension reduction algorithm; The intercepted second point cloud data is projected proportionally into the two-dimensional grid map to form a grid map around the vehicle body from a bird's-eye view.

8. The unstructured road condition perception method according to claim 1, characterized in that: The detecting of obstacles and drivable road areas in the target area in the grid map around the vehicle body based on the deep neural network to determine the drivable road areas under different road conditions specifically includes: The target area in the grid map around the vehicle body under the bird's-eye view is detected by using the Mask R-CNN algorithm, and the target area is divided into a rigid obstacle area, a flexible obstacle area and a drivable area according to the passability; the drivable area includes a free driving area and a poor driving area.

9. The unstructured road condition perception method according to claim 8, characterized in that: The roughness in the bad driving area is: Among them, i represents the row coordinate of the pixel point, j represents the column coordinate of the pixel point, m represents the total number of rows in the grid map, and n represents the total number of columns in the grid map. Represents the G channel value of the pixel, Represents the B channel value of the pixel.

10. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the unstructured road condition perception method according to any one of claims 1 to 9.

Images