Method for making semantic surfel map based on laser radar

Abstract

The present invention provides a method for making a semantic facet map based on laser radar. The method completes the construction of the semantic facet map through three steps: data collection of the semantic facet map, making each level of the semantic facet map, using facets to represent a point cloud map, and combining semantic information to form the semantic facet map. In addition, when making each level of the semantic facet map, unnecessary noise data is filtered and semantic features are parameterized, thereby achieving better robustness of the semantic facet map during operation. At the same time, the operation mode is optimized, the program operation time is shortened, and the delay is further reduced.

Description

Technical Field

[0001] The present invention relates to an algorithm for making an autonomous driving map, and in particular to a method for making a semantic facet map based on a laser radar. Background Art

[0002] According to the Intelligent Connected Vehicle Technology Roadmap Figure 2 .0》plan, the penetration rate of new L2 / L3 autonomous driving vehicles will reach 50% in 2025. In 2021, L3 autonomous driving vehicles will enter the first year of mass production. As one of the core technologies of the perception layer of L3 and above autonomous driving, autonomous driving maps are indispensable prior information in unmanned driving systems and are a prerequisite for tasks such as relocation, path planning, and navigation decisions.

[0003] In the actual mapping process, urban and suburban environments are constantly changing, and these changes occur at different time scales, including dynamic and ephemeral objects (such as parked cars), seasonal changes (vegetation, snow, dust), and human impacts such as construction. Therefore, it is crucial that the mapping method is robust to these real environment changes.

[0004] Secondly, since the autonomous driving system must make real-time decisions based on environmental changes during the vehicle's driving, low latency is extremely critical. Most mapping algorithms simply perform simplified parameterization on point cloud data, which will result in the subsequent matching of objects with the same geometric features being matched with different objects in the map library. This problem is particularly evident in urban environments with complex scenes and high repetition. How to extract detailed information for mapping and subsequent map matching and positioning in this scenario has become a difficult problem that smart cars need to solve urgently. Summary of the invention

[0005] Purpose of the invention: In view of the deficiencies in the prior art, the present invention provides a method for producing a semantic facet map based on lidar, which completes the construction of the semantic facet map through three steps: data collection of the semantic facet map, production of each level of the semantic facet map, characterization of the point cloud map with facets, and combining semantic information to form the semantic facet map. In addition, when producing each level of the semantic facet map, unnecessary noise data is filtered out and the semantic features are parameterized, thereby achieving better robustness of the semantic facet map during operation. At the same time, the operation mode is optimized, the program operation time is shortened, and the delay is further reduced.

[0006] Technical solution: A method for producing a semantic surfel map based on laser radar, comprising:

[0007] S1: Data collection of semantic surfel maps;

[0008] S1.1: Use laser radar to obtain laser point cloud information in urban environments. The point cloud data content includes the three-dimensional coordinates and laser reflection intensity of each point of the scanned object in the urban environment;

[0009] S1.2: Use RTK real-time dynamic positioning technology to obtain the real coordinate information of the vehicle in real time;

[0010] S1.3: Preprocess the collected laser point cloud information, organize the obtained laser point cloud data and their corresponding trajectory information, and complete the original data set;

[0011] S2: Create semantic surfel maps by level;

[0012] S2.1: Project the points in the three-dimensional space onto the two-dimensional depth map to obtain the two-dimensional depth map V D ;

[0013] S2.2: The depth map V D Calculate the normal vector of each point in the vector graph N. D ;

[0014] S2.3: Use deep learning methods for semantic segmentation. Use the laser point cloud semantic segmentation method to perform 2D image full convolution semantic segmentation on the depth image and then restore the semantic conversion of all points from 2D to 3D from the original point cloud.

[0015] Reconstruct the point cloud based on the effective depth image, and eliminate the artifacts of undesired discretization and reasoning based on the KNN nearest neighbor classification method search. After processing the point cloud data, classify the results;

[0016] S2.4: Filter the classification results and parameterize the filtered semantic features;

[0017] S2.5: Record the real coordinates of the vehicle in the world coordinate system and express the real coordinates of the vehicle trajectory in a 4×4 matrix;

[0018] S3: Use surfels to represent point cloud maps and combine them with semantic information to form semantic surfel maps;

[0019] S3.1: Based on the basic elements of the semantic surface map obtained in the above steps, the real coordinate information of the vehicle is obtained in real time using RTK real-time dynamic positioning technology. After obtaining the position and posture of each frame, the current frame is then fused into the map;

[0020] S3.2: For each point, the corresponding face element is calculated, and then it is determined whether the face element is close enough to an existing face element. If so, it is merged into the existing face element. Otherwise, the face element is retained as a new face element.

[0021] Preferably, S2.1 is specifically as follows: firstly, the three-dimensional points are projected onto the spherical surface, and all the points are represented by (θ, ψ, depth), and then (θ, ψ) is mapped to the two-dimensional coordinates (u, v), and the depth is mapped to the pixel value, thereby obtaining the depth map V D ;

[0022] Converting three-dimensional coordinates into two-dimensional coordinates specifically includes:

[0023] For a rotating scanning laser radar, its vertical field of view FOV is divided into two parts: FOV_up and FOV_down. The value of FOV_up is a positive number and the value of FOV_down is a negative number, so FOV = FOV_up + (-FOV_down);

[0024] Spherical coordinates are represented by three parameters: distance, azimuth, and zenith. Each point represented by three-dimensional Cartesian coordinates in the commonly used LiDAR point cloud is actually converted from the spherical coordinate system. The coordinates of each point represented by three-dimensional Cartesian coordinates in the LiDAR point cloud are (X, Y, Z). After unfolding the cylinder obtained by rotating the LiDAR for one circle with the x-axis as the direction of the front view, an image is obtained: the origin of the coordinate is at the center of the image, the vertical coordinate of the pixel in the image is obtained by the projection of the pitch angle, the range is [FOV_down, FOV_up], and the horizontal coordinate is obtained by the projection of the deflection angle, the range is [-π, π];

[0025]

[0026] pitch1=sin -1 (Z / R) (2)

[0027] yaw1=tan -1 (Y / X) (3)

[0028] In formula (1), R is the distance from each point collected by the laser radar to the origin, X, Y, Z are the coordinates of the point, in formula (2), pitch1 is the pitch angle, Z is the point cloud coordinate in the Z-axis direction, and in formula (3), yaw1 is the deflection angle;

[0029] The image coordinate system uses the upper left corner as the origin. Transform the front view obtained above and move the origin to the upper left corner:

[0030] pitch2=FOV_Up-pitch1 (4)

[0031] yaw2=yaw1+π (5)

[0032] In formula (4), pitch2 is the converted pitch angle, FOV_Up is the vertical resolution of the upper field of view, and in formula (5), yaw1 is the deflection angle, and yaw2 is the converted deflection angle;

[0033] Projecting a three-dimensional point cloud into a two-dimensional image, this dimensionality reduction operation will inevitably lead to information loss. In order to minimize the information loss caused by projection, we need to choose a projection image of appropriate size. For an 80-line LiDAR, the height of the projected image is generally set to 80, and the width of the image is set according to the maximum horizontal resolution of the LiDAR. The horizontal resolution of an 80-line LiDAR is 0.35-0.4 degrees, so the maximum number of points generated by a laser in one rotation is 1028. In convolutional neural networks, the input feature map is generally downsampled by a factor of 2 multiple times, so the width of the image needs to be set to a power of 2, which can be set to 1024 here. Due to the different field of view angles and horizontal resolutions of different types of LiDARs, the size of the projected image will also be set to different values ​​as needed. In order to adapt to these changes, yaw2 and pitch2 are normalized. After normalization, multiply by the width and height of the projected image to obtain the coordinates of this point projected to the depth image.

[0034] pitch3=(FOV_Up-pitch2) / (FOV_Up-FOV_Down) (6)

[0035] yaw3=(yaw2+π) / 2π (7)

[0036]

[0037] In formula (6), pitch3 is the normalized and scaled pitch angle, FOV_Down is the vertical resolution of the lower field of view, in formula (7), yaw3 is the normalized and scaled yaw angle, in formula (8), u and v are the normalized and scaled coordinate values, z is the coordinate of the point cloud on the Z axis, R is the distance between the point scanned by the laser and the coordinate origin, row_scale is the width of the converted depth image, and col_scale is the length of the depth map.

[0038] Preferably, the S2.2 is specifically as follows: calculate the normal vector of a random point, which is the outer product of the vector from the point to the adjacent point on the right and the vector from the point to the adjacent point on the upper side after the cross product of the two. The calculation formula is as follows:

[0039] N D (u, v) = [V D (u+1,v)-V D (u,v+1)]×[V D (u,v+1)-V D (u,v)] (9)

[0040] N in formula (9) D is the normal vector of each point, u, v are the two-dimensional coordinates calculated in formula (8), V D refers to the depth map coordinates, and “×” refers to the cross product.

[0041] Preferably, S2.3 is specifically: using a deep learning method to perform semantic segmentation, using the neural network RangeNet++ to perform 2D image full convolution semantic segmentation on the depth image, and then recovering the semantic conversion from 2D to 3D of all points from the original point cloud, regardless of the depth map discretization used, and finally reconstructing the point cloud based on the valid depth image, and based on KNN search, eliminating undesirable discretization and reasoning artifacts.

[0042] After processing the point cloud data, it is identified and classified by a convolutional neural network, and the following results are obtained, including: traffic signal signs, columnar objects, roads, sidewalks, moving vehicles, stopped vehicles, buildings, and people.

[0043] The columnar objects include tree trunks, electric poles, street lamp posts, fire hydrants, trash cans and other facilities fixedly placed on the edge of the road.

[0044] The preferred option is to refine the semantic map in order to meet the needs of vehicle relocation or navigation. Considering the robustness, it is necessary to filter out not only moving objects but also semantic information that is prone to change. Considering the real-time, accuracy and storage, it is necessary to select feature objects.

[0045] S2.4 specifically includes: filtering the classification results obtained in S2.3, and retaining the point cloud data of the traffic signal sign, columnar object, road and building;

[0046] The filtering of the classification results is specifically to perform secondary classification of these categories by a dynamic object filtering method, the method is as follows:

[0047] When the new laser point passes through the position of the old laser point, the old laser point is a dynamic point; each time the new laser overlaps with the old laser, the number of misses is increased by one, and thresholds φ and ω are set. When the accumulated number of misses is greater than φ, it is marked as a high-dynamic object; when the accumulated number of misses is less than φ and greater than ω, it is marked as a low-dynamic object; when the accumulated number of misses is less than ω, it is marked as an object to be processed;

[0048] The above classification method is used to filter out moving objects, including moving vehicles and people;

[0049] The remaining classification results are processed according to the environmental object classification, and the objects to be processed are divided into semi-static objects and static objects according to the dynamic degree of the objects. Semi-static objects include stopped vehicles, and static objects include traffic signs, columnar objects, roads and buildings;

[0050] The semi-static objects are filtered out through the above classification method, and traffic signal signs, columnar objects, roads and buildings are retained;

[0051] Parameterized semantic feature traffic signal signboard, since the point cloud of traffic signals is generally relatively independent, Euclidean clustering can be directly used for secondary segmentation and extracted through the RANSAC method;

[0052] For parameterized semantic feature cylindrical objects, Euclidean clustering is used to fit a 3D straight line, which is represented by the coordinates and direction vector of the point on the rod closest to the origin.

[0053] The specific algorithm is as follows: First, create a Kd-tree representation P for the input point cloud dataset, then establish an empty cluster list C and a queue Q of points to be checked, and then for each point Pi∈P, perform the following steps: Add Pi to the current queue Q, for each point Pi∈Q do: search for the set of neighbors of the Pik point in a cylinder with a radius of r, for each neighbor point pik∈Pik, check whether the point has been processed, if not, add it to Q, after processing all points in the queue Q, add Q to the cluster list C, and reset Q to an empty list, when all points Pi∈P have been processed and are now part of the point cluster list C, the algorithm terminates.

[0054] Real coordinate information, using RTK tools to measure the vehicle position in real time and dynamically, record the real coordinate value of the vehicle in the world coordinate system, expressed in a 4×4 matrix.

[0055] Maps are in the form of point cloud maps, raster maps, and polygon maps. Point cloud maps are the mainstream maps with high precision, but as the scale increases, memory consumption is serious, which is not conducive to calculation and storage. The advantage of raster maps is that memory consumption is better than point cloud maps in large-scale scenes, but the disadvantage is poor precision. Finally, there is the polygon map. The advantage of the polygon map is that it uses as little data as possible to express the largest possible scene, so the resource usage is low. Considering the storage problem, the polygon map is used.

[0056] A facet is the basic element of a facet map. A facet is an (elliptical) circular plane with direction and size in space. Therefore, the basic components of a facet are: the three-dimensional coordinates of the facet center point; the facet normal vector; and the facet size (circle radius).

[0057] According to the requirements of constituting facets and combining maps at all levels, we propose a map updating strategy: first, the corresponding facet is calculated for each point, and then it is determined whether the facet is close enough to an existing facet. If so, it is merged into the existing facet; otherwise, the facet is retained as a new facet.

[0058] Preferably, the S3.2 is as follows: for any point v in the current frame s , let the corresponding surface element be s, the spatial coordinates of the surface element are the coordinates of the point, the normal vector is the normal vector of the point, and the radius of the surface element is r s Calculated as follows:

[0059]

[0060] In this formula, the molecular ||v s || represents the distance from the point to the origin of the laser radar, so the numerator is the distance from the point to the origin multiplied by a proportional coefficient Where p is the pixel density per unit angle in the spherical coordinate system, which is a fixed parameter. The clamp(*) in the denominator is an interval-limited function, which limits the value range of the first term in the brackets to between 0.5 and 1.0; the first term in the brackets represents the cosine value of the angle between the laser line direction and the point normal vector direction, v s Represents the coordinates of the midpoint of the surface element, n s Represents the normal vector of the corresponding point.

[0061] Since the angle must be between 0 and 90 degrees, the cosine value range is 1-0. After clamp(*), the actual range of the denominator is 1.0 to 0.5. It can be inferred that the more parallel the normal vector is to the laser line, the smaller the angle, the larger the denominator, and the smaller the radius. In this formula, the numerator is equivalent to providing an initial radius, and the denominator adjusts this radius according to the angle between the normal vector and the laser line. If the laser hits the place more perpendicular to the laser line, the smaller the angle between the normal vector and the laser line, the larger the denominator, and the smaller the corresponding surface element radius; conversely, the surface element radius is larger.

[0062] To determine whether the face element is close enough to an existing face element, a beam of light is emitted along the direction of the laser line. The first face element hit in the existing map is the face element s' that is closest to the origin of the coordinates on the light path. The criterion for determining whether s and s' are close enough is:

[0063]

[0064] ||n s ×n s ,||<sinθ M (12)

[0065] In formula (11), s' is the first surface element reached by extending the line of sight of the point in the current frame. is the transpose of the normal vector of the face element, v s , is the center coordinate of the surface element, v s is the coordinate of the midpoint of the surface element in the current frame, δ M is the distance threshold, in formula (12) n s is the normal vector of the face in the current frame, n s , is the normal vector of the first face element hit along the line of sight of the point in the current frame, θ M is the angle threshold;

[0066] This standard means that the positions of s and s' need to be close enough and their normal vectors point in the same direction. If they are not close enough, the probability of s' is reduced by a value and the new face s is inserted into the face map. If they are close enough, the probability of s' is increased by a value and s' is updated with the information of s, which is face fusion. The update formula is as follows:

[0067]

[0068]

[0069]

[0070] In formula (13) is the updated coordinate position, γ is the threshold, v s is the center coordinate of the surface element in the current frame, is the center coordinate of the first face element reached by extending the line of sight of the point in the current frame. In formula (14), is the updated normal vector, γ is the threshold, n s is the normal vector in the current frame, is the normal vector of the first face element hit along the line of sight of the point in the current frame. In formula (15), is the updated radius, r s is the radius of the surface element in the current frame.

[0071] After processing all the point cloud data information, the production of the semantic surface map is completed.

[0072] Beneficial effects: The present invention constructs a semantic facet map, which ensures that each node contains multi-dimensional information of two-dimensional depth, vector, semantics and real coordinates of vehicle trajectory; considering the robustness of the algorithm and the real-time nature of subsequent positioning, the traditional feature extraction method is used to refine semantic information on the basis of the deep learning method, and the noise data is filtered out by secondary filtering to provide accurate prior information for subsequent vehicle positioning, navigation or decision-making. The facet form is used to represent point clouds and is applied to large-scale outdoor scenes, which greatly reduces memory while ensuring accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0074] Figure 1 It is the overall flow chart of the present invention.

[0075] Figure 2 Schematic diagram of each level of semantic facet map.

[0076] Figure 3 Schematic diagram of semantic segmentation.

[0077] Figure 4 Schematic diagram of semantic facet. DETAILED DESCRIPTION

[0078] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0079] In the description of the present invention, it is necessary to understand that the terms "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0080] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.

[0081] A method for making a semantic surfel map based on laser radar, comprising:

[0082] S1: Data collection of semantic surfel maps;

[0083] S Chuan: Laser point cloud information is obtained by using laser radar in urban environments. The point cloud data content includes the three-dimensional coordinates and laser reflection intensity of each point of the scanned object in the urban environment;

[0084] S1.2: Use RTK real-time dynamic positioning technology to obtain the real coordinate information of the vehicle in real time;

[0085] S1.3: Preprocess the collected laser point cloud information, organize the obtained laser point cloud data and their corresponding trajectory information, and complete the original data set;

[0086] S2: Create semantic surfel maps by level;

[0087] S2.1: Project the points in the three-dimensional space onto the two-dimensional depth map to obtain the two-dimensional depth map V D ;

[0088] First, project the three-dimensional points onto the sphere, with all points represented by (θ, ψ, depth). Then, map (θ, ψ) to two-dimensional coordinates (u, v), and map depth to pixel values, thus obtaining the depth map V. D ;

[0089] Converting three-dimensional coordinates into two-dimensional coordinates specifically includes:

[0090] For a rotating scanning laser radar, its vertical field of view FOV is divided into two parts: FOV_up and FOV_down. The value of FOV_up is a positive number and the value of FOV_down is a negative number, so FOV = FOV_up + (-FOV_down);

[0091] Spherical coordinates are represented by three parameters: distance, azimuth, and zenith. Each point represented by three-dimensional Cartesian coordinates in the commonly used LiDAR point cloud is actually converted from the spherical coordinate system. The coordinates of each point represented by three-dimensional Cartesian coordinates in the LiDAR point cloud are (X, Y, Z). After unfolding the cylinder obtained by rotating the LiDAR for one circle with the x-axis as the direction of the front view, an image is obtained: the origin of the coordinate is at the center of the image, the vertical coordinate of the pixel in the image is obtained by the projection of the pitch angle, the range is [FOV_down, FOV_up], and the horizontal coordinate is obtained by the projection of the deflection angle, the range is [-π, π];

[0092]

[0093] pitch1=sin -1 (Z / R) (2)

[0094] yaw1=tan -1 (Y / X) (3)

[0095] In formula (1), R is the distance from each point collected by the laser radar to the origin, X, Y, and Z are the coordinates of the point, in formula (2), pitch1 is the pitch angle, Z is the point cloud coordinate in the Z-axis direction, and in formula (3), yaw1 is the deflection angle;

[0096] The image coordinate system uses the upper left corner as the origin. Transform the front view obtained above and move the origin to the upper left corner:

[0097] pitch2=FOV_Up-pitch1 (4)

[0098] yaw2=yaw1+π (5)

[0099] In formula (4), pitch2 is the converted pitch angle, FOV_Up is the vertical resolution of the upper field of view, and in formula (5), yaw1 is the deflection angle, and yaw2 is the converted deflection angle;

[0100] Projecting a three-dimensional point cloud into a two-dimensional image, this dimensionality reduction operation will inevitably lead to information loss. In order to minimize the information loss caused by projection, we need to choose a projection image of appropriate size. For an 80-line LiDAR, the height of the projected image is generally set to 80, and the width of the image is set according to the maximum horizontal resolution of the LiDAR. The horizontal resolution of an 80-line LiDAR is 0.35-0.4 degrees, so the maximum number of points generated by a laser in one rotation is 1028. In convolutional neural networks, the input feature map is generally downsampled by a factor of 2 multiple times, so the width of the image needs to be set to a power of 2, which can be set to 1024 here. Due to the different field of view angles and horizontal resolutions of different types of LiDARs, the size of the projected image will also be set to different values ​​as needed. In order to adapt to these changes, yaw2 and pitch2 are normalized. After normalization, multiply by the width and height of the projected image to obtain the coordinates of this point projected to the depth image.

[0101] pitch3=(FOV_Up-pitch2) / (FOV_Up-FOV_Down) (6)

[0102] yaw3=(yaw2+π) / 2π (7)

[0103]

[0104] In formula (6), pitch3 is the normalized and scaled pitch angle, FOV_Down is the vertical resolution of the lower field of view, in formula (7), yaw3 is the normalized and scaled yaw angle, in formula (8), u and v are the normalized and scaled coordinate values, z is the coordinate of the point cloud on the Z axis, R is the distance between the point scanned by the laser and the coordinate origin, row_scale is the width of the converted depth image, and col_scale is the length of the depth map.

[0105] S2.2: The depth map V D Calculate the normal vector of each point in the vector graph N. D ;

[0106] Calculate the normal vector of a random point, which is the outer product of the vector from the point to the adjacent point on the right and the vector from the point to the adjacent point on the upper side. The calculation formula is as follows:

[0107] N D (u, v) = [V D (u+1,v)-V D (u,v+1)]×[V D (u,v+1)-V D (u,v)] (9)

[0108] N in formula (9)D is the normal vector of each point, u, v are the two-dimensional coordinates calculated in formula (8), V D refers to the depth map coordinates, and “×” refers to the cross product.

[0109] S2.3: Use deep learning methods for semantic segmentation. Use the laser point cloud semantic segmentation method to perform 2D image full convolution semantic segmentation on the depth image and then restore the semantic conversion of all points from 2D to 3D from the original point cloud.

[0110] Reconstruct the point cloud based on the effective depth image, and eliminate the artifacts of undesired discretization and reasoning based on the KNN nearest neighbor classification method search. After processing the point cloud data, classify the results;

[0111] We use deep learning methods for semantic segmentation, and use the neural network RangeNet++ to perform 2D image fully convolution semantic segmentation on the depth image. Then we recover the semantic conversion from 2D to 3D of all points from the original point cloud, regardless of the depth map discretization used. Finally, we reconstruct the point cloud based on the valid depth image, and based on KNN search, we eliminate undesirable discretization and reasoning artifacts.

[0112] After processing the point cloud data, it is identified and classified by a convolutional neural network, and the following results are obtained, including: traffic signal signs, columnar objects, roads, sidewalks, moving vehicles, stopped vehicles, buildings, and people.

[0113] The columnar objects include tree trunks, electric poles, street lamp posts, fire hydrants, trash cans and other facilities fixedly placed on the edge of the road.

[0114] S2.4: Filter the classification results and parameterize the filtered semantic features;

[0115] The preferred option is to refine the semantic map in order to meet the needs of vehicle relocation or navigation. Considering the robustness, it is necessary to filter out not only moving objects but also semantic information that is prone to change. Considering the real-time, accuracy and storage, it is necessary to select feature objects.

[0116] Filter the classification results obtained in S2.3, and retain the point cloud data of the traffic signal sign, columnar object, road and building;

[0117] The filtering of the classification results is specifically to perform secondary classification of these categories by a dynamic object filtering method, the method is as follows:

[0118] If the new laser point passes through the position of the old laser point, the old laser point is a dynamic point; if the two lasers overlap, it means that the second laser missed and the object left, which is a miss. Every time the new laser overlaps with the old laser, the number of misses increases by one. Set thresholds φ and ω. When the accumulated number of misses is greater than φ, it is marked as a high-dynamic object. When the accumulated number of misses is less than φ and greater than ω, it is marked as a low-dynamic object. When the accumulated number of misses is less than ω, it is marked as an object to be processed.

[0119] The above classification method is used to filter out moving objects, including moving vehicles and people;

[0120] The remaining classification results are processed according to the environmental object classification, and the objects to be processed are divided into semi-static objects and static objects according to the dynamic degree of the objects. Semi-static objects include stopped vehicles, and static objects include traffic signs, columnar objects, roads and buildings;

[0121] The semi-static objects are filtered out through the above classification method, and traffic signal signs, columnar objects, roads and buildings are retained;

[0122] Parameterized semantic feature traffic signal signboard, since the point cloud of traffic signals is generally relatively independent, Euclidean clustering can be directly used for secondary segmentation and extracted through the RANSAC method;

[0123] For parameterized semantic feature cylindrical objects, Euclidean clustering is used to fit a 3D straight line, which is represented by the coordinates and direction vector of the point on the rod closest to the origin.

[0124] The specific algorithm is as follows: First, create a Kd-tree representation P for the input point cloud dataset, then establish an empty cluster list C and a queue Q of points to be checked, and then for each point Pi∈P, perform the following steps: Add Pi to the current queue Q, for each point Pi∈Q do: search for the set of neighbors of the Pik point in a cylinder with a radius of r, for each neighbor point pik∈Pik, check whether the point has been processed, if not, add it to Q, after processing all points in the queue Q, add Q to the cluster list C, and reset Q to an empty list, when all points Pi∈P have been processed and are now part of the point cluster list C, the algorithm terminates.

[0125] S2.5: Record the real coordinates of the vehicle in the world coordinate system and express the real coordinates of the vehicle trajectory in a 4×4 matrix;

[0126] Real coordinate information, using RTK tools to measure the vehicle position in real time and dynamically, record the real coordinate value of the vehicle in the world coordinate system, expressed in a 4×4 matrix.

[0127] S3: Maps are in the form of point cloud maps, raster maps, and surface maps. Point cloud maps are the mainstream maps with high precision, but as the scale increases, memory consumption is serious, which is not conducive to calculation and storage. The advantage of raster maps is that they consume less memory than point cloud maps in large-scale scenes, but the disadvantage is that the precision is poor. Finally, there is the surface map. The advantage of the surface map is that it uses as little data as possible to express the largest possible scene, so the resource usage is low. Considering the storage problem, the surface map is used.

[0128] A facet is the basic element of a facet map. A facet is an (elliptical) circular plane with direction and size in space. Therefore, the basic components of a facet are: the three-dimensional coordinates of the facet center point; the facet normal vector; and the facet size (circle radius).

[0129] Use facets to represent point cloud maps and combine them with semantic information to form semantic facet maps;

[0130] S3.1: Based on the basic elements of the semantic surface map obtained in the above steps, the real coordinate information of the vehicle is obtained in real time using RTK real-time dynamic positioning technology. After obtaining the position and posture of each frame, the current frame is then fused into the map;

[0131] S3.2: For each point, the corresponding face element is calculated, and then it is determined whether the face element is close enough to an existing face element. If so, it is merged into the existing face element. Otherwise, the face element is retained as a new face element.

[0132] For any point v in the current frame s , let the corresponding surface element be s, the spatial coordinates of the surface element are the coordinates of the point, the normal vector is the normal vector of the point, and the radius of the surface element is r s Calculated as follows:

[0133]

[0134] In this formula, the molecular ||v s || represents the distance from the point to the origin of the laser radar, so the numerator is the distance from the point to the origin multiplied by a proportional coefficient Where p is the pixel density per unit angle in the spherical coordinate system, which is a fixed parameter. The clamp(*) in the denominator is an interval-limited function, which limits the value range of the first term in the brackets to between 0.5 and 1.0; the first term in the brackets represents the cosine value of the angle between the laser line direction and the point normal vector direction, v s Represents the coordinates of the midpoint of the surface element, n s Represents the normal vector of the corresponding point.

[0135] Since the angle must be between 0 and 90 degrees, the cosine value range is 1-0. After clamp(*), the actual range of the denominator is 1.0 to 0.5. It can be inferred that the more parallel the normal vector is to the laser line, the smaller the angle, the larger the denominator, and the smaller the radius. In this formula, the numerator is equivalent to providing an initial radius, and the denominator adjusts this radius according to the angle between the normal vector and the laser line. If the laser hits the place more perpendicular to the laser line, the smaller the angle between the normal vector and the laser line, the larger the denominator, and the smaller the corresponding surface element radius; conversely, the surface element radius is larger.

[0136] To determine whether the face element is close enough to an existing face element, a beam of light is emitted along the direction of the laser line. The first face element hit in the existing map is the face element s' that is closest to the origin of the coordinates on the light path. The criterion for determining whether s and s' are close enough is:

[0137]

[0138] ||n s ×n s′ ||<sinθ M (12)

[0139] In formula (11), s' is the first surface element reached by extending the line of sight of the point in the current frame. is the transpose of the normal vector of the face element, v s′ is the center coordinate of the surface element, v s is the coordinate of the midpoint of the surface element in the current frame, δ M is the distance threshold, in formula (12) n s is the normal vector of the face in the current frame, n s′ is the normal vector of the first face element hit along the line of sight of the point in the current frame, θ M is the angle threshold;

[0140] This standard means that the positions of s and s' need to be close enough and their normal vectors point in the same direction. If they are not close enough, the probability of s' is reduced by a value and the new face s is inserted into the face map. If they are close enough, the probability of s' is increased by a value and s' is updated with the information of s, which is face fusion. The update formula is as follows:

[0141]

[0142]

[0143]

[0144] In formula (13) is the updated coordinate position, γ is the threshold, vs is the center coordinate of the surface element in the current frame, is the center coordinate of the first face element reached by extending the line of sight of the point in the current frame. In formula (14), is the updated normal vector, γ is the threshold, n s is the normal vector in the current frame, is the normal vector of the first face element hit along the line of sight of the point in the current frame. In formula (15), is the updated radius, r s is the radius of the surface element in the current frame.

[0145] Example 1: Use the urban scene data in the KITTI dataset platform to complete the map construction.

[0146] In order to verify the reliability of the algorithm, the urban scene in the KITTI dataset platform was selected for experimental simulation. The KITTI dataset was jointly created by the Karlsruhe Institute of Technology in Germany and Toyota America Technical Research Institute. It is one of the commonly used datasets in the international autonomous driving scene. The data acquisition platform of the KITTI dataset is equipped with 2 grayscale cameras, 2 color cameras, a Velodyne 64-line 3D laser radar, 4 optical lenses, and a GPS navigation system. The horizontal resolution of the 64-line laser radar is 0.2° to 0.4°, so the length of the designed depth image is 1028 and the width is set to 64. Since there are world coordinate systems and laser radar coordinate systems, an external parameter matrix is ​​required to convert the two. The external parameter matrix is ​​Tr_velo_to_cam, with a size of 3x4, which contains the rotation matrix R and the translation vector T. Multiplying the external parameter matrix by the point cloud coordinates can obtain the coordinates of the point cloud in the world coordinate system. The world coordinate system reflects the real position coordinates of the object. Here, the position of the first frame is set as the origin of the world coordinate system. Next is the step of entering the algorithm. First, the 3D point cloud is converted into a 1028*64 depth image according to the settings, and then the vector of each point in the image is calculated to obtain a vector map. The result is as follows Figure 2 After the above steps, the semantic information is extracted using deep learning methods, and then the point cloud is reconstructed. In this way, the point cloud has been given semantic information, such as Figure 3 As shown. Finally, the point cloud information is filtered out and the two semantic information is parameterized to obtain a refined semantic point cloud map; finally, the point cloud map is represented by the surface element, which greatly reduces the memory consumption, and the semantic surface element map is completed by combining the RTK real coordinate information. The demonstration results are shown in Figure 4 shown.

[0147] Example 2: Mapping using Jiangsu University scene data

[0148] In addition, the campus of Jiangsu University was selected as the experimental scene, and the intelligent driving test vehicle of Jiangsu University was used as the experimental data platform. The vehicle is equipped with a positioning system based on CORS differential technology combined with GPS and IMU. The perception system consists of an RS-Ruby Lite 80-line laser radar, two ibeo 4-line laser radars, a Delphi millimeter-wave radar, a SICK single-line laser radar, and two Gige fusion industrial cameras. The total length of the test lane is 500 meters, and the RS-Ruby Lite 80-line laser radar is used. The laser radar has 80 laser emitters and can cover a 360-degree horizontal viewing angle and a 40-degree vertical field of view. A fixed mark is set at the starting point, and RTK is used to collect the position information of the map node and the test point to be located from the fixed mark in the experiment. This information is regarded as the position information of the point in the world coordinate system in the experiment. Starting from one end of the road, a set of point cloud information is collected every 20 meters as a map node and the distance from the fixed mark at the starting point is measured to collect the node position information and generate the initial point cloud data map library.

[0149] According to the method described above, a two-dimensional depth map V is generated for each node in the point cloud database. D 、Vector graph N D , semantic information, and complete the feature layer and semantic information layer of the multi-level map. The specific details are as follows: first, the point cloud is projected into a sphere, and then all points are represented by (θ, ψ, depth), and then (θ, ψ) is mapped to two-dimensional coordinates (u, v), and depth is mapped to pixel values, thus obtaining the depth map V D , and then scale and normalize to get the coordinates. Next, we will take the depth map V D Calculate the normal vector of each point in the vector graph, and the normal vectors of all points form a vector graph, denoted by N D The semantic information is extracted using RangeNet++, and the subsequent 3D reconstruction is parameterized to obtain a point cloud map library.

[0150] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0151] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for producing a semantic surfel map based on laser radar, characterized in that: include: S1: Data collection of semantic surfel maps; S1.1: Use laser radar to obtain laser point cloud information in urban environments. The point cloud data content includes the three-dimensional coordinates and laser reflection intensity of each point of the scanned object in the urban environment; S1.2: Use RTK real-time dynamic positioning technology to obtain the real coordinate information of the vehicle in real time; S1.3: Preprocess the collected laser point cloud information, organize the obtained laser point cloud data and their corresponding trajectory information, and complete the original data set; S2: Create semantic surfel maps by level; S2.1: Project the points in the three-dimensional space onto the two-dimensional depth map to obtain the two-dimensional depth map V D ; S2.2: The depth map V D Calculate the normal vector of each point in the vector graph N. D ; S2.3: Use deep learning methods for semantic segmentation. Use the laser point cloud semantic segmentation method to perform 2D image full convolution semantic segmentation on the depth image and then restore the semantic conversion of all points from 2D to 3D from the original point cloud. Reconstruct the point cloud based on the effective depth image, and eliminate the artifacts of undesired discretization and reasoning based on the KNN nearest neighbor classification method search. After processing the point cloud data, classify the results; S2.4: Filter the classification results and parameterize the filtered semantic features; S2.5: Record the real coordinates of the vehicle in the world coordinate system and express the real coordinates of the vehicle trajectory in a 4×4 matrix; S3: Use surfels to represent point cloud maps and combine them with semantic information to form semantic surfel maps; S3.1: Based on the basic elements of the semantic surface map obtained in the above steps, the real coordinate information of the vehicle is obtained in real time using RTK real-time dynamic positioning technology. After obtaining the position and posture of each frame, the current frame is then fused into the map; S3.2: For each point, the corresponding face element is calculated, and then it is determined whether the face element is close enough to an existing face element. If so, it is merged into the existing face element. Otherwise, the face element is retained as a new face element.

2. The method for producing a semantic surfel map based on laser radar according to claim 1, characterized in that: The S2.1 is as follows: first, project the three-dimensional points onto the sphere, with all points represented by (θ, ψ, depth), then map (θ, ψ) to two-dimensional coordinates (u, v), and map depth to pixel values, thereby obtaining a depth map V. D ; Converting three-dimensional coordinates into two-dimensional coordinates specifically includes: For a rotating scanning laser radar, its vertical field of view FOV is divided into two parts: FOV_up and FOV_down. The value of FOV_up is a positive number and the value of FOV_down is a negative number, so FOV = FOV_up + (-FOV_down); Each point in the laser radar point cloud with three-dimensional Cartesian coordinates is (X, Y, Z). After unfolding the cylinder obtained by rotating the laser radar for one circle with the x-axis direction as the front view, an image is obtained: the origin of the coordinate is at the center of the image, the ordinate of the pixel in the image is obtained by the projection of the pitch angle, the range is [FOV_down, FOV_up], and the abscissa is obtained by the projection of the deflection angle, the range is [-π, π]; pitch1=sin -1 (Z / R) (2) yaw1=tan -1 (Y / X) (3) In formula (1), R is the distance from each point collected by the laser radar to the origin, X, Y, Z are the coordinates of the point, in formula (2), pitch1 is the pitch angle, Z is the point cloud coordinate in the Z-axis direction, and in formula (3), yaw1 is the deflection angle; The image coordinate system uses the upper left corner as the origin. Transform the front view obtained above and move the origin to the upper left corner: pitch2=FOV_Up-pitch1 (4) yaw2=yaw1+π (5) In formula (4), pitch2 is the converted pitch angle, FOV_Up is the vertical resolution of the upper field of view, and in formula (5), yaw1 is the deflection angle, and yaw2 is the converted deflection angle; Normalize yaw2 and pitch2, and after normalization, multiply them by the width and height of the projected image to obtain the coordinates of this point projected onto the depth image. pitch3=(FOV_Up-pitch2) / (FOV_Up-FOV_Down) (6) yaw3=(yaw2+π) / 2π (7) In formula (6), pitch3 is the normalized and scaled pitch angle, FOV_Down is the vertical resolution of the lower field of view, in formula (7), yaw3 is the normalized and scaled yaw angle, in formula (8), u and v are the normalized and scaled coordinate values, z is the coordinate of the point cloud on the Z axis, R is the distance between the point scanned by the laser and the coordinate origin, row_scale is the width of the converted depth image, and col_scale is the length of the depth map.

3. The method for producing a semantic surfel map based on laser radar according to claim 1, characterized in that: S2.2 is specifically as follows: Calculate the normal vector of a random point, which is the outer product of the vector from the point to the adjacent point on the right and the vector from the point to the adjacent point on the upper side after the cross product of the two. The calculation formula is as follows: N D (u,v)=[V D (u+1,v)-V D (u,v+1)]×[V D (u,v+1)-V D (u,v)] (9) N in formula (9) D is the normal vector of each point, u, v are the two-dimensional coordinates calculated in formula (8), V D refers to the depth map coordinates, and "×" refers to the cross product.

4. The method for producing a semantic surfel map based on laser radar according to claim 1, characterized in that: Specifically, S2.3 is as follows: after processing the point cloud data, the convolutional neural network is used for identification and classification to obtain the following results, including: traffic signal signs, columnar objects, roads, sidewalks, moving vehicles, stopped vehicles, buildings, and people.

5. The method for producing a semantic surfel map based on laser radar according to claim 4, characterized in that: S2.4 specifically includes: filtering the classification results obtained in S2.3, and retaining the point cloud data of the traffic signal sign, columnar object, road and building; The filtering of the classification results is specifically to perform secondary classification of these categories by a dynamic object filtering method, the method is as follows: When the new laser point passes through the position of the old laser point, the old laser point is a dynamic point; each time the new laser overlaps with the old laser, the number of misses is increased by one, and thresholds φ and ω are set. When the accumulated number of misses is greater than φ, it is marked as a high-dynamic object; when the accumulated number of misses is less than φ and greater than ω, it is marked as a low-dynamic object; when the accumulated number of misses is less than ω, it is marked as an object to be processed; The above classification method is used to filter out moving objects, including moving vehicles and people; The remaining classification results are processed according to the environmental object classification, and the objects to be processed are divided into semi-static objects and static objects according to the dynamic degree of the objects. Semi-static objects include stopped vehicles, and static objects include traffic signs, columnar objects, roads and buildings; The semi-static objects are filtered out through the above classification method, and traffic signal signs, columnar objects, roads and buildings are retained; The parameterized semantic features of traffic signal signs are segmented twice using Euclidean clustering and extracted using the RANSAC method; For parameterized semantic feature cylindrical objects, Euclidean clustering is used to fit a 3D straight line, which is represented by the coordinates and direction vector of the point on the rod closest to the origin.

6. The method for producing a semantic surfel map based on laser radar according to claim 1, characterized in that: S3.2 is specifically as follows: for any point v in the current frame s , let the corresponding surface element be s, the spatial coordinates of the surface element are the coordinates of the point, the normal vector is the normal vector of the point, and the radius of the surface element is r s Calculated as follows: In formula (10), the numerator ‖v s ‖ is the distance from the point to the origin of the laser radar, p is the pixel density per unit angle in the spherical coordinate system, and v s is the coordinate of the midpoint of the surface element, n s is the normal vector of the corresponding point; To determine whether the face element is close enough to an existing face element, a beam of light is emitted along the direction of the laser line. The first face element hit in the existing map is the face element s' that is closest to the origin of the coordinates on the light path. The criterion for determining whether s and s' are close enough is: ‖n s ×n s′ ‖<sinθ M (12) In formula (11), s' is the first surface element reached by extending the line of sight of the point in the current frame. is the transpose of the normal vector of the face element, v s′ is the center coordinate of the surface element, v s is the coordinate of the midpoint of the surface element in the current frame, δ M is the distance threshold, in formula (12) n s is the normal vector of the face in the current frame, n s′ is the normal vector of the first face element hit along the line of sight of the point in the current frame, θ M is the angle threshold; If it is judged to be close enough, the probability of s' is accumulated to a value, and s' is updated with the information of s. The update formula is as follows: In formula (13) is the updated coordinate position, γ is the threshold, v s is the center coordinate of the surface element in the current frame, is the center coordinate of the first face element reached by extending the line of sight of the point in the current frame. In formula (14), is the updated normal vector, γ is the threshold, n s is the normal vector in the current frame, is the normal vector of the first face element hit along the line of sight of the point in the current frame. In formula (15), is the updated radius, r s is the radius of the surface element in the current frame.

Images