Lidar-based Lane Line Extraction Method, Device, Medium and Equipment

Abstract

The present invention discloses a method for extracting lane lines based on lidar, including: acquiring image data collected by a camera and lidar point cloud data collected by a lidar; performing time synchronization on the image data and the lidar point cloud data; for the time-synchronized image frame and lidar point cloud frame, mapping the pixel colors on the image frame to the corresponding point clouds of the lidar point cloud frame; acquiring the pose information of the lidar point cloud frame, and splicing the lidar point cloud frame according to the pose information to obtain a colored point cloud map; extracting ground point information according to the colored point cloud map; and extracting lane lines according to the ground point information and a color threshold. The present invention realizes low-cost lane line extraction, does not need to rely on a deep learning model, has a small amount of calculation, and can realize real-time lane line extraction.

Description

Technical Field

[0001] The present invention relates to the technical field of robots, and in particular, to a lane line extraction method, device, medium and equipment based on a lidar. Background Art

[0002] There are mainly two existing lane line extraction methods. One is a pure vision extraction method, which equips a robot with a camera, and the camera can be a monocular camera, a binocular camera or a fisheye camera, and deep learning training is performed on the image information collected by the camera to extract lane lines. The other is a laser reflection intensity extraction method. By using a high-beam lidar, such as a 128-beam lidar, based on the principle that the reflection intensities of different materials are different, lane lines can be extracted according to the different reflection intensities between the ground and the lane lines.

[0003] However, the first extraction method relies on a deep learning model, which requires manual annotation during training, has high time and labor costs, and a large amount of calculation, so the robot cannot extract lane lines in real time; the high-beam lidar used in the second extraction method has the problem of high cost. Summary of the Invention

[0004] Embodiments of the present invention provide a lane line extraction method, device, medium and equipment based on a lidar to solve the problems of large calculation amount and inability to extract in real time, and high cost existing in the existing lane line extraction methods.

[0005] A lane line extraction method based on a lidar, the method comprising:

[0006] Obtaining image data collected by a camera and laser point cloud data collected by a lidar;

[0007] Performing time synchronization on the image data and the laser point cloud data;

[0008] For the time-synchronized image frame and laser point cloud frame, mapping the pixel colors on the image frame to the corresponding point cloud on the laser point cloud frame;

[0009] Obtaining the pose information of the laser point cloud frame, and splicing the laser point cloud frame according to the pose information to obtain a colored point cloud map;

[0010] Extracting ground point information according to the colored point cloud map;

[0011] Extracting lane lines according to the ground point information and a color threshold.

[0012] Optionally, the performing time synchronization on the image data and the laser point cloud data includes:

[0013] For the i-th laser point cloud frame in the laser point cloud data, obtain the acquisition time of the i-th laser point cloud frame;

[0014] Obtain the two adjacent image frames closest to the acquisition time from the image data and their acquisition times, and denote them as the k-th image frame and the (k + 1)-th image frame respectively;

[0015] If the difference between the acquisition time of the (k + 1)-th image frame and the acquisition time of the i-th laser point cloud frame is greater than the difference between the acquisition time of the i-th laser point cloud frame and the acquisition time of the k-th image frame, then synchronize the i-th laser point cloud frame with the k-th image frame;

[0016] If the difference between the acquisition time of the (k + 1)-th image frame and the acquisition time of the i-th laser point cloud frame is less than the difference between the acquisition time of the i-th laser point cloud frame and the acquisition time of the k-th image frame, then synchronize the i-th laser point cloud frame with the (k + 1)-th image frame.

[0017] Optionally, for the image frames and laser point cloud frames after time synchronization, mapping the pixel colors on the image frames to the corresponding point clouds on the laser point cloud frames includes:

[0018] Calculate the image height field of view angle according to the width of the image frame and the first internal parameter coefficient of the camera;

[0019] Calculate the image width field of view angle according to the height of the image frame and the second internal parameter coefficient of the camera;

[0020] Convert the radar coordinates of each point in the radar coordinate system to the camera coordinates in the camera coordinate system;

[0021] Determine whether the camera coordinates fall within the range formed by the image height field of view angle and the image width field of view angle;

[0022] If so, obtain the color value corresponding to this point from the image frame, and assign the color value to the point cloud corresponding to the laser point cloud frame.

[0023] Optionally, the determination of whether the camera coordinates fall within the range formed by the image height field of view angle and the image width field of view angle includes:

[0024] Calculate the first sine value and the second sine value of this point according to the converted camera coordinates of each point in the radar coordinate system;

[0025] Determine whether the first sine value is less than the positive image height field of view angle, greater than the negative image height field of view angle, and whether the second sine value is less than the positive image width field of view angle and greater than the negative image width field of view angle.

[0026] Optionally, the extraction of ground point information based on the color point cloud map includes:

[0027] Obtain the point cloud scanned by the same laser beam from the color point cloud map;

[0028] Arrange the measurement distances corresponding to the point cloud scanned by the same laser beam in chronological order;

[0029] Traverse each laser beam emitted by the lidar to obtain an M*N azimuth matrix, where M represents the number of laser beams of the lidar and N represents the number of point clouds;

[0030] Calculate the first angle between two point clouds in the same column of the azimuth matrix, and the first angle represents the angle between the laser beam to the latter point cloud and the line connecting the two point clouds;

[0031] Obtain ground points according to the first angle and construct ground point information.

[0032] Optionally, the obtaining of ground points according to the first angle and constructing ground point information includes:

[0033] Compare the first angle with a preset angle threshold;

[0034] When the first angle is less than the preset angle threshold, the two point clouds are considered ground points, otherwise, the two point clouds are not ground points.

[0035] Optionally, the extraction of lane lines according to the ground point information and color threshold includes:

[0036] Obtain the color value of the ground point and compare the color value of the ground point with a preset color range;

[0037] If the color value of the ground point falls within the preset color range, the ground point is a lane line, otherwise it is not a lane line;

[0038] Traverse each ground point and obtain all ground points that fall within the preset color range to obtain lane lines.

[0039] A lane line extraction device based on lidar, the device includes:

[0040] An acquisition module for acquiring image data collected by a camera and laser point cloud data collected by a lidar;

[0041] A synchronization module for synchronizing the image data and the laser point cloud data in time;

[0042] A mapping module, configured to map the pixel colors on the image frame to the corresponding point clouds on the lidar point cloud frame for the image frame and the lidar point cloud frame after time synchronization;

[0043] A stitching module, configured to obtain the pose information of the lidar point cloud frame, and stitch the lidar point cloud frame according to the pose information to obtain a colored point cloud map;

[0044] A ground extraction module, configured to extract ground point information according to the colored point cloud map;

[0045] A lane line extraction module, configured to extract lane lines according to the ground point information and a color threshold.

[0046] A computer-readable storage medium storing a computer program, where when the computer program is executed by a processor, the above-mentioned lidar-based lane line extraction method is implemented.

[0047] A terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where when the processor executes the computer program, the above-mentioned lidar-based lane line extraction method is implemented.

[0048] In an embodiment of the present invention, a low-beam lidar is adopted. By acquiring image data collected by a camera and lidar point cloud data collected by the lidar; performing time synchronization on the image data and the lidar point cloud data; for the image frame and the lidar point cloud frame after time synchronization, mapping the pixel colors on the image frame to the corresponding point clouds on the lidar point cloud frame; obtaining the pose information of the lidar point cloud frame, and stitching the lidar point cloud frame according to the pose information to obtain a colored point cloud map; extracting ground point information according to the colored point cloud map; extracting lane lines according to the ground point information and a color threshold; thus realizing low-cost lane line extraction, without relying on a deep learning model, with a small amount of computation, and enabling real-time lane line extraction. Description of the Drawings

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required to be used in the description of the embodiments of the present invention. Obviously, the following drawings are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other drawings based on these drawings without creative efforts.

[0050] Figure 1 It is a flowchart of the implementation of the lidar-based lane line extraction method provided by an embodiment of the present invention;

[0051] Figure 2 It is a schematic diagram of a first included angle provided by an embodiment of the present invention;

[0052] Figure 3 It is a schematic diagram of a lane line extraction device based on lidar provided by an embodiment of the present invention;

[0053] Figure 4 It is a schematic diagram of a computer device in an embodiment of the present invention. Detailed implementation manners

[0054] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0055] The embodiment of the present invention uses a low-beam lidar, combines the image data collected by the camera and the laser point cloud data collected by the lidar, and synchronizes the time of the image data and the laser point cloud data; for the synchronized image frame and laser point cloud frame, map the pixel color on the image frame to the corresponding point cloud on the laser point cloud frame; obtain the pose information of the laser point cloud frame, splice the laser point cloud frame according to the pose information to obtain a color point cloud map; extract ground point information according to the color point cloud map; extract lane lines according to the ground point information and color threshold; thus realizing low-cost lane line extraction, without relying on deep learning models, with small computational complexity, and real-time lane line extraction can be achieved.

[0056] In the embodiment of the present invention, the lane line extraction method based on lidar is applied to a robot that can navigate autonomously. The robot is equipped with a camera, a 3D lidar, an inertial measurement unit IMU, and a wheel odometer. The camera can be a monocular camera, a binocular camera, or a fish-eye camera.

[0057] Before implementing the lane line extraction based on lidar provided by the embodiment of the present invention, it is first necessary to calibrate the internal parameters of the camera to obtain the internal parameters and distortion coefficients of the camera. Taking a monocular camera as an example, first prepare a 9×7 camera calibration board, move the calibration board in the visible area of the current monocular camera, including rotating and translating the calibration board. The world coordinate system is at the upper left corner of the calibration board. If the coordinates of the feature points on the calibration board are known and pixel coordinates Substitute into the following formula: where R represents the rotation from the world coordinate system to the camera coordinate system, and t represents the translation information from the world coordinate system to the camera coordinate system. Obtain the coordinates of multiple feature points and substitute them into the above formula, and solve the parameters f x 、f y 、cx , c y , where f x represents the scaling factor of the horizontal direction x in the image coordinate system, and f y represents the scaling factor of the vertical direction y in the image coordinate system, and c x represents the translation in the x direction relative to the center of the image coordinate system, and c y represents the translation in the y direction relative to the center of the image coordinate system. These four parameters constitute the camera internal parameters.

[0058] The tangential distortion parameters are solved according to the following formula:

[0059] X corrected = X(1 + k 1 r 2 + k 2 r 4 + k 3 r 6 ) + 2p 1 XY + p 2 (r 2 + 2X 2 )

[0060] Y corrected = Y(1 + k 1 r 2 + k 2 r 4 + k 3 r 6 ) + 2p 2 XY + p 1 (r 2 + 2Y 2 )

[0061] Among them, (X, Y) represents the pixel coordinates before distortion correction, X corrected and Y corrected respectively represent the pixel coordinates after distortion correction, r represents the distance between the point in the real world and the camera coordinate system, and p 1 , p 2 respectively represent the tangential distortion parameters, and k 1 , k 2 , k 3 represent the constant coefficients in the formula.

[0062] The following provides a detailed description of the lane line extraction method based on lidar provided by the embodiments of the present invention. Figure 1 This is the lane line extraction method based on lidar provided by the embodiments of the present invention. As Figure 1 shown, the lane line extraction method based on lidar includes:

[0063] In step S101, obtain the image data collected by the camera and the laser point cloud data collected by the lidar.

[0064] Here, in the embodiments of the present invention, the image data is obtained from the camera mounted on the robot, and the laser point cloud data is obtained from the 3D lidar. Among them, the image data is stored in a message queue, and the laser point cloud data is stored in another message queue.

[0065] In step S102, synchronize the image data and the laser point cloud data in terms of time.

[0066] Here, if laser point cloud data and camera data are both required simultaneously, it is necessary to synchronize the two, and find the camera data closest to the laser point cloud data at the current moment. Optionally, step S102 includes:

[0067] In step S201, for the i-th laser point cloud frame in the laser point cloud data, obtain the acquisition time of the i-th laser point cloud frame.

[0068] Traverse the message queue of the laser point cloud data. For each laser point cloud frame, there is a corresponding acquisition time indicating the acquisition time of the i-th laser point cloud frame in the message queue of the laser point cloud data.

[0069] In step S202, obtain the two image frames closest to the acquisition time and their acquisition times before and after from the image data, and denote them as the k-th image frame and the (k + 1)-th image frame respectively.

[0070] From the message queue of the image data, find the previous image frame and the next image frame closest to the acquisition time of the i-th laser point cloud frame. Here, they are denoted as the k-th image frame and the (k + 1)-th image frame, and the corresponding acquisition times are and

[0071] In step S203, if the difference between the acquisition time of the (k + 1)-th image frame and the acquisition time of the i-th laser point cloud frame is greater than the difference between the acquisition time of the i-th laser point cloud frame and the acquisition time of the k-th image frame, then synchronize the i-th laser point cloud frame with the k-th image frame.

[0072] In step S204, if the difference between the acquisition time of the (k + 1)-th image frame and the acquisition time of the i-th laser point cloud frame is less than the difference between the acquisition time of the i-th laser point cloud frame and the acquisition time of the k-th image frame, then synchronize the i-th laser point cloud frame with the (k + 1)-th image frame.

[0073] Here, the embodiments of the present invention perform time alignment through the following formula:

[0074]

[0075] That is, calculate the time differences between the acquisition time of the i-th laser point cloud frame and the acquisition times of the two adjacent image frames respectively, compare the two differences with each other, determine which image frame the i-th laser point cloud frame is closer to, and then perform time synchronization between the image frame and the i-th laser point cloud frame.

[0076] Traverse each laser point cloud frame in the laser point cloud message queue to complete the time synchronization between each laser point cloud frame and the image frame.

[0077] After completing the time synchronization, the embodiments of the present invention perform external parameter calibration based on the synchronized laser point cloud frames and image frames. Similarly, prepare a calibration board and place it in a place where both the camera and the lidar can see it. Rotate and translate the calibration board to obtain multiple data frames. The camera detects the corner points on the calibration board and extracts the plane normal vector of the calibration board. At the same time, use the lidar to perform plane extraction to obtain the laser point coordinates and normal vectors on the plane, and construct an error function: n cT (R cl P l +t cl )+d c =0. Wherein, n cT represents the normal vector in the camera coordinate system, R cl represents the rotation relationship from the camera coordinate system to the lidar coordinate system, p l represents the laser point on the calibration board plane in the lidar coordinate system, t cl represents the translation relationship from the camera coordinate system to the lidar coordinate system, and d c represents the distance from the origin of the camera coordinate system to the calibration board plane.

[0078] By minimizing the above error function, the optimal rotation relationship R cl and translation relationship t cl are obtained, and further the external parameter calibration from the camera to the lidar is completed.

[0079] In step S103, for the synchronized image frames and laser point cloud frames, map the pixel colors on the image frames to the corresponding point clouds of the laser point cloud frames.

[0080] Optionally, as a preferred example of the present invention, step 103 includes:

[0081] In step S301, calculate the image height field of view angle according to the width of the image frame and the first internal parameter coefficient of the camera.

[0082] Here, the first internal reference coefficient is the scaling factor f of the horizontal direction x in the image coordinate system x , and the embodiment of the present invention uses the formula to calculate the image height field of view angle. Wherein, W represents the width of the image frame

[0083] In step S302, according to the height of the image frame and the second internal reference coefficient of the camera, calculate the image width field of view angle

[0084] The second internal reference coefficient is the scaling factor f of the vertical direction y in the image coordinate system y , and the embodiment of the present invention uses the formula to calculate the image width field of view angle. Wherein, H represents the height of the image frame

[0085] In step S303, convert the radar coordinates of each point in the radar coordinate system into camera coordinates in the camera coordinate system

[0086] For each point in the radar coordinate system, obtain its radar coordinates and convert them into camera coordinates. The conversion formula is: P c =R cl P l +t cl , R cl represents the rotation relationship from the camera coordinate system to the lidar coordinate system, p l represents the laser point on the calibration plate plane in the lidar coordinate system, t cl represents the translation relationship from the camera coordinate system to the lidar coordinate system, p c represents the coordinates of the point in the camera coordinate system, that is, the camera coordinates

[0087] In step S304, determine whether the camera coordinates fall within the range formed by the image height field of view angle and the image width field of view angle

[0088] Here, the range FOV formed by the image height field of view angle and the image width field of view angle represents the field of view range of the camera. Since the lidar rotates 360°, there will definitely be some points that are co-visible with the camera. If they are within the co-visible range, that is, within the camera's field of view range, they are retained. If they are not within the co-visible range, that is, outside the camera's field of view range, they are discarded. Optionally, as a preferred example of the present invention, step S304 further includes

[0089] In step S3041, calculate the first sine value and the second sine value of each point according to the converted camera coordinates in the radar coordinate system

[0090] Here, the first sine value is The second sine value is

[0091] In step S3042, it is determined whether the first sine value is less than the positive image height field of view angle and greater than the negative image height field of view angle, and whether the second sine value is less than the positive image width field of view angle and greater than the negative image width field of view angle.

[0092] The range formed by the image height field of view angle and the image width field of view angle means greater than the negative image height field of view angle and less than the positive image height field of view angle, and greater than the negative image width field of view angle and less than the positive image width field of view angle.

[0093] Determine the first sine value F x Whether it is less than the positive image height field of view angle FOV H And greater than the negative image height field of view angle -FOV H , and the second sine value F y Whether it is less than the positive image width field of view angle FOV w And greater than the negative image width field of view angle -FOV w .

[0094] In step S305, if so, obtain the color value corresponding to this point from the image frame, and assign the color value to the point cloud corresponding to the laser point cloud frame.

[0095] Here, if the first sine value F x Is less than the positive image height field of view angle FOV H And greater than the negative image height field of view angle -FOV H , and the second sine value F y Is less than the positive image width field of view angle FOV w And greater than the negative image width field of view angle -FOV w , it is considered that the camera coordinates fall within the range formed by the image height field of view angle and the image width field of view angle. Obtain the corresponding pixel color value from the image frame according to the camera coordinates, and assign the color value to the point cloud corresponding to the laser point cloud frame. Optionally, the color value is preferably an RGB color. At this time, the point cloud can reflect the color of the real world.

[0096] If the first sine value F x Is less than the negative image height field of view angle -FOV H Or the first sine value is greater than the positive image height field of view angle FOV H Or the second sine value F y Is less than the negative image width field of view angle -FOV w Or the second sine value F y Is greater than the positive image width field of view angle FOV w , that is, Fx <-FOV H or F x >FOV H or F y <-FOV w or F y >FOV w If so, it is considered that the camera coordinates do not fall within the range formed by the image height field of view angle and the image width field of view angle, and the camera coordinates are discarded.

[0097] Traverse all camera coordinates, map the colors corresponding to the pixels of the image frame to the point cloud, and complete the coloring of the laser point cloud frame.

[0098] In step S104, obtain the pose information of the laser point cloud frame, and splice the laser point cloud frame according to the pose information to obtain a colored point cloud map.

[0099] Here, in the embodiment of the present invention, mapping is performed on the colored laser point cloud frame according to the pose information. Optionally, in the embodiment of the present invention, the pose information of the laser point cloud frame is calculated according to the rotation quaternion of the measurement value provided by the inertial measurement unit IMU and the displacement of the measurement value provided by the wheel odometer. Specifically, assume that the measurement value of the IMU of the i-th laser point cloud frame at the current moment is A i , and the measurement value of the wheel odometer is B i , and the measurement value of the IMU of the (i - 1)-th laser point cloud frame at the previous moment is A i-1 , and the measurement value of the wheel odometer is B i-1 , then the pose information of the i-th laser point cloud frame at the current moment is obtained through the following formula:

[0100] △B = B i - B i-1

[0101] A = A 1 -1 A i

[0102]

[0103] Among them, Podom is the pose information of the i-th laser point cloud frame at the current moment. All laser point cloud frames are spliced using the pose information, so as to obtain a point cloud map with real-world colors, that is, a colored point cloud map.

[0104] In step S105, extract ground point information according to the colored point cloud map.

[0105] Since the color point cloud map established through step S104 can reflect the colors of the real world, the colors of the lane lines are also presented on the color point cloud map. In the embodiments of the present invention, ground point information is first extracted from the color point cloud map to reduce the amount of data processing, improve the speed of lane line extraction, and achieve real-time lane line extraction. Optionally, step S105 further includes:

[0106] In step S501, obtain the point cloud scanned by the same laser beam from the color point cloud map.

[0107] In step S502, arrange the measured distances corresponding to the point cloud scanned by the same laser beam in chronological order.

[0108] In step S503, traverse each laser beam emitted by the lidar to obtain an M*N azimuth matrix, where M represents the number of laser beams of the lidar and N represents the number of point clouds.

[0109] Here, for a 16-line lidar, M = 16, and the number of point clouds N can be set according to specific circumstances and is not limited here. Exemplarily, in the embodiments of the present invention, the point cloud scanned by each laser beam of the lidar is arranged to obtain a 16*2000 azimuth matrix, where each element in the azimuth matrix is the measured distance obtained by laser beam scanning, and all the elements in the same row are the measured distances corresponding to the point cloud scanned by the same laser beam.

[0110] In step S504, calculate the first angle between the two point clouds according to the measured distances of the two point clouds in the same column of the azimuth matrix.

[0111] The first angle is the angle between the laser beam to the latter point cloud and the line connecting the two point clouds, representing the inclination angle of a preset obstacle. For ease of understanding, Figure 2 is a schematic diagram of the first angle provided by the embodiments of the present invention, where represents the first angle, and C1 and C2 represent two point clouds in the same column. In the embodiments of the present invention, the first angle is calculated according to the measured distances of the two point clouds in the same column in the azimuth matrix.

[0112] In step S505, obtain ground points according to the first angle and construct ground point information.

[0113] Here, in the embodiments of the present invention, an angle threshold is preset in advance, and ground points are obtained according to the comparison result between the first angle and the angle threshold. Optionally, step S505 further includes:

[0114] In step S5051, compare the first angle with a preset angle threshold.

[0115] The preset angle threshold can be obtained through testing by an experimental device, and the specific influencing factors are related to the installation angle of the laser. Optionally, the preset angle threshold is preferably 5 degrees. Compare the first angle with 5 degrees.

[0116] In step S5052, when the first included angle is less than the preset angle threshold, it is considered that the two point clouds are ground points; otherwise, the two point clouds are not ground points.

[0117] Here, if the first included angle is less than 5 degrees, it is considered that the two point clouds are ground points; otherwise, the two point clouds are not ground points. Traverse the two point clouds on the same column to obtain ground point information.

[0118] In step S106, extract the lane lines according to the ground point information and the color threshold.

[0119] After obtaining the ground point information, the embodiment of the present invention extracts the lane lines based on the ground point information and the color threshold. Step S106 further includes:

[0120] In step S601, obtain the color value of the ground point, and compare the color value of the ground point with a preset color range.

[0121] Here, the embodiment of the present invention pre-sets the color range according to the conventional color of the lane line. Then, compare the color value of each ground point in the ground point information extracted through step S105 with the preset color range.

[0122] In step S602, if the color value of the ground point falls within the preset color range, the ground point is a lane line; otherwise, it is not a lane line.

[0123] In step S603, traverse each ground point, and obtain all the ground points that fall within the preset color range to obtain the lane lines.

[0124] If the color value of the ground point falls within the given preset color range, it is a lane line; otherwise, it is not a lane line. Traverse all the ground points and extract the lane lines therefrom.

[0125] In summary, the embodiments of the present invention adopt a low-beam lidar. By acquiring the image data collected by a camera and the lidar point cloud data; synchronizing the image data and the lidar point cloud data in time; for the image frame and the lidar point cloud frame after time synchronization, mapping the pixel colors on the image frame to the corresponding point clouds on the lidar point cloud frame; obtaining the pose information of the lidar point cloud frame, and stitching the lidar point cloud frame according to the pose information to obtain a color point cloud map; extracting ground point information according to the color point cloud map; and extracting lane lines according to the ground point information and a color threshold, thereby realizing low-cost lane line extraction, without relying on a deep learning model, with a small amount of computation, and enabling real-time lane line extraction.

[0126] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The order of execution of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0127] In one embodiment, the present invention further provides a lane line extraction device based on a lidar. The lane line extraction device based on a lidar corresponds one-to-one with the lane line extraction method based on a lidar in the above embodiment. As Figure 3 shown, the lane line extraction device based on a lidar includes an acquisition module 31, a synchronization module 32, a mapping module 33, a stitching module 34, a ground extraction module 35, and a lane line extraction module 36. The detailed descriptions of each functional module are as follows:

[0128] The acquisition module 31 is configured to acquire the image data collected by a camera and the lidar point cloud data;

[0129] The synchronization module 32 is configured to synchronize the image data and the lidar point cloud data in time;

[0130] The mapping module 33 is configured to, for the image frame and the lidar point cloud frame after time synchronization, map the pixel colors on the image frame to the corresponding point clouds on the lidar point cloud frame;

[0131] The stitching module 34 is configured to obtain the pose information of the lidar point cloud frame, and stitch the lidar point cloud frame according to the pose information to obtain a color point cloud map;

[0132] The ground extraction module 35 is configured to extract ground point information according to the color point cloud map;

[0133] The lane line extraction module 36 is configured to extract lane lines according to the ground point information and a color threshold.

[0134] Optionally, the synchronization module 32 includes:

[0135] The first acquisition unit is configured to acquire the acquisition time of the \(i\)th laser point cloud frame in the laser point cloud data.

[0136] The second acquisition unit is configured to acquire the two image frames before and after closest to the acquisition time from the image data, and their acquisition times, which are respectively denoted as the \(k\)th image frame and the \((k + 1)\)th image frame.

[0137] The first synchronization unit is configured to synchronize the \(i\)th laser point cloud frame with the \(k\)th image frame if the difference between the acquisition time of the \((k + 1)\)th image frame and the acquisition time of the \(i\)th laser point cloud frame is greater than the difference between the acquisition time of the \(i\)th laser point cloud frame and the acquisition time of the \(k\)th image frame.

[0138] The second synchronization unit is configured to synchronize the \(i\)th laser point cloud frame with the \((k + 1)\)th image frame if the difference between the acquisition time of the \((k + 1)\)th image frame and the acquisition time of the \(i\)th laser point cloud frame is less than the difference between the acquisition time of the \(i\)th laser point cloud frame and the acquisition time of the \(k\)th image frame.

[0139] Optionally, the mapping module 33 includes:

[0140] The first calculation unit is configured to calculate the image height field of view angle according to the width of the image frame and the first internal parameter coefficient of the camera.

[0141] The second calculation unit is configured to calculate the image width field of view angle according to the height of the image frame and the second internal parameter coefficient of the camera.

[0142] The conversion unit is configured to convert the radar coordinates of each point in the radar coordinate system into the camera coordinates in the camera coordinate system.

[0143] The judgment unit is configured to judge whether the camera coordinates fall within the range formed by the image height field of view angle and the image width field of view angle.

[0144] The mapping unit is configured to, if so, acquire the color value corresponding to the point from the image frame, and assign the color value to the point cloud corresponding to the laser point cloud frame.

[0145] Optionally, the judgment unit is specifically configured to:

[0146] Calculate the first sine value and the second sine value of each point according to the converted camera coordinates of each point in the radar coordinate system.

[0147] Judge whether the first sine value is less than the positive image height field of view angle, greater than the negative image height field of view angle, and whether the second sine value is less than the positive image width field of view angle and greater than the negative image width field of view angle.

[0148] Optionally, the ground extraction module 35 includes:

[0149] A point cloud acquisition unit, configured to acquire point clouds scanned by the same laser beam from the color point cloud map;

[0150] An arrangement unit, configured to arrange the measured distances corresponding to the point clouds scanned by the same laser beam in chronological order;

[0151] A matrix acquisition unit, configured to traverse each laser beam emitted by the lidar to obtain an M*N azimuth matrix, where M represents the number of laser beams of the lidar and N represents the number of point clouds;

[0152] An included angle calculation unit, configured to calculate a first included angle between two point clouds according to the measured distances of the two point clouds in the same column of the azimuth matrix, where the first included angle represents the included angle between the laser beam to the latter point cloud and the line connecting the two point clouds;

[0153] A construction unit, configured to obtain ground points according to the first included angle and construct ground point information.

[0154] Optionally, the construction unit is specifically configured to:

[0155] Compare the first included angle with a preset angle threshold;

[0156] When the first included angle is less than the preset angle threshold, the two point clouds are considered ground points, otherwise, the two point clouds are not ground points.

[0157] Optionally, the lane line extraction module 36 is specifically configured to:

[0158] Obtain the color value of the ground point, and compare the color value of the ground point with a preset color range;

[0159] If the color value of the ground point falls within the preset color range, the ground point is a lane line, otherwise it is not a lane line;

[0160] Traverse each ground point, and obtain all ground points that fall within the preset color range to obtain lane lines.

[0161] For the specific limitations of the lane line extraction device based on lidar, reference can be made to the limitations of the lane line extraction method based on lidar in the above text, which will not be elaborated here. Each module in the above lane line extraction device based on lidar can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0162] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structural diagram may be as shown in Figure 4 . The computer device includes a processor, a memory, a network interface, and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a method for extracting lane lines based on lidar.

[0163] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented:

[0164] Obtain image data collected by a camera and lidar point cloud data collected by a lidar;

[0165] Perform time synchronization on the image data and the lidar point cloud data;

[0166] For the time-synchronized image frame and lidar point cloud frame, map the pixel colors on the image frame to the corresponding point clouds on the lidar point cloud frame;

[0167] Obtain the pose information of the lidar point cloud frame, and splice the lidar point cloud frame according to the pose information to obtain a colored point cloud map;

[0168] Extract ground point information according to the colored point cloud map;

[0169] Extract lane lines according to the ground point information and a color threshold.

[0170] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided by the present invention can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0171] Those skilled in the art can clearly understand that for the convenience and brevity of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0172] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention and should all be included in the protection scope of the present invention.

Claims

1. A method for extracting lane lines based on lidar, characterized in that, the method includes: Obtain image data collected by a camera and lidar point cloud data collected by a lidar; Perform time synchronization on the image data and the lidar point cloud data; For the image frame and the lidar point cloud frame after time synchronization, map the pixel colors on the image frame to the corresponding point clouds on the lidar point cloud frame; Obtain the pose information of the lidar point cloud frame, and splice the lidar point cloud frame according to the pose information to obtain a colored point cloud map; Extract ground point information according to the colored point cloud map; The extracting ground point information according to the colored point cloud map includes: Obtain the point clouds scanned by the same lidar beam from the colored point cloud map; Arrange the measured distances corresponding to the point clouds scanned by the same lidar beam in chronological order; Traverse each lidar beam emitted by the lidar to obtain an M*N azimuth matrix, where M represents the number of lidar beams and N represents the number of point clouds; Calculate the first angle between two point clouds in the same column of the azimuth matrix, and the first angle represents the angle between the lidar beam to the latter point cloud and the line connecting the two point clouds; Obtain ground points according to the first angle and the angle threshold, and construct ground point information; where the angle threshold is pre-set data; Extract lane lines according to the ground point information and the color threshold.

2. The method for extracting lane lines based on lidar according to claim 1, characterized in that, the performing time synchronization on the image data and the lidar point cloud data includes: For the i-th lidar point cloud frame in the lidar point cloud data, obtain the acquisition time of the i-th lidar point cloud frame; Obtain the two image frames before and after the acquisition time closest to the acquisition time from the image data, and record them as the k-th image frame and the k+1-th image frame respectively; If the difference between the acquisition time of the k+1-th image frame and the acquisition time of the i-th lidar point cloud frame is greater than the difference between the acquisition time of the i-th lidar point cloud frame and the acquisition time of the k-th image frame, synchronize the i-th lidar point cloud frame with the k-th image frame; If the difference between the acquisition time of the k+1-th image frame and the acquisition time of the i-th lidar point cloud frame is less than the difference between the acquisition time of the i-th lidar point cloud frame and the acquisition time of the k-th image frame, synchronize the i-th lidar point cloud frame with the k+1-th image frame.

3. The method for extracting lane lines based on lidar according to claim 1, characterized in that, the for the image frame and the lidar point cloud frame after time synchronization, mapping the pixel colors on the image frame to the corresponding point clouds on the lidar point cloud frame includes: Calculate the image height field of view angle according to the width of the image frame and the first internal parameter coefficient of the camera; Calculate the image width field of view angle according to the height of the image frame and the second internal parameter coefficient of the camera; Convert the radar coordinates of each point in the radar coordinate system into camera coordinates in the camera coordinate system; Judge whether the camera coordinates fall within the range formed by the image height field of view angle and the image width field of view angle; If so, obtain the color value corresponding to the point from the image frame, and assign the color value to the point cloud corresponding to the lidar point cloud frame.

4. The method for extracting lane lines based on lidar according to claim 3, wherein, the determination of whether the camera coordinates fall within the range formed by the image height field of view angle and the image width field of view angle includes: Calculate the first sine value and the second sine value of each point after conversion in the camera coordinates under the radar coordinate system. Here, the first sine value is The second sine value is where x c , y c , z c are the coordinate point values of the x, y, and z axes of the camera respectively; determining whether the first sine value is less than the positive image height field of view angle and greater than the negative image height field of view angle, and whether the second sine value is less than the positive image width field of view angle and greater than the negative image width field of view angle.

5. The method for extracting lane lines based on lidar according to claim 1, wherein, the obtaining of ground points according to the first included angle and the angle threshold and the construction of ground point information include: comparing the first included angle with a preset angle threshold; when the first included angle is less than the preset angle threshold, the two point clouds are considered as ground points, otherwise, the two point clouds are not ground points.

6. The method for extracting lane lines based on lidar according to claim 1, wherein, the extraction of lane lines according to the ground point information and the color threshold includes: obtaining the color value of the ground point, and comparing the color value of the ground point with a preset color range; if the color value of the ground point falls within the preset color range, the ground point is a lane line, otherwise it is not a lane line; traverse each ground point, obtain all ground points that fall within the preset color range, and obtain the lane line.

7. A device for extracting lane lines based on lidar, wherein, the device includes: an acquisition module for acquiring image data collected by a camera and lidar point cloud data collected by a lidar; a synchronization module for synchronizing the time of the image data and the lidar point cloud data; a mapping module for mapping the pixel colors on the image frame to the corresponding point clouds on the lidar point cloud frame for the time-synchronized image frame and lidar point cloud frame; a stitching module for obtaining the pose information of the lidar point cloud frame, and stitching the lidar point cloud frame according to the pose information to obtain a colored point cloud map; a ground extraction module for extracting ground point information according to the colored point cloud map; the extraction of ground point information according to the colored point cloud map includes: obtaining the point clouds scanned by the same lidar beam from the colored point cloud map; arranging the measured distances corresponding to the point clouds scanned by the same lidar beam in chronological order; traverse each lidar beam emitted by the lidar to obtain an M*N azimuth matrix, where M represents the number of lidar beams and N represents the number of point clouds; calculating the first included angle between two point clouds in the same column of the azimuth matrix, where the first included angle represents the angle between the lidar beam to the latter point cloud and the line connecting the two point clouds; obtaining ground points according to the first included angle and the angle threshold, and constructing ground point information; where the angle threshold is preset data; a lane line extraction module for extracting lane lines according to the ground point information and the color threshold.

8. A computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, it implements the lidar-based lane line extraction method according to any one of claims 1 to 6.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the lidar-based lane line extraction method according to any one of claims 1 to 6.

Images