Lane line detection method and device

Abstract

The application discloses a lane line detection method and device. The method comprises the following steps: acquiring a point cloud map corresponding to a target environment and a target image, the target environment containing lane lines; inputting the target image into a lane line detection model; detecting the lane lines in the target image through the lane line detection model; outputting first lane lines; extracting ground points from the point cloud map to obtain ground point clouds; determining a plurality of points in the ground point clouds that meet a preset condition as candidate lane line points; transforming the first lane lines into the point cloud map to obtain second lane lines; in the case that the distance between the candidate lane line points and the second lane lines does not exceed a first preset range, determining the candidate lane line points as target lane line points; and determining the target lane lines according to the target lane line points. In this way, the accuracy of lane line detection can be improved.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to a lane line detection method and apparatus. Background Technology

[0002] Autonomous driving technology is generally divided into perception, localization, decision-making, planning, and control. The perception module needs to perform lane line detection, and the detection results can be used for higher-level functions such as localization and assisted path planning. In addition, the lane line detection results are also an indispensable part of high-precision maps.

[0003] Most current lane detection methods are based on images or point cloud data. Image-based lane detection often suffers from low accuracy due to wear and tear on the lane lines, which can blend into the road surface. Furthermore, the need for 2D-to-3D conversion of the detection results also reduces accuracy. Additionally, image-based lane detection is significantly affected by lighting conditions, such as at night or when entering or exiting tunnels. Point cloud-based lane detection is also prone to inaccuracies due to issues such as lane line damage, similarity between road markings and lane lines, vehicle occlusion, and inaccurate road segmentation.

[0004] Therefore, lane line detection based on images or point cloud data has low accuracy. Summary of the Invention

[0005] This application provides a lane line detection method and apparatus that can improve the accuracy of lane line detection.

[0006] In a first aspect, embodiments of this application provide a lane line detection method, the method comprising:

[0007] Obtain the point cloud map and target image corresponding to the target environment, which includes lane lines.

[0008] The target image is input into the lane detection model, which detects lane lines in the target image and outputs the first lane line.

[0009] Ground points are extracted from the point cloud map to obtain the ground point cloud.

[0010] Multiple points in the ground point cloud that meet preset conditions are identified as candidate lane line points.

[0011] Transform the first lane line onto the point cloud map to obtain the second lane line.

[0012] If the distance between the candidate lane line point and the second lane line does not exceed the first preset range, the candidate lane line point is determined as the target lane line point.

[0013] Determine the target lane line based on the target lane line point.

[0014] Secondly, embodiments of this application provide a lane line detection device, the device comprising:

[0015] The acquisition module is used to acquire the point cloud map and target image corresponding to the target environment, which includes lane lines.

[0016] The input module is used to input the target image into the lane detection model, which then detects lane lines in the target image and outputs the first lane line.

[0017] The extraction module is used to extract ground points from the point cloud map to obtain the ground point cloud.

[0018] The first determining module is used to identify multiple points in the ground point cloud that meet preset conditions as candidate lane line points.

[0019] The transformation module is used to transform the first lane line onto the point cloud map to obtain the second lane line.

[0020] The second determining module is used to determine the candidate lane line point as the target lane line point when the distance between the candidate lane line point and the second lane line does not exceed a first preset range.

[0021] The third determination module is used to determine the target lane line based on the target lane line points.

[0022] Thirdly, embodiments of this application provide an electronic device, the device comprising: a processor and a memory storing computer program instructions.

[0023] When the processor executes the computer program instructions, it implements the lane detection method as shown in any embodiment of the first aspect.

[0024] Fourthly, embodiments of this application provide a computer storage medium storing computer program instructions, which, when executed by a processor, implement the lane detection method shown in any embodiment of the first aspect.

[0025] Fifthly, embodiments of this application provide a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to perform the lane detection method shown in any embodiment of the first aspect.

[0026] The lane line detection method and apparatus of this application can acquire a point cloud map and a target image corresponding to a target environment containing lane lines. The target image is then input into a lane line detection model, which detects lane lines in the target image and outputs a first lane line. Ground points are extracted from the point cloud map to obtain a ground point cloud. Multiple points in the ground point cloud that meet preset conditions are identified as candidate lane line points. The first lane line is then transformed into the point cloud map to obtain a second lane line. Finally, if the distance between the candidate lane line points and the second lane line does not exceed a first preset range, the candidate lane line points are determined as target lane line points, thus obtaining the target lane line. Therefore, lane line detection based on image and point cloud data is more accurate than lane line detection based on a single data source. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is one of the flowcharts of a lane line detection method provided in one embodiment of this application.

[0029] Figure 2 This is a schematic diagram illustrating data synchronization according to one embodiment of this application.

[0030] Figure 3 This is a schematic diagram illustrating candidate lane line point selection according to an embodiment of this application.

[0031] Figure 4 This is a second flowchart of a lane line detection method provided in one embodiment of this application.

[0032] Figure 5 This is a schematic diagram of the structure of a lane line detection device according to an embodiment of this application.

[0033] Figure 6 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0034] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0035] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0036] As mentioned in the background section, most current lane detection methods used in driver assistance systems are based on image or point cloud data.

[0037] Image-based lane detection methods can be broadly categorized into two types: algorithms based on traditional geometric features and algorithms based on deep learning. Traditional geometric feature-based lane detection algorithms typically detect lane lines by defining regions of interest, image grayscale processing, edge extraction, Hough transform, or RANSAC fitting. Deep learning-based image lane detection algorithms generally detect lane lines through image preprocessing, normalization, lane segmentation, lane fitting, and lane classification.

[0038] Lane detection methods based on point cloud data typically extract road marking attributes by combining features such as the reflection intensity of laser point clouds, the angle between the installation location and the ground, and the measured distance. Least squares are then used to fit the lane line equation, and tracking algorithms are employed to track the lane lines. During lane line extraction, road surface and curb extraction are usually performed first. Some algorithms also convert point cloud reflection intensity values ​​into images and then use image processing algorithms for lane line extraction.

[0039] However, image-based lane detection often suffers from low accuracy due to lane line wear and the resulting color closely resembling the road surface. Furthermore, the need for 2D-to-3D conversion of the detection results further reduces accuracy. Additionally, image-based lane detection is significantly affected by lighting conditions, such as at night or when entering or exiting tunnels. Point cloud-based lane detection typically faces challenges such as lane line damage, weak reflection intensity, similarity between road markings and lane lines, sparse point clouds, vehicle occlusion, and inaccurate road segmentation. Therefore, relying solely on single-frame laser point cloud data often fails to achieve sufficiently refined and accurate lane line localization and detection.

[0040] Based on this, the embodiments of this application provide a lane line detection method, which is described below in conjunction with... Figure 1-4 A detailed introduction will be provided.

[0041] Figure 1 A schematic flowchart of a lane line detection method provided in one embodiment of this application is shown.

[0042] like Figure 1 As shown, the execution entity of this lane line detection method can be a lane line detection device, and the lane line detection method can include the following steps:

[0043] S110: Obtain the point cloud map and target image corresponding to the target environment.

[0044] S120: Input the target image into the lane detection model, detect lane lines in the target image using the lane detection model, and output the first lane line.

[0045] S130, extract ground points from the point cloud map to obtain the ground point cloud.

[0046] S140, identify multiple points in the ground point cloud that meet preset conditions as candidate lane line points.

[0047] S150, transform the first lane line into the point cloud map to obtain the second lane line.

[0048] S160, if the distance between the candidate lane line point and the second lane line does not exceed the first preset range, the candidate lane line point is determined as the target lane line point.

[0049] S170, determine the target lane line based on the target lane line point.

[0050] Therefore, a point cloud map and a target image corresponding to the target environment containing lane lines can be obtained. The target image is then input into a lane line detection model, which detects lane lines in the target image and outputs the first lane line. Ground points are extracted from the point cloud map to obtain a ground point cloud. Multiple points in the ground point cloud that meet preset conditions are then identified as candidate lane line points. The first lane line is then transformed into the point cloud map to obtain the second lane line. Finally, if the distance between the candidate lane line points and the second lane line does not exceed a first preset range, the candidate lane line points are determined as the target lane line points, thus obtaining the target lane line. In this way, lane line detection based on image and point cloud data is more accurate than lane line detection based on a single data source.

[0051] Regarding S110, the target environment may include lane lines. Point cloud maps can be obtained by processing point cloud data acquired by radar or lidar. Target images can be acquired by cameras.

[0052] In some implementations, to improve the accuracy of lane line detection, the acquisition of the point cloud map corresponding to the target environment may include:

[0053] Acquire multi-frame point cloud data of the target environment and pose data of multiple vehicles.

[0054] Time synchronization was performed on multiple frames of point cloud data and multiple vehicle pose data to obtain a one-to-one correspondence between the multiple frames of point cloud data and the multiple vehicle pose data.

[0055] Based on multiple vehicle pose data and target vehicle pose data, determine the transformation matrix that transforms the multiple vehicle pose data into the target vehicle pose data.

[0056] Based on the transformation matrix, the multi-frame point cloud data is transformed into the vehicle coordinate system corresponding to the target point cloud data to obtain the point cloud map.

[0057] Here, multiple frames of point cloud data can be acquired using LiDAR, and multiple vehicle pose data can be acquired using a carrier phase differential (RTK) / Inertial Navigation System (INS) module. The vehicle pose data can be the vehicle's pose in a global coordinate system. For example, the global coordinate system can be a coordinate system with the vehicle's starting position as the origin, the vehicle's initial direction of travel as the positive X-axis, and the horizontal leftward direction perpendicular to the X-axis as the positive Y-axis.

[0058] Multi-frame point cloud data can be continuous, while multiple vehicle pose data can be vehicle pose data corresponding to multiple consecutive time points. Time synchronization of the multi-frame point cloud data and the multiple vehicle pose data ensures a one-to-one correspondence between them, meaning each frame of point cloud data corresponds to one vehicle pose data point. The corresponding point cloud data and vehicle pose data can be data from the same time point.

[0059] The target vehicle pose data can correspond to the target point cloud data. The target point cloud data can be the last frame in a multi-frame point cloud dataset, and the target vehicle pose data can be the latest vehicle pose data among multiple vehicle pose data sets. For example, the target point cloud data can be the point cloud data of the current frame, and the target vehicle pose data can be the vehicle pose data at the current moment.

[0060] For example, four frames of point cloud data can be stitched together to obtain a local point cloud map, i.e., a point cloud map. Specifically, the four frames of point cloud data can be stored in a cache queue. Then, a transformation matrix is ​​obtained using the vehicle pose data corresponding to these four frames of point cloud data. Based on the transformation matrix, all four frames of point cloud data are transformed to the vehicle coordinate system corresponding to the current frame of point cloud data. The cache queue can store only four frames of point cloud data. Each time new point cloud data arrives, the oldest frame of point cloud data is discarded.

[0061] In this way, a point cloud map is obtained by stitching together multiple frames of point cloud data, realizing a dense point cloud. Lane line detection based on the dense point cloud can improve the accuracy of lane line detection.

[0062] In some implementations, to further improve the accuracy of lane line detection, the aforementioned time synchronization of multi-frame point cloud data and multiple vehicle pose data to obtain a one-to-one correspondence between the multi-frame point cloud data and multiple vehicle pose data may include:

[0063] The vehicle pose data corresponding to each frame of point cloud data in a multi-frame point cloud dataset is determined by the following steps:

[0064] Determine the first pose timestamp and the second pose timestamp immediately preceding and following the point cloud timestamp.

[0065] The target pose timestamp, located between the first pose timestamp and the second pose timestamp, is determined using interpolation.

[0066] Determine the vehicle pose information corresponding to the target pose timestamp, which is the vehicle pose information corresponding to the point cloud data.

[0067] Here, in order to further improve the accuracy of lane line detection, point cloud data, vehicle pose data and target image can be synchronized in time. Since the camera is triggered to take pictures by LiDAR, it can be assumed that the target image captured by the camera and the point cloud data captured by LiDAR are synchronized in time. Therefore, it is only necessary to synchronize multiple frames of point cloud data and multiple vehicle pose data in time.

[0068] For example, the acquisition frequency of point cloud data and target image can both be 10Hz, while the acquisition frequency of vehicle pose data can be 100Hz.

[0069] Time synchronization of multiple frames of point cloud data and multiple vehicle pose data can be achieved by separately determining the vehicle pose data corresponding to each frame of point cloud data.

[0070] Specifically, point cloud data may include point cloud timestamps, and vehicle pose data may include pose timestamps. For any frame of point cloud data, the first pose timestamp and the second pose timestamp immediately preceding and following the point cloud timestamp of that frame can be determined first. Then, through interpolation, the target pose timestamp located between the first pose timestamp and the second pose timestamp can be determined. The vehicle pose information corresponding to this target pose timestamp is the vehicle pose information corresponding to the point cloud data.

[0071] For example, it can be as follows Figure 2 As shown, first determine the first pose timestamp 220 before the point cloud timestamp 210 and the second pose timestamp 230 after the point cloud timestamp 210. Then, the target pose timestamp between the first pose timestamp 220 and the second pose timestamp 230 can be determined using the interpolation method.

[0072] In this way, vehicle pose data corresponding to each frame of point cloud data can be determined by timestamps and interpolation, thereby achieving time synchronization. Lane detection based on time-synchronized data can further improve the accuracy of lane detection.

[0073] In step S120, the target image is input into a lane detection model. The lane detection model detects lane lines in the target image and outputs lane line pixels, thus obtaining the first lane line. The lane detection model can be an ERFNET semantic segmentation network.

[0074] In some implementations, in order to obtain a more realistic target image, the method may further include the following steps before S120:

[0075] Obtain the calibration intrinsic parameters of the camera, which is the camera used to acquire images of the target.

[0076] The target image is subjected to distortion correction based on the calibrated intrinsic parameters.

[0077] Here, the camera can be a camera that captures images of the target. The Zhang Zhengyou calibration method can be used to calibrate the camera's intrinsic parameters, obtaining the camera's calibration intrinsic parameters. Then, distortion correction processing can be performed on the target image based on these calibration intrinsic parameters.

[0078] Thus, distortion correction yields a more realistic target image, thereby improving the accuracy of lane detection.

[0079] In some implementations, to facilitate lane line detection using a lane line detection model, the method may further include the following steps before S120:

[0080] The target image is normalized.

[0081] Specifically, the pixel information of the target image can be normalized. The range of each pixel value in the target image captured by the camera can be between 0 and 255. Normalization means dividing the pixel value by 255 to obtain a value between 0 and 1.

[0082] In this way, after normalizing the target image, it is easier to perform lane line detection using the lane line detection model.

[0083] Since lane lines are located on the ground, ground points can be extracted from the point cloud map to obtain a ground point cloud.

[0084] In some implementations, to obtain a more accurate ground point cloud, S130 may include:

[0085] Centered on the origin of the vehicle coordinate system, the target area is determined in the point cloud map according to the first preset size.

[0086] The target area is divided into grids according to the second preset size, resulting in multiple first grids.

[0087] Determine the difference between the maximum and minimum elevation directions of multiple points within each first grid cell.

[0088] If the difference between the maximum and minimum values ​​corresponding to the first grid does not exceed a preset difference, the points in the first grid are determined as ground points, and a ground point cloud is obtained.

[0089] Here, the target region can be a rectangular region of interest. The origin of the vehicle coordinate system can be used as the center of the rectangular region of interest, and the first preset size can be 50m above and below the center and 20m to the left and right, with the size of the rectangular region of interest being 100m × 40m.

[0090] The target area can then be divided into grids. For example, the second preset size can be 20cm × 20m.

[0091] For example, the preset difference can be 6cm. If the difference between the maximum and minimum elevation values ​​of multiple points in a first grid does not exceed 6cm, then it can be determined that all points in the first grid are ground points.

[0092] In addition, each point in the point cloud data has four pieces of information: x, y, z, and I, which represent the distance values ​​from the point to the center of the lidar in the three directions, the reflection intensity of the point, and the elevation direction, which can be the distance value in the z direction.

[0093] Thus, by dividing the data into grids and determining the difference between the maximum and minimum elevation values ​​of multiple points in each grid, a more accurate ground point cloud can be obtained.

[0094] Involving S140, multiple points in the ground point cloud that meet preset conditions can be identified as candidate lane line points.

[0095] In some implementations, to more accurately determine candidate lane line points, S140 may include:

[0096] The ground point cloud is divided into raster cells according to the third preset size, resulting in multiple second raster cells.

[0097] Determine the sum of the reflection intensities at multiple points within each second grid cell.

[0098] For each row of second grid cells, given that the sum of the reflection intensities of the N consecutive second grid cells increases sequentially, determine the first sequence number of the last grid cell among the N second grid cells.

[0099] For each row of second grid cells, if the sum of the reflection intensities of the M second grid cells decreases sequentially, determine the second row number corresponding to the last grid cell among the M second grid cells.

[0100] If the difference between the first and second sequence numbers does not exceed the second preset range, the points within the target grid are determined as candidate lane line points.

[0101] Here, both N and M can be positive integers greater than the preset value.

[0102] For example, the preset value can be 4. The third preset size can be 20cm × 5cm, that is, 20cm in the vertical direction and 5cm in the horizontal direction. This grid division ensures that the amount of calculation is not large and also ensures the lateral accuracy of the lane lines.

[0103] Specifically, for each row of second grid cells, the sum of the reflection intensities of multiple points within each second grid cell in the window can be calculated using the sliding window method. Calculating the sum of reflection intensities amplifies the features, facilitating subsequent filtering. The sum of reflection intensities and the index of the starting grid cell within the window, i.e., the arrangement number, can be saved.

[0104] Then, the queue storing the sum of reflection intensities can be traversed to find the endpoint of the continuously increasing grid and the index of that endpoint grid, i.e., the first sequence number, can be recorded. Here, a grid is considered to be continuously increasing only if the number of grids with a continuously increasing sum of reflection intensities exceeds a preset value.

[0105] Furthermore, it is necessary to find the endpoint of the continuously decreasing grid and record the index of that endpoint grid, which is the second sequence number. Here, a grid is considered to be continuously decreasing only if the number of grids whose sum of reflection intensity continuously decreases exceeds a preset value.

[0106] Then, it can be determined whether the index difference between the endpoint grid of the continuously ascending grid and the endpoint grid of the continuously descending grid, that is, the difference between the first and second permutation numbers, exceeds a second preset range. If it does not exceed the second preset range, then the point in the target grid can be determined as a candidate lane line point. The target grid may include the grids corresponding to the first and second permutation numbers respectively, as well as the grids corresponding to the permutation numbers between the first and second permutation numbers.

[0107] For example, the second preset range can be less than or equal to 5 and greater than or equal to 3.

[0108] Thus, it is possible to Figure 3 As shown, grids 310 to 320 are continuously ascending grids, with grid 310 being the starting grid and grid 320 being the ending grid. The first arrangement number can be 7. Grids 330 to 340 are continuously descending grids, with grid 330 being the starting grid and grid 340 being the ending grid. The second arrangement number can be 12. Therefore, the difference in arrangement numbers between grids 320 and 340 can be 5, which does not exceed the second preset range. Thus, it can be determined that the points in the grids between grids 320 and 340, as well as the grids between grids 320 and 340, are all candidate lane line points.

[0109] Regarding S150, the image data and point cloud data can be fused by transforming the first lane line into a point cloud map.

[0110] In some implementations, to obtain a more accurate second lane line, S150 may include:

[0111] By using inverse perspective transformation, the first lane line is transformed into a bird's-eye view to obtain the third lane line.

[0112] Projecting the third lane line onto the point cloud map yields the second lane line.

[0113] Here, the multiple lane line pixels corresponding to the first lane line can be transformed into points on the bird's-eye view using the principle of inverse perspective transformation to obtain the third lane line. Then, the multiple points corresponding to the third lane line on the bird's-eye view can be projected onto the point cloud map to obtain the second lane line.

[0114] Projection can be achieved by using inverse perspective transformation parameters to obtain the distance values ​​in the x and y directions of points on a bird's-eye view, and then setting the distance value in the z direction to 0. This will give you a 3D point. Then, you can use coordinate transformation parameters to transform this 3D point to the point cloud coordinate system, thus completing the projection.

[0115] Thus, by using inverse perspective transformation and projection transformation, a more accurate second lane line can be obtained.

[0116] In S160, the distance between each candidate lane line point and the second lane line can be determined. If the distance does not exceed the first preset range, the candidate lane line point can be used as the target lane line point.

[0117] For example, the first preset range can be greater than 20cm and less than 40cm.

[0118] For S170, the target lane line can be obtained by fitting multiple target lane line points.

[0119] In some implementations, to obtain more lane line information, S150 may include:

[0120] If the distance between the candidate lane line point and the second lane line does not exceed the first preset range, the attribute information of the target lane line is determined to be the attribute information of the second lane line.

[0121] Here, the attribute information may include line type and / or color. The attribute information of the second lane line is determined based on the target image. Since the attribute information of the lane line cannot be determined based on point cloud data, the attribute information such as line type and / or color obtained from the target image can be assigned to the target lane line.

[0122] In this way, more lane line information can be obtained by assigning attribute information such as line type and / or color determined based on image data to the target lane line.

[0123] It should be noted that the embodiments of this application do not limit the order of S120 and S130, nor do they limit the order of S140 and S150.

[0124] This application provides a lane line detection method with good stability and high accuracy. By introducing an RTK / INS module to stitch together multi-frame point cloud data to construct a local point cloud map, it ensures that a sufficient number of laser point clouds participate in the lane line extraction process. Then, the results of image lane line detection are used to assist the local point cloud map in lane line detection. By comprehensively considering the advantages and disadvantages of LiDAR and camera data, lane line extraction can be performed more accurately and efficiently, providing a basis for high-precision map production and lane line-based positioning.

[0125] To better describe the overall solution, based on the above embodiments, a specific example is given, such as... Figure 4 As shown, the lane line detection method may include S401-S410, which will be explained in detail below.

[0126] The S401 collects multiple frames of point cloud data through LiDAR, acquires multiple vehicle pose data through the RTK / INS module, and obtains target images through a camera.

[0127] S402 determines the transformation matrix through multiple vehicle pose data, and stitches together multiple frames of point cloud data using the transformation matrix to obtain a point cloud map.

[0128] S403 extracts ground points from the point cloud map to obtain the ground point cloud.

[0129] S404: Select multiple points from the ground point cloud that meet preset conditions as candidate lane line points.

[0130] S405 performs distortion correction on the target image.

[0131] S406, normalize the target image.

[0132] S407: Input the distortion-removed and normalized target image into the lane detection model and output the first lane line.

[0133] S408, through inverse perspective transformation and projection, the first lane line is transformed into the point cloud map to obtain the second lane line.

[0134] S409, determine the candidate lane line points whose distance from the second lane line does not exceed the first preset distance range as the target lane line points.

[0135] S410, determine the target lane line based on the target lane line point.

[0136] It should be noted that the embodiments of this application do not limit the order of S402-S404 and S405-S407.

[0137] The lane detection method provided in this application can stitch together point cloud data over a period of time and vehicle pose information at corresponding times to form a local map. Candidate lane line points are then extracted from the local map, and the lane line detection based on the image is projected onto the point cloud coordinate system. More accurate points are then extracted from the candidate lane line points using the lane lines detected based on the image. This method combines the distance advantage of point clouds, utilizes a local point cloud map mode to achieve a dense point cloud, ensuring rich point cloud information, and integrates the distance results, line type, and color information of lane lines detected based on the image, providing accurate lane line information for the production of high-precision maps.

[0138] Based on the same inventive concept, this application also provides a lane line detection device. The following describes... Figure 5 The lane line detection device provided in the embodiments of this application will be described in detail.

[0139] Figure 5 A schematic diagram of the structure of a lane line detection device provided in one embodiment of this application is shown.

[0140] like Figure 5 As shown, the lane line detection device may include:

[0141] Module 501 is used to acquire the point cloud map and target image corresponding to the target environment, which includes lane lines.

[0142] Input module 502 is used to input the target image into the lane detection model, detect lane lines in the target image through the lane detection model, and output the first lane line.

[0143] Extraction module 503 is used to extract ground points from the point cloud map to obtain a ground point cloud.

[0144] The first determining module 504 is used to determine multiple points in the ground point cloud that meet preset conditions as candidate lane line points.

[0145] Transformation module 505 is used to transform the first lane line into the point cloud map to obtain the second lane line.

[0146] The second determining module 506 is used to determine the candidate lane line point as the target lane line point when the distance between the candidate lane line point and the second lane line does not exceed a first preset range.

[0147] The third determining module 507 is used to determine the target lane line based on the target lane line points.

[0148] Therefore, a point cloud map and a target image corresponding to the target environment containing lane lines can be obtained. The target image is then input into a lane line detection model, which detects lane lines in the target image and outputs the first lane line. Ground points are extracted from the point cloud map to obtain a ground point cloud. Multiple points in the ground point cloud that meet preset conditions are then identified as candidate lane line points. The first lane line is then transformed into the point cloud map to obtain the second lane line. Finally, if the distance between the candidate lane line points and the second lane line does not exceed a first preset range, the candidate lane line points are determined as the target lane line points, thus obtaining the target lane line. In this way, lane line detection based on image and point cloud data is more accurate than lane line detection based on a single data source.

[0149] In some implementations, to improve the accuracy of lane line detection, the acquisition module 501 may include:

[0150] The acquisition submodule is used to acquire multi-frame point cloud data and multiple vehicle pose data of the target environment.

[0151] The synchronization submodule is used to synchronize the time of multiple frames of point cloud data and multiple vehicle pose data, thereby obtaining a one-to-one correspondence between the multiple frames of point cloud data and the multiple vehicle pose data.

[0152] The first determining submodule is used to determine the transformation matrix for transforming the multiple vehicle pose data into the target vehicle pose data based on multiple vehicle pose data and target vehicle pose data. The target vehicle pose data corresponds to the target point cloud data, which is the last frame in the multi-frame point cloud data.

[0153] The first transformation submodule is used to transform multi-frame point cloud data into the vehicle coordinate system corresponding to the target point cloud data according to the transformation matrix, so as to obtain a point cloud map.

[0154] In some implementations, to further improve the accuracy of lane line detection, the point cloud data includes point cloud timestamps, the vehicle pose data includes pose timestamps, and the synchronization submodule may include:

[0155] The first determining unit is used to determine the first pose timestamp and the second pose timestamp immediately preceding and following the point cloud timestamp.

[0156] The second determining unit is used to determine the target pose timestamp located between the first pose timestamp and the second pose timestamp using an interpolation method.

[0157] The third determining unit is used to determine the vehicle pose information corresponding to the target pose timestamp, which is the vehicle pose information corresponding to the point cloud data.

[0158] In some embodiments, to obtain a more realistic target image, the apparatus may further include:

[0159] The distortion correction module is used to obtain the calibration intrinsic parameters of the camera before inputting the target image into the lane detection model, detecting lane lines in the target image through the lane detection model, and outputting the lane line pixels in the target image. The camera is the one that acquires the target image.

[0160] The target image is subjected to distortion correction based on the calibrated intrinsic parameters.

[0161] In some embodiments, to facilitate lane line detection using a lane line detection model, the device may further include:

[0162] The normalization module is used to normalize the target image before inputting it into the lane detection model, detecting lane lines in the target image through the lane detection model, and outputting the lane line pixels in the target image to obtain the first lane line.

[0163] In some implementations, to obtain a more accurate ground point cloud, the extraction module 503 may include:

[0164] The second determination submodule is used to determine the target area in the point cloud map with the origin of the vehicle coordinate system as the center and according to the first preset size.

[0165] The first partitioning submodule is used to divide the target area into grids according to a second preset size, resulting in multiple first grids.

[0166] The third determining submodule is used to determine the difference between the maximum and minimum elevation directions of multiple points within each first grid cell.

[0167] The fourth determination submodule is used to determine the points in the first grid as ground points, and obtain the ground point cloud, provided that the difference between the maximum and minimum values ​​corresponding to the first grid does not exceed a preset difference.

[0168] In some implementations, to more accurately determine candidate lane line points, the first determining module 504 may include:

[0169] The second partitioning submodule is used to divide the ground point cloud into raster grids according to a third preset size, resulting in multiple second raster grids.

[0170] The fifth determining submodule is used to determine the sum of the reflection intensities of multiple points in each second grid cell.

[0171] The sixth submodule is used to determine the first sequence number of the last grid cell in each row of N second grid cells, given that the sum of the reflection intensities of the corresponding N consecutive second grid cells increases sequentially. N is a positive integer greater than a preset value.

[0172] The seventh submodule is used to determine the second sequence number of the last grid cell in each row of M second grid cells, provided that the sum of the reflection intensities of the M second grid cells decreases sequentially. M is a positive integer greater than a preset value.

[0173] The eighth determination submodule is used to determine the points in the target grid as candidate lane line points when the difference between the first and second arrangement numbers does not exceed the second preset range. The target grid includes the grids corresponding to the first and second arrangement numbers respectively, and the grids corresponding to the arrangement numbers between the first and second arrangement numbers.

[0174] In some implementations, to obtain a more accurate second lane line, the transformation module 505 may include:

[0175] The second transformation submodule is used to transform the first lane line into a bird's-eye view through inverse perspective transformation to obtain the third lane line.

[0176] The projection submodule is used to project the third lane line onto the point cloud map to obtain the second lane line.

[0177] In some embodiments, to obtain more lane line information, the device may further include:

[0178] The fourth determining module is used to determine the attribute information of the target lane line as the attribute information of the second lane line when the distance between the candidate lane line point and the second lane line does not exceed the first preset range. The attribute information includes line type and / or color.

[0179] Figure 6 A schematic diagram of the structure of an electronic device provided in one embodiment of this application is shown.

[0180] like Figure 6 As shown, the electronic device 6 is a structural diagram of an exemplary hardware architecture of an electronic device that can implement the lane line detection method and lane line detection device according to the embodiments of this application. This electronic device may refer to the electronic device in the embodiments of this application.

[0181] The electronic device 6 may include a processor 601 and a memory 602 storing computer program instructions.

[0182] Specifically, the processor 601 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0183] Memory 602 may include mass storage for data or instructions. For example, and not limitingly, memory 602 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 602 may include removable or non-removable (or fixed) media. Where appropriate, memory 602 may be internal or external to an integrated gateway disaster recovery device. In a particular embodiment, memory 602 is non-volatile solid-state memory. In a particular embodiment, memory 602 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Thus, generally, memory 602 includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this application.

[0184] The processor 601 implements any of the lane detection methods in the above embodiments by reading and executing computer program instructions stored in the memory 602.

[0185] In one example, the electronic device may also include a communication interface 603 and a bus 604. Wherein, for example... Figure 6 As shown, the processor 601, memory 602, and communication interface 603 are connected through bus 604 and complete communication with each other.

[0186] The communication interface 603 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0187] Bus 604 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 604 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0188] The electronic device can execute the lane line detection method in the embodiments of this application, thereby achieving a combination Figures 1 to 5 The described lane line detection method and apparatus.

[0189] Furthermore, in conjunction with the lane detection methods in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions, which, when executed by a processor, implement any of the lane detection methods in the above embodiments.

[0190] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0191] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0192] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0193] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0194] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A lane line detection method, characterized in that, The method includes: Obtain a point cloud map and a target image corresponding to the target environment, wherein the target environment includes lane lines. The target image is input into the lane detection model, which detects lane lines in the target image and outputs the first lane line. Ground points are extracted from the point cloud map to obtain a ground point cloud. Multiple points in the ground point cloud that meet preset conditions are identified as candidate lane line points. The first lane line is transformed into the point cloud map to obtain the second lane line. If the distance between the candidate lane line point and the second lane line does not exceed a second preset distance range, the candidate lane line point is determined as the target lane line point. Determine the target lane line based on the target lane line points; The step of obtaining the point cloud map corresponding to the target environment includes: acquiring multi-frame point cloud data and multiple vehicle pose data of the target environment; synchronizing the multi-frame point cloud data and the multiple vehicle pose data in time to obtain a one-to-one correspondence between the multi-frame point cloud data and the multiple vehicle pose data; determining a transformation matrix to transform the multiple vehicle pose data into the target vehicle pose data based on the multiple vehicle pose data and the target vehicle pose data, wherein the target vehicle pose data corresponds to the target point cloud data, and the target point cloud data is the last frame in the multi-frame point cloud data; and transforming the multi-frame point cloud data into the vehicle coordinate system corresponding to the target point cloud data according to the transformation matrix to obtain the point cloud map.

2. The lane line detection method according to claim 1, characterized in that, The point cloud data includes point cloud timestamps, and the vehicle pose data includes pose timestamps. The step of synchronizing the multi-frame point cloud data and the multiple vehicle pose data to obtain a one-to-one correspondence between the multi-frame point cloud data and the multiple vehicle pose data includes: The vehicle pose data corresponding to each frame of point cloud data in the multi-frame point cloud data is determined by the following steps: Determine the first pose timestamp and the second pose timestamp immediately preceding and following the point cloud timestamp. The target pose timestamp, located between the first pose timestamp and the second pose timestamp, is determined using interpolation. The vehicle pose information corresponding to the target pose timestamp is determined to be the vehicle pose information corresponding to the point cloud data.

3. The lane line detection method according to claim 1, characterized in that, Before inputting the target image into the lane detection model, detecting lane lines in the target image through the lane detection model, and outputting the lane line pixels in the target image to obtain the first lane line, the method further includes: Obtain the calibration intrinsic parameters of the camera, wherein the camera is the one used to capture the target image. The target image is subjected to distortion correction based on the calibration intrinsic parameters.

4. The lane line detection method according to claim 1, characterized in that, Before inputting the target image into the lane detection model, detecting lane lines in the target image through the lane detection model, and outputting the lane line pixels in the target image to obtain the first lane line, the method further includes: The target image is then normalized.

5. The lane line detection method according to claim 1, characterized in that, The step of extracting ground points from the point cloud map to obtain a ground point cloud includes: Centered on the origin of the vehicle coordinate system, the target area is determined in the point cloud map according to a first preset size. The target area is divided into grids according to a second preset size to obtain multiple first grids. Determine the difference between the maximum and minimum elevation directions of multiple points within each of the first grid cells. If the difference between the maximum and minimum values ​​corresponding to the first grid does not exceed a preset difference, the points in the first grid are determined to be ground points, and the ground point cloud is obtained.

6. The lane line detection method according to claim 1, characterized in that, The step of determining multiple points in the ground point cloud that meet preset conditions as candidate lane line points includes: The ground point cloud is divided into grids according to a third preset size to obtain multiple second grids. Determine the sum of the reflection intensities at multiple points within each of the second grid cells. For each row of the second grid, if the sum of the reflection intensities corresponding to N consecutive second grids increases sequentially, determine the first sequence number of the last grid in the N second grids, where N is a positive integer greater than a preset value. For each row of the second grid, if the sum of the reflection intensities corresponding to the M second grids decreases sequentially, determine the second sequence number corresponding to the last grid among the M second grids, where M is a positive integer greater than a preset value. If the difference between the first arrangement number and the second arrangement number does not exceed the second preset range, the point in the target grid is determined as the candidate lane line point. The target grid includes the grids corresponding to the first arrangement number and the second arrangement number, respectively, and the grids corresponding to the arrangement number between the first arrangement number and the second arrangement number.

7. The lane line detection method according to claim 1, characterized in that, The step of transforming the first lane line into the point cloud map to obtain the second lane line includes: By using inverse perspective transformation, the first lane line is transformed into a bird's-eye view to obtain the third lane line. The third lane line is projected onto the point cloud map to obtain the second lane line.

8. The lane line detection method according to claim 1, characterized in that, The method further includes: If the distance between the candidate lane line point and the second lane line does not exceed a first preset range, the attribute information of the target lane line is determined to be the attribute information of the second lane line, and the attribute information includes line type and / or color.

9. A lane line detection device, characterized in that, The device includes: The acquisition module is used to acquire a point cloud map and a target image corresponding to the target environment, wherein the target environment includes lane lines. The input module is used to input the target image into the lane detection model, detect lane lines in the target image through the lane detection model, and output the first lane line. The extraction module is used to extract ground points from the point cloud map to obtain a ground point cloud. The first determining module is used to determine multiple points in the ground point cloud that meet preset conditions as candidate lane line points. The transformation module is used to transform the first lane line into the point cloud map to obtain the second lane line. The second determining module is used to determine the candidate lane line point as the target lane line point when the distance between the candidate lane line point and the second lane line does not exceed a first preset range. The third determining module is used to determine the target lane line based on the target lane line points; The acquisition module includes: an acquisition submodule for acquiring multi-frame point cloud data and multiple vehicle pose data of the target environment; a synchronization submodule for time synchronization of the multi-frame point cloud data and the multiple vehicle pose data to obtain a one-to-one correspondence between the multi-frame point cloud data and the multiple vehicle pose data; a first determination submodule for determining a transformation matrix to transform the multiple vehicle pose data into the target vehicle pose data based on the multiple vehicle pose data and the target vehicle pose data, wherein the target vehicle pose data corresponds to the target point cloud data, and the target point cloud data is the last frame in the multi-frame point cloud data; and a first transformation submodule for transforming the multi-frame point cloud data into the vehicle coordinate system corresponding to the target point cloud data according to the transformation matrix to obtain the point cloud map.

Images