A fast method for extracting curved lane skeleton

Abstract

The present invention relates to a method for fast extraction of curved lane skeletons. The present invention targets left-turn and right-turn lane line conditions, combines the perspective phenomenon of road images and the curvature of the curve, and uses a Gaussian filter kernel size that depends on the perspective relationship of the road image. The filtering direction is perpendicular to the tangent direction of the curve. Finally, non-maximum suppression and least squares fitting are combined to quickly extract the curved lane skeleton line, greatly improving the efficiency of curved lane skeleton extraction. The problem of inaccurate curve skeleton extraction in the past is solved, and the produced data set can be used for lane training data sets in autonomous driving or assisted driving, to warn vehicles of lane deviation during driving, to remind drivers to turn, etc. The present invention solves the problems of unsatisfactory existing curved lane line skeleton extraction technology, difficulty in marking curved lane lines in autonomous driving related data sets, small amount of data in data sets, and inaccurate curve marking results.

Description

Technical Field

[0001] The invention relates to a method for quickly extracting a curved lane skeleton, and belongs to the technical field of computer vision image processing. Background Art

[0002] In the process of autonomous driving or assisted driving, it is crucial to accurately detect the lane lines on both sides of the road currently being driven. Vehicle driving related datasets can be used to conduct experimental tests or theoretical verification of target detection related algorithms. However, most datasets are currently established by manual annotation, which is time-consuming and labor-intensive, and the amount of data is limited. It cannot cover most lane scenarios, and the existing datasets have less annotation content for curved lanes, and the amount of data is not enough to meet the needs.

[0003] The existing methods for automatically establishing lane line datasets include traditional classical algorithms and deep learning algorithms. There are three main classical algorithms: the first is to detect lane lines based on the mathematical model of lane lines. It is necessary to establish a lane line model and solve the model-related parameters. Commonly used lane line models include straight line or spline curve models and binomial models. This method is only applicable to lane line detection that conforms to this lane model and has poor versatility. The second is to detect lane lines based on lane line features. For example, lane line edges, lane line colors, lane textures, etc. are used as extraction features of the detection algorithm. The applicability of this method is more single and more suitable for structured roads. The algorithm itself is more sensitive to changing lighting environments and has poor robustness. The third is a method of extracting lane lines from lane skeletons. This method performs Gaussian filtering on the obtained lane interest area according to the perspective relationship of the road scene by adaptively changing the kernel size, and then undergoes non-maximum suppression, etc., and can finally accurately extract the lane line skeleton on the straight road, but the extraction effect of the central skeleton of the curved lane line is not ideal.

[0004] In the process of extracting the skeleton of curved lane lines, if the filter kernel size is changed according to the perspective relationship of the road scene, the filtering direction is not perpendicular to the tangent direction of the lane line, and the filtering grayscale value of the central skeleton of the curved lane line is not the highest, not only the lane skeleton of the current driving curve cannot be correctly extracted, but also the lane lines on both sides of other roads may be mistakenly extracted, which seriously affects the driving safety of the vehicle. Therefore, accurately and quickly extracting the skeleton of the curved lane line is particularly important for research on autonomous driving and assisted driving. Summary of the invention

[0005] Aiming at the problems that the existing curved lane line skeleton extraction technology is not ideal, the curved lane line labeling in autonomous driving technology is difficult, the data set has a small amount of data, and the curved lane labeling results are not accurate enough, a curved lane skeleton extraction method with adaptively changing Gaussian filter kernel direction and size is proposed. Summary of the invention:

[0007] The present invention obtains road images in real time through a vehicle-mounted camera installed in front of the vehicle. Based on the original method of changing the Gaussian filter kernel size according to the perspective relationship of the road image, a method is proposed in which the filtering direction of the filter is perpendicular to the tangent direction of the curve lane line. Finally, the central skeleton line of the curve is quickly extracted by combining non-maximum suppression and least squares curve fitting, which greatly improves the efficiency of curve lane marking.

[0008] Terminology explanation:

[0009] 1. Gaussian filtering, also known as Gaussian smoothing, is a widely used processing effect in image processing software. It is usually used to reduce image noise and reduce the level of detail. Gaussian smoothing is used in the pre-processing stage of computer vision algorithms to enhance the image effect at different scales. From a mathematical point of view, the Gaussian blur process of an image is the convolution of the image with the normal distribution. Since the normal distribution is also called the Gaussian distribution, this technique is called Gaussian blur. Convolving the image with the circular box blur will produce a more accurate out-of-focus imaging effect.

[0010] 2. Non-Maximum Suppression (NMS), that is, suppressing elements that are not maximum values, can be understood as local maximum search. This local represents a neighborhood, and the neighborhood has two variable parameters: one is the dimension of the neighborhood, and the other is the size of the neighborhood.

[0011] 3. Least square method, also known as least square method, is a mathematical optimization technique. It finds the best function matching data by minimizing the sum of squares of errors. The least square method can be used to easily obtain unknown data and minimize the sum of squares of errors between the obtained data and the actual data. The least square method can also be used for curve fitting, and some other optimization problems can also be expressed by the least square method by minimizing energy or maximizing entropy.

[0012] The technical solution of the present invention is:

[0013] A method for quickly extracting a curved lane skeleton comprises the following steps:

[0014] 1) Real-time acquisition of left and right turning videos of vehicles along the road, frame processing, pre-processing of the images of vehicles turning left and right along the road, and selecting the area of ​​interest of the lane line of the vehicle's initial frame turning left and right along the road;

[0015] 2) In the region of interest obtained in step 1), Gaussian filtering is performed through a Gaussian filter to obtain a filtering result; the width of the Gaussian filter kernel changes adaptively with the lane line width, and the filtering direction of the Gaussian filter is perpendicular to the tangent direction of the curved lane line; the grayscale value of the skeleton part of the center line of the curved lane line is the highest, and the grayscale values ​​on both sides of the skeleton of the center line decrease in sequence, presenting a Gaussian distribution;

[0016] 3) Perform non-maximum suppression on the filtering result obtained in step 2) to achieve binary segmentation of the image and obtain the central skeleton line of each lane line;

[0017] 4) performing least square curve fitting on the result in step 3) to obtain a fitting curve expression that is closest to the skeleton line;

[0018] 5) The curve obtained by least squares fitting is marked in the ROI image, that is, the skeleton of the curved lane is extracted.

[0019] Preferably, according to the present invention, in step 1), the images of the vehicle turning left or right along the road are preprocessed, specifically referring to: clipping and smoothing preprocessing of the images of the vehicle turning left or right along the road.

[0020] Further preferably, the width of the Gaussian filter kernel changes adaptively with the lane line width, which means:

[0021] Gaussian filter kernel width W kernel The lane line width w along the tangent direction v perpendicular to the curved lane line is changed in direct proportion to w, as shown in formula (I):

[0022] W kernel ∝w (I).

[0023] Preferably, according to the present invention, in step 2), the filtering direction of the Gaussian filter is perpendicular to the tangent direction of the lane line of the curve, and the change of the filtering direction of the Gaussian filter conforms to formula (II):

[0024]

[0025] In formula (II), v represents the filtering direction of the Gaussian filter, (x0, y0) is the coordinate of the current pixel, and (x1, y1) is the coordinate of the next pixel along the curve direction.

[0026] According to the preferred embodiment of the present invention, in step 3), non-maximum suppression is performed on the filtering result obtained in step 2) to achieve binary segmentation of the image and obtain the skeleton lines of each lane line segment, specifically:

[0027] The filtering result, i.e. the filtered image, is subjected to non-maximum suppression through a sliding window along the tangent direction perpendicular to the lane line of the curve. The size of the sliding window changes adaptively with the width of the lane line. All the pixels in the sliding window are traversed to obtain the maximum grayscale value of the pixels in each sliding window, and the grayscale values ​​of the remaining pixels are set to zero. Finally, the pixels in the skeleton part of the center line of the lane line of the curve are subjected to non-maximum suppression processing and become the maximum grayscale value, and the grayscale values ​​of the remaining pixels are zero.

[0028] Further preferably, the grayscale value of the pixel is processed as shown in formula (III):

[0029]

[0030] In formula (III), G(i,j) is the gray value of each pixel in the image, G max For the maximum grayscale value of each pixel in the sliding window, the processing process is: slide the sliding window with a step size of 1 pixel along the tangent direction of the lane line, traverse the pixels at each sliding point, find the maximum grayscale value, and compare the grayscale value of each pixel with the current maximum value in the window. If the pixel values ​​are the same, retain the pixel value, otherwise, set the grayscale value to zero.

[0031] Preferably, according to the present invention, in step 1), manually selecting the region of interest of the lane line where the vehicle turns left or right along the road specifically refers to: cropping and removing useless information in the lane image to obtain the region of interest.

[0032] The beneficial effects of the present invention are:

[0033] 1. The present invention proposes a method for quickly extracting the skeleton of a curved lane, which solves the problems that the existing curved lane line skeleton extraction technology is not ideal, the curved lane line labeling in the autonomous driving related data set is difficult, the data set has a small amount of data, and the curved lane labeling results are not accurate enough.

[0034] 2. The present invention utilizes image brightness information for skeleton extraction, which can better adapt to the annotation of curve lines under different lighting and road conditions, greatly improves the efficiency of curve lane annotation, and effectively constructs an expanded curve data set.

[0035] 3. The method for fast extraction of curved lane skeletons of the present invention is aimed at left-turn and right-turn lane lines, combined with the perspective phenomenon of the road image and the curvature of the curve. The size of the Gaussian filter kernel depends on the perspective relationship of the road image, and the filtering direction is perpendicular to the tangent direction of the curve. Finally, the non-maximum suppression and least squares fitting are combined to quickly extract the curved lane skeleton line, greatly improving the extraction efficiency of the curved lane skeleton. The problem of inaccurate extraction of curved skeletons in the past is solved. The produced data set can be used for lane training data sets in autonomous driving or assisted driving, and the vehicle is warned of lane deviation during driving, reminding the driver to turn, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flowchart of a method for quickly extracting a curved lane skeleton according to the present invention;

[0037] Figure 2 The original road image recorded by the vehicle-mounted camera;

[0038] Figure 3 It is a schematic diagram showing that the filtering direction in the ROI area is perpendicular to the tangent direction of the curve;

[0039] Figure 4 A schematic diagram for collecting input images of vehicles turning left;

[0040] Figure 5 The lane center skeleton annotation result image of the area of ​​interest under the vehicle left turn condition;

[0041] Figure 6 A schematic diagram for collecting input images of a vehicle turning right;

[0042] Figure 7 The lane center skeleton annotation result for the area of ​​interest when the vehicle turns left. DETAILED DESCRIPTION

[0043] The present invention will be further defined below in conjunction with the accompanying drawings and embodiments, but is not limited thereto.

[0044] Example 1

[0045] A fast extraction method for curved lane skeleton, such as Figure 1 As shown, the steps include:

[0046] 1) Use the on-board camera installed in front of the vehicle to obtain real-time video of the vehicle turning left or right along the road, such as Figure 2 As shown in the figure, after the frame processing, the image of the vehicle turning left or right along the road is cropped and smoothed to fully reduce the interference of other noise information in the image on the curved lane line filtering result. The region of interest (ROI) of the lane line of the vehicle turning left or right along the road in the initial frame is selected;

[0047] 2) In the region of interest obtained in step 1), Gaussian filtering is performed through a Gaussian filter to obtain a filtering result; since the curved lane line has a certain curvature and a perspective relationship exists, the width of the Gaussian filter kernel changes adaptively with the lane line width, and the filtering direction of the Gaussian filter is perpendicular to the tangent direction of the curved lane line, such as Figure 3 As shown; based on the original method of changing the Gaussian filter kernel size according to the perspective relationship of the road image, the present invention proposes a method in which the filtering direction of the filter needs to be perpendicular to the tangent direction of the curve lane line.

[0048] The grayscale value of the skeleton part of the center line of the lane line of the curve is the highest, and the grayscale values ​​on both sides of the skeleton of the center line decrease successively, showing a Gaussian distribution; the two-dimensional Gaussian function conforms to the following formula:

[0049] Where (x, y) is the coordinates of the point and σ is the standard deviation.

[0050] 3) Perform non-maximum suppression on the filtering result obtained in step 2) to achieve binary segmentation of the image and obtain the central skeleton line of each lane line;

[0051] 4) Perform least squares curve fitting on the result in step 3) to obtain the fitting curve expression closest to the skeleton line; the central skeleton line obtained by maximum value suppression in step 3) is mostly points in discrete areas. The coordinate values ​​of this series of discrete points are used to construct an n-order polynomial to fit the lane curve expression. By performing curve fitting by least squares method, the discrete points of the central skeleton can be fitted to a curve with the smallest error. Each segment is marked in its corresponding lane segment ROI. After the skeleton curve of the curved lane in each area is stored in its respective segment ROI area, it is integrated and marked with line segments on the overall lane image according to the cropping relationship when the ROI is created.

[0052] 5) The curve obtained by least squares fitting is marked in the ROI image, that is, the skeleton of the curved lane is extracted. The result of step 4) is used as the input point for confirming the lane region of interest in the next frame. Combined with the video frame rate and vehicle speed, the start frame is manually marked to predict the change range of the relative position of the lane ROI in the image of the subsequent frames, so as to realize the automatic extraction of the curved lane skeleton between consecutive frames.

[0053] Example 2

[0054] The difference between the method for quickly extracting the skeleton of a curved lane according to Embodiment 1 is that:

[0055] In step 1), the region of interest of the lane line where the vehicle turns left or right along the road is manually selected, specifically, the useless information in the lane image is removed by cropping to obtain the region of interest. Most lane images recorded by vehicle-mounted cameras contain a lot of useless information, such as Figure 2 As shown in the figure, the buildings and sky in the upper part of the image and the hood of the vehicle at the bottom of the image are useless information. By cropping the image, this part of useless information is removed. The size of the entire image before cropping is 1280×720, and the size of the remaining part after cropping is about 1280×270. The area after cropping is as follows Figure 3 shown.

[0056] The width of the Gaussian filter kernel changes adaptively with the lane width, which means:

[0057] Gaussian filter kernel width W kernel The lane line width w along the tangent direction v perpendicular to the curved lane line is changed in direct proportion to w, as shown in formula (I):

[0058] W kernel ∝w (I).

[0059] In step 2), the filtering direction of the Gaussian filter is perpendicular to the tangent direction of the lane line of the curve, and the change of the filtering direction of the Gaussian filter conforms to formula (II):

[0060]

[0061] In formula (II), v represents the filtering direction of the Gaussian filter, (x0, y0) is the coordinate of the current pixel, and (x1, y1) is the coordinate of the next pixel along the curve direction.

[0062] In step 3), non-maximum suppression is performed on the filtering result obtained in step 2) to realize binary segmentation of the image and obtain the skeleton lines of each lane line segment, specifically:

[0063] The filtering result, i.e. the filtered image, is subjected to non-maximum suppression through a sliding window along the tangent direction perpendicular to the lane line of the curve. The size of the sliding window changes adaptively with the lane line width like the Gaussian filter kernel. All pixel points in the sliding window are traversed to obtain the maximum grayscale value of the pixel points in each sliding window, and the grayscale values ​​of the remaining pixel points are set to zero. Finally, the pixel points of the skeleton part of the center line of the lane line of the curve are subjected to non-maximum suppression processing and become the grayscale maximum value, and the grayscale values ​​of the remaining pixel points are zero.

[0064] The gray value of the pixel is processed as shown in formula (III):

[0065]

[0066] In formula (III), G(i,j) is the gray value of each pixel in the image, G max For the maximum grayscale value of each pixel in the sliding window, the processing process is as follows: slide the sliding window of appropriate size and step length of 1 pixel along the tangent direction of the lane line, traverse the pixels at each sliding point, find the maximum grayscale value, and compare the grayscale value of each pixel with the current maximum value in the window. If the pixel values ​​are the same, retain the pixel value, otherwise, set the grayscale value to zero.

[0067] Figure 4 A schematic diagram for collecting input images of vehicles turning left; Figure 5 The lane center skeleton annotation result image of the area of ​​interest under the vehicle left turn condition; Figure 6 A schematic diagram for collecting input images of a vehicle turning right; Figure 7 The result map of the lane center skeleton annotation of the region of interest under the vehicle left turn condition is shown. Therefore, the present invention can quickly and accurately extract the curve skeleton.

Claims

1. A method for quickly extracting the skeleton of a curved lane, characterized in that: The steps include: 1) Real-time acquisition of left and right turning videos of vehicles along the road, frame processing, pre-processing of the images of vehicles turning left and right along the road, and selecting the area of ​​interest of the lane line of the vehicle's initial frame turning left and right along the road; 2) In the region of interest obtained in step 1), Gaussian filtering is performed through a Gaussian filter to obtain a filtering result; the width of the Gaussian filter kernel changes adaptively with the lane line width, and the filtering direction of the Gaussian filter is perpendicular to the tangent direction of the curved lane line; the grayscale value of the skeleton part of the center line of the curved lane line is the highest, and the grayscale values ​​on both sides of the skeleton of the center line decrease in sequence, presenting a Gaussian distribution; 3) Perform non-maximum suppression on the filtering result obtained in step 2) to achieve binary segmentation of the image and obtain the central skeleton line of each lane line; 4) performing least square curve fitting on the result in step 3) to obtain a fitting curve expression that is closest to the skeleton line; 5) The curve obtained by least squares fitting is marked in the ROI image, that is, the skeleton of the curved lane is extracted; In step 2), the filtering direction of the Gaussian filter is perpendicular to the tangent direction of the lane line of the curve, and the change of the filtering direction of the Gaussian filter conforms to formula (II): In formula (II), v represents the filtering direction of the Gaussian filter, (x0, y0) is the coordinate of the current pixel, and (x1, y1) is the coordinate of the next pixel along the curve direction.

2. The method for quickly extracting the skeleton of a curved lane according to claim 1, characterized in that: The width of the Gaussian filter kernel changes adaptively with the lane width, which means: Gaussian filter kernel width W kernel The lane line width w along the tangent direction v perpendicular to the curved lane line is changed in direct proportion to w, as shown in formula (I): IN kernel ∝in(I)。 3. The method for quickly extracting the skeleton of a curved lane according to claim 1, characterized in that: In step 3), non-maximum suppression is performed on the filtering result obtained in step 2) to realize binary segmentation of the image and obtain the skeleton lines of each lane line segment, specifically: The filtering result, i.e. the filtered image, is subjected to non-maximum suppression through a sliding window along the tangent direction perpendicular to the lane line of the curve. The size of the sliding window changes adaptively with the width of the lane line. All the pixels in the sliding window are traversed to obtain the maximum grayscale value of the pixels in each sliding window, and the grayscale values ​​of the remaining pixels are set to zero. Finally, the pixels in the skeleton part of the center line of the lane line of the curve are subjected to non-maximum suppression processing and become the maximum grayscale value, and the grayscale values ​​of the remaining pixels are zero.

4. The method for quickly extracting the skeleton of a curved lane according to claim 3, characterized in that: The gray value of the pixel is processed as shown in formula (III): In formula (III), G(i,j) is the gray value of each pixel in the image, G max For the maximum grayscale value of each pixel in the sliding window, the processing process is: slide the sliding window with a step size of 1 pixel along the tangent direction of the lane line, traverse the pixels at each sliding point, find the maximum grayscale value, and compare the grayscale value of each pixel with the current maximum value in the window. If the pixel values ​​are the same, retain the pixel value, otherwise, set the grayscale value to zero.

5. The method for quickly extracting the skeleton of a curved lane according to claim 1, characterized in that: In step 1), the region of interest of the lane line where the vehicle turns left or right along the road is manually selected, specifically, the region of interest is obtained by cropping and removing useless information in the lane image.

6. A method for quickly extracting curved lane skeleton according to any one of claims 1 to 5, characterized in that: In step 1), the images of the vehicle turning left or right along the road are preprocessed, specifically, the images of the vehicle turning left or right along the road are clipped and smoothed.

Images