A real-time lane line detection method based on an improved FAST corner detection algorithm

Abstract

An improved FPGA-based FAST corner detection algorithm is used as a filter for the image processing method of real-time lane line detection of a road surface, first, the RGB image collected by a camera is converted into a gray image, then the gray data is first subjected to minimum filtering processing, wherein the minimum filtering processing method is to take the current pixel point as the center, find the minimum value of the pixel value in the neighborhood of 3 pixels x 3 pixels of the center, replace the pixel value of the current pixel point with the minimum value, then use the improved FAST corner detection algorithm as the second filtering algorithm, finally, the lane line of the road surface is detected through the Sobel edge detection algorithm, the image information is acquired in real time by the camera, the real-time processing and transmission of the image information are completed through the field programmable logic gate array ZYNQ architecture, and finally the result of lane line detection is displayed in real time. The present application can accurately detect the lane line on the actual road surface, and will not detect the noise on the road surface.

Description

Technical Field

[0001] This invention relates to the field of real-time lane detection technology, and more specifically to a real-time lane detection method based on an improved FAST corner detection algorithm. Background Technology

[0002] In recent years, with the rapid development of robotics and autonomous driving technologies, road recognition technology has become widely known. Road recognition is a technology that uses cameras to capture lane lines and analyze their features to make a final judgment. Lane lines are characterized by a significant color difference from the road surface, making them easy to distinguish. Lane line features can be acquired through edge detection. However, real road surfaces are uneven and affected by factors such as light and rain, resulting in uneven color in the images captured by the camera. Furthermore, due to rain, sunlight, and vehicle traffic, some lane lines may have black spots or even cracks. These factors all contribute to higher image noise and more interference, thus reducing the accuracy of edge detection. Currently, commonly used edge detection algorithms include Sobel edge detection and Canny edge detection. Although Canny edge detection has good experimental results, its steps are complex and require a large amount of computation. Therefore, it is not suitable for deployment on hardware. However, Sobel edge detection is a hardware-friendly algorithm due to its simple steps, low computational cost, and fast running speed. However, it is sensitive to noise and lacks noise suppression capabilities. More importantly, real outdoor road conditions are complex, with various noises mixed together, and large areas of patches of different colors. These factors can all affect the Sobel edge detection algorithm's judgment, leading to the detection of more false edges.

[0003] Furthermore, for safety reasons, autonomous driving requires rapid assessment and processing of the road ahead. Traditional software programs execute serially, resulting in slow processing speeds. However, FPGAs, as hardware devices, possess high-speed parallel data processing capabilities, offering significant advantages in handling real-time data. However, they also suffer from long development cycles, high design costs, and limited computing and storage resources. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide an image processing method for real-time lane line detection on road surfaces by using an improved FAST corner detection algorithm based on FPGA as a filter, which overcomes the shortcomings of the prior art. The method has good real-time performance.

[0005] The following technical solution is adopted to solve the above technical problems:

[0006] A real-time lane detection method based on an improved FAST corner detection algorithm is an image processing method for real-time lane detection on road surfaces, using an improved FAST corner detection algorithm as a filter based on FPGA. First, the RGB image captured by the camera is converted into a grayscale image. Then, the grayscale data is subjected to minimum value filtering. The minimum value filtering method is to find the minimum value of the pixel value in the 3×3 pixel neighborhood of the current pixel and replace the current pixel value with this minimum value. Then, the improved FAST corner detection algorithm is used as a second filtering algorithm. Finally, the Sobel edge detection algorithm is used to detect the lane lines on the road surface. The image information is acquired in real time by the camera, and the image information is processed and transmitted in real time through the ZYNQ field-programmable gate array architecture. Finally, the lane detection results can be displayed on the monitor in real time via HDMI transmission.

[0007] The above-mentioned real-time lane detection method based on the improved FAST corner detection algorithm is as follows:

[0008] Step 1: Convert the RGB image to grayscale.

[0009] The camera sends real-time image data to the ZYNQ field-programmable gate array. The FPGA on the ZYNQ processes the image into a single-channel 8-bit image, converting the RGB image into a grayscale image. The formula (1) for converting the RGB image into YCbCr is as follows, where Y represents the luminance channel, R represents the red channel of the input RGB image, G represents the green channel of the input RGB image, B represents the blue channel of the input RGB image, and extracting the Y channel yields the grayscale image. Cb is the blue component, and Cr is the red component.

[0010]

[0011] Because FPGAs are not good at handling decimals, decimal operations need to be broken down into integer operations and shift operations. The specific formulas are as follows:

[0012]

[0013] Step 2: Perform minimum value filtering on the pixel data of the grayscale image.

[0014] Find the smallest pixel value within a 3-pixel × 3-pixel neighborhood centered on the current pixel p, and then replace the pixel value of the current pixel p with this smallest value.

[0015] Step 3: Use the improved FAST corner detection algorithm as the second filtering algorithm.

[0016] (1) Taking pixel p as the center, take the pixel values ​​of 16 pixels p1, p2, ..., p16 on a circle with a radius of 3 pixels, and denote them as I. p1 I p2 I p3 I p4 I p5 I p6 I p7 I p8 I p9 I p10 I p11 I p12 I p13 I p14 I p15 I p16 And the pixel value of pixel p is I p And set the threshold for corner detection as t;

[0017] (2) Calculate the pixel difference between the 16 points p1 to p16 and the center point p, and make the following judgments: First, if at least 3 of the points p1, p5, p9, and p13 are darker or brighter, then proceed to the second judgment; otherwise, exclude the center point p directly. Second, if there are 12 consecutive points that are darker or brighter, then the center point p will be considered as a candidate point.

[0018] The criteria for determining Darker and Brighter are shown in the following formula:

[0019]

[0020] (3) Non-maximum suppression of the image: Calculate the FAST score s at the candidate point. The calculation method is to take the difference between the pixel value of the above 16 points and the pixel value of the center point p, and take the absolute value. Then add these 16 values ​​to get the score s. If there are multiple candidate points in the 3-pixel × 3-pixel neighborhood of the candidate point p, then determine whether the score s of the center p is the largest among all candidate points in the neighborhood. If so, retain it; otherwise, suppress it.

[0021] The formula for calculating the score s is as follows, where V represents the score and t represents the threshold:

[0022]

[0023] Step 4: Use Sobel edge detection to detect lane lines.

[0024] Sobel edge detection is performed using the Sobel operator, which is a matrix that calculates the weighted sum of gray values ​​in a 3×3 pixel neighborhood centered on the current pixel. It is divided into x-direction and y-direction matrices, g and g', respectively. x and g y The specific weighted template is shown in the matrix below:

[0025]

[0026] Then, the pixel matrix of that neighborhood is multiplied by these two operators to obtain G. x and G y Next, calculate G according to the following formula. x and G y The square root of G is obtained:

[0027]

[0028]

[0029]

[0030] Finally, G is compared with the set Sobel threshold St. If G is greater than St, the pixel is considered an edge pixel; if G is less than St, the pixel is considered not an edge pixel.

[0031] The specific method of step 2 is as follows: Since minimum filtering needs to be performed in a 3-pixel × 3-pixel neighborhood, in addition to taking the input pixel as the center pixel, it is also necessary to obtain the gray values ​​of the 8 adjacent pixels around the center pixel for comparison. In the hardware FPGA, continuous shift RAM is used to buffer the image captured by the camera as pixel data, and the pixel data that is already in the shift RAM is shifted to construct a matrix so that the gray values ​​of 9 pixels can be read out in one clock cycle. A 3×3 matrix is ​​constructed using 3 shift RAMs. The 9 gray values ​​in the matrix are then compared and the minimum value is taken as the output as the input of the next sub-module.

[0032] When implementing the FAST corner detection algorithm, a circle with a radius of 3 pixels centered on the current pixel is considered as a 7-pixel × 7-pixel neighborhood centered on the current pixel. All 16 reference points on the circle with a radius of 3 pixels can be obtained within this neighborhood. At this time, a 7×7 matrix is ​​constructed using shift RAM. It is directly determined whether there are 12 consecutive reference points that are darker or brighter. If so, the center point is determined to be a corner point, and its grayscale value is modified to 0. Otherwise, its original grayscale value is retained, and the grayscale value of the center point at this time is output as the input of the Sobel edge detection module.

[0033] When using Sobel edge detection for lane line detection, the algorithm needs to construct a matrix, which is then built using shiftRAMs. This is followed by the calculation of G. x and G y When calculating the square root, the Cordic IP in Vivado is called.

[0034] This invention analyzes the principles of actual outdoor road surfaces and various image processing algorithms, and proposes an improved FAST corner detection filter based on FPGA for real-time detection of road lane lines. It combines the Sobel edge detection algorithm with a minimum filtering algorithm, using the improved FAST corner detection algorithm as a filter to suppress various road surface noises and detect lane lines. Furthermore, a pipelined design approach is adopted, giving the proposed algorithm excellent real-time performance. Using a camera, images can be captured and displayed in real time, with a video display resolution of 1280×720 and a frame rate of 60Hz.

[0035] from Figure 2 As can be seen from the images on the left, the actual outdoor road surface is uneven and affected by various other factors, resulting in uneven color in the road surface images captured by the camera. Furthermore, some lane markings show spots or even cracks, which is very detrimental to normal lane marking detection. If the Sobel edge detection algorithm is directly used to detect the actual road surface, it will be affected by the above factors, leading to poor final results. Figure 2 As shown in the middle column, this invention combines the Sobel edge detection algorithm with a minimum filtering algorithm and an improved FAST corner detection algorithm. First, the minimum filtering algorithm reduces large noise blocks and eliminates small noise blocks. Second, the improved FAST corner detection algorithm accurately detects smaller noise points, which can be eliminated one by one. Finally, the results of these two processing steps are input into the Sobel edge detection algorithm for lane line detection. Thus, the lane line detection effect is as follows: Figure 2 As shown in the right column, there is a significant improvement compared to the middle column. It can be seen that lane markings on the actual road surface are accurately detected, and road noise is not mistakenly detected. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the FAST corner detection algorithm of the present invention, where p is the current pixel point, and the points numbered 1-16 represent 16 points on a circle with a radius of 3 pixels centered on the current pixel point p.

[0037] Figure 2 The diagram shows a comparison of the effects of this invention. The leftmost column is the original image of lane lines in real life, the middle column is the lane line detection result image using the Sobel edge detection algorithm directly, and the rightmost column is the lane line detection result image of this invention.

[0038] Figure 3 This is the flowchart of the design process for this method. Detailed Implementation

[0039] An image processing method based on FPGA, using an improved FAST corner detection algorithm as a filter for real-time lane line detection on road surfaces, is presented. First, the RGB image is converted to grayscale. Then, the grayscale data undergoes minimum value filtering. Next, the improved FAST corner detection algorithm is used as a second filtering algorithm. Finally, the Sobel edge detection algorithm is applied to detect the lane lines on the road surface. Image information is acquired in real-time using a camera, and real-time processing and transmission of the image information are achieved through a ZYNQ architecture. The lane line detection results are then displayed on a monitor in real-time via HDMI transmission.

[0040] Methodology: An image processing method based on an improved FAST corner detection algorithm using FPGA as a filter for real-time lane line detection on road surfaces.

[0041] Step 1: Convert the RGB image to grayscale.

[0042] Image processing on a Field Programmable Gate Array (FPGA) requires single-channel 8-bit images, necessitating the conversion of RGB images to grayscale. The formula for converting an RGB image to YCbCr is as follows, where Y represents the luminance channel; extracting the Y channel yields the grayscale image. Cb represents the blue component, and Cr represents the red component.

[0043]

[0044] However, FPGAs are not good at handling decimals, so decimal operations need to be broken down into integer operations and shift operations. This saves development board space and improves processing speed. Therefore, the final formula is as follows:

[0045]

[0046] Step 2: Perform minimum value filtering on the pixel data of the grayscale image.

[0047] Minimum filtering is a simple and fast algorithm for removing image noise. Its principle is straightforward: find the smallest pixel value within a neighborhood of the current pixel, and then replace the current pixel's value with this minimum value. Minimum filtering can remove salt noise from images and highlight dark spots.

[0048] The minimum filter was chosen for the initial filtering of the road surface because most road surfaces are dark-colored, while lane lines are light-colored. Our primary focus is filtering noisy road surfaces to initially homogenize areas with uneven grayscale. Simultaneously, we must avoid affecting the grayscale values ​​of the lane lines. Therefore, the minimum filter was selected.

[0049] Since minimum filtering needs to be performed within a 3×3 pixel neighborhood, in addition to using the incoming pixel as the center pixel, it's also necessary to obtain the grayscale values ​​of the eight neighboring pixels around the center pixel for comparison. In hardware, this can be achieved by using consecutive shift RAMs to buffer the incoming pixels and shift the existing pixels, thus constructing a matrix. Here, we use three shift RAMs to construct a 3×3 matrix. After the matrix is ​​constructed, the grayscale values ​​of the nine pixels within it can be read out within one clock cycle. Then, these nine grayscale values ​​are compared, and the minimum value is taken as the output, serving as the input for the next submodule. The image after minimum filtering is free of salt noise and has relatively uniform grayscale.

[0050] (-1，0) (0，0) (1，0) (-1，-1) (0，-1) (1，-1)

[0051] Step 3: Continue filtering using the improved FAST corner detection.

[0052] The proposers of FAST defined FAST corner points as follows: if a pixel differs significantly from a sufficient number of pixels in its surrounding neighborhood, then that pixel may be a corner point.

[0053] like Figure 1 As shown, there are 16 pixels (p1, p2, ..., p16) on a circle with a radius of 3 centered at pixel p. Calculate the pixel difference between these 16 pixels (p1 to p16) and the center pixel p, and make the following judgments: First, if at least 3 of the pixels 1, 5, 9, and 13 are either darker or brighter, then proceed to the second judgment. Otherwise, exclude them directly. Second, if there are 12 consecutive pixels that are either darker or brighter, then the center pixel will be considered a candidate pixel. The judgment condition is shown in the following formula:

[0054]

[0055] Non-maximum suppression is applied to the image: The FAST score (s: the sum of the absolute values ​​of the differences between the 16 points and the center) of each feature point is calculated. Within a 3-pixel × 3-pixel neighborhood centered on feature point p, if there are multiple feature points, the s value of each feature point is evaluated. If p has the largest response value among all neighboring feature points, it is retained; otherwise, it is suppressed. The score calculation formula is as follows (where V represents the score and t represents the threshold):

[0056]

[0057] The purpose of choosing the FAST corner detection algorithm here is to further remove points with uneven gray levels on the road surface. After the minimum filtering in the previous step, most of the gray levels on the road surface have become relatively uniform. However, there are still some points with gray levels larger than the surrounding pixels. The FAST corner detection algorithm can easily detect these points. The FAST corner detection algorithm used in this paper is modified. It retains the corner detection part of the original FAST algorithm. However, as mentioned above, the detected corner points are the noise points we need to suppress. Therefore, the detected corner points are assigned a value of 0, while the non-corner parts retain their original gray values. Furthermore, no response value is calculated, and no non-maximum suppression is performed. The goal is to set the gray values ​​of all found corner points to 0, thereby achieving the effect of noise suppression.

[0058] When implementing the FAST corner detection algorithm, a circle with a radius of 3 pixels centered on the current pixel is considered a 7×7 pixel neighborhood centered on the current pixel. All 16 reference points on this 3-pixel radius circle can be obtained within this neighborhood. The shift RAM mentioned in the previous step is then used to construct a 7×7 matrix. Similarly, the pixel values ​​of the 16 reference points and the center point—a total of 17 points—can be obtained within one clock cycle. To simplify the judgment process, it is directly determined whether there are 12 consecutive darker or brighter reference points. If so, the center point is identified as a corner point, and its grayscale value is modified to 0. Otherwise, its original grayscale value is retained. The grayscale value of the center point is output as the input to the Sobel edge detection module. After this filtering step, only the lane lines in the image have a relatively high grayscale value; the road surface's grayscale value is almost entirely 0.

[0059] Step 4: Use Sobel edge detection to detect lane lines.

[0060] Sobel edge detection is performed using the Sobel operator. The Sobel operator is a matrix that calculates the weighted sum of gray values ​​in a 3×3 pixel neighborhood centered on the current pixel. It consists of two matrices, g and y, in the x and y directions respectively. x and gy The specific weighted template is shown in the matrix below:

[0061]

[0062] Then, the pixel matrix of that neighborhood is multiplied by these two operators to obtain G. x and G y Next, calculate G. x and G y The square root of . These formulas are as follows:

[0063]

[0064]

[0065]

[0066] Finally, G is compared with a set threshold to determine whether the current pixel is an edge.

[0067] Since this algorithm also requires matrix construction, shift RAMs are used for the construction process. When calculating G... x and G y When calculating the square root, the Cordic IP in Vivado is called. This simplifies the calculation and improves accuracy. After the two filtering steps described above, the lane lines are more prominent in the entire image, as shown below. Figure 2 As shown, lane lines can be easily detected even with a relatively small threshold.

Claims

1. A real-time lane detection method based on an improved FAST corner detection algorithm, characterized in that: This is an image processing method for real-time lane line detection on road surfaces, based on an improved FAST corner detection algorithm using FPGA as a filter. First, the RGB image captured by the camera is converted into a grayscale image. Then, the grayscale data is subjected to minimum value filtering. The minimum value filtering method is to find the minimum value of the pixel value in the 3×3 pixel neighborhood of the current pixel and replace the current pixel value with this minimum value. Then, the improved FAST corner detection algorithm is used as the second filtering algorithm. Finally, the Sobel edge detection algorithm is used to detect the lane lines on the road surface. The image information is acquired in real time by the camera, and the image information is processed and transmitted in real time through the ZYNQ field-programmable gate array architecture. Finally, the lane line detection results can be displayed on the monitor in real time via HDMI transmission. The improved FAST corner detection algorithm is as follows: (1) Taking pixel p as the center, take the pixel values ​​of 16 pixels p1, p2, ..., p16 on a circle with a radius of 3 pixels, and denot them as I. p1 I p2 I p3 I p4 I p5 I p6 I p7 I p8 I p9 I p10 I p11 I p12 I p13 I p14 I p15 I p16 And the pixel value of pixel p is I p And set the threshold for corner detection as t; (2) Calculate the pixel difference between the 16 points from p1 to p16 and the center point p, and make the following judgments: First, if at least 3 of the points p1, p5, p9, and p13 are darker or brighter, then proceed to the second judgment; otherwise, exclude the center point p directly. Second, if there are 12 consecutive points that are darker or brighter, then the center point p will be considered as a candidate point. The criteria for determining Darker and Brighter are shown in the following formula: (3) (3) Non-maximum suppression of the image: Calculate the FAST score s at the candidate point. The calculation method is to subtract the pixel value of the above 16 points from the pixel value of the center point p, take the absolute value, and then add these 16 values ​​to get the score s. If there are multiple candidate points in the 3-pixel × 3-pixel neighborhood centered on the candidate point p, then determine whether the score s of the center p is the largest among all candidate points in the neighborhood, and retain it; otherwise, suppress it. The formula for calculating the score s is as follows, where V represents the score and t represents the threshold: (4)。 2. The real-time lane detection method based on the improved FAST corner detection algorithm according to claim 1, characterized in that... The specific method is as follows: Step 1: Convert the RGB image to grayscale. The camera sends real-time image data to the ZYNQ field-programmable gate array. The FPGA on the ZYNQ processes the image into a single-channel 8-bit image, converting the RGB image into a grayscale image. The formula (1) for converting the RGB image into YCbCr is as follows, where Y represents the luminance channel, R represents the red channel of the input RGB image, G represents the green channel of the input RGB image, B represents the blue channel of the input RGB image, and extracting the Y channel yields the grayscale image. Cb is the blue component, and Cr is the red component. (1) Because FPGAs are not good at handling decimals, decimal operations need to be broken down into integer operations and shift operations. The specific formulas are as follows: (2) Step 2: Perform minimum value filtering on the pixel data of the grayscale image. Find the smallest pixel value within a 3-pixel × 3-pixel neighborhood centered on the current pixel p, and then replace the pixel value of the current pixel p with this smallest value. Step 3: Use the improved FAST corner detection algorithm as the second filtering algorithm; Step 4: Use Sobel edge detection to detect lane lines. Sobel edge detection is performed using the Sobel operator, which is a matrix that calculates the weighted sum of gray values ​​in a 3×3 pixel neighborhood centered on the current pixel. It is divided into x-direction and y-direction matrices, g and g', respectively. x and g y The specific weighted template is shown in the matrix below: Then, the pixel matrix of that neighborhood is multiplied by these two operators to obtain G. x and G y Next, calculate G according to the following formula. x and G y The square root of G is obtained as: Finally, G is compared with the set Sobel threshold St. If G is greater than St, the pixel is considered an edge pixel; if G is less than St, the pixel is considered not an edge pixel.

3. The real-time lane detection method based on the improved FAST corner detection algorithm according to claim 2, characterized in that... The specific method of step 2 is as follows: Since minimum filtering needs to be performed in a 3-pixel × 3-pixel neighborhood, in addition to taking the input pixel as the center pixel, it is also necessary to obtain the gray values ​​of the 8 adjacent pixels around the center pixel for comparison. In the hardware FPGA, continuous shift RAM is used to buffer the image captured by the camera as pixel data, and the pixel data that is already in the shift RAM is shifted to construct a matrix so that the gray values ​​of 9 pixels can be read out in one clock cycle. A 3×3 matrix is ​​constructed using 3 shift RAMs. The 9 gray values ​​in the matrix are then compared and the minimum value is taken as the output as the input of the next submodule. When implementing the FAST corner detection algorithm, a circle with a radius of 3 pixels centered on the current pixel is considered as a 7-pixel × 7-pixel neighborhood centered on the current pixel. All 16 reference points on the circle with a radius of 3 pixels can be obtained within this neighborhood. At this time, a 7×7 matrix is ​​constructed using shift RAM. It is directly determined whether there are 12 consecutive reference points that are darker or brighter. If so, the center point is determined to be a corner point, and its grayscale value is modified to 0. Otherwise, its original grayscale value is retained, and the grayscale value of the center point at this time is output as the input of the Sobel edge detection module.

4. A real-time lane detection method based on an improved FAST corner detection algorithm according to claim 2 or 3, characterized in that: When using Sobel edge detection for lane line detection, the algorithm needs to construct a matrix, which is then built using shift RAMs. This is followed by the calculation of G. x and G y When calculating the square root, the Cordic IP in Vivado is called.

Images