Lane detection method and device, vehicle, storage medium and electronic equipment

Abstract

The invention discloses a lane detection method and device, a vehicle, a storage medium and electronic equipment, and relates to the technical field of intelligent driving. The lane detection method comprises the following steps: acquiring a road image in front of a vehicle; performing image threshold segmentation on the road image to obtain a lane line image; converting the lane line image into a top view visual angle to obtain a lane line top view image; determining an order based on the distribution of lane line pixel points in the lane line top view image; and fitting lane line pixel points in the lane line top view image by adopting an n-degree polynomial corresponding to the order to obtain a lane line curve. Therefore, the obtained lane line curve is more accurate, the method can be suitable for the condition that the lane has a plurality of continuous bends, can meet the actual needs, and improves the robustness and accuracy of lane line detection.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving technology, and specifically to a lane detection method, a lane detection device, a vehicle, a machine-readable storage medium, and an electronic device. Background Art

[0002] Lane detection system is an important part of vehicle intelligent assistance system, which is real-time, robust and practical. This system uses image processing technology and computer vision technology to identify the edge of lane lines from road images, thereby realizing functions such as lane departure warning, lane keeping assistance and automatic lane following. Lane detection system is suitable for passenger cars and engineering vehicles such as skid loaders, wheeled excavators, etc.

[0003] Existing lane detection usually samples second-order curves to fit lane lines. However, in actual vehicle driving, there may be a situation where the lane has multiple consecutive bends, which the second-order curve cannot meet, resulting in inaccurate lane line detection. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a lane detection method, a lane detection device, a vehicle, a machine-readable storage medium and an electronic device to solve the problem of inaccurate lane line detection in the prior art.

[0005] In order to achieve the above-mentioned object, the first aspect of the present application provides a lane detection method, comprising:

[0006] Acquire a road image in front of the vehicle;

[0007] Performing image threshold segmentation on the road image to obtain a lane line image;

[0008] Converting the lane line image into a bird's-eye view to obtain a lane line bird's-eye view image;

[0009] Determining an order based on the distribution of lane line pixel points in the lane line overhead image;

[0010] The lane line pixel points in the lane line overhead image are fitted using an n-order polynomial corresponding to the order to obtain a lane line curve.

[0011] In the embodiment of the present application, the determining the order based on the distribution of lane line pixels in the lane line overhead image includes:

[0012] Determining an initial sliding window position based on pixel distribution in the lane line overhead image;

[0013] Sliding a preset sliding window from the initial sliding window position toward the bottom edge of the lane line overhead image to obtain a plurality of lane line sliding window areas, and determining lane line pixel points of each lane line sliding window area respectively; wherein the average value of the horizontal coordinates of the lane line pixels in the current sliding window is used as the horizontal coordinate of the center position of the next sliding window;

[0014] The order is determined based on the lane line pixel points within the lane line sliding window area.

[0015] In the embodiment of the present application, determining the order based on the lane line pixel points within the lane line sliding window area includes:

[0016] Based on the lane line pixel points in the lane line sliding window areas, a plurality of sliding window centroids are obtained;

[0017] Connecting the plurality of sliding window centroids to form a first curve;

[0018] Detecting the number of inflection points on the first curve to obtain a first number of inflection points;

[0019] Based on the first number of inflection points, an order is determined.

[0020] In the embodiment of the present application, determining the order based on the first number of inflection points includes:

[0021] Determine whether the first inflection point number is 0;

[0022] When the number of the first inflection points is determined to be 0, the order is determined to be 2;

[0023] When it is determined that the first inflection point number is not 0, the order is determined to be the first inflection point number plus 1.

[0024] In the embodiment of the present application, it also includes:

[0025] Continuously obtain lane line curves corresponding to multiple frames of road images;

[0026] Based on the lane line curves corresponding to the multiple frames of road images, the curve equation coefficient corresponding to the current lane line curve is corrected to obtain the lane line curve corresponding to the current road image.

[0027] In the embodiment of the present application, the determining the order based on the distribution of lane line pixels in the lane line overhead image includes:

[0028] Drawing a lane line curve corresponding to a previous frame of road image in the lane line overhead image;

[0029] According to the preset boundary range value, the boundaries are determined on the left and right sides of the position where the lane line curve of the previous frame is located in the lane line overhead image, respectively, to obtain the current lane line boundary range;

[0030] Determine the lane line pixel point within the current lane line boundary range;

[0031] Based on the distribution of lane line pixel points, the order is determined using a sliding window method.

[0032] In the embodiment of the present application, the determining the order based on the distribution of lane line pixels in the lane line overhead image includes:

[0033] Cropping the lane line overhead image to obtain a preprocessed lane line overhead image;

[0034] Determining the order based on the distribution of lane line pixels in the pre-processed lane line overhead image;

[0035] The step of fitting the lane line pixel points in the lane line overhead image using the n-th order polynomial corresponding to the order to obtain the lane line curve includes:

[0036] The lane line pixel points in the pre-processed lane line overhead image are fitted using an n-th order polynomial corresponding to the order to obtain a lane line curve.

[0037] In the embodiment of the present application, it also includes:

[0038] Based on a preset scale parameter, converting lane line pixel points in the lane line overhead image into actual lane line points;

[0039] Using an n-order polynomial corresponding to the order to fit the points on the actual lane line to obtain an actual lane line curve;

[0040] Based on the actual lane line curve, determine the coordinates of the lane line starting point;

[0041] Lane information is calculated based on the coordinates of the lane line starting point, where the lane information includes any one or more of lane curvature, lane width, and a distance that the vehicle deviates from the center of the lane.

[0042] In an embodiment of the present application, the actual lane line curve includes a first actual lane line curve and a second actual lane line, and the lane line starting point coordinates include a first lane line starting point coordinates and a second lane line starting point coordinates;

[0043] The calculating lane information based on the lane line starting point coordinates includes:

[0044] Based on the coordinates of the first lane line starting point and the coordinates of the second lane line starting point, respectively, a first lane curvature and a second lane line curve are calculated according to a preset lane line curvature calculation formula, and an average of the first lane curvature and the second lane line curve is obtained to obtain a lane curvature;

[0045] Calculate the lane width based on the coordinates of the starting point of the first lane line and the coordinates of the starting point of the second lane line;

[0046] Calculate the actual lane center position based on the coordinates of the first lane line starting point and the coordinates of the second lane line starting point;

[0047] The center position of the vehicle is calculated based on a preset scale parameter and the midpoint coordinates of the bottom edge of the lane line initial point in the lane line overhead image;

[0048] The difference between the actual lane center position and the vehicle center position is calculated to obtain the vehicle deviation from the lane center distance.

[0049] In an embodiment of the present application, the lane line curve includes a first lane line curve and a second lane line curve, and the method further includes:

[0050] Performing interval sampling on the first lane line and the second lane line respectively to obtain a plurality of sampling points;

[0051] Connecting multiple sampling points to form a polygonal figure, and displaying the polygonal figure on a blank image, wherein the blank image has the same size as the lane line overhead image;

[0052] Convert the blank image into a camera perspective to obtain a lane image;

[0053] The lane image is superimposed on the road image in front of the vehicle to obtain a lane display image.

[0054] In an embodiment of the present application, converting the lane line image into a top view perspective to obtain a lane line top view image includes:

[0055] The lane line image is converted into a bird's-eye view perspective by using a homography matrix to obtain a lane line bird's-eye view image.

[0056] In the embodiment of the present application, it also includes:

[0057] The camera that captures the road image captures a rectangular reference object in front of the vehicle to obtain an imaging image;

[0058] Determine the pixel coordinates of the four vertices of the rectangular reference object in the imaged image to obtain the pixel coordinates of the four vertices of the rectangular reference object;

[0059] Selecting four points in the imaging image to form a first rectangle, wherein four vertices of the first rectangle correspond to four vertices of the rectangular reference object respectively;

[0060] A homography matrix is ​​calculated based on the pixel coordinates of the four vertices of the first rectangle and the pixel coordinates of the four vertices of the rectangular reference object.

[0061] In the embodiment of the present application, it also includes:

[0062] Calculating the length and width of the first rectangle to obtain a reference length and a reference width;

[0063] Calculate the ratio of the length value of the rectangular reference object to the reference length value to obtain a first ratio value;

[0064] Calculate the ratio of the width value of the rectangular reference object to the reference width value to obtain a second ratio value;

[0065] A scale parameter is obtained based on the first scale value and the second scale value.

[0066] A second aspect of the present application provides a lane detection device, comprising:

[0067] An acquisition module, used for acquiring a road image in front of the vehicle;

[0068] A recognition module, used for performing image threshold segmentation on the road image to obtain a lane line image;

[0069] A conversion module, used to convert the lane line image into a bird's-eye view to obtain a lane line bird's-eye view image;

[0070] A determination module, configured to determine the order based on the distribution of lane line pixels in the lane line overhead image;

[0071] The fitting module is used to fit the lane line pixel points in the lane line overhead image using an n-order polynomial corresponding to the order to obtain a lane line curve.

[0072] A third aspect of the present application provides a vehicle that uses the above-mentioned lane detection method to perform lane detection.

[0073] A fourth aspect of the present application provides an electronic device, the electronic device comprising:

[0074] at least one processor;

[0075] a memory connected to the at least one processor;

[0076] The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the above lane detection method by executing the instructions stored in the memory.

[0077] A fifth aspect of the present application provides a machine-readable storage medium having instructions stored thereon, which, when executed by a processor, configure the processor to execute the above-mentioned lane detection method.

[0078] Through the above technical solution, the road image in front of the vehicle is acquired; the image threshold segmentation is performed on the road image to obtain a lane line image; the lane line image is converted into a bird's-eye view to obtain a lane line bird's-eye view image; the order is determined based on the distribution of lane line pixels in the lane line bird's-eye view image; the lane line pixels in the lane line bird's-eye view image are fitted using an n-order polynomial corresponding to the order to obtain a lane line curve. By converting the lane line image into a bird's-eye view perspective, the lane line data is made more accurate. According to the distribution of lane line pixels in the lane line bird's-eye view image, an n-order polynomial of a corresponding order can be determined, so that the lane curve obtained by fitting the lane line pixels using the n-order polynomial can better conform to the actual lane change trend, making the obtained lane line curve more accurate, and can be applied to the situation where the lane has multiple consecutive bends, which can meet actual needs and improve the robustness and accuracy of lane line detection.

[0079] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:

[0081] Figure 1 A schematic diagram of a process flow of a lane detection method according to an embodiment of the present application is schematically shown;

[0082] Figure 2 A schematic diagram of a threshold segmentation result according to an embodiment of the present application is schematically shown;

[0083] Figure 3 A schematic diagram schematically shows a result after projection transformation according to an embodiment of the present application;

[0084] Figure 4 Schematically showing different view conversion schematics according to an embodiment of the present application;

[0085] Figure 5 A histogram according to an embodiment of the present application is schematically shown;

[0086] Figure 6 A schematic diagram of lane detection using a sliding window method according to an embodiment of the present application is shown schematically;

[0087] Figure 7 A schematic diagram of lane detection based on a lane line curve of a previous frame according to an embodiment of the present application is schematically shown;

[0088] Figure 8 A schematic diagram of a lane display effect according to an embodiment of the present application is schematically shown;

[0089] Fig. 9 A schematic diagram of the steps of lane detection according to an embodiment of the present application is schematically shown;

[0090] Fig.10 A schematic diagram of the structure of a lane detection device according to an embodiment of the present application is shown;

[0091] Fig.11 The internal structure diagram of a computer device according to an embodiment of the present application is schematically shown.

[0092] Description of Reference Numerals

[0093] 410 - acquisition module; 420 - identification module; 430 - conversion module; 440 - determination module; 450 - fitting module; A01 - processor; A02 - network interface; A03 - internal memory; A04 - display screen; A05 - input device; A06 - non-volatile storage medium; B01 - operating system; B02 - computer program. DETAILED DESCRIPTION

[0094] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0095] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back...), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0096] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0097] Figure 1 The following schematically shows a flow chart of a lane detection method according to an embodiment of the present application. Figure 1 As shown, an embodiment of the present application provides a lane detection method, which may include the following steps:

[0098] Step 210: Acquire a road image in front of the vehicle;

[0099] In this embodiment, the road image in front of the vehicle refers to the road image captured by the vehicle during driving, which can be obtained by real-time capture by a camera installed on the vehicle, or by extracting each frame of the captured video. The camera can be installed above the vehicle cab with the optical axis facing forward and downward.

[0100] In some embodiments, in order to obtain a more accurate road image, the image captured by the camera can be distorted to correct the image deformation caused by factors such as the lens and shooting angle. The image is distorted based on the camera internal parameters and five distortion parameters to obtain the road image in front of the vehicle. Specifically, the following steps are included:

[0101] The first step is to project the three-dimensional space point onto the image plane and set its coordinates to (x, y).

[0102] In the second step, the points on the image plane are corrected for radial distortion and tangential distortion. The corrected coordinates are:

[0103]

[0104] in, k1, k2, k3 are radial distortion parameters, p1, p2 are tangential distortion parameters, x correct is the corrected horizontal coordinate, y correct is the corrected vertical coordinate.

[0105] The third step is to convert the corrected points to the pixel plane through the camera intrinsic parameters to obtain the correct pixel coordinates (u, v), where:

[0106]

[0107] Among them, f x ,f y ,c x ,c y is the camera internal parameter.

[0108] It should be noted that if the camera intrinsic parameters and distortion parameters are unknown, they can be calibrated through OpenCV functions. The calibration steps are:

[0109] Prepare a calibration plate: Use a chessboard or other known pattern as a calibration plate and take multiple pictures in different poses.

[0110] Feature point detection: Use corner detection algorithms (such as Harris, FAST, SIFT, SURF, etc.) to extract feature points (usually the corners of the chessboard) from each image. Feature point detection can be achieved through OpenCV's findChessboardCorners.

[0111] Calibration: Use the least squares method or other optimization algorithms to calculate and estimate the camera's internal parameters, external parameters, and distortion coefficients to obtain the camera's internal parameters and distortion parameters. Calibration can be achieved using the calibrateCamera function of OpenCV.

[0112] Step 220: performing image threshold segmentation on the road image to obtain a lane line image;

[0113] In this embodiment, the image threshold segmentation can be performed by binary threshold segmentation to divide the pixels in the road image into two categories: foreground and background, where the grayscale value of the foreground pixel is 255 (white) and the grayscale value of the background pixel is 0 (black). A threshold value T can be selected, and for each pixel in the road image, if its grayscale value is greater than T, its grayscale value is set to 255, otherwise it is set to 0. The image threshold segmentation can be implemented using existing technology, which will not be described in detail here.

[0114] In order to improve the segmentation effect, enhance robustness and adaptability, and reduce the misclassification rate, multiple methods can be combined for image threshold segmentation. Since the lane lines are two forward lines on the lane image, you can first use the edge detection method to detect them. For example, you can use the sobel operator in the x-direction to convolve the road image to detect the lane lines. Considering that S (saturation) in the HLS color space is more robust to the effects of shadows and lighting, because saturation usually reflects the brightness of the object's color and has nothing to do with the object's color or the brightness of the color, you can use a method that combines the sobel edge detection in the x-direction and the saturation threshold to perform threshold segmentation on the road image. The above combination method is to take the union. Image threshold segmentation is as follows: Figure 2 As shown, Figure 2 The schematic diagram of the threshold segmentation result according to the embodiment of the present application is schematically shown. The upper left corner is the effect of threshold segmentation after Sobel edge detection in the x direction, the upper right corner is the effect of segmentation based on saturation threshold, the lower left corner is the original image, and the lower right corner is the final effect of combining the two threshold segmentation methods. It can be seen from the effects in the above figures that the threshold segmentation of the road image by combining the Sobel edge detection in the x direction and the saturation threshold method can improve the segmentation effect.

[0115] The above method of combining Sobel edge detection in the x direction and saturation threshold to perform threshold segmentation on the road image belongs to the prior art and will not be described in detail here.

[0116] Step 230: converting the lane line image into a top view perspective to obtain a lane line top view image;

[0117] In this embodiment, the lane line image is a camera perspective. In order to detect the lane more accurately, the lane line image can be converted into a bird's-eye view perspective. The bird's-eye view perspective is a perspective perpendicular to the ground, so that the lane line data obtained is more accurate.

[0118] In some embodiments, in order to accurately convert the lane line image into a top-view perspective, the lane line image may be converted into a top-view perspective through a homography matrix to obtain a lane line top-view image.

[0119] In this embodiment, for a point A in the physical space, its coordinates on the image plane can be expressed as (x1, y1). The imaging perspective can be converted into a top-down perspective through the homography matrix H. At this time, the coordinates of point A on the image plane can be expressed as (x2, y2), then:

[0120]

[0121] in, is the homography matrix H.

[0122] The homography matrix can be used to describe the mapping relationship between two planes. The image plane and the top view plane can be aligned through the homography matrix to achieve accurate image conversion.

[0123] like Figure 3 As shown, Figure 3 The schematic diagram of the result after the projection transformation according to the embodiment of the present application is schematically shown. The image after threshold segmentation is projected and transformed through the homography matrix to be converted into a top view perspective, and the left figure is the top view perspective; the top view perspective is converted into the original perspective through the inverse transformation of the homography matrix, as shown in the right figure. It can be seen from the above that the bidirectional conversion between the top view perspective and the original perspective can be achieved through the homography matrix.

[0124] Since the homography matrix has scale equivalence, it has only 8 degrees of freedom and needs to select at least four groups of corresponding points for solving. In some embodiments, the solution process of the homography matrix is ​​also included, which specifically includes the following steps:

[0125] First, a rectangular reference object in front of the vehicle is photographed by a camera that photographs the road image to obtain an imaging image;

[0126] In this embodiment, the rectangular reference object may be a rectangular cloth of known size, placed horizontally in front of the vehicle, and the camera can completely capture the four corners of the rectangular cloth, thereby obtaining an imaging image through the camera of the road image.

[0127] Then, the pixel coordinates of the four vertices of the rectangular reference object in the imaging image are determined to obtain the pixel coordinates of the four vertices of the rectangular reference object;

[0128] In this embodiment, after the imaging image is obtained, the pixel coordinates of the four corners of the rectangular reference object on the imaging image are recorded to obtain the pixel coordinates of the four vertices.

[0129] Then, four points are selected in the imaging image to form a first rectangle, wherein four vertices of the first rectangle correspond to four vertices of the rectangular reference object respectively;

[0130] In this embodiment, the size of the imaging image can be kept unchanged, and then four points are selected horizontally and vertically at the four corners. The positions of these four points correspond to the four vertices of the rectangular reference object, so that the four selected points can be used as four vertices to form a rectangle, and the size of the rectangle is proportional to the size of the rectangular reference object.

[0131] Finally, based on the pixel coordinates of the four vertices of the first rectangle and the pixel coordinates of the four vertices of the rectangular reference object, a homography matrix is ​​calculated.

[0132] In this embodiment, four points are selected in the image, and the pixel coordinates of the four points are recorded to obtain the pixel coordinates of the four vertices of the first rectangle. According to the pixel coordinates of the four vertices of the first rectangle and the pixel coordinates of the four vertices of the rectangular reference object, the homography matrix is ​​obtained by least square method or other linear algebra method. The above-mentioned method of solving the homography matrix by least square method or other linear algebra method belongs to the prior art and will not be described in detail here.

[0133] By shooting a rectangular reference object in front of the vehicle with a camera that shoots the road image, the pixel coordinates of the four vertices of the rectangular reference object in the imaging image are determined, and four points are selected in the imaging image, and the four vertices correspond to the four vertices of the rectangular reference object respectively, so that four groups of corresponding points can be obtained. Finally, based on these four groups of corresponding points, the homography matrix can be calculated quickly and accurately.

[0134] In some embodiments, in the process of determining the homography matrix, the ratio relationship between the image and the rectangular reference object may be further determined to obtain a ratio parameter, that is, the method further includes the following steps:

[0135] First, the length value and the width value of the first rectangle are calculated respectively to obtain a reference length value and a reference width value;

[0136] In this embodiment, after selecting four points in the imaging image, the length and width of the first rectangle can be calculated respectively according to the pixel horizontal coordinate difference and the pixel vertical coordinate difference between the four selected points, that is, the reference length value and the reference width value are obtained.

[0137] Then, the ratio of the length value of the rectangular reference object to the reference length value is calculated to obtain a first ratio value;

[0138] In this embodiment, the length value of the rectangular reference object is the actual length of the rectangular reference object, that is, the actual horizontal distance. Then, the ratio of the actual horizontal distance to the pixel horizontal coordinate difference, that is, the ratio of the length value of the rectangular reference object to the reference length value, is calculated to obtain a first ratio value, which can be used as a x express.

[0139] Then, the ratio of the width value of the rectangular reference object to the reference width value is calculated to obtain a second ratio value;

[0140] In this embodiment, the width value of the rectangular reference object is the actual width of the rectangular reference object, that is, the actual longitudinal distance. Then, the ratio of the actual longitudinal distance to the pixel longitudinal coordinate difference, that is, the ratio of the width value of the rectangular reference object to the reference width value, is calculated to obtain a second ratio value, which can be used as a y express.

[0141] Finally, a ratio parameter is obtained based on the first ratio value and the second ratio value.

[0142] In this embodiment, the ratio parameter includes a first ratio value and a second ratio value.

[0143] like Figure 4 As shown, Figure 4 The schematic diagram of different view conversions according to the embodiment of the present application is shown schematically. In the world coordinate system, a rectangular cloth is spread on the ground. When viewed from the front camera, it will become a trapezoidal image, which is larger near and smaller far away. This makes it impossible to judge the distance between two pixel coordinates in the image. This trapezoidal image can be converted into a rectangular image through perspective projection. At this time, the ratio of the length of the rectangular cloth to the length of the rectangular image between the rectangular image and the actual rectangular cloth is a constant value a. y , the width of the rectangular cloth is a constant value a greater than the width of the rectangular image x .

[0144] By calculating the ratio of the actual lateral distance to the pixel lateral coordinate difference and the ratio of the actual longitudinal distance to the pixel longitudinal coordinate difference based on the horizontal and vertical coordinate differences between the four selected points and the length and width of the actual rectangular reference object, a proportional parameter is obtained. The proportional parameter can represent the proportional relationship between the pixel point in the image and the corresponding position in the actual object, which helps to convert the pixel point in the image to the corresponding position in reality, so as to facilitate the subsequent lane information calculation.

[0145] Step 240: determining an order based on the distribution of lane line pixels in the lane line overhead image;

[0146] In this embodiment, after threshold segmentation and projection transformation, a binary lane line image, that is, a lane line overhead image, can be obtained, and then the lane line can be further detected and fitted based on the lane line overhead image. The number of windows and the window size can be customized, and each window covers a certain range of pixels. In each window, the pixels that may belong to the lane line are determined, and the pixels can be screened by pixel value. Then, an n-order polynomial fitting is performed on the pixels in the window, specifically, a fitting method such as the least squares method can be used to find an n-order polynomial function so that the error between the function and the pixels in the window is minimized. n-order polynomial fitting curves of different orders can be used according to the lane curve conditions, that is, different lane curve conditions correspond to different orders, and the curve conditions can be determined specifically by the distribution of lane line pixels in the lane line overhead image.

[0147] In some embodiments, the determining the order based on the distribution of lane line pixels in the lane line overhead image includes:

[0148] First, based on the pixel distribution in the lane line overhead image, an initial sliding window position is determined;

[0149] In this embodiment, a pixel histogram of the binary lane line can be drawn according to the distribution of pixels in the lane line overhead image. Then, according to the pixel histogram, for the left and right parts of the lane line overhead image, the positions with the most pixels are taken as the starting horizontal coordinates of the lane line, and the bottom edge of the lane line overhead image is taken as the starting vertical coordinate to obtain the initial sliding window position.

[0150] The above pixel histogram can be specifically traversing the binarized lane line overhead image, and for each pixel point, checking whether its value is a pixel value representing the lane line. If it is a lane line pixel, then according to its position information in the lane line overhead image, the corresponding counter is increased by one, so that the number of lane line pixels at different horizontal coordinate positions can be counted. The horizontal coordinate can be set as the pixel position, and the vertical coordinate can be set as the number of lane line pixels at the corresponding position, and the pixel histogram of the binary lane line can be obtained. Figure 5 As shown, Figure 5 A histogram according to an embodiment of the present application is schematically shown, in which the horizontal axis is the pixel value and the vertical axis is the frequency. The binarized lane line image pixel histogram will show an obvious bimodal distribution. One peak represents the background pixel (usually black, with a pixel value of 0), and the other peak represents the lane line pixel (usually white, with a pixel value of 255). The position and width range of the lane line can be preliminarily determined based on the position and width of the peak representing the lane line in the histogram.

[0151] Then, a preset sliding window is slid from the initial sliding window position toward the bottom edge of the lane line overhead image to obtain multiple lane line sliding window areas, and lane line pixel points of each lane line sliding window area are determined respectively; wherein the average value of the horizontal coordinates of the lane line pixels in the current sliding window is used as the horizontal coordinate of the center position of the next sliding window;

[0152] In this embodiment, after the initial sliding window position is determined, the size and number of windows are customized, and the sliding window operation is performed upward. All pixel points are traversed in each sliding window respectively, and the horizontal and vertical coordinates of the lane line pixel points in each sliding window are used as the pixel coordinates of the lane line to obtain the lane line pixel points in each lane line sliding window area, and the average horizontal coordinate of the lane line pixels in the current window is used as the horizontal coordinate of the center position of the next window.

[0153] Finally, based on the lane line pixel points within the lane line sliding window area, the order is determined.

[0154] By adopting the sliding window operation, the lane line pixels can be quickly and accurately determined in the lane line overhead image, which helps to determine the order based on the lane line pixel distribution.

[0155] In some embodiments, the determining the order based on the lane line pixel points within the lane line sliding window area includes the following steps:

[0156] The first step is to obtain multiple sliding window centroids based on lane line pixel points in the lane line sliding window areas;

[0157] In this embodiment, the centroid coordinates of the lane line pixels in each sliding window are calculated, that is, the mean of the horizontal coordinates and the mean of the vertical coordinates of the lane line pixels in each sliding window are calculated to obtain the window centroid coordinates.

[0158] The second step is to connect the centroids of the plurality of sliding windows to form a first curve;

[0159] The third step is to detect the number of inflection points on the first curve to obtain a first number of inflection points;

[0160] In this embodiment, the lane change trend can be determined by the first curve. The inflection point on the first curve refers to the point on the first curve when the first curve changes from concave to convex, or from convex to concave. In specific implementation, the horizontal coordinate of the sliding window centroid can be traversed from top to bottom according to the vertical coordinate from small to large. If the horizontal coordinate changes from small to large and then from large to small, it corresponds to an inflection point; if the horizontal coordinate changes from large to small and then from small to large, it also corresponds to an inflection point. When traversing, the above can be in the order from top to bottom along the vertical coordinate, or in the order from bottom to top along the vertical coordinate.

[0161] The fourth step is to determine the order based on the first number of inflection points;

[0162] Specifically, when determining the order, it can be determined whether the first number of inflection points is 0; when it is determined that the first number of inflection points is 0, the order is determined to be 2; when it is determined that the first number of inflection points is not 0, the order is determined to be the first number of inflection points plus 1.

[0163] For example, if the number of inflection points of the first curve is m, and if m=0, a second-order curve is used to fit the curve equation of the lane line in the pixel coordinate system; if the number of inflection points m≥1, an n-order curve is used to fit the curve equation of the lane line in the pixel coordinate system, which is expressed as:

[0164] u(v)=a n v n +a n-1 v n-1 +…+a1v+a0,

[0165] Where n = m + 1, u is the horizontal coordinate of the pixel coordinate system, v is the vertical coordinate of the pixel coordinate system, a0, a1, ..., a n is the coefficient of the n-th order curve, n≥2.

[0166] By connecting the sliding window centroids of each sliding window to form a first curve, the lane change trend can be determined through the first curve, and then the order can be quickly and accurately determined, which helps to obtain a suitable fitting function to fit all pixel points of the lane line, so as to obtain an accurate lane line curve.

[0167] The above-mentioned step of determining the order is obtained by traversing each pixel point in the lane line overhead image through the sliding window method to determine the lane pixel point, such as Figure 6 As shown, Figure 6 A schematic diagram of lane detection using a sliding window method according to an embodiment of the present application is schematically shown. In some embodiments, considering the situation of multiple consecutive frames of images, if the lane line curve of the previous frame image has been obtained using the sliding window method, for subsequent frames of images, the order can be determined based on the lane line curve of the previous frame image to speed up the detection speed.

[0168] Please see Figure 7 , Figure 7 A schematic diagram of lane detection based on a lane line curve of a previous frame according to an embodiment of the present application is schematically shown. In some embodiments, the order is determined based on the distribution of lane line pixels in the lane line overhead image, including the following steps:

[0169] The first step is to draw a lane line curve corresponding to the previous frame of road image in the lane line overhead image;

[0170] In this embodiment, the lane line curve corresponding to the previous frame of road image can be obtained by using steps 210 to 250. If the lane line curve corresponding to the previous frame of road image has been obtained by using the sliding window method, the curve can be drawn in the current lane line overhead image.

[0171] In the second step, according to the preset boundary range value, the boundaries are determined on the left and right sides of the position where the lane line curve of the previous frame is located in the lane line overhead image, respectively, to obtain the current lane line boundary range;

[0172] In this embodiment, the preset boundary range value may be pre-set and may be a lateral boundary value, that is, left and right boundaries may be established on both sides of the lane line curve of the previous frame, and the left and right boundary ranges are the current lane line boundary ranges.

[0173] Step 3: Determine the lane line pixel points within the boundary range of the current lane line;

[0174] In this embodiment, the lane line pixel point can be determined according to the lane line pixel value within the current lane line boundary range.

[0175] In the fourth step, based on the distribution of lane line pixels, a sliding window method is used to determine the order.

[0176] In this embodiment, the window centroid coordinates can be calculated according to the sliding window method, and then the order of the fitting curve can be determined. Specifically, it includes: performing a sliding window operation within the boundary range of the current lane line to obtain multiple current lane line sliding window areas; then based on the coordinates of the current lane line pixel points in each current lane line sliding window area, obtaining the sliding window centroid corresponding to each current lane line sliding window area; calculating the centroid coordinates of the lane line pixels in each sliding window, that is, calculating the mean of the horizontal coordinates and the mean of the vertical coordinates of the lane line pixel points in each current sliding window, and obtaining the window centroid coordinates corresponding to each current lane line sliding window area. Connect the sliding window centroids corresponding to the current lane line sliding window areas to form a second curve; detect the number of inflection points on the second curve to obtain the second number of inflection points; determine the current order based on the second number of inflection points. It should be noted that the specific process of determining the order is the same as the step of determining the order by traversing each pixel point in the lane line overhead image through the sliding window method, and will not be repeated here.

[0177] By detecting lane lines around the lane line curve fitted in the previous frame, setting a horizontal boundary, searching for lane line pixel points within the left and right boundary ranges, and then calculating the centroid coordinates according to the sliding window method to determine the order, the traversal of lane line pixel points can be reduced, thus speeding up the detection speed.

[0178] In some embodiments, in order to improve the detection efficiency, the lane line overhead image may be preprocessed. Accordingly, the order is determined based on the distribution of lane line pixels in the lane line overhead image, including:

[0179] First, the lane line overhead image is cropped to obtain a preprocessed lane line overhead image;

[0180] Then, based on the distribution of lane line pixel points in the pre-processed lane line overhead image, the order is determined.

[0181] In this embodiment, considering that there are still some background parts in the lane line overhead image, in order to improve the detection efficiency, the lane line overhead image can be further cropped, for example: it can be divided horizontally from the middle of the lane line overhead image, and then the lower half of the image is taken. It is also possible to divide the lane line overhead image horizontally into multiple parts, and take the bottom part of the image as the preprocessed lane line overhead image. For example, the lane line overhead image is evenly divided into three parts horizontally, and one of the bottom three parts is taken as the preprocessed lane line overhead image. The specific method can be determined according to the actual situation. The preprocessed lane line overhead image after cropping removes the background part, thereby improving the detection efficiency.

[0182] Step 250: Using an n-th order polynomial corresponding to the order, the lane line pixel points in the lane line overhead image are fitted to obtain a lane line curve.

[0183] In this embodiment, after the order is determined, the corresponding polynomial can be determined, and through the fitting process, a continuous lane line curve can be finally obtained, which represents the shape of the lane line in the image. It should be noted that the specific process of fitting using an n-order polynomial mentioned in this embodiment belongs to the prior art and will not be described in detail in this embodiment.

[0184] It should be noted that, considering that the lane line is generally divided into two left and right parts, which can be detected separately, the lane line overhead image can be divided into two left and right parts, specifically, it can be divided into two parts from the middle of the bottom of the lane line overhead image along the lane extension direction, so that each part includes a lane line. For each part, step 240 and step 250 are executed to fit the lane line pixels of each part. After the fitting curves are obtained for each part, the lane line curves on the left and right sides of the lane are obtained, that is, the lane line curves are obtained.

[0185] In some embodiments, in order to improve detection efficiency, the lane line overhead view image can be cropped to obtain a preprocessed lane line overhead view image. Accordingly, the lane line pixel points in the lane line overhead view image are fitted using an n-order polynomial corresponding to the order to obtain a lane line curve, including: fitting the lane line pixel points in the preprocessed lane line overhead view image using an n-order polynomial corresponding to the order to obtain a lane line curve.

[0186] In the above implementation process, the road image in front of the vehicle is acquired; the image threshold segmentation is performed on the road image to obtain a lane line image; the lane line image is converted into a bird's-eye view to obtain a lane line bird's-eye view image; the order is determined based on the distribution of lane line pixels in the lane line bird's-eye view image; the lane line pixels in the lane line bird's-eye view image are fitted using an n-order polynomial corresponding to the order to obtain a lane line curve. By converting the lane line image into a bird's-eye view, the lane line data is made more accurate. According to the distribution of lane line pixels in the lane line bird's-eye view image, an n-order polynomial of a corresponding order can be determined, so that the lane curve obtained by fitting the lane line pixels using the n-order polynomial can better conform to the actual lane change trend, making the obtained lane line curve more accurate, and can be applied to the situation where the lane has multiple consecutive bends, which can meet actual needs and improve the robustness and accuracy of lane line detection.

[0187] In some embodiments, it also includes:

[0188] First, the lane line curves corresponding to multiple frames of road images are continuously acquired;

[0189] In this embodiment, the number of the frames can be determined according to actual conditions, for example, it can be 5 consecutive frames. The lane line curve corresponding to the road image can be obtained by using the above steps 210 to 250.

[0190] Then, based on the lane line curves corresponding to the multiple frames of road images, the curve equation coefficient corresponding to the current lane line curve is corrected to obtain the lane line curve corresponding to the current road image.

[0191] In this embodiment, after the lane line curve is obtained, the lane line curve corresponds to a curve equation, and the curve equation contains coefficients. The curve equation coefficients corresponding to multiple lane line curves can be weighted averaged as the final coefficient of the lane line curve corresponding to the current road image, thereby obtaining the lane line curve corresponding to the current road image.

[0192] By taking the weighted average of the curve equation coefficients obtained from multiple consecutive frames of images as the final coefficients of the curve equation in the current frame pixel coordinate system, the lane line curve corresponding to the current road image is made more accurate, further improving the accuracy of lane line detection.

[0193] In some embodiments, lane information may be further calculated, including:

[0194] Firstly, based on a preset scale parameter, the lane line pixel points in the lane line overhead image are converted into actual lane line points;

[0195] In this embodiment, the above-mentioned preset scale parameters can be determined in the process of determining the homography matrix by calculating the ratio of the actual horizontal distance to the pixel horizontal coordinate difference, and the ratio of the actual vertical distance to the pixel vertical coordinate difference according to the horizontal and vertical coordinate differences between the four selected points and the length and width of the actual rectangular reference object, thereby obtaining the scale parameters. The scale parameters can represent the proportional relationship between the pixel points in the image and the corresponding positions in the actual object. The lane line pixel points can be converted into actual lane line points through the scale parameters.

[0196] Then, using an n-order polynomial corresponding to the order, fitting the points on the actual lane line to obtain an actual lane line curve;

[0197] In this embodiment, the order of the actual lane line curve is the same as the order of the lane line curve. After the lane line curve is obtained, its order can be determined. Correspondingly, the order of the actual lane line curve is also determined, and then the n-order polynomial of the corresponding order is used to fit the points on the actual lane line to obtain the actual lane line curve. According to the ratio a of the actual lateral distance and the pixel lateral coordinate difference x, the ratio of the actual longitudinal distance to the difference in the longitudinal coordinates of the pixel a y , convert all pixel coordinates of the left and right lane lines into world two-dimensional plane coordinates. The origin position and coordinate axis direction of this coordinate system are the same as those of the image coordinate system, but the scale is different. Use the n-order curve to fit the curve equation of the lane line in the world two-dimensional plane coordinate system:

[0198] X(Y)=A n Y n +A n-1 Y n-1 +…+A1Y+A0,

[0199] Among them, X is the horizontal coordinate of the world two-dimensional plane coordinate system, Y is the vertical coordinate of the world two-dimensional plane coordinate system, A0, A1, ..., A n is the coefficient of the n-th order curve, n≥2.

[0200] Then, based on the actual lane line curve, the coordinates of the lane line starting point are determined;

[0201] In this embodiment, the starting point of the lane line is located at the bottom edge of the image, at the ordinate Y of the world two-dimensional plane coordinate system. b For: Y b =u b *a y , where u b Y is the ordinate of the lane line starting point in the image pixel coordinate system. b Substituting the actual lane line curve, that is, substituting the Y in the curve equation of the lane line in the world two-dimensional plane coordinate system, the horizontal coordinate of the lane line in the world two-dimensional plane coordinate system can be obtained, and then the coordinates of the starting point of the lane line can be obtained.

[0202] Finally, based on the coordinates of the lane line starting point, the lane information is calculated, where the lane information includes any one or more of lane curvature, lane width, and a distance from the center of the lane where the vehicle is offset.

[0203] In some embodiments, the actual lane line curve includes a first actual lane line curve and a second actual lane line, and the lane line starting point coordinates include a first lane line starting point coordinates and a second lane line starting point coordinates;

[0204] In this embodiment, since the lane line curve includes two left and right curves, correspondingly, two actual lane line curves on the left and right can be obtained, namely the first actual lane line curve and the second actual lane line. Correspondingly, the lane line starting point coordinates corresponding to the first actual lane line curve are the first lane line starting point coordinates, and the lane line starting point coordinates corresponding to the second actual lane line curve are the second lane line starting point coordinates.

[0205] Accordingly, the lane information is calculated based on the lane line starting point coordinates, including:

[0206] The first step is to calculate the first lane curvature and the second lane line curve according to a preset lane line curvature calculation formula based on the coordinates of the first lane line starting point and the coordinates of the second lane line starting point, and calculate the average of the first lane curvature and the second lane line curve to obtain the lane curvature;

[0207] In this embodiment, the preset lane line curvature calculation formula is:

[0208]

[0209] Where R is the radius of curvature, K is the curvature, X' is the first-order derivative of X, and X' = nA n Y n-1 +(n-1)A n-1 Y n-2 +…+2A2Y+Z1, X″ is the second-order derivative of X, X″=n(n-1)AnYn-2+(n-1)(n-2)An-1Yn-3+…+2A2.

[0210] The Y coordinates of the first lane line starting point and the Y coordinates of the second lane line starting point are respectively b Substituting Y into the lane line curvature calculation formula can obtain the corresponding curve curvature, that is, the first lane curvature and the second lane line curve, that is, the curvature of the left and right lane lines, and then take the average value of the curvature of the left and right lane lines as the curvature of the entire lane.

[0211] Step 2: Calculate the lane width based on the coordinates of the starting point of the first lane line and the coordinates of the starting point of the second lane line;

[0212] In this embodiment, the horizontal coordinate difference between the coordinates of the starting point of the first lane line and the coordinates of the starting point of the second lane line is the lane width.

[0213] Step 3: Calculate the actual lane center position based on the coordinates of the starting point of the first lane line and the coordinates of the starting point of the second lane line;

[0214] In this embodiment, the average value of the horizontal coordinates of the coordinates of the starting point of the first lane line and the starting point of the second lane line is calculated to be the average value of the horizontal coordinates of the left and right lane lines, which is the actual lane center position.

[0215] Step 4: Calculate the center position of the vehicle based on the preset scale parameter and the midpoint coordinates of the bottom edge of the lane line initial point in the lane line overhead image;

[0216] In this embodiment, when the camera is located at the center of the vehicle, the horizontal coordinate of the midpoint of the bottom edge of the lane line top view image is multiplied by the ratio a of the difference between the actual horizontal distance and the horizontal coordinate of the pixel. x , the center position of the vehicle can be obtained.

[0217] The fifth step is to calculate the difference between the actual lane center position and the vehicle center position to obtain the vehicle deviation lane center distance.

[0218] In this embodiment, the difference between the actual lane center position and the vehicle center position is the lateral offset distance of the vehicle chassis relative to the lane center. Further, a threshold for deviation from the lane center can be set according to the width of the vehicle. The range of the threshold is less than half of the lane width minus half of the vehicle width. Once the deviation from the lane center is greater than the set threshold, an early warning is issued.

[0219] It should be noted that the above calculation of lane curvature, lane width and vehicle offset distance from lane center may be implemented individually or in multiple steps, depending on specific needs.

[0220] By converting the coordinates of all pixel points of the left and right lane lines into points in the world two-dimensional plane coordinates according to the scale parameter, lane information such as lane curvature, lane width and vehicle offset distance from the lane center can be accurately calculated, thereby improving the accuracy of lane detection.

[0221] In one embodiment, the lane line curve includes a first lane line curve and a second lane line curve, and the method further includes:

[0222] Firstly, the first lane line and the second lane line are sampled at intervals to obtain a plurality of sampling points;

[0223] In this embodiment, the first lane line curve and the second lane line curve are left and right lane line curves, and the above sampling can be the left and right lane line curves fitted in the pixel coordinate system, with sampling performed once at every longitudinal interval of one pixel.

[0224] Then, multiple sampling points are connected to form a polygonal figure, and the polygonal figure is displayed on a blank image, and the blank image has the same size as the lane line overhead image;

[0225] In this embodiment, all sampling points are enclosed into a closed polygon, which is displayed on a blank image of the same size as the lane line top view image.

[0226] Then, the blank image is converted into a camera perspective to obtain a lane image;

[0227] In this embodiment, the blank image can be converted into the camera perspective by inverse transformation of the homography matrix.

[0228] Finally, the lane image is superimposed on the road image in front of the vehicle to obtain a lane display image.

[0229] In this embodiment, the lane image can be superimposed on the road image in front of the vehicle, so that the lane can be displayed on the road image in front of the vehicle. At the same time, the lane curvature radius, lane width, distance from the center of the lane, etc. can also be displayed on the road image in front of the vehicle, for example, in the upper left corner of the road image. Figure 8 As shown, Figure 8 A schematic diagram of a lane display effect according to an embodiment of the present application is schematically shown.

[0230] The left and right lane line curves fitted in the pixel coordinate system are sampled once every longitudinal pixel, and all the sampling points are enclosed into a closed polygon. The polygon is displayed on a blank image of the same size as the original image. The blank image is then converted into a camera perspective through the inverse transformation of the homography matrix and superimposed on the original image to achieve lane display, so that drivers can view the road more easily.

[0231] Please see Fig. 9 , Fig. 9 The schematic diagram of the lane detection steps according to the embodiment of the present application is schematically shown. The camera is installed above the cab of the engineering vehicle with the optical axis facing forward and downward. The camera is first expressed and the distortion is corrected. Then the image taken by the camera is segmented by thresholds in a combination of multiple methods. Then, the image is transformed into a top view by projection. Then, the lane line pixel points are found, the lane line curve is fitted, and finally, the lane curvature, width and offset are calculated.

[0232] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0233] This embodiment provides a vehicle that uses the above lane detection method to perform lane detection.

[0234] In this embodiment, the vehicle may be a passenger car, an engineering vehicle, such as a skid loader, a wheeled excavator, etc. The lane detection method may be used to accurately perform lane detection to assist driving.

[0235] Please see Fig.10 , Fig.10 The schematic diagram of the structure of a lane detection device according to an embodiment of the present application is shown schematically. This embodiment provides a lane detection device, including an acquisition module 410, an identification module 420, a conversion module 430, a determination module 440 and a fitting module 450, wherein:

[0236] An acquisition module 410 is used to acquire a road image in front of the vehicle;

[0237] The recognition module 420 is used to perform image threshold segmentation on the road image to obtain a lane line image;

[0238] A conversion module 430 is used to convert the lane line image into a top view perspective to obtain a lane line top view image;

[0239] A determination module 440, configured to determine an order based on a distribution of lane line pixels in the lane line overhead image;

[0240] The fitting module 450 is used to fit the lane line pixel points in the lane line overhead image using an n-order polynomial to obtain a lane line curve.

[0241] The lane detection device includes a processor and a memory. The acquisition module 410, the recognition module 420, the conversion module 430, the determination module 440 and the fitting module 450 are all stored in the memory as program units, and the processor executes the program units stored in the memory to realize corresponding functions.

[0242] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the accuracy of lane line detection can be improved by adjusting kernel parameters.

[0243] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0244] An embodiment of the present invention provides a machine-readable storage medium on which a program is stored. When the program is executed by a processor, the lane detection method is implemented.

[0245] An embodiment of the present invention provides a processor, which is used to run a program, wherein the lane detection method is executed when the program is running.

[0246] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.11 As shown. The computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05 and a memory (not shown in the figure) connected through a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A06. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor A01, a lane detection method is implemented. The display screen A04 of the computer device can be a liquid crystal display or an electronic ink display, and the input device A05 of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0247] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0248] In one embodiment, the lane detection device provided by the present application can be implemented in the form of a computer program. The computer program can be Fig.11 The computer device shown in the figure is run. The memory of the computer device can store various program modules constituting the lane detection device, for example, Fig.10 The acquisition module 410, the identification module 420, the conversion module 430, the determination module 440 and the fitting module 450 are shown. The computer program composed of various program modules enables the processor to execute the steps of the lane detection method of each embodiment of the present application described in this specification.

[0249] Fig.11 The computer device shown can be Fig.10The acquisition module 410 in the lane detection device shown executes step 210. The computer device may execute step 220 via the identification module 420. The computer device may execute step 230 via the conversion module 430. The computer device may execute step 240 via the determination module 440. The computer device may execute step 250 via the fitting module 450.

[0250] An embodiment of the present application provides an electronic device, the electronic device comprising: at least one processor; a memory connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, the at least one processor implements the above lane detection method by executing the instructions stored in the memory, and the processor implements the following steps when executing the instructions:

[0251] Acquire a road image in front of the vehicle;

[0252] Performing image threshold segmentation on the road image to obtain a lane line image;

[0253] Converting the lane line image into a bird's-eye view to obtain a lane line bird's-eye view image;

[0254] Determining an order based on the distribution of lane line pixel points in the lane line overhead image;

[0255] The lane line pixel points in the lane line overhead image are fitted using an n-order polynomial corresponding to the order to obtain a lane line curve.

[0256] In one embodiment, the determining the order based on the distribution of lane line pixels in the lane line overhead image includes:

[0257] Determining an initial sliding window position based on pixel distribution in the lane line overhead image;

[0258] Sliding a preset sliding window from the initial sliding window position toward the bottom edge of the lane line overhead image to obtain a plurality of lane line sliding window areas, and determining lane line pixel points of each lane line sliding window area respectively; wherein the average value of the horizontal coordinates of the lane line pixels in the current sliding window is used as the horizontal coordinate of the center position of the next sliding window;

[0259] The order is determined based on the lane line pixel points within the lane line sliding window area.

[0260] In one embodiment, determining the order based on the lane line pixel points within the lane line sliding window area includes:

[0261] Based on the lane line pixel points in the lane line sliding window areas, a plurality of sliding window centroids are obtained;

[0262] Connecting the plurality of sliding window centroids to form a first curve;

[0263] Detecting the number of inflection points on the first curve to obtain a first number of inflection points;

[0264] Based on the first number of inflection points, an order is determined.

[0265] In one embodiment, determining the order based on the first number of inflection points includes:

[0266] Determine whether the first inflection point number is 0;

[0267] When the number of the first inflection points is determined to be 0, the order is determined to be 2;

[0268] When it is determined that the first inflection point number is not 0, the order is determined to be the first inflection point number plus 1.

[0269] In one embodiment, it further includes:

[0270] Continuously obtain lane line curves corresponding to multiple frames of road images;

[0271] Based on the lane line curves corresponding to the multiple frames of road images, the curve equation coefficient corresponding to the current lane line curve is corrected to obtain the lane line curve corresponding to the current road image.

[0272] In one embodiment, the determining the order based on the distribution of lane line pixels in the lane line overhead image includes:

[0273] Drawing a lane line curve corresponding to a previous frame of road image in the lane line overhead image;

[0274] According to the preset boundary range value, the boundaries are determined on the left and right sides of the position where the lane line curve of the previous frame is located in the lane line overhead image, respectively, to obtain the current lane line boundary range;

[0275] Determine the lane line pixel point within the current lane line boundary range;

[0276] Based on the distribution of lane line pixel points, the order is determined using a sliding window method.

[0277] In one embodiment, the determining the order based on the distribution of lane line pixels in the lane line overhead image includes:

[0278] Cropping the lane line overhead image to obtain a preprocessed lane line overhead image;

[0279] Determining the order based on the distribution of lane line pixels in the pre-processed lane line overhead image;

[0280] The step of fitting the lane line pixel points in the lane line overhead image using the n-th order polynomial corresponding to the order to obtain the lane line curve includes:

[0281] The lane line pixel points in the pre-processed lane line overhead image are fitted using an n-th order polynomial corresponding to the order to obtain a lane line curve.

[0282] In one embodiment, it further includes:

[0283] Based on a preset scale parameter, converting lane line pixel points in the lane line overhead image into actual lane line points;

[0284] Using an n-order polynomial corresponding to the order to fit the points on the actual lane line to obtain an actual lane line curve;

[0285] Based on the actual lane line curve, determine the coordinates of the lane line starting point;

[0286] Lane information is calculated based on the coordinates of the lane line starting point, where the lane information includes any one or more of lane curvature, lane width, and a distance that the vehicle deviates from the center of the lane.

[0287] In one embodiment, the actual lane line curve includes a first actual lane line curve and a second actual lane line, and the lane line starting point coordinates include a first lane line starting point coordinates and a second lane line starting point coordinates;

[0288] The calculating lane information based on the lane line starting point coordinates includes:

[0289] Based on the coordinates of the first lane line starting point and the coordinates of the second lane line starting point, respectively, a first lane curvature and a second lane line curve are calculated according to a preset lane line curvature calculation formula, and an average of the first lane curvature and the second lane line curve is obtained to obtain a lane curvature;

[0290] Calculate the lane width based on the coordinates of the starting point of the first lane line and the coordinates of the starting point of the second lane line;

[0291] Calculate the actual lane center position based on the coordinates of the first lane line starting point and the coordinates of the second lane line starting point;

[0292] The center position of the vehicle is calculated based on a preset scale parameter and the midpoint coordinates of the bottom edge of the lane line initial point in the lane line overhead image;

[0293] The difference between the actual lane center position and the vehicle center position is calculated to obtain the vehicle deviation from the lane center distance.

[0294] In one embodiment, the lane line curve includes a first lane line curve and a second lane line curve, and the method further includes:

[0295] Performing interval sampling on the first lane line and the second lane line respectively to obtain a plurality of sampling points;

[0296] Connecting multiple sampling points to form a polygonal figure, and displaying the polygonal figure on a blank image, wherein the blank image has the same size as the lane line overhead image;

[0297] Convert the blank image into a camera perspective to obtain a lane image;

[0298] The lane image is superimposed on the road image in front of the vehicle to obtain a lane display image.

[0299] In one embodiment, converting the lane line image into a bird's-eye view perspective to obtain a lane line bird's-eye view image includes:

[0300] The lane line image is converted into a bird's-eye view perspective by using a homography matrix to obtain a lane line bird's-eye view image.

[0301] In one embodiment, it further includes:

[0302] The camera that captures the road image captures a rectangular reference object in front of the vehicle to obtain an imaging image;

[0303] Determine the pixel coordinates of the four vertices of the rectangular reference object in the imaged image to obtain the pixel coordinates of the four vertices of the rectangular reference object;

[0304] Selecting four points in the imaging image to form a first rectangle, wherein four vertices of the first rectangle correspond to four vertices of the rectangular reference object respectively;

[0305] A homography matrix is ​​calculated based on the pixel coordinates of the four vertices of the first rectangle and the pixel coordinates of the four vertices of the rectangular reference object.

[0306] In one embodiment, it further includes:

[0307] Calculating the length and width of the first rectangle to obtain a reference length and a reference width;

[0308] Calculate the ratio of the length value of the rectangular reference object to the reference length value to obtain a first ratio value;

[0309] Calculate the ratio of the width value of the rectangular reference object to the reference width value to obtain a second ratio value;

[0310] Based on the first ratio value and the second ratio value, obtain a ratio parameter. It should be understood by those skilled in the art that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the application can adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0311] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0312] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0313] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0314] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0315] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0316] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0317] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0318] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A lane detection method, characterized in that: include: Acquire a road image in front of the vehicle; Performing image threshold segmentation on the road image to obtain a lane line image; Converting the lane line image into a bird's-eye view to obtain a lane line bird's-eye view image; Determining an order based on the distribution of lane line pixel points in the lane line overhead image; The lane line pixel points in the lane line overhead image are fitted using an n-order polynomial corresponding to the order to obtain a lane line curve.

2. The method according to claim 1, characterized in that The determining the order based on the distribution of lane line pixels in the lane line overhead image includes: Determining an initial sliding window position based on pixel distribution in the lane line overhead image; Sliding a preset sliding window from the initial sliding window position toward the bottom edge of the lane line overhead image to obtain a plurality of lane line sliding window areas, and determining lane line pixel points of each lane line sliding window area respectively; wherein the average value of the horizontal coordinates of the lane line pixels in the current sliding window is used as the horizontal coordinate of the center position of the next sliding window; The order is determined based on the lane line pixel points within the lane line sliding window area.

3. The method according to claim 2, characterized in that The determining the order based on the lane line pixel points in the lane line sliding window area includes: Based on the lane line pixel points in the lane line sliding window areas, a plurality of sliding window centroids are obtained; Connecting the plurality of sliding window centroids to form a first curve; Detecting the number of inflection points on the first curve to obtain a first number of inflection points; Based on the first number of inflection points, an order is determined.

4. The method according to claim 3, characterized in that The determining the order based on the first number of inflection points includes: Determine whether the first inflection point number is 0; When the number of the first inflection points is determined to be 0, the order is determined to be 2; When it is determined that the first inflection point number is not 0, the order is determined to be the first inflection point number plus 1.

5. The method according to claim 1, characterized in that Also includes: Continuously obtain lane line curves corresponding to multiple frames of road images; Based on the lane line curves corresponding to the multiple frames of road images, the curve equation coefficient corresponding to the current lane line curve is corrected to obtain the lane line curve corresponding to the current road image.

6. The method according to claim 1, characterized in that The determining the order based on the distribution of lane line pixels in the lane line overhead image includes: Drawing a lane line curve corresponding to a previous frame of road image in the lane line overhead image; According to the preset boundary range value, the boundaries are determined on the left and right sides of the position where the lane line curve of the previous frame is located in the lane line overhead image, respectively, to obtain the current lane line boundary range; Determine the lane line pixel point within the current lane line boundary range; Based on the distribution of lane line pixel points, the order is determined using a sliding window method.

7. The method according to claim 1, characterized in that The determining the order based on the distribution of lane line pixels in the lane line overhead image includes: Cropping the lane line overhead image to obtain a preprocessed lane line overhead image; Determining the order based on the distribution of lane line pixels in the pre-processed lane line overhead image; The step of fitting the lane line pixel points in the lane line overhead image using the n-th order polynomial corresponding to the order to obtain the lane line curve includes: The lane line pixel points in the pre-processed lane line overhead image are fitted using an n-th order polynomial corresponding to the order to obtain a lane line curve.

8. The method according to claim 1, characterized in that Also includes: Based on a preset scale parameter, converting lane line pixel points in the lane line overhead image into actual lane line points; Using an n-order polynomial corresponding to the order to fit the points on the actual lane line to obtain an actual lane line curve; Based on the actual lane line curve, determine the coordinates of the lane line starting point; Lane information is calculated based on the coordinates of the lane line starting point, where the lane information includes any one or more of lane curvature, lane width, and a distance that the vehicle deviates from the center of the lane.

9. The method according to claim 8, characterized in that The actual lane line curve includes a first actual lane line curve and a second actual lane line, and the lane line starting point coordinates include a first lane line starting point coordinates and a second lane line starting point coordinates; The calculating lane information based on the lane line starting point coordinates includes: Based on the coordinates of the first lane line starting point and the coordinates of the second lane line starting point, respectively, a first lane curvature and a second lane line curve are calculated according to a preset lane line curvature calculation formula, and an average of the first lane curvature and the second lane line curve is obtained to obtain a lane curvature; Calculate the lane width based on the coordinates of the starting point of the first lane line and the coordinates of the starting point of the second lane line; Calculate the actual lane center position based on the coordinates of the first lane line starting point and the coordinates of the second lane line starting point; The center position of the vehicle is calculated based on a preset scale parameter and the midpoint coordinates of the bottom edge of the lane line initial point in the lane line overhead image; The difference between the actual lane center position and the vehicle center position is calculated to obtain the vehicle deviation from the lane center distance.

10. The method according to claim 1, characterized in that The lane line curve includes a first lane line curve and a second lane line curve, and the method further includes: Performing interval sampling on the first lane line and the second lane line respectively to obtain a plurality of sampling points; Connecting multiple sampling points to form a polygonal figure, and displaying the polygonal figure on a blank image, wherein the blank image has the same size as the lane line overhead image; Convert the blank image into a camera perspective to obtain a lane image; The lane image is superimposed on the road image in front of the vehicle to obtain a lane display image.

11. The method according to claim 1, characterized in that: The converting the lane line image into a bird's-eye view to obtain a lane line bird's-eye view image includes: The lane line image is converted into a bird's-eye view perspective by using a homography matrix to obtain a lane line bird's-eye view image.

12. The method according to claim 11, characterized in that Also includes: The camera that captures the road image captures a rectangular reference object in front of the vehicle to obtain an imaging image; Determine the pixel coordinates of the four vertices of the rectangular reference object in the imaged image to obtain the pixel coordinates of the four vertices of the rectangular reference object; Selecting four points in the imaging image to form a first rectangle, wherein four vertices of the first rectangle correspond to four vertices of the rectangular reference object respectively; A homography matrix is ​​calculated based on the pixel coordinates of the four vertices of the first rectangle and the pixel coordinates of the four vertices of the rectangular reference object.

13. The method according to claim 12, characterized in that Also includes: Calculating the length and width of the first rectangle to obtain a reference length and a reference width; Calculate the ratio of the length value of the rectangular reference object to the reference length value to obtain a first ratio value; Calculate the ratio of the width value of the rectangular reference object to the reference width value to obtain a second ratio value; A scale parameter is obtained based on the first scale value and the second scale value.

14. A lane detection device, characterized in that: include: An acquisition module, used for acquiring a road image in front of the vehicle; A recognition module, used for performing image threshold segmentation on the road image to obtain a lane line image; A conversion module, used to convert the lane line image into a bird's-eye view to obtain a lane line bird's-eye view image; A determination module, configured to determine the order based on the distribution of lane line pixels in the lane line overhead image; The fitting module is used to fit the lane line pixel points in the lane line overhead image using an n-order polynomial corresponding to the order to obtain a lane line curve.

15. A vehicle, characterized in that: Lane detection is performed using the lane detection method described in any one of claims 1 to 13.

16. An electronic device, characterized in that: The electronic device includes: at least one processor; a memory connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the lane detection method according to any one of claims 1 to 13 by executing the instructions stored in the memory.

17. A machine-readable storage medium having instructions stored thereon, characterized in that: When the instruction is executed by a processor, the processor is configured to perform the lane detection method according to any one of claims 1 to 13.

Images