A method and device for fitting lane lines

Abstract

A method and device for fitting lane lines, the method comprising: obtaining a road surface image, and performing semantic segmentation on the road surface image to obtain a binary feature map; eliminating misclassified pixels contained in the first pixel point in the binary feature map according to a preset rule; performing perspective processing on the binary feature map after eliminating the misclassified pixels to obtain a top view, and generating a histogram of the top view; performing clustering processing on the second pixel point in the top view according to the generated histogram and using a preset sliding window algorithm; the second pixel point is a pixel point obtained by perspective of the first pixel point; fitting the second pixel point set after clustering processing to obtain two curves, and performing inverse perspective transformation on the obtained two curves to determine a pixel point set in the road surface image for representing two lane lines.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and more particularly to a method and device for fitting lane lines. Background Art

[0002] Lane detection is one of the research hotspots in the field of autonomous driving and is also a key technology for ADAS (Advanced Driver Assistance Systems). Lane detection is a particularly important link that can assist drivers in reducing traffic accidents caused by lane deviation. By placing a camera above the windshield inside the car to collect information about the front environment during driving, a vision-based processing algorithm is used to extract lane information from the camera output image and use it in subsequent tasks such as track planning, lane keeping, and behavior prediction.

[0003] Lane detection includes two steps: feature extraction and curve fitting. Currently, feature extraction is mainly divided into two directions. One is the traditional lane extraction algorithm based on machine vision, which uses lane edge information, color and texture features to capture the effective pixel points of the lane in the image. The other is the lane extraction algorithm based on deep learning. Relying on the powerful learning ability of the convolutional neural network, it continuously optimizes the neural network parameters to improve the network's perception of the scene and the generalization ability of the network itself, extracts effective pixels, and thus realizes the semantic segmentation of the lane. The former is easily affected by changes in lighting, weather changes, and severe occlusion of road features, resulting in the inability to correctly extract lanes. Compared with the former, the latter has higher robustness and applicability, is not affected by weather and lighting, and can also identify most occlusion situations. After generating a set of lane pixel points, the extracted lane is fitted using a curve model to obtain a lane geometric parameter model.

[0004] However, both methods may misclassify some background pixels in the image as lane line pixels, resulting in incorrect lane line direction or shape fitting. Summary of the invention

[0005] The present application provides a method and device for fitting lane lines. After obtaining a set of pixel points for fitting lane lines based on semantic segmentation, a sliding window algorithm is used to accurately fit the lane lines to obtain more accurate lane lines.

[0006] The present application provides a method for fitting lane lines, the method comprising:

[0007] Acquire a road surface image, and perform semantic segmentation on the road surface image to obtain a binary feature map; in the binary feature map, a first pixel point with a predetermined grayscale value is used to represent an estimated lane line in the road surface image;

[0008] Eliminate misclassified pixels contained in the first pixel in the binary feature map according to a preset rule;

[0009] Performing perspective processing on the binary feature map after removing the misclassified pixels to obtain a top view, and generating a histogram of the top view;

[0010] Performing clustering processing on second pixel points in the top view according to the generated histogram and using a preset sliding window algorithm; the second pixel points are pixel points obtained by perspective viewing the first pixel points;

[0011] The second pixel point set after clustering processing is fitted to obtain two curves, and the two obtained curves are subjected to inverse perspective transformation to determine a pixel point set for representing two lane lines in the road surface image.

[0012] In an exemplary embodiment, the step of acquiring a road surface image and performing semantic segmentation on the road surface image to obtain a binary feature map includes:

[0013] Sampling the acquired video sequence to obtain a plurality of continuous frames, wherein each frame serves as a road surface picture;

[0014] The road surface image is input into a pre-trained LaneNet network model for semantic segmentation to obtain the binary feature map; wherein, in the binary feature map, the pixel point with a pixel value of 255 is the first pixel point.

[0015] In an exemplary embodiment, the step of inputting the road surface image into a pre-trained LaneNet network model for semantic segmentation to obtain the binary feature map includes:

[0016] Inputting the road surface image into the weight-shared convolutional neural network feature extraction layer in the LaneNet network model to extract a local feature map containing geometric information;

[0017] Inputting the extracted local feature map into the deep feature extraction layer network structure in the LaneNet network model to extract an abstract high-semantic feature map containing global information;

[0018] Deconvolution is performed on the high semantic feature map to obtain a binary feature map of lane line pixel points in the road surface image.

[0019] In an exemplary embodiment, the step of removing misclassified pixels contained in the first pixel in the binary feature map according to a preset rule includes:

[0020] Performing connected domain calculation on the pixel points in the binary feature map to obtain multiple connected regions;

[0021] Calculate the area and tilt angle of the minimum circumscribed rectangle of each connected area respectively;

[0022] Eliminate pixels in the connected area that do not meet preset conditions in the binary feature map.

[0023] In an exemplary embodiment, the connected domain calculation is performed on the pixel points in the binary feature map to obtain multiple connected regions, including:

[0024] S51, searching for a first pixel without a domain identifier from the binary feature map in a predetermined order, until the binary feature map is traversed, and taking the pixels with the same domain identifier as a connected domain; whenever a first pixel without a domain identifier is found, taking the new connected domain sequence number as the domain identifier of the first pixel, and taking the first pixel as the domain initial pixel to perform S52;

[0025] S52, if the domain initial pixel has M adjacent first pixels, use the domain identifier of the domain initial pixel as the domain identifier of the M first pixels, and put the M first pixels into the first queue to perform S53; if none of the adjacent pixels of the domain initial pixel is a first pixel, directly return to S51; wherein M is an integer greater than 0 and less than or equal to 8;

[0026] S53, performing a judgment operation on each first pixel in the first queue in turn until there is no first pixel in the first queue and then returning to step S51;

[0027] The step of performing a judgment operation on each first pixel in the first queue includes:

[0028] If the first pixel has M' adjacent first pixel points without domain identifiers, use the domain identifier of the secondary pixel point as the domain identifier of the M' first pixel points, and add the M' first pixel points to the first queue; and perform a judgment operation on the next first pixel point in the first queue;

[0029] If the first pixel does not have an adjacent pixel without a domain identifier, a judgment operation is performed on the next first pixel in the first queue.

[0030] In an exemplary embodiment, respectively calculating the area and tilt angle of the minimum circumscribed rectangle of each connected region includes:

[0031] Calculating the area and tilt angle of the minimum circumscribed rectangle of each connected region is achieved through the following steps:

[0032] Determine the outermost pixel points of the current connected area and connect the determined outermost pixel points to form a polygon, which is a figure including all the pixel points in the current connected area;

[0033] Using each side of the polygon as a side of a circumscribed rectangle to form a plurality of circumscribed rectangles;

[0034] Calculate the area and tilt angle of each circumscribed rectangle;

[0035] The circumscribed rectangle with the smallest area value among the multiple circumscribed rectangles is selected as the circumscribed rectangle of the current connected area.

[0036] In an exemplary embodiment, clustering the left and right lane line pixel points in the top view according to the generated histogram using a preset sliding window algorithm includes:

[0037] For the set P(n1,n2,.....n 640 );

[0038] Determine a middle column in the plurality of sets of pixel values, and using the middle column as a starting point, determine a first column number C0 at which the sum of pixel values ​​from the middle column is zero;

[0039] From the first column number C0, the column numbers C corresponding to the first maximum point on the left and the first maximum point on the right are determined respectively. l and the number of columns C r ;

[0040] According to the determined number of columns C l and the number of columns C r , and a preset sliding window algorithm to determine the pixel point set for fitting the left lane line and the right lane line.

[0041] In an exemplary embodiment, determining the pixel point set for fitting the left lane line and the right lane line according to the determined number of columns Cl and the number of columns Cr and a preset sliding window algorithm includes:

[0042] The number of columns C l and the number of columns C r Set as the center of the initial sliding window for the left lane line and the right lane line respectively;

[0043] If there is no pixel point with a pixel value of 255 in the preset sliding window, the center of the sliding window remains unchanged and moves to the previous rectangle among the multiple pre-divided rectangles; if there is a pixel point with a pixel value of 255 in the preset sliding window, it is determined whether the column coordinates of the center point of the pixel point set in the sliding window coincide with the column coordinates of the center of the rectangle;

[0044] If they do not overlap, the column coordinates of the center of the rectangle are moved to the column coordinates of the center point of the pixel point set in the sliding window, and moved to the previous rectangle;

[0045] When the moved rectangle is at the top in the top view, the calculation ends.

[0046] In an exemplary embodiment, the number of columns C determined by the data l and the number of columns C r , and a preset sliding window algorithm to determine the pixel point set for fitting the left lane line and the right lane line, and also include:

[0047] Count the pixels of all sliding windows in the top view;

[0048] When the pixel points in the sliding window do not meet the preset discrete degree index, the pixel points in the sliding window are removed, and the set of pixel points in the sliding window after removal is used as the pixel point set of the left lane line and the right lane line;

[0049] Wherein, the dispersion index is:

[0050]

[0051] In the above formula, μ is the mean value of the pixel coordinates in the sliding window, x i is the value corresponding to the coordinates of each pixel in the sliding window, and N is the number of pixels.

[0052] The present application also provides a device for fitting lane lines, the device comprising: a memory and a processor; characterized in that the memory is used to store a program for fitting lane lines, and the processor is used to read and execute the program for fitting lane lines, and execute any one of the methods in the above embodiments.

[0053] Compared with the related art, the present application provides a method and device for fitting lane lines, the method comprising: obtaining a road surface image, and performing semantic segmentation on the road surface image to obtain a binary feature map; removing the misclassified pixels contained in the first pixel in the binary feature map according to a preset rule; performing perspective processing on the binary feature map after removing the misclassified pixels to obtain a top view, and generating a histogram of the top view; clustering the second pixel in the top view according to the generated histogram using a preset sliding window algorithm; the second pixel is the pixel obtained by perspective of the first pixel; fitting the second pixel set after clustering to obtain two curves, and mapping the obtained two curves to the top view for inverse perspective transformation, and determining the pixel set used to represent the two lane lines in the road surface image. After obtaining the pixel set of fitted lane lines based on semantic segmentation, this embodiment uses a sliding window algorithm to accurately fit the lane lines, so as to obtain more accurate lane lines.

[0054] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by implementing the present application. Other advantages of the present application can be realized and obtained by the schemes described in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings are used to provide an understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0056] Figure 1 A flow chart of a method for fitting lane lines according to an embodiment of the present application;

[0057] Figure 2 A schematic diagram of a lane line semantic segmentation result based on a laneNet network in an example embodiment;

[0058] Figure 3 A schematic diagram of removing mis-segmented areas in an exemplary embodiment;

[0059] Figure 4 is a schematic diagram of perspective transformation in an exemplary embodiment;

[0060] Figure 5 A schematic diagram of clustering of left and right lane lines in an example embodiment;

[0061] Figure 6 A schematic diagram of the left and right lane line fitting results in an example embodiment

[0062] Figure 7 A schematic diagram of calculating a connected area in an exemplary embodiment;

[0063] Figure 8 A flow chart of a lane line fitting method in an exemplary embodiment;

[0064] Fig. 9 Schematic diagram of a device for fitting lane lines according to an embodiment of the present application. DETAILED DESCRIPTION

[0065] The present application describes multiple embodiments, but the description is exemplary rather than restrictive, and it is obvious to those skilled in the art that there may be more embodiments and implementations within the scope of the embodiments described in the present application. Although many possible feature combinations are shown in the drawings and discussed in the specific embodiments, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with any other feature or element in any other embodiment, or may replace any other feature or element in any other embodiment.

[0066] The present application includes and contemplates combinations of features and elements known to those of ordinary skill in the art. The embodiments, features and elements disclosed in the present application may also be combined with any conventional features or elements to form a unique invention scheme defined by the claims. Any features or elements of any embodiment may also be combined with features or elements from other invention schemes to form another unique invention scheme defined by the claims. Therefore, it should be understood that any feature shown and / or discussed in the present application may be implemented individually or in any appropriate combination. Therefore, except for the limitations made according to the attached claims and their equivalents, the embodiments are not subject to other restrictions. In addition, various modifications and changes may be made within the scope of protection of the attached claims.

[0067] In addition, when describing representative embodiments, the specification may have presented the method and / or process as a specific sequence of steps. However, to the extent that the method or process does not rely on the specific order of the steps described herein, the method or process should not be limited to the steps of the specific order described. As will be understood by those of ordinary skill in the art, other sequences of steps are also possible. Therefore, the specific sequence of the steps set forth in the specification should not be interpreted as a limitation to the claims. In addition, the claims for the method and / or process should not be limited to the steps of performing them in the order written, and those skilled in the art can easily understand that these sequences can be changed and still remain within the spirit and scope of the embodiments of the present application.

[0068] The present disclosure provides a method for fitting lane lines. Figure 1 As shown, the method includes steps S100-S140;

[0069] S100. Obtain a road surface image, and perform semantic segmentation on the road surface image to obtain a binary feature map;

[0070] S110. Eliminate misclassified pixels contained in the first pixel in the binary feature map according to a preset rule;

[0071] S120. Performing perspective processing on the binary feature map after removing the misclassified pixels to obtain a top view, and generating a histogram of the top view;

[0072] S130. Performing clustering processing on second pixel points in the top view according to the generated histogram and using a preset sliding window algorithm; the second pixel points are pixel points obtained by perspective viewing the first pixel points;

[0073] S140. Fitting the second pixel point set after clustering processing to obtain two curves, and mapping the obtained two curves to the top view to perform inverse perspective transformation, and determining a pixel point set for representing two lane lines in the road surface image.

[0074] In this embodiment, the road surface image can be obtained by capturing an image with a camera disposed above the windshield in the vehicle, and the captured image can be an image including lane information. The binary feature map is a first pixel point with a predetermined grayscale value, and the first pixel point is used to represent the estimated lane line in the road surface image; that is, the binary feature map is a map including white and black pixel points, wherein the white pixel point represents the lane line and the black is the background color.

[0075] In this embodiment, the fitted curve includes a straight line, a curved line, or any combination thereof.

[0076] In an exemplary embodiment, the road surface image is obtained and the road surface image is semantically segmented to obtain a binary feature map, including: sampling the collected video sequence to obtain multiple continuous frames, wherein each frame is a road surface image; inputting the road surface image into a pre-trained LaneNet network model for semantic segmentation to obtain the binary feature map; wherein in the binary feature map, the pixel point with a pixel value of 255 is the first pixel point. Figure 2 As shown, semantic segmentation is performed to obtain a binary feature map, where white is the lane line and black is the background. In this embodiment, the road surface image can be obtained through the following steps: Step 1, during the actual driving of the car, the camera set above the windshield inside the car collects the video sequence of the road surface in front, Step 2, the collected video is sampled in the time dimension to obtain the road surface information image of the road surface in front of the continuous frame; Step 3, all the images are uniformly scaled to 640*480 size.

[0077] In an exemplary embodiment, the step of inputting the road surface image into a pre-trained LaneNet network model for semantic segmentation to obtain the binary feature map includes:

[0078] Step 1. Input the road surface image into the weight-shared convolutional neural network feature extraction layer in the LaneNet network model to extract a local feature map containing geometric information;

[0079] Step 2. Input the extracted local feature map into the deep feature extraction layer network structure in the LaneNet network model to extract an abstract high-semantic feature map containing global information;

[0080] Step 3. Perform deconvolution processing on the high semantic feature map to obtain a binary feature map of lane line pixels in the road surface image.

[0081] In an exemplary embodiment, the step of removing misclassified pixels contained in the first pixel in the binary feature map according to a preset rule includes:

[0082] Performing connected domain calculation on the pixel points in the binary feature map to obtain multiple connected regions;

[0083] Calculate the area and tilt angle of the minimum circumscribed rectangle of each connected area respectively;

[0084] Eliminate pixels in the connected area that do not meet preset conditions in the binary feature map.

[0085] In an exemplary embodiment, the connected domain calculation is performed on the pixel points in the binary feature map to obtain multiple connected regions, including:

[0086] S51, searching for a first pixel without a domain identifier from the binary feature map in a predetermined order, until the binary feature map is traversed, and taking the pixels with the same domain identifier as a connected domain; whenever a first pixel without a domain identifier is found, taking the new connected domain sequence number as the domain identifier of the first pixel, and taking the first pixel as the domain initial pixel to perform S52;

[0087] S52, if the domain initial pixel has M adjacent first pixels, use the domain identifier of the domain initial pixel as the domain identifier of the M first pixels, and put the M first pixels into the first queue to perform S53; if none of the adjacent pixels of the domain initial pixel is a first pixel, directly return to S51; wherein M is an integer greater than 0 and less than or equal to 8;

[0088] S53, performing a judgment operation on each first pixel in the first queue in turn until there is no first pixel in the first queue and then returning to step S51;

[0089] The step of performing a judgment operation on each first pixel in the first queue includes:

[0090] If the first pixel has M' adjacent first pixel points without domain identifiers, use the domain identifier of the secondary pixel point as the domain identifier of the M' first pixel points, and add the M' first pixel points to the first queue; and perform a judgment operation on the next first pixel point in the first queue;

[0091] If the first pixel does not have an adjacent pixel without a domain identifier, a judgment operation is performed on the next first pixel in the first queue.

[0092] In an exemplary embodiment, respectively calculating the area and tilt angle of the minimum circumscribed rectangle of each connected region includes: calculating the area and tilt angle of the minimum circumscribed rectangle of each connected region by the following steps:

[0093] Determine the outermost pixel points of the current connected area and connect the determined outermost pixel points to form a polygon, which is a figure including all the pixel points in the current connected area;

[0094] Using each side of the polygon as a side of a circumscribed rectangle to form a plurality of circumscribed rectangles;

[0095] Calculate the area and tilt angle of each circumscribed rectangle;

[0096] The circumscribed rectangle with the smallest area value among the multiple circumscribed rectangles is selected as the circumscribed rectangle of the current connected area.

[0097] In this embodiment, the binary feature map of lane line pixels in the road surface image is as follows: Figure 3 As shown in Figure a, after obtaining n connected regions, the minimum enclosing rectangle of each region is calculated respectively. The process is as follows:

[0098] 1. Find the convex polygon formed by connecting the outermost pixels of the connected area. This polygon includes all the points in the set, such as Figure 3 As shown in Figure b, p0-p12 is the set of pixel points in the connected domain, and the polygon formed by connecting the outermost point set (p10, p12, p0, p1, p3) is the desired one.

[0099] 2. Since one side of the minimum circumscribed rectangle of a polygon must be collinear with one of the lines of the polygon, traverse each side of the above convex polygon and establish the circumscribed rectangle respectively, as shown in the following figure. Figure 3Finally, compare the areas of each circumscribed rectangle and select the rectangle with the smallest circumscribed rectangle area as the minimum enclosing rectangle of the polygon.

[0100] 3. Calculate the area and tilt angle of the minimum bounding rectangle of each connected area according to the above rules, and remove the invalid area. The results of the retained area are as shown in the attached figure. Figure 3 Figure d in .

[0101] In this embodiment, the processed binary feature map is subjected to perspective change to obtain a top view, and a histogram of the top view is generated at the same time. The implementation process includes:

[0102] Projection transformation is performed on the binary image after removing invalid connected areas. In reality, two parallel lane lines will appear non-parallel on the image and intersect at a point at infinity. After transformation, the two lane lines appear parallel, which is also convenient for fitting the lane line equation. The principle of projection transformation is shown in formula (1):

[0103]

[0104] Among them, M is a perspective matrix of size 3*3 is the target point; in this embodiment, it is assumed that the value of Z is 1, First, we need to calculate the perspective matrix M based on the four coordinate points on the original image. The calculation process is as follows:

[0105] 1. In the attached Figure 4 In the figure a, select the upper left coordinate point A (x1, y1), the lower left coordinate point B (x2, y2), the upper right coordinate point C (x3, y3), and the lower right coordinate point D (x4, y4) before deformation to form a trapezoid. Move the four coordinate points until the AB line and the CD line remain parallel, as shown in the attached figure. Figure 4 As shown in Figure b, we get four coordinate points A1(x5,x5), B1(x6,y6), C1(x7,x7), and D1(x8,y8);

[0106] 2. Based on equation (1), equation (2) is derived, and the four pairs of coordinate systems before and after the transformation are substituted into equation (2) to solve the eight unknowns in the perspective matrix M.

[0107]

[0108] After calculating the M matrix, perform perspective transformation on each pixel in the connected area. The result is shown in the attached figure. Figure 4 As shown in Figure c, two approximately parallel lane lines are obtained.

[0109] The clustering process of the left and right lane line pixel points in the top view according to the generated histogram and using a preset sliding window algorithm includes:

[0110] The first step is to sum up the pixel values ​​of each column in the generated histogram to obtain a set of multiple pixel values ​​P (n1, n2, ..... n 640 );

[0111] The second step is to determine the middle column in the set of multiple columns of pixel values, and take the middle column as the starting point to determine the first column number C0 whose distance from the middle column to the sum of pixel values ​​is zero; the specific process of determining the column number C0 can be: first, calculate whether the sum of pixels in the middle column is 0, if it is zero, record it as the first column number C0, if it is not zero, it is necessary to determine whether the segmented lane line is distributed on the left or right side of the image, if it is on the left side, then search from the middle column to the left for a column whose sum of pixels is 0 as the first column number C0, otherwise search from the right for a column whose sum of pixels is 0 as the first column number C0.

[0112] Step 3: Determine the column numbers C corresponding to the first maximum point on the left and the first maximum point on the right from the first column number C0 l and the number of columns C r ;

[0113] Step 4: According to the determined number of columns C l and the number of columns C r , and a preset sliding window algorithm to determine the pixel point set for fitting the left lane line and the right lane line.

[0114] In an exemplary embodiment, the step of determining the pixel point set for fitting the left lane line and the right lane line according to the determined number of columns Cl and the number of columns Cr and a preset sliding window algorithm includes:

[0115] The number of columns C l and the number of columns C r Set as the center of the initial sliding window for the left lane line and the right lane line respectively;

[0116] If there is no pixel point with a pixel value of 255 in the preset sliding window, the center of the sliding window remains unchanged and moves to the previous rectangle among the multiple rectangles divided in advance; wherein the parameters of the rectangle are preset, and the length and width of the rectangle can also be adjusted according to the lane line fitting effect. In this embodiment, the length of the preset rectangle satisfies l<2*(C0-C l ), the length of the rectangle satisfies the above conditions so that the sliding window can distinguish the left and right lane line pixels when sliding up and down, and the width w of the rectangle is equal to the width of the original captured image divided by 12.

[0117] If there is a pixel point with a pixel value of 255 in the preset sliding window, determine whether the column coordinates of the center point of the pixel point set in the sliding window coincide with the column coordinates of the center of the rectangle;

[0118] If they do not overlap, the column coordinates of the center of the rectangle are moved to the column coordinates of the center point of the pixel point set in the sliding window, and moved to the previous rectangle;

[0119] When the moved rectangle is at the top in the top view, the calculation ends.

[0120] In an exemplary embodiment, the number of columns C determined according to l and the number of columns C r , and a preset sliding window algorithm to determine the pixel point set for fitting the left lane line and the right lane line, and also include:

[0121] Count the pixels of all sliding windows in the top view;

[0122] When the pixel points in the sliding window do not meet the preset discrete degree index, the pixel points in the sliding window are removed, and the set of pixel points in the sliding window after removal is used as the pixel point set of the left lane line and the right lane line;

[0123] Wherein, the dispersion index is:

[0124]

[0125] In the above formula, μ is the mean value of the pixel coordinates in the sliding window, x i is the value corresponding to the coordinates of each pixel in the sliding window, and N is the number of pixels.

[0126] In this embodiment, the left and right lane lines in the perspective transformed image are adaptively clustered. First, the pixel values ​​of each column in the image are summed to obtain a set P (n1, n2, ..... n 640 ), starting from the middle column, find the column number C0 whose sum of pixel values ​​closest to the middle column is zero, and find the column number corresponding to the first extreme point along C0 to the left and right, respectively, recorded as C l and C r ; then follow C l and C r Use the sliding window algorithm to query the set of left lane lines and right lane lines from the bottom of the image upwards. The query rules are as follows:

[0127] 1. Set the sliding window height h to 40 and the width to 80. Slide upward along the bottom of the image. l and C rSet it as the center of the initial sliding window of the left and right lane lines. If no pixel of interest appears in the window, keep the center unchanged and move vertically upward. Otherwise, calculate whether the column coordinates of the center point of the pixel point set in the sliding window coincide with the column coordinates of the center of the rectangle. If they do not coincide, move the window left and right until the column coordinates coincide with the center point of the pixel point set, until the window reaches the top of the image.

[0128] 2. Lane lines are generally smooth and continuous, with uniform thickness and no sudden changes in direction. Therefore, it is necessary to count the pixel distribution of all windows in the image. Pixels with too many pixels or high discreteness need to be removed. The calculation formula of the discreteness index S is as follows:

[0129]

[0130] Where μ is the pixel mean, x i is the value corresponding to each pixel in the window, and N is the number of pixels. The sum of the pixels in the left and right sliding windows left after elimination is the pixel of interest on the left and right lane lines, such as Figure 5 shown.

[0131] In step 140, the left and right lane line pixel points after clustering are fitted to obtain two curves, and the obtained two curves are mapped to the top view for inverse transmission transformation to obtain two lane lines in the road surface image; in this step, the process of obtaining lane lines through clustering, for example: respectively Figure 5 The pixel point set of the white left lane line and the white right lane line in the middle is fitted with a binary linear equation using the least squares method to obtain the equations of the left and right lane lines. Then, the left and right lane lines are inversely transformed by combining the perspective transformation matrix in step 4, which are the left and right lane lines of the current driving lane on the actual road. The result is as follows Figure 6 shown.

[0132] In this embodiment, based on a lane line fitting method fused with a sliding window, after obtaining a lane line pixel point set based on semantic segmentation, accurate fitting of the lane line shape and direction is achieved through traditional image post-processing and a sliding window algorithm.

[0133] Example 1

[0134] The process of performing lane line semantic segmentation based on the LaneNet network model and obtaining a binary feature map is as follows:

[0135] First, the uniformly scaled image is input into the weight-sharing convolutional neural network feature extraction layer, and the input image is downsampled using convolution and pooling operations to extract local feature maps with rich geometric information. The local feature maps are then input into the deep feature extraction layer network structure to gradually extract abstract high-semantic feature maps with global information from high-resolution features. The high-semantic feature maps are then deconvolved and concatenated with local features for upsampling. This process is repeated until the feature map and input image are the same size. The resulting binary feature map is the lane segmentation result.

[0136] Example 2

[0137] The connected domains of the pixels in the binary feature map are calculated to obtain multiple connected regions. The process is as follows:

[0138] Assume that in the binary feature image, the pixel value of 255 is the foreground pixel (white), and the pixel value of 0 is the background (black). A(x,y) is the pixel value at the coordinate (x,y). Traversing each pixel in the binary feature image, the process is as follows:

[0139] Step 200. Starting from the first row from the top and the first column from the left, traverse each pixel in the binary feature image in turn, and determine whether the pixel value of each traversed pixel is equal to 255;

[0140] Step 210. If the pixel value of the pixel point is not equal to 255, continue to traverse the next pixel point and jump to step 200;

[0141] Step 220. If the pixel value is A(x, y)==255, the pixel is determined to be the first pixel, and a corresponding domain label label is established for the first pixel, and the current value A(x, y)=label; and step 230 is executed;

[0142] Step 230: Determine the first connected area using the 8-neighborhood connectivity method;

[0143] In this step, using the 8-neighborhood connectivity method to determine the first connected area includes:

[0144] Step 2301. Determine the first connected region as all the pixel points in the region that are reached from the current point in eight directions: up, down, left, right, upper left, lower left, upper right, and lower right;

[0145] Step 2302: Determine whether there is a pixel point with a pixel value equal to 255 among all the pixel points in the area reached in all eight directions. If so, assign a label value to the pixel point with a pixel value equal to 255 and store the pixel point.

[0146] Step 2303. Repeat steps 2301 and 2302 for the pixel points whose pixel values ​​are equal to 255 in the arrival area;

[0147] Step 2304. Arrive at the last pixel point in the area with a pixel value equal to 255, execute steps 2301 and 2302, determine that there is no pixel point with a pixel value equal to 255 in the area reached in eight directions, and end the calculation.

[0148] Step 240. Traverse all the pixels in the binary feature map to obtain N connected regions. The result is as follows: Figure 3 As shown in a.

[0149] Example 3

[0150] The connected domains of the pixels in the binary feature map are calculated to obtain multiple connected regions. The process is as follows:

[0151] S51, searching for a first pixel without a domain identifier from the binary feature map in a predetermined order, until the binary feature map is traversed, and taking the pixels with the same domain identifier as a connected domain; whenever a first pixel without a domain identifier is found, taking the new connected domain sequence number as the domain identifier of the first pixel, and taking the first pixel as the domain initial pixel to perform S52;

[0152] S52, if the domain initial pixel has M adjacent first pixels, use the domain identifier of the domain initial pixel as the domain identifier of the M first pixels, and put the M first pixels into the first queue to perform S53; if none of the adjacent pixels of the domain initial pixel is a first pixel, directly return to S51; wherein M is an integer greater than 0 and less than or equal to 8;

[0153] S53, performing a judgment operation on each first pixel in the first queue in turn until there is no first pixel in the first queue and then returning to step S51;

[0154] The step of performing a judgment operation on each first pixel in the first queue includes:

[0155] If the first pixel has M' adjacent first pixel points without domain identifiers, use the domain identifier of the secondary pixel point as the domain identifier of the M' first pixel points, and add the M' first pixel points to the first queue; and perform a judgment operation on the next first pixel point in the first queue;

[0156] If the first pixel does not have an adjacent pixel without a domain identifier, a judgment operation is performed on the next first pixel in the first queue. In this embodiment, the connected domain is calculated for the pixels in the binary feature map to obtain multiple connected regions, such as Figure 7 As shown, the process of determining multiple connected areas is as follows: according to a predetermined order, the eight-neighborhood connectivity method is used to determine the pixels in the eight directions of the binary feature map, and the first pixel without a domain identifier is searched; that is, assuming that 1 is the first pixel, the search finds that there are no white pixels in the adjacent 8 directions, and returns to step 51 to continue searching, and finds 2 (the second pixel), and establishes label as A; the search finds that there are 3 white points with adjacent pixel values ​​of 255 in the 8 adjacent directions of 2, namely, pixel points 21, 22, and 23, and the labels are all set to A and put into the queue; the next first pixel in the first queue is judged, 21 judges that there is a white pixel point 23 with a pixel value of 255 in the adjacent 8 directions, but pixel point 23 already has a label, so the next 22 in the queue is judged. 22 has white 23, 221, and 222 with adjacent pixel values ​​of 255, and it is determined that pixel points 221 and 222 have no labels, so pixel points 221 and 222 are added to the queue. The pixel point 23 is judged. All the white pixels with a pixel value of 255 in the eight adjacent positions of pixel 23 have labels; therefore, the judgment of 221 is performed... and so on, until the search and judgment of the connected domain starting with 2 is completed (there are no pixels in the first queue), return to step 51 to continue searching, find 3, and continue searching similar to the above steps.

[0157] Example 4

[0158] The method flow of fitting lane lines is as follows: Figure 8 As shown:

[0159] Step 1: Select an image captured by a camera located above the inside of a car windshield as the input image;

[0160] Step 2: Input the input image into the trained LaneNet network module for semantic segmentation to obtain a binary feature map of pixels that may be lane lines, where white represents lane lines and black represents background.

[0161] Step 3: Remove the misclassified pixels of the lane lines in the binary feature map according to relevant rules;

[0162] Step 4: Perform perspective change on the processed binary feature map to obtain a top view, and generate a histogram of the top view;

[0163] Step 5: Cluster the left and right lane lines in the top view by combining the above histogram knot and sliding window algorithm;

[0164] Step 6: Perform a binary linear polynomial fitting on the clustered left and right lane line sets to obtain two complete smooth curves, which are mapped to the top view. The top view is then inversely transformed to obtain the left and right lane lines of the current lane during actual driving.

[0165] The present application also provides a device for fitting lane lines, such as Fig. 9 As shown, the device includes: a memory and a processor; it is characterized in that the memory 910 is used to store a program for fitting lane lines, and the processor 920 is used to read and execute the program for fitting lane lines, and execute any one of the methods in the above embodiments.

[0166] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

Claims

1. A method for fitting lane lines, characterized in that: The method comprises: Acquire a road surface image, and perform semantic segmentation on the road surface image to obtain a binary feature map; in the binary feature map, a first pixel point with a predetermined grayscale value is used to represent an estimated lane line in the road surface image; Eliminate misclassified pixels contained in the first pixel in the binary feature map according to a preset rule; Performing perspective processing on the binary feature map after removing the misclassified pixels to obtain a top view, and generating a histogram of the top view; Performing clustering processing on second pixel points in the top view according to the generated histogram and using a preset sliding window algorithm; the second pixel points are pixel points obtained by perspective viewing the first pixel points; Fitting the second pixel point set after clustering processing to obtain two curves, and performing inverse perspective transformation on the obtained two curves to determine a pixel point set for representing two lane lines in the road surface image; The removing of misclassified pixels contained in the first pixel in the binary feature map according to a preset rule includes: Performing connected domain calculation on the pixel points in the binary feature map to obtain multiple connected regions; Calculate the area and tilt angle of the minimum circumscribed rectangle of each connected area respectively; Eliminate pixels in the connected area of ​​the binary feature map that do not meet preset conditions; The connected domain calculation is performed on the pixel points in the binary feature map to obtain multiple connected regions, including: S51, searching for a first pixel without a domain identifier from the binary feature map in a predetermined order, until the binary feature map is traversed, and taking the pixels with the same domain identifier as a connected domain; whenever a first pixel without a domain identifier is found, taking the new connected domain sequence number as the domain identifier of the first pixel, and taking the first pixel as the domain initial pixel to perform S52; S52, if the domain initial pixel has M adjacent first pixels, use the domain identifier of the domain initial pixel as the domain identifier of the M first pixels, and put the M first pixels into the first queue to perform S53; if none of the adjacent pixels of the domain initial pixel is a first pixel, directly return to S51; wherein M is an integer greater than 0 and less than or equal to 8; S53, performing a judgment operation on each first pixel in the first queue in turn until there is no first pixel in the first queue and then returning to step S51; The step of performing a judgment operation on each first pixel in the first queue includes: If the first pixel has M' adjacent first pixel points without domain identifiers, the domain identifier of the secondary pixel point is used as the domain identifier of the M' first pixel points, and the M' first pixel points are added to the first queue; and a judgment operation is performed on the next first pixel point in the first queue; If the first pixel does not have an adjacent pixel without a domain identifier, a judgment operation is performed on the next first pixel in the first queue.

2. The method for fitting lane lines according to claim 1, characterized in that: The step of acquiring a road surface image and performing semantic segmentation on the road surface image to obtain a binary feature map includes: Sampling the acquired video sequence to obtain a plurality of continuous frames, wherein each frame serves as a road surface picture; The road surface image is input into a pre-trained LaneNet network model for semantic segmentation to obtain the binary feature map; wherein, in the binary feature map, the pixel point with a pixel value of 255 is the first pixel point.

3. The method for fitting lane lines according to claim 2, characterized in that: The step of inputting the road surface image into a pre-trained LaneNet network model for semantic segmentation to obtain the binary feature map includes: Inputting the road surface image into the weight-shared convolutional neural network feature extraction layer in the LaneNet network model to extract a local feature map containing geometric information; Inputting the extracted local feature map into the deep feature extraction layer network structure in the LaneNet network model to extract an abstract high-semantic feature map containing global information; Deconvolution is performed on the high semantic feature map to obtain a binary feature map of lane line pixel points in the road surface image.

4. The method for fitting lane lines according to claim 1, characterized in that: The step of respectively calculating the area and the tilt angle of the minimum circumscribed rectangle of each connected region includes: Calculating the area and tilt angle of the minimum circumscribed rectangle of each connected region is achieved through the following steps: Determine the outermost pixel points of the current connected area and connect the determined outermost pixel points to form a polygon, which is a figure including all the pixel points in the current connected area; Using each side of the polygon as a side of a circumscribed rectangle to form a plurality of circumscribed rectangles; Calculate the area and tilt angle of each circumscribed rectangle; The circumscribed rectangle with the smallest area value among the multiple circumscribed rectangles is selected as the circumscribed rectangle of the current connected area.

5. The method for fitting lane lines according to claim 4, characterized in that: The clustering process of the left and right lane line pixel points in the top view according to the generated histogram and using a preset sliding window algorithm includes: For the set P(n1,n2,.....n 640 ); Determine the middle column in each column of pixel value sets, and use the middle column as a starting point to determine the first column number C0 whose sum of pixel values ​​from the middle column is zero; From the first column number C0, the column numbers C corresponding to the first maximum point on the left and the first maximum point on the right are determined respectively. l and the number of columns C r ; According to the determined number of columns C l and the number of columns C r , and a preset sliding window algorithm to determine the pixel point set for fitting the left lane line and the right lane line.

6. The method for fitting lane lines according to claim 5, characterized in that: The method of determining the pixel point set for fitting the left lane line and the right lane line according to the determined number of columns Cl and the number of columns Cr and a preset sliding window algorithm includes: The number of columns C l and the number of columns C r Set as the center of the initial sliding window for the left lane line and the right lane line respectively; If there is no pixel point with a pixel value of 255 in the preset sliding window, the center of the sliding window remains unchanged and moves to the previous rectangle among the multiple pre-divided rectangles; if there is a pixel point with a pixel value of 255 in the preset sliding window, it is determined whether the column coordinates of the center point of the pixel point set in the sliding window coincide with the column coordinates of the center of the rectangle; If they do not overlap, the column coordinates of the center of the rectangle are moved to the column coordinates of the center point of the pixel point set in the sliding window, and moved to the previous rectangle; When the moved rectangle is at the top in the top view, the calculation ends.

7. The method for fitting lane lines according to claim 6, characterized in that: The number of columns C determined according to l and the number of columns C r , and a preset sliding window algorithm to determine the pixel point set for fitting the left lane line and the right lane line, and also include: Count the pixels of all sliding windows in the top view; When the pixel points in the sliding window do not meet the preset discrete degree index, the pixel points in the sliding window are removed, and the set of pixel points in the sliding window after removal is used as the pixel point set of the left lane line and the right lane line; Wherein, the dispersion index is: In the above formula, μ is the mean value of the pixel coordinates in the sliding window, x i is the value corresponding to the coordinates of each pixel in the sliding window, and N is the number of pixels.

8. A device for fitting lane lines, characterized in that: The device includes: a memory and a processor; characterized in that the memory is used to store a program for fitting lane lines, and the processor is used to read and execute the program for fitting lane lines, and execute the method described in any one of claims 1-7.

Images