Methods, apparatus, equipment, and computer storage media for determining lane lines

By identifying and segmenting the pixel information of the outer contour of lane lines, and using weighted least squares fitting, the problem of low lane line detection accuracy on roads with small radii of curvature is solved, achieving high-precision lane line detection and vehicle control.

CN117351441BActive Publication Date: 2026-11-17CHANGSHA INTELLIGENT DRIVING INST CORP LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202210716010.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-23
Publication Date
2026-11-17
Estimated Expiration
2042-06-23

AI Technical Summary

Technical Problem

In existing technologies, when using neural network models to detect lane lines and fitting them using the least squares method, the error in fitting lane lines is large for roads with small radii of curvature, resulting in low accuracy of the output lane lines, which cannot meet the requirements of autonomous driving systems.

Method used

By identifying the pixel information of the outer contour of the lane line in the target image, the fitting error is calculated. If the error is greater than a preset threshold, the outer contour is segmented into segments, and the weighted least squares method is used to fit the outer contour. A preset weight factor is configured to obtain the target lane line.

Benefits of technology

In scenarios with a small road curvature radius, the accuracy and reliability of lane line detection are improved, ensuring the accuracy of vehicle control.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117351441B_ABST
    Figure CN117351441B_ABST
Patent Text Reader

Abstract

This application provides a method, apparatus, device, and computer storage medium for determining lane lines. The method includes identifying pixel information of the outer contour of a lane line in a target image, the pixel information of the outer contour including first position information of each pixel point of the lane line's outer contour; calculating second position information of the lane line fitted based on the first position information, and the error between the second and first position information; if the error is greater than or equal to a preset error threshold, segmenting the outer contour into a preset number of segments; and fitting the outer contour using a weighted least squares method based on the position information of the pixels of each segment and a preset weighting factor configured for each segment to obtain a first target lane line. According to this application embodiment, using the weighted least squares method to fit curve-type lane lines can accurately detect lane lines that can be used to control vehicles, even in scenarios with a very small road curvature radius.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application belongs to the field of autonomous driving technology, and in particular relates to a method, apparatus, device and computer storage medium for determining lane lines. Background Technology

[0002] With the rapid development of autonomous driving technology, more and more vehicles are equipped with autonomous driving systems. Lane detection is an important component of autonomous driving systems. Accurately detecting lane lines during vehicle driving can ensure driving safety.

[0003] In existing technologies, neural network models are mainly used to detect lane lines, and then the least squares method is used to fit the lane lines directly. However, this method has a large error in fitting lane lines for roads with small radii of curvature, and the accuracy of the output lane lines is low. Summary of the Invention

[0004] This application provides a method, apparatus, device, and computer storage medium for determining lane lines, which can solve the problem in the prior art that the lane line fitting for roads with small radii of curvature has large errors and the accuracy of the output lane lines is low.

[0005] In a first aspect, embodiments of this application provide a method for determining lane lines, the method comprising:

[0006] Identify the pixel information of the outer contour of the lane line in the target image. The pixel information of the outer contour includes the first position information of each pixel point of the outer contour of the lane line.

[0007] Calculate the second position information of the lane line fitted based on the first position information, and the error between it and the first position information;

[0008] If the error is greater than or equal to the preset error threshold, the outer contour line is divided into a preset number of segments.

[0009] Based on the position information of the pixels of each segment line and the preset weight factor configured for each segment line, the outer contour line is fitted using the weighted least squares method to obtain the first target lane line.

[0010] In one implementation, the first position information of the pixel includes x-coordinate values ​​and y-coordinate values; the pixel information for identifying the outer contour of the lane line in the target image includes:

[0011] Identify the positional information of the pixels on the first outer contour line of the lane line in the target image;

[0012] The first pixel whose y-coordinate value is a first preset value is determined within the first outer contour line;

[0013] Calculate the first expected value of the x-coordinate of the first pixel;

[0014] Calculate the first variance between the x-coordinate value of each first pixel and the first expected value;

[0015] If the largest first variance is greater than the preset width threshold, the pixel values ​​of pixels whose y-coordinate values ​​are less than the first preset value in the first outer contour of the target image are modified to the preset pixel values, thereby obtaining the pixel information of the outer contour of the lane line in the recognized target image.

[0016] In one implementation, the method further includes:

[0017] If the largest first variance is less than or equal to a preset width threshold, determine the second pixel point whose y-coordinate value is a second preset value in the first outer contour line;

[0018] Calculate the second expected value of the x-coordinate of the second pixel;

[0019] Calculate the second variance between the x-coordinate value of each second pixel and the second expected value;

[0020] If the largest second variance is greater than the preset width threshold, the pixel values ​​of pixels whose y-coordinate values ​​are less than the second preset value in the first outer contour of the target image are modified to the preset pixel values, thereby obtaining the pixel information of the outer contour of the lane line in the recognized target image.

[0021] In one implementation, when the largest first variance is greater than a preset width threshold, the pixel values ​​of pixels whose y-coordinate values ​​are less than a first preset value in the first outer contour line of the target image are modified to preset pixel values ​​to obtain pixel information of the outer contour line of the lane line in the identified target image, and the method further includes:

[0022] When the largest first variance is greater than the preset width threshold, the pixel values ​​of the pixels whose y-coordinate values ​​of the first outer contour line on the target image are less than the first preset value are modified to the preset pixel values ​​to obtain the pixel information of the second outer contour line of the lane line in the recognized target image. The second outer contour line includes multiple lines.

[0023] For each second outer contour line, a straight line is fitted to obtain multiple fitted lane lines, and the fitted lane lines are straight lines.

[0024] Find the third pixel with the largest y-coordinate value among the pixels of each fitted lane line;

[0025] The fourth pixel with the smaller y-coordinate value is obtained from the third pixel of any two fitted lane lines. The fourth pixel is the pixel on the first fitted lane line among any two fitted lane lines.

[0026] Obtain the x-coordinate value of the fifth pixel point on the extension line of the second fitted lane line among any two fitted lane lines, where the y-coordinate value of the fifth pixel point is equal to that of the fourth pixel point.

[0027] Calculate the first distance between the x-coordinate value of the fourth pixel and the x-coordinate value of the fifth pixel;

[0028] If the first distance is less than the first preset distance threshold, the second outer contour lines corresponding to any two fitted lane lines are merged to obtain the pixel information of the outer contour lines of the lane lines in the identified target image.

[0029] In one implementation, the method further includes:

[0030] Apply mean filtering to the outer contour lines.

[0031] In one implementation, the target image includes N consecutively acquired frames, where N is an integer greater than 1; the method further includes:

[0032] Obtain multiple first target lane lines detected in the target images of frame i and frame i+1, where i≤N and i is an integer greater than 1;

[0033] Calculate the second distance between any first target lane line among the multiple first target lane lines detected in the target image of frame i and each first target lane line among the first target lane lines detected in the target image of frame i+1.

[0034] When the minimum second distance is less than the second preset distance threshold, the first target lane line corresponding to the minimum second distance is fused with the first target lane line detected in the target image of the (i+1)th frame to obtain the fused first target lane line.

[0035] When the minimum second distance detected after a preset number of consecutive detections is greater than the second preset distance threshold, any first target lane line detected in the target image of the i-th frame is deleted.

[0036] In one implementation, the method further includes:

[0037] If the error is less than the preset error threshold, the second target lane line is obtained by fitting the outer contour line using the least squares method based on the first position information of the outer contour pixels of the lane line.

[0038] In one implementation, the coordinate system of the first target lane line includes a pixel coordinate system; after obtaining the first target lane line by fitting the outer contour line using a weighted least squares method based on the position information of the pixels of each segment line and a preset weight factor configured for each segment line, the method further includes:

[0039] Transform the pixel coordinate system of the first target lane line into the vehicle body coordinate system;

[0040] Curve fitting is performed on the first target lane line after coordinate system transformation, and the fitted lane line result is output for vehicle control during vehicle driving.

[0041] In one implementation, the method further includes, before acquiring the target image:

[0042] Acquire road images;

[0043] A grayscale image, including lane lines, is obtained from a road image using a pixel-segmentation-based neural network.

[0044] The grayscale image is binarized to obtain the binarized target image.

[0045] Secondly, embodiments of this application provide an apparatus for determining lane lines, the apparatus comprising:

[0046] The recognition module is used to recognize the pixel information of the outer contour of the lane line in the target image. The pixel information of the outer contour includes the first position information of each pixel point of the outer contour of the lane line.

[0047] The calculation module is used to calculate the second position information of the lane line fitted based on the first position information, and the error between the second position information and the first position information.

[0048] The segmentation module is used to segment the outer contour line into a preset number of segments when the error is greater than or equal to a preset error threshold.

[0049] The fitting module is used to fit the outer contour line using the weighted least squares method based on the position information of the pixels of each segment line and the preset weight factor configured for each segment line, so as to obtain the first target lane line.

[0050] In one embodiment, the first position information of a pixel includes x-coordinate values ​​and y-coordinate values; the device for determining lane lines further includes a determining module and a modifying module;

[0051] The recognition module is also used to identify the position information of the pixels of the first outer contour line of the lane line in the target image;

[0052] The determination module is used to determine the first pixel point whose y-coordinate value is a first preset value in the first outer contour line;

[0053] The calculation module is also used to calculate the first expected value of the x-coordinate of the first pixel;

[0054] The calculation module is also used to calculate the first variance between the x-coordinate value of each first pixel and the first expected value;

[0055] The modification module is used to modify the pixel values ​​of pixels whose y-coordinate values ​​are less than a first preset value to preset pixel values ​​when the largest first variance is greater than a preset width threshold, so as to obtain the pixel information of the outer contour of the lane line in the recognized target image.

[0056] In one embodiment, the determining module is further configured to determine a second pixel whose y-coordinate value is a second preset value when the largest first variance is less than or equal to a preset width threshold.

[0057] The calculation module is also used to calculate the second expected value of the x-coordinate of the second pixel;

[0058] The calculation module is also used to calculate the second variance between the x-coordinate value of each second pixel and the second expected value;

[0059] The modification module is also used to modify the pixel values ​​of pixels whose y-coordinate values ​​are less than a second preset value in the first outer contour of the target image to preset pixel values ​​when the largest second variance is greater than a preset width threshold, so as to obtain the pixel information of the outer contour of the lane line in the recognized target image.

[0060] In one embodiment, the device for determining lane lines further includes an acquisition module and a determination module;

[0061] The modification module is also used to modify the pixel values ​​of pixels whose y-coordinate values ​​are less than the first preset value to the preset pixel values ​​when the largest first variance is greater than the preset width threshold, so as to obtain the pixel information of the second outer contour of the lane line in the recognized target image. The second outer contour includes multiple lines.

[0062] The fitting module is also used to perform straight line fitting on each second outer contour line to obtain multiple fitted lane lines, and the fitted lane lines are straight lines;

[0063] The acquisition module is used to obtain the third pixel with the largest y-coordinate value among the pixels of each fitted lane line;

[0064] The acquisition module is also used to acquire the fourth pixel with the smaller y-coordinate value among the third pixels of any two fitted lane lines. The fourth pixel is a pixel on the first fitted lane line among any two fitted lane lines.

[0065] The acquisition module is also used to acquire the x-coordinate value of the fifth pixel point that is equal to the y-coordinate value of the fourth pixel point on the extension line of the second fitted lane line among any two fitted lane lines.

[0066] The calculation module is also used to calculate the first distance between the x-coordinate value of the fourth pixel and the x-coordinate value of the fifth pixel;

[0067] The merging module is used to merge the second outer contour lines corresponding to any two fitted lane lines when the first distance is less than a first preset distance threshold, so as to obtain the pixel information of the outer contour lines of the lane lines in the recognized target image.

[0068] In one embodiment, the device for determining lane lines further includes a mean filtering module;

[0069] The mean filtering module is used to perform mean filtering on the outer contour lines.

[0070] In one implementation, the target image includes N consecutively acquired frames, where N is an integer greater than 1; the device for determining lane lines further includes a fusion module and a deletion module.

[0071] The acquisition module is also used to acquire multiple first target lane lines detected in the target images of the i-th frame and the (i+1)-th frame, where i≤N and i is an integer greater than 1;

[0072] The calculation module is also used to calculate the second distance between any first target lane line among the multiple first target lane lines detected in the target image of the i-th frame and each first target lane line among the first target lane lines detected in the target image of the (i+1)-th frame.

[0073] The fusion module is used to fuse any first target lane line corresponding to the minimum second distance with the first target lane line detected in the target image of the (i+1)th frame when the minimum second distance is less than the second preset distance threshold, so as to obtain the fused first target lane line.

[0074] The deletion module is used to delete any first target lane line detected in the target image of the i-th frame when the minimum second distance detected a preset number of times is greater than the second preset distance threshold.

[0075] In one embodiment, the fitting module is further configured to, based on the first position information of the outer contour pixels of the lane line, fit the outer contour line using the least squares method to obtain the second target lane line, provided that the error is less than a preset error threshold.

[0076] In one embodiment, the lane marking determination device further includes a conversion module and an output module;

[0077] The conversion module is used to fit the outer contour line using the weighted least squares method based on the position information of the pixel points of each segment line and the preset weight factor configured for each segment line, and then convert the pixel coordinate system of the first target lane line into the vehicle coordinate system.

[0078] The output module is used to perform curve fitting on the first target lane line after coordinate system transformation, and output the fitted lane line result for vehicle control during vehicle driving.

[0079] In one embodiment, the device for determining lane lines further includes a data acquisition module and a processing module;

[0080] The acquisition module is used to acquire road images before acquiring the target image;

[0081] The acquisition module is also used to acquire a grayscale image of the road image, including lane lines, through a pixel-segmentation-based neural network.

[0082] The processing module is used to binarize the grayscale image to obtain the binarized target image.

[0083] Thirdly, embodiments of this application provide an electronic device, which includes: a processor and a memory storing computer program instructions;

[0084] When the processor executes computer program instructions, it implements the method for determining lane lines as described in any embodiment of the first aspect.

[0085] Fourthly, embodiments of this application provide a computer storage medium storing computer program instructions, which, when executed by a processor, implement the method for determining lane lines as described in any embodiment of the first aspect.

[0086] Fifthly, embodiments of this application provide a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to perform a method for determining lane lines as described in any embodiment of the first aspect.

[0087] The method, apparatus, device, and computer storage medium for determining lane lines in this application embodiment identify the pixel information of the outer contour of lane lines in a target image. The pixel information of the outer contour includes first position information of each pixel point of the lane line's outer contour. A second position information of the lane line fitted based on the first position information is calculated, along with the error between the second and first position information. This allows determination of whether the outer contour of the lane line is a straight line or a curve. If the error is greater than or equal to a preset error threshold, the outer contour is a curve. In this case, the outer contour is segmented into a preset number of segments. Based on the position information of the pixels in each segment and a preset weighting factor configured for each segment, a weighted least squares method is used to fit the outer contour to obtain the first target lane line. Thus, by using the weighted least squares method to fit curve-type lane lines, lane lines suitable for vehicle control can be detected with high precision even in scenarios with a small road curvature radius. Attached Figure Description

[0088] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0089] Figure 1 This is a flowchart illustrating a method for determining lane lines according to an embodiment of this application;

[0090] Figure 2 This is a schematic diagram of a road image within a park provided in one embodiment of this application;

[0091] Figure 3 This is a schematic diagram of a binary image provided in one embodiment of this application;

[0092] Figure 4 This is a schematic diagram of the result of searching the outer contour of a road image according to an embodiment of this application;

[0093] Figure 5 This is a schematic diagram of segmenting the outer contour line of a curve according to an embodiment of this application;

[0094] Figure 6 This is a schematic diagram illustrating the comparison of lane line fitting results provided in one embodiment of this application;

[0095] Figure 7 This is a schematic diagram illustrating the length of the outer contour line provided in one embodiment of this application;

[0096] Figure 8 This is one of the schematic diagrams of the overlapping area of ​​the outer contour lines provided in one embodiment of this application;

[0097] Figure 9 This is a second schematic diagram of the overlapping area of ​​the outer contour lines provided in one embodiment of this application;

[0098] Figure 10 This is a schematic diagram of the result of truncating the outer contour line according to an embodiment of this application;

[0099] Figure 11 This is a schematic diagram of lane line interruption provided in one embodiment of this application;

[0100] Figure 12 This is a schematic diagram illustrating the calculation of lane distances according to an embodiment of this application;

[0101] Figure 13 This is a flowchart illustrating a method for determining lane lines according to an embodiment of this application;

[0102] Figure 14This is a schematic diagram of the structure of a lane line determination device provided in one embodiment of this application;

[0103] Figure 15 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0104] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0105] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0106] Currently, vision-based lane detection algorithms mainly employ two methods: the Hough transform method and neural network models. Since the Hough transform method is a traditional approach, it is susceptible to interference, has poor robustness, cannot handle occluded lane lines, and cannot classify lane lines. Therefore, mainstream vision-based lane detection methods utilize neural network models.

[0107] In existing technologies, neural network models are mainly used to detect lane lines, and then the least squares method is used to fit the lane lines directly. However, this method is only suitable for highway scenarios because the minimum radius of curvature of a highway is 650 meters. But the radius of curvature of roads in parks is much smaller than that of highways. Therefore, the lane lines fitted directly by the least squares method have a large error when curves and cannot be used for vehicle control.

[0108] To address the aforementioned problems, this application provides a method, apparatus, device, and computer storage medium for determining lane lines. This method identifies the pixel information of the outer contour of a lane line in a target image. The pixel information includes first position information of each pixel on the outer contour. A second position information of the lane line fitted based on the first position information is calculated, along with the error between the second and first position information. This allows the determination of whether the outer contour is a straight line or a curve. If the error is greater than or equal to a preset error threshold, the outer contour is considered a curve. In this case, the outer contour is segmented into a preset number of segments. Based on the position information of the pixels in each segment and a preset weighting factor configured for each segment, a weighted least squares method is used to fit the outer contour to obtain the first target lane line. Thus, by using the weighted least squares method to fit curve-type lane lines, lane lines suitable for vehicle control can be detected with high accuracy even in scenarios with small road curvature radii. The method for determining lane lines provided in this application is described below.

[0109] Figure 1 A flowchart illustrating a method for determining lane lines according to an embodiment of this application is shown.

[0110] like Figure 1 As shown, the method for determining lane lines may specifically include the following steps:

[0111] S110, Identify the pixel information of the outer contour of the lane line in the target image. The pixel information of the outer contour may include the first position information of the outer contour pixels of the lane line.

[0112] The target image can be a road image including the outer contour lines of lane lines, or an image captured in real time by a camera in a vehicle. The outer contour lines can be one or multiple. The first position information of the pixels on the outer contour lines of the lane lines can be represented using Cartesian coordinates.

[0113] In some embodiments, the method for determining lane lines may further include, before identifying pixel information of the outer contour of lane lines in a target image:

[0114] Acquire road images;

[0115] A grayscale image, including lane lines, is obtained from a road image using a pixel-segmentation-based neural network.

[0116] The grayscale image is binarized to obtain the binarized target image.

[0117] Road images can be captured in real time by cameras inside vehicles. The road images are input into a pixel-segmentation-based neural network, and the grayscale image containing lane lines output by the neural network is subjected to operations such as erosion and dilation. Then, the grayscale image is binarized using a threshold to obtain the binarized target image.

[0118] As an example, a camera is installed inside a vehicle, and its intrinsic and extrinsic parameters are calibrated to acquire road images in real time, converting the YUV format images to RGB format images. Figure 2 As shown, the image is a road image within the park. This road image is input into a pixel-segmentation-based neural network, and the grayscale image containing lane lines output by the neural network is eroded and dilated, then binarized using a threshold to obtain a binary image I. b ,like Figure 3 As shown in the image, the white area represents the set of lane line pixels. Further analysis of this binary image I... b Search for the outer contour line.

[0119] In this embodiment of the application, before identifying the pixel information of the outer contour of the lane lines in the target image, a road image is acquired, and a grayscale image including the lane lines is obtained through a pixel segmentation-based neural network. Then, the grayscale image is binarized. In this way, the real-time acquired road image can be processed by neural networks and binarization, and the resulting binarized target image is convenient for subsequent outer contour line search.

[0120] As an example, a camera in a vehicle captures road images in real time, searches for the outer contour C of the captured road images, and saves C. j Let P represent the j-th outer contour, and each outer contour contains N 2D pixels P = {(x1, y1), (x2, y2), ..., (x... n ,y n )}.like Figure 4 As shown, the search for the outer contour of the road image yielded three outer contour lines: C1, C2, and C3. The white pixels represent the 2D pixels corresponding to these outer contour lines. The positional information of these pixels is expressed in Cartesian coordinates, with the Cartesian coordinate axes oriented as shown below. Figure 4 As shown.

[0121] S120, calculate the second position information of the lane line fitted based on the first position information, and the error between the second position information and the first position information.

[0122] Based on the first position information, a straight line is fitted to the outer contour line, and the fitting error between the second position information of the fitted lane line and the first position information is calculated. The second position information can be the position information of each pixel point of the fitted lane line, which can be represented in Cartesian coordinates.

[0123] As an example, the outer contour lines of a road image containing lane lines are fitted with straight lines to obtain the fitted straight line y = kx + b, and the fitting error e is calculated. l Fitting error e l The calculation formula is as follows:

[0124]

[0125] Where, x j and y j These are the x and y coordinates of the j-th pixel of the outer contour line.

[0126] In some embodiments, the method for determining lane lines may further include:

[0127] If the error is less than the preset error threshold, the second target lane line is obtained by fitting the outer contour line using the least squares method based on the first position information of the outer contour pixels of the lane line.

[0128] The second target lane line can be a lane line obtained by fitting a straight-line type outer contour line.

[0129] As an example, the fitting error e is calculated from the straight line fitted to the outer contour. l Afterwards, when e l When the error is less than the preset fitting error threshold, the outer contour line is marked as a straight line, and a third-order polynomial is directly fitted using the least squares method. The fitting process is shown below:

[0130]

[0131] According to the definition of least squares, the β that minimizes the objective function e is the optimal polynomial coefficient. Therefore, we take the partial derivatives of the objective function e with respect to β0, β1, β2, and β3.

[0132]

[0133] make

[0134]

[0135] Simplifying and rearranging the above equation, we get:

[0136]

[0137] The above equation is a system of four linear equations. By directly decomposing it using LU, we can obtain the coefficients of the cubic polynomial: β0, β1, β2, β3, which is the curve equation of the second target lane line.

[0138] In this embodiment, when the error is less than a preset error threshold, the second target lane line is obtained by fitting the outer contour line using the least squares method based on the first position information of the outer contour pixel points of the lane line. In this way, the type of the outer contour line of the lane line can be distinguished first, and the lane line of the straight type can be fitted using the least squares method, while the lane line of the curve type can be fitted using the weighted least squares method, thereby reducing the amount of calculation and improving efficiency.

[0139] S130, when the error is greater than or equal to the preset error threshold, the outer contour line is divided into a preset number of segments.

[0140] The preset error threshold can be any threshold set by the user as needed, and the preset number can be any number set by the user as needed. For example, it can be that the outer contour line is divided into 10 segments, and each segment can be of equal length.

[0141] As an example, the fitting error e is calculated from the straight line fitted to the outer contour. l Afterwards, when e l If the error exceeds a preset fitting error threshold, the outer contour line will be marked as a curve type. For example... Figure 5 As shown, the outer contour of the curve type is segmented along the Y-axis, and d = |y max -y min The distance is divided into 10 equal parts, denoted by y0, y1, ..., y9, where y0 corresponds to the pixel closest to the front of the vehicle in the captured image, and y9 corresponds to the pixel furthest from the front of the vehicle in the captured image.

[0142] S140: Based on the position information of the pixels of each segment line and the preset weight factor configured for each segment line, the outer contour line is fitted using the weighted least squares method to obtain the first target lane line.

[0143] The preset weighting factor can be a weighting factor that the user pre-assigns to each segment line based on experience. It can be assigned according to the distance between the pixels of the segment line and the vehicles in the target image. For example, the larger the y-coordinate value of the pixel of the segment line in the target image, the closer it is to the vehicles in the target image; therefore, a larger y-coordinate value for the pixel of the segment line and a larger preset weighting factor are configured. Then, based on the preset weighting factor, a weighted least squares method is used to fit the outer contour line to obtain the fitted first target lane line.

[0144] As an example, such as Figure 5As shown, for each segment of the outer contour in the image, a weight factor is assigned based on the distance between the pixel of the segment and the vehicle in the acquired image. Then, the outer contour is fitted using the weighted least squares method. 0.5 is assigned as a weight factor to segment y0, 0.25 as a weight factor to segment y1, and so on, with a 1 / 2 ratio, so that each segment receives a weight factor. Since lane lines closer to the front of the vehicle are more important for vehicle control, the closer the segment is to the front of the vehicle, the larger its weight factor.

[0145] All pixels in the current outer contour are assigned a weight factor based on their segment. Since the weight factor is constant, the differentiation method is the same as the standard least squares method used to fit straight-line outer contours. The difference lies in the subsequent system of linear equations:

[0146]

[0147] By directly solving the above linear equation system using LU decomposition, we can obtain the coefficients of the cubic polynomial: β0, β1, β2, β3, which is the curve equation of the first target lane line.

[0148] As another example, after processing the image of a park road through a neural network and thresholding it, the outer contour line is obtained. This outer contour line is then fitted with a cubic polynomial using both least squares and weighted least squares methods, resulting in the following... Figure 6 The results shown show the output of the least squares method on the left and the output of the weighted least squares method on the right. It can be seen that the output of the weighted least squares method is smoother and more accurate than that of the least squares method, and can be used for vehicle control.

[0149] In this embodiment, the pixel information of the outer contour of the lane line in the target image is identified. This pixel information includes the first position information of each pixel on the outer contour of the lane line. The second position information of the lane line fitted based on the first position information is calculated, along with the error between the second and first position information. This allows determination of whether the outer contour of the lane line is a straight line or a curve. If the error is greater than or equal to a preset error threshold, the outer contour is a curve. In this case, the outer contour is segmented into a preset number of segments. Based on the position information of the pixels on each segment and a preset weighting factor configured for each segment, the outer contour is fitted using a weighted least squares method to obtain the first target lane line. Thus, by fitting a curve-type lane line using the weighted least squares method, lane lines suitable for vehicle control can be detected with high accuracy even in scenarios with a small road curvature radius.

[0150] In some embodiments, the first position information of a pixel may include x-coordinate values ​​and y-coordinate values; S110 may specifically include:

[0151] Identify the positional information of the pixels on the first outer contour line of the lane line in the target image;

[0152] The first pixel whose y-coordinate value is a first preset value is determined within the first outer contour line;

[0153] Calculate the first expected value of the x-coordinate of the first pixel;

[0154] Calculate the first variance between the x-coordinate value of each first pixel and the first expected value;

[0155] If the largest first variance is greater than the preset width threshold, the pixel values ​​of pixels whose y-coordinate values ​​are less than the first preset value in the first outer contour of the target image are modified to the preset pixel values, thereby obtaining the pixel information of the outer contour of the lane line in the recognized target image.

[0156] The position information of a pixel is represented by the x-coordinate value along the first direction and the y-coordinate value along the second direction, which is 90 degrees from the first direction. For example, it can be a pixel coordinate system.

[0157] The first outer contour line can be one or more lane lines identified from the target image. The first preset value can be a priori value calculated based on parameters such as the installation height of the in-vehicle camera, the camera's field of view, and road curves. The first pixel can be a pixel in the first outer contour line whose y-coordinate value is the first preset value. A first expected value for the x-coordinate of the first pixel is calculated, and a first variance between the x-coordinate value of each first pixel and the first expected value is calculated. The largest variance is obtained. If the largest first variance is greater than a preset width threshold, the pixel values ​​of pixels in the first outer contour line whose y-coordinate values ​​are less than the first preset value are modified to the preset pixel values. The preset pixel values ​​can be pre-set pixel values, such as 0, meaning white pixels in the first outer contour line with y-coordinate values ​​less than the first preset value are modified to black pixels. The preset width threshold can be a width threshold pre-set by the user, such as the width of the lane line itself.

[0158] Furthermore, if the length of the first outer contour line is less than a preset length threshold, the first outer contour line is deleted. For example, such as... Figure 7 As shown, C1 is the first outer contour line currently being processed, and the length of this first outer contour line is the maximum y-coordinate value of the pixel points within the first outer contour line. max With the minimum y-coordinate value y min If the difference is less than the preset length threshold, the first outer contour line is deleted and the next first outer contour line is processed.

[0159] Alternatively, the first pixel can also be a pixel whose y-coordinate value is within a preset range of the first preset value in the first outer contour line.

[0160] As an example, when the radius of curvature of a curve is small, multiple lane lines tend to intersect at the vanishing point, creating an overlapping area of ​​the outer contour lines. This can lead to errors in lane line fitting in subsequent steps, so it is necessary to truncate the overlapping area. First, the outer contour line C... j All pixels in the dataset that satisfy the following conditions are used as the sample set S. b The sample.

[0161] S b ={P1, P2, ..., P n},|P n .yy min -OVERLAP_Y1|<3

[0162] Among them, P n .y represents the y-coordinate value corresponding to the nth pixel, and OVERLAP_Y1 is the first preset value, such as... Figure 8 As shown, set S b The set S is composed of the coordinates of all white pixels within the rectangle corresponding to OVERLAP_Y1, and is calculated using the following formula. b Expected x-coordinate values ​​of pixels on the inner and outer contour lines:

[0163]

[0164] Calculate set S using the following formula b The variance of the x-coordinate value of each pixel within the range to the expected value:

[0165]

[0166] To obtain the maximum variance σ b max indicates taking the maximum value:

[0167] σ b =max{σ b1 , σ b2 , …, σ bN}

[0168] If the outer contour line C j If it does not overlap with other outer contour lines, then outer contour line C j It corresponds to only one lane line. In this case, in set S b The expected value of the x-coordinate of the pixel on the outer contour line is the midpoint of the outer contour line, and the maximum variance σ between the x-coordinate of the pixel on the outer contour line and the expected value is... b It will not exceed the preset width threshold, i.e., σ bIt will not exceed the width of the lane lines themselves. At the maximum variance σ b If the width exceeds the preset threshold, then the current outer contour line C... j In set S b There are overlapping areas among the pixels within the current outer contour line C. j The y-coordinate of the pixel with the largest y-coordinate value in the overlapping region is: Cut_yj = OVERLAP_Y1 + y min The pixel values ​​of pixels whose y-coordinate values ​​are less than Cut_yj in the outer contour line are cleared to zero, that is, the overlapping area is truncated, and the pixel information of the outer contour line of the lane line is obtained after truncating.

[0169] In this embodiment, by identifying the position information of the pixels of the first outer contour line of the lane lines in the target image, the first pixel with a y-coordinate value of a first preset value is determined, and the first expected value of the x-coordinate value of the first pixel is calculated. Then, the first variance between the x-coordinate value and the first expected value of each first pixel is calculated. If the largest first variance is greater than a preset width threshold, the pixel values ​​of pixels with y-coordinate values ​​less than the first preset value in the first outer contour line of the target image are modified to the preset pixel values, thus obtaining the pixel information of the outer contour line of the lane lines in the identified target image. In this way, it is possible to detect whether the lane lines in the target image overlap at the vanishing point. If they overlap, the minimum overlap area in the y-axis direction is calculated, and the overlap area is truncated to ensure the correctness of subsequent curve fitting.

[0170] In some embodiments, S110 may further include:

[0171] If the largest first variance is less than or equal to a preset width threshold, determine the second pixel point whose y-coordinate value is a second preset value in the first outer contour line;

[0172] Calculate the second expected value of the x-coordinate of the second pixel;

[0173] Calculate the second variance between the x-coordinate value of each second pixel and the second expected value;

[0174] If the largest second variance is greater than the preset width threshold, the pixel values ​​of pixels whose y-coordinate values ​​are less than the second preset value in the first outer contour of the target image are modified to the preset pixel values, thereby obtaining the pixel information of the outer contour of the lane line in the recognized target image.

[0175] The second preset value is less than the first preset value. The second preset value can be a priori value calculated based on parameters such as the installation height of the in-vehicle camera, the camera's field of view, and road curves. The second pixel can be a pixel in the first outer contour whose y-coordinate value is the second preset value. Alternatively, the second pixel can also be a pixel in the first outer contour whose y-coordinate value falls within a preset range of the second preset value.

[0176] As an example, at the maximum variance σ b If the width is less than or equal to the preset width threshold, then the current outer contour line C j In set S b The pixels within the outline C have no overlapping areas. j All pixels in the dataset that satisfy the following conditions are used as the sample set S. a The sample.

[0177] S a ={P1, P2, ..., P n},|P n .yy min -OVERLAP_Y0|<3

[0178] Among them, P n .y represents the y-coordinate value corresponding to the nth pixel, and OVERLAP_Y0 is the second preset value, such as... Figure 9 As shown, set S a The set S is composed of the coordinates of all white pixels within the rectangle corresponding to OVERLAP_Y0, and is calculated using the following formula. a Expected x-coordinate values ​​of pixels on the inner and outer contour lines:

[0179]

[0180] Calculate set S using the following formula a The variance of the x-coordinate value of each pixel within the range to the expected value:

[0181]

[0182] To obtain the maximum variance σ a max indicates taking the maximum value:

[0183] σ a =max{σ a1 , σ a2 ,…,σ aN}

[0184] At the largest variance σ a If the width exceeds the preset threshold, then the current outer contour line C... j In set Sa There are overlapping areas among the pixels within the current outer contour line C. j The y-coordinate of the pixel with the largest y-coordinate value in the overlapping region is: Cut_yj = OVERLAP_Y0 + y min The pixel values ​​of pixels with y-coordinate values ​​less than Cut_yj in the outer contour line are cleared to zero, thus truncating the overlapping area, as shown below:

[0185] P x,y =0, y≤Cut y

[0186] Among them, P x,y This represents the pixel value corresponding to the pixel with pixel coordinates (x, y), and the result after truncating the overlapping region is as follows: Figure 10 As shown, the pixel information of the outer contour of the lane line is obtained after truncation. If the maximum variance σ... a If the width is less than or equal to the preset width threshold, then the current outer contour line C ends. j The processing.

[0187] In some embodiments, when there are multiple first outer contour lines, after determining the maximum y-coordinate value Cut_yj of the pixel points in the overlapping area of ​​each first outer contour line, the maximum y-coordinate value Cut_y = max(Cut_yj, Cut_y) in Cut_yj is determined until all first outer contour lines have been processed.

[0188] In this embodiment, if the largest first variance is less than or equal to a preset width threshold, a second pixel with a y-coordinate value of a second preset value is determined in the first outer contour line. A second expected value for the x-coordinate value of the second pixel is calculated, and the second variance between the x-coordinate value and the second expected value of each second pixel is calculated. If the largest second variance is greater than the preset width threshold, the pixel values ​​of pixels with y-coordinate values ​​less than the second preset value in the first outer contour line of the target image are modified to the preset pixel values, thus obtaining the pixel information of the lane line's outer contour line in the recognized target image. This allows for the determination of whether the positions of the first and second preset values ​​overlap. If the positions of the first preset values ​​do not overlap, the positions of the second preset values ​​are checked for overlap. If they overlap, the minimum overlap region in the y-axis direction is calculated, and the overlap region is truncated, further ensuring the correctness of subsequent curve fitting.

[0189] In some embodiments, when the largest first variance is greater than a preset width threshold, the pixel values ​​of pixels whose y-coordinate values ​​are less than a first preset value in the first outer contour line of the target image are modified to preset pixel values ​​to obtain the pixel information of the outer contour line of the lane line in the recognized target image, which may further include:

[0190] When the largest first variance is greater than the preset width threshold, the pixel values ​​of the pixels whose y-coordinate values ​​of the first outer contour line on the target image are less than the first preset value are modified to the preset pixel values ​​to obtain the pixel information of the second outer contour line of the lane line in the recognized target image. The second outer contour line includes multiple lines.

[0191] For each second outer contour line, a straight line is fitted to obtain multiple fitted lane lines, and the fitted lane lines are straight lines.

[0192] Find the third pixel with the largest y-coordinate value among the pixels of each fitted lane line;

[0193] The fourth pixel with the smaller y-coordinate value is obtained from the third pixel of any two fitted lane lines. The fourth pixel is the pixel on the first fitted lane line among any two fitted lane lines.

[0194] Obtain the x-coordinate value of the fifth pixel point on the extension line of the second fitted lane line among any two fitted lane lines, where the y-coordinate value of the fifth pixel point is equal to that of the fourth pixel point.

[0195] Calculate the first distance between the x-coordinate value of the fourth pixel and the x-coordinate value of the fifth pixel;

[0196] If the first distance is less than the first preset distance threshold, the second outer contour lines corresponding to any two fitted lane lines are merged to obtain the pixel information of the outer contour lines of the lane lines in the identified target image.

[0197] The second outer contour line is a set of multiple outer contour lines of the lane lines identified from the target image. A straight line is fitted to each of these second outer contour lines to obtain multiple fitted lane lines, which are straight lines. For example, fitting the second outer contour lines yields the corresponding straight line equation L. j y = k j x+b j The third pixel can be the pixel with the largest y-coordinate value among the pixels of each fitted lane line. The fourth pixel can be the pixel with the smaller y-coordinate value among the third pixels of any two fitted lane lines. The fourth pixel can be located on the first fitted lane line among any two fitted lane lines. On the extension line of the second fitted lane line among any two fitted lane lines, the x-coordinate value of the fifth pixel with the same y-coordinate value as the fourth pixel is determined, and the first distance between the x-coordinate values ​​of the fourth pixel and the fifth pixel is calculated. If the first distance is less than a first preset distance threshold, the second outer contour lines corresponding to any two fitted lane lines are merged. The first preset distance threshold can be any pre-set threshold.

[0198] As an example, such as Figure 11 As shown, due to external influences such as lane line occlusion or unclear segments, the outer contour lines output by the neural network are discontinuous. This causes lane lines C2 and C3, which originally belong to the same lane, to be divided into multiple segments after visual detection. Therefore, it is necessary to merge the outer contour lines that originally belong to the same lane into a single outer contour line. To detect whether any two pairs of outer contour lines in the image belong to the same lane, straight line fitting is first performed on the multiple outer contour lines to obtain multiple fitted lane lines. For example... Figure 12 As shown, assume that the two outer contour lines C being detected are... j and C i The straight line equations of the fitted lane lines are respectively L j and L i Obtain line L j Find the pixel A with the largest y-coordinate among the pixels, and obtain the line L. i Among the pixels, pixel B has the largest y-coordinate value. Then, taking the y-coordinate value of pixel B (which has the smaller y-coordinate value between pixels A and B) as the y-coordinate value, substitute it into the equation of line L. j Solve for the corresponding x j And calculate x j x coordinates of pixel B i Distance: X d =|x j -x i | represents the distance between two straight lines at the same position along the Y-axis and along the x-axis. At a distance of X... d If the distance exceeds the first preset distance threshold, it indicates that the distance between the two detected outer contour lines is too great and they do not belong to the same lane. At distance X... d If the distance is less than the first preset threshold, the two currently detected outer contour lines will be merged.

[0199] Additionally, as another example, at a distance X d If the distance is less than the first preset threshold, calculate L. j and L i Cosine similarity of corresponding vectors:

[0200]

[0201] If the cosine similarity is less than the preset similarity threshold, the two detected outer contour lines will be marked as the same lane line.

[0202] In this embodiment, when the largest first variance is greater than a preset width threshold, the pixel values ​​of pixels whose y-coordinate values ​​are less than a first preset value are modified to preset pixel values ​​to obtain pixel information of the second outer contour of the lane lines in the identified target image. Next, a straight line fitting is performed on each second outer contour to obtain multiple fitted lane lines. The fitted lane lines are straight lines. The third pixel with the largest y-coordinate value among the pixels of each fitted lane line is obtained. Then, the fourth pixel with the smaller y-coordinate value among the third pixels of any two fitted lane lines is obtained, where the fourth pixel is a pixel on the first fitted lane line among the two fitted lane lines. Then, the x-coordinate value of the fifth pixel on the extension line of the second fitted lane line among the two fitted lane lines, which has the same y-coordinate value as the fourth pixel, is obtained. A first distance between the x-coordinate values ​​of the fourth and fifth pixels is calculated. If the first distance is less than a first preset distance threshold, the second outer contours corresponding to any two fitted lane lines are merged. In this way, by merging discontinuous outer contour lines, multiple lane lines that originally belonged to the same lane line are connected, thus enhancing the robustness of lane line detection.

[0203] In some embodiments, the method for determining lane lines may further include:

[0204] Apply mean filtering to the outer contour lines.

[0205] The mean filtering of the outer contour line can specifically include: obtaining the pixels with equal y-coordinate values ​​in each outer contour line; calculating the mean of the x-coordinate values ​​of the pixels with equal y-coordinate values ​​in each outer contour line to obtain the new coordinate values ​​of the pixels in the outer contour line; and generating an outer contour line that includes the new set of pixels.

[0206] As an example, mean filtering is performed on the x-axis of each of the multiple outer contours in an image. This involves taking the average of the x-coordinates of all pixels with the same y-coordinate in the current outer contour to obtain the new pixel coordinates P(μ). xk ,k), p in the formula below j p represents a pixel. j .x represents the x-axis coordinate of that pixel, p j .y represents the y-coordinate of that pixel:

[0207]

[0208] Then, clear all pixels of the current outer contour line, and use the pixels obtained after mean filtering as the pixel set of the current outer contour line, which is then applied to the curve fitting in the subsequent steps.

[0209] In this embodiment, by applying mean filtering to the outer contour line, the accuracy of subsequent fitting of the outer contour line can be improved, the accuracy of lane line detection can be improved, and the computational load of subsequent curve fitting can be reduced.

[0210] In some embodiments, the target image may include N consecutively acquired images, where N is an integer greater than 1. The method for determining lane lines may further include:

[0211] Obtain multiple first target lane lines detected in the target images of frame i and frame i+1, where i≤N and i is an integer greater than 1;

[0212] Calculate the second distance between any first target lane line among the multiple first target lane lines detected in the target image of frame i and each first target lane line among the first target lane lines detected in the target image of frame i+1.

[0213] When the minimum second distance is less than the second preset distance threshold, the first target lane line in the i-th frame corresponding to the minimum second distance is fused with the first target lane line detected in the target image of the (i+1)-th frame to obtain the fused first target lane line.

[0214] When the minimum second distance detected after a preset number of consecutive detections is greater than the second preset distance threshold, any first target lane line detected in the target image of the i-th frame is deleted.

[0215] Multiple first target lane lines detected in the i-th frame target image are matched with multiple first target lane lines detected in the (i+1)-th frame target image. Specifically, the distance between the first target lane lines detected in the two frames is calculated and compared with a second preset distance threshold. The (i+1)-th frame target image can be, for example, the target image of the current frame, and the i-th frame target image can be, for example, the target image of the previous frame. The second preset distance threshold can be any pre-set threshold. When any first target lane line successfully matches a first target lane line detected in the (i+1)-th frame target image, the two lane lines are merged. If no match is found for a preset number of consecutive times, any first target lane line detected in the i-th frame target image is deleted. The preset number of times can be any set number, for example, 5 times.

[0216] As an example, the current frame image F c Each lane line in the image is compared to the previous frame image F. p Each lane line in the algorithm is matched, that is, the distance D between two lane lines is calculated. l If the k-th lane line in the current frame is L... k Its cubic polynomial coefficients are c3, c2, c1, and c0, where the start and end points in the x-direction are s and e, respectively. It is also assumed that the previous frame image F...p There are M lanes in total, then the j-th lane is P. j The corresponding parameter is: C 3j C 2j C 1j C j0 S j E j If j≤M, then L k With P j The distance is D lj :

[0217] D lj =|sS j |+|eE j |+|c3-C 3j |+|c2-C 2j |+|c1-C 1j |+|c0-C 0j |

[0218] Take the smallest distance D among M distances. min When D min If the distance is less than the preset distance threshold, the match is successful, and the minimum distance D is recorded. min The two corresponding lane lines are merged, and the merged lane line result is updated to F. p If not matched, then lane line L will be moved. k As a new lane line update to F p If the previous frame image F p If the result is empty, then the current frame image F will be... c The result is directly updated to F p until the current frame image F c All lane markings have been processed. Additionally, the previous frame image F has been deleted. p Lane lines that have not been matched for 5 consecutive frames.

[0219] In this embodiment, by acquiring multiple first target lane lines detected in the target images of frame i and frame i+1, and calculating the second distance between any first target lane line detected in frame i and each first target lane line detected in frame i+1, the currently detected lane line can be matched with the detection result of the previous frame. Then, when the minimum second distance is less than a second preset distance threshold, the first target lane line corresponding to the minimum second distance is fused with the first target lane line detected in frame i+1 to obtain the fused first target lane line. When the minimum second distance is greater than the second preset distance threshold after a preset number of consecutive detections, the first target lane line detected in frame i is deleted. In this way, lane lines successfully matched between the current frame and the previous frame can be fused, further improving the stability of the detection results.

[0220] In some embodiments, the coordinate system of the first target lane line may include a pixel coordinate system; after S140, the method for determining the lane line may further include:

[0221] Transform the pixel coordinate system of the first target lane line into the vehicle body coordinate system;

[0222] Curve fitting is performed on the first target lane line after coordinate system transformation, and the fitted lane line result is output for vehicle control during vehicle driving.

[0223] The pixel coordinate system of the first target lane line is converted into the vehicle coordinate system. The converted first target lane line is then subjected to curve fitting again, and the fitted lane line result is sent to the control module for vehicle control.

[0224] In this embodiment, the pixel coordinate system of the first target lane line is converted into the vehicle coordinate system, and the first target lane line after coordinate system conversion is subjected to curve fitting. Curve fitting can play a smoothing role, and the output fitted lane line is easier for vehicle control.

[0225] Therefore, to better describe the overall solution, a specific example will be given based on the above embodiments. For example... Figure 13The flowchart shown illustrates the method for determining lane lines. It begins by processing the outer contour lines in the target image, detecting overlapping regions. If overlapping regions exist, they are processed, the minimum overlap in the y-axis direction is calculated, and these overlapping regions are truncated. If not, mean filtering is directly applied to the outer contour lines. After processing the overlapping regions, mean filtering is applied to the outer contour lines. It is then checked whether any of the mean-filtered outer contour lines belong to the same lane line. If so, discontinuing and reconnecting the lines merges those belonging to the same lane line. Next, it is checked whether the fitted lane lines are straight lines. If so, least squares is used to fit the outer contour lines directly. If not, weighted least squares is used to fit the outer contour lines, resulting in the fitted lane lines. Next, the pixel coordinate system of the fitted lane line result is converted into the vehicle coordinate system. Then, the fitted lane line result of the current frame is matched with the fitted lane line result of the previous frame. If they match, the lane lines of the two frames are fused and the lane lines are fitted again. Finally, the lane line result is output to the control module for vehicle control.

[0226] In this embodiment, the accuracy of subsequent curve fitting is ensured by detecting whether lane lines overlap at the vanishing point and truncating the overlapping areas. Mean filtering of the pixel set of the outer contour line before fitting improves lane line detection accuracy and reduces the computational cost of curve fitting. Merging discontinuous outer contour lines connects multiple lane lines that originally belonged to the same lane, enhancing the robustness of lane line detection. Least squares is used to fit straight-line lane lines, while weighted least squares is used to fit curve-type lane lines, thus solving the problem of accurately detecting lane lines suitable for vehicle control in scenarios with small radii of curvature within a park. Matching the lane lines detected in the current frame with the detection results of the previous frame and fusing the matched lane lines further improves the stability of the detection results.

[0227] Figure 14 This is a schematic diagram of the structure of a lane line determination device 1400 according to an exemplary embodiment.

[0228] like Figure 14 As shown, the lane marking determination device 1400 may include:

[0229] The recognition module 1401 is used to recognize the pixel information of the outer contour of the lane line in the target image. The pixel information of the outer contour includes the first position information of each pixel point of the outer contour of the lane line.

[0230] Calculation module 1402 is used to calculate the second position information of the lane line fitted based on the first position information, and the error between the second position information and the first position information;

[0231] The segmentation module 1403 is used to segment the outer contour line into a preset number of segments when the error is greater than or equal to a preset error threshold.

[0232] The fitting module 1404 is used to fit the outer contour line using the weighted least squares method based on the position information of the pixel points of each segment line and the preset weight factor configured for each segment line, so as to obtain the first target lane line.

[0233] In one embodiment, the first position information of a pixel may include x-coordinate values ​​and y-coordinate values; the device 1400 for determining lane lines may also include a determining module and a modifying module.

[0234] The recognition module 1401 is also used to recognize the position information of the pixels of the first outer contour line of the lane line in the target image;

[0235] The determination module is used to determine the first pixel point whose y-coordinate value is a first preset value in the first outer contour line;

[0236] The calculation module 1402 is also used to calculate the first expected value of the x-coordinate of the first pixel;

[0237] The calculation module 1402 is also used to calculate the first variance between the x-coordinate value of each first pixel and the first expected value;

[0238] The modification module is used to modify the pixel values ​​of pixels whose y-coordinate values ​​are less than a first preset value to preset pixel values ​​when the largest first variance is greater than a preset width threshold, so as to obtain the pixel information of the outer contour of the lane line in the recognized target image.

[0239] In one embodiment, the determining module is further configured to determine a second pixel whose y-coordinate value is a second preset value when the largest first variance is less than or equal to a preset width threshold.

[0240] The calculation module 1402 is also used to calculate the second expected value of the x-coordinate of the second pixel;

[0241] The calculation module 1402 is also used to calculate the second variance between the x-coordinate value of each second pixel and the second expected value;

[0242] The modification module is also used to modify the pixel values ​​of pixels whose y-coordinate values ​​are less than a second preset value in the first outer contour of the target image to preset pixel values ​​when the largest second variance is greater than a preset width threshold, so as to obtain the pixel information of the outer contour of the lane line in the recognized target image.

[0243] In one embodiment, the lane line determination device 1400 may further include an acquisition module and a determination module;

[0244] The modification module is also used to modify the pixel values ​​of pixels whose y-coordinate values ​​are less than the first preset value to the preset pixel values ​​when the largest first variance is greater than the preset width threshold, so as to obtain the pixel information of the second outer contour of the lane line in the recognized target image. The second outer contour includes multiple lines.

[0245] The fitting module 1404 is also used to perform straight line fitting on each second outer contour line to obtain multiple fitted lane lines, and the fitted lane lines are straight lines.

[0246] The acquisition module is used to obtain the third pixel with the largest y-coordinate value among the pixels of each fitted lane line;

[0247] The acquisition module is also used to acquire the fourth pixel with the smaller y-coordinate value among the third pixels of any two fitted lane lines. The fourth pixel is a pixel on the first fitted lane line among any two fitted lane lines.

[0248] The acquisition module is also used to acquire the x-coordinate value of the fifth pixel point that is equal to the y-coordinate value of the fourth pixel point on the extension line of the second fitted lane line among any two fitted lane lines.

[0249] The calculation module 1402 is also used to calculate the first distance between the x-coordinate value of the fourth pixel and the x-coordinate value of the fifth pixel;

[0250] The merging module is used to merge the second outer contour lines corresponding to any two fitted lane lines when the first distance is less than a first preset distance threshold, so as to obtain the pixel information of the outer contour lines of the lane lines in the recognized target image.

[0251] In one embodiment, the lane line determination device 1400 may further include a mean filtering module;

[0252] The mean filtering module is used to perform mean filtering on the outer contour lines.

[0253] In one embodiment, the target image may include N consecutively acquired images, where N is an integer greater than 1; the lane line determination device 1400 may also include a fusion module and a deletion module.

[0254] The acquisition module is also used to acquire multiple first target lane lines detected in the target images of the i-th frame and the (i+1)-th frame, where i≤N and i is an integer greater than 1;

[0255] The calculation module 1402 is also used to calculate the second distance between any first target lane line among the multiple first target lane lines detected in the target image of the i-th frame and each first target lane line among the first target lane lines detected in the target image of the (i+1)-th frame.

[0256] The fusion module is used to fuse any first target lane line corresponding to the minimum second distance with the first target lane line detected in the target image of the (i+1)th frame when the minimum second distance is less than the second preset distance threshold, so as to obtain the fused first target lane line.

[0257] The deletion module is used to delete any first target lane line detected in the target image of the i-th frame when the minimum second distance detected a preset number of times is greater than the second preset distance threshold.

[0258] In one embodiment, the fitting module 1404 is further configured to, based on the first position information of the outer contour pixels of the lane line, fit the outer contour line using the least squares method to obtain the second target lane line when the error is less than a preset error threshold.

[0259] In one embodiment, the lane line determination device 1400 may further include a conversion module and an output module;

[0260] The conversion module is used to fit the outer contour line using the weighted least squares method based on the position information of the pixel points of each segment line and the preset weight factor configured for each segment line, and then convert the pixel coordinate system of the first target lane line into the vehicle coordinate system.

[0261] The output module is used to perform curve fitting on the first target lane line after coordinate system transformation, and output the fitted lane line result for vehicle control during vehicle driving.

[0262] In one embodiment, the lane line determination device 1400 may further include a data acquisition module and a processing module;

[0263] The acquisition module is used to acquire road images before acquiring the target image;

[0264] The acquisition module is also used to acquire a grayscale image of the road image, including lane lines, through a pixel-segmentation-based neural network.

[0265] The processing module is used to binarize the grayscale image to obtain the binarized target image.

[0266] Therefore, by identifying the pixel information of the outer contour of the lane line in the target image, including the first position information of each pixel point of the lane line's outer contour, and calculating the second position information of the lane line fitted based on the first position information, and the error between the second and first position information, it is possible to determine whether the outer contour of the lane line is a straight line or a curve. If the error is greater than or equal to a preset error threshold, the outer contour is a curve-type lane line. In this case, the outer contour is segmented into a preset number of segments, and based on the position information of the pixels of each segment and a preset weighting factor configured for each segment, the outer contour is fitted using a weighted least squares method to obtain the first target lane line. In this way, by fitting a curve-type lane line using the weighted least squares method, lane lines that can be used to control vehicles can be detected with high accuracy even in scenarios where the road curvature radius is very small.

[0267] Figure 15 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.

[0268] An electronic device may include a processor 1501 and a memory 1502 storing computer program instructions.

[0269] Specifically, the processor 1501 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0270] Memory 1502 may include mass storage for data or instructions. For example, and not limitingly, memory 1502 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 1502 may include removable or non-removable (or fixed) media. Where appropriate, memory 1502 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 1502 is non-volatile solid-state memory.

[0271] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the methods according to one aspect of this disclosure.

[0272] The processor 1501 implements any of the methods for determining lane lines in the above embodiments by reading and executing computer program instructions stored in the memory 1502.

[0273] In one example, the electronic device may also include a communication interface 1503 and a bus 1510. For example, Figure 15 As shown, the processor 1501, memory 1502, and communication interface 1503 are connected through bus 1510 and complete communication with each other.

[0274] The communication interface 1503 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0275] Bus 1510 includes hardware, software, or both, that couples components of a device that defines lane lines together. For example, and not as a limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 1510 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0276] This electronic device can perform the lane line determination method in this application embodiment based on the pixel information of the outer contour of the lane line in the target image. The pixel information of the outer contour includes the first position information of each pixel point of the lane line outer contour, thereby achieving a combination of... Figure 1 The method for determining lane lines is described.

[0277] Furthermore, in conjunction with the lane line determination methods described in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the lane line determination methods described in the above embodiments.

[0278] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0279] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0280] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0281] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0282] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A method for determining lane lines, characterized in that, include: Identify the pixel information of the outer contour of the lane line in the target image, wherein the pixel information of the outer contour includes the first position information of each pixel point of the outer contour of the lane line. The first position information of the pixel includes the x coordinate value and the y coordinate value; The pixel information of the outer contour of the lane line in the target image is identified, including: Identify the positional information of the pixels on the first outer contour line of the lane line in the target image; The first pixel whose y-coordinate value is a first preset value is determined in the first outer contour line; Calculate the first expected value of the x-coordinate of the first pixel; Calculate the first variance between the x-coordinate value of each first pixel and the first expected value; If the largest first variance is greater than the preset width threshold, the pixel values ​​of the pixels whose y-coordinate values ​​are less than the first preset value in the first outer contour of the target image are modified to the preset pixel values ​​to obtain the pixel information of the outer contour of the lane line in the identified target image. Calculate the error between the second position information of the lane line fitted based on the first position information and the first position information; If the error is greater than or equal to a preset error threshold, the outer contour line is divided into a preset number of segments. Based on the position information of the pixels of each segment line and the preset weight factor configured for each segment line, the outer contour line is fitted using the weighted least squares method to obtain the first target lane line.

2. The method according to claim 1, characterized in that, The method further includes: If the largest first variance is less than or equal to the preset width threshold, determine the second pixel point whose y-coordinate value is the second preset value in the first outer contour line; Calculate the second expected value of the x-coordinate of the second pixel; Calculate the second variance between the x-coordinate value of each second pixel and the second expected value; If the largest second variance is greater than the preset width threshold, the pixel values ​​of pixels whose y-coordinate values ​​are less than the second preset value in the first outer contour of the target image are modified to the preset pixel values ​​to obtain the pixel information of the outer contour of the lane line in the identified target image.

3. The method according to claim 1, characterized in that, The step of modifying the pixel values ​​of pixels whose y-coordinate values ​​are less than the first preset value to the preset pixel values ​​when the largest first variance is greater than the preset width threshold, thereby obtaining the pixel information of the outer contour of the lane line in the identified target image, further includes: When the largest first variance is greater than a preset width threshold, the pixel values ​​of the pixels with y-coordinate values ​​less than the first preset value on the first outer contour line of the target image are modified to preset pixel values ​​to obtain pixel information of the second outer contour line of the lane line in the identified target image. The second outer contour line includes multiple lines. Each second outer contour line is fitted with a straight line to obtain multiple fitted lane lines, wherein the fitted lane lines are straight lines. Obtain the third pixel with the largest y-coordinate value among the pixels of each fitted lane line; Obtain the fourth pixel with the smaller y-coordinate value among the third pixels of any two fitted lane lines. The fourth pixel is a pixel on the first fitted lane line among any two fitted lane lines. Obtain the x-coordinate value of the fifth pixel point on the extension line of the second fitted lane line among any two fitted lane lines, where the y-coordinate value of the fifth pixel point is equal to that of the fourth pixel point. Calculate the first distance between the x-coordinate value of the fourth pixel and the x-coordinate value of the fifth pixel; If the first distance is less than the first preset distance threshold, the second outer contour lines corresponding to any two fitted lane lines are merged to obtain the pixel information of the outer contour lines of the lane lines in the identified target image.

4. The method according to claim 1, characterized in that, The target image includes N consecutively acquired frames, where N is an integer greater than 1. The method further includes: Obtain multiple first target lane lines detected in the target images of frame i and frame i+1, where i < N and i is an integer greater than 1; Calculate the second distance between any first target lane line among the multiple first target lane lines detected in the i-th frame target image and each of the first target lane lines detected in the (i+1)-th frame target image; When the minimum second distance is less than the second preset distance threshold, the arbitrary first target lane line corresponding to the minimum second distance is fused with the first target lane line detected in the target image of the (i+1)th frame to obtain the fused first target lane line. When the minimum second distance detected a preset number of times is greater than the second preset distance threshold, the arbitrary first target lane line detected in the i-th frame target image is deleted.

5. The method according to claim 1, characterized in that, The method further includes: If the error is less than a preset error threshold, the second target lane line is obtained by fitting the outer contour line using the least squares method based on the first position information of the outer contour pixels of the lane line.

6. The method according to claim 1, characterized in that, The coordinate system of the first target lane line includes a pixel coordinate system; after the first target lane line is obtained by fitting the outer contour line using a weighted least squares method based on the position information of the pixels of each segment line and a preset weight factor configured for each segment line, the method further includes: Transform the pixel coordinate system of the first target lane line into the vehicle body coordinate system; The first target lane line after coordinate system transformation is subjected to curve fitting, and the fitted lane line result is output for vehicle control during vehicle driving.

7. The method according to claim 1, characterized in that, Before identifying the pixel information of the outer contour of the lane lines in the target image, the method further includes: Acquire road images; A grayscale image, including lane lines, is obtained from the road image using a pixel-segmentation-based neural network. The grayscale image is binarized to obtain the binarized target image.

8. A device for determining lane lines, characterized in that, The device includes: The recognition module is used to recognize the pixel information of the outer contour of the lane line in the target image. The pixel information of the outer contour includes the first position information of each pixel point of the outer contour of the lane line. The first position information of the pixel point includes the x coordinate value and the y coordinate value. The recognition module is also used to identify the position information of the pixels of the first outer contour line of the lane line in the target image; The determination module is used to determine the first pixel point whose y-coordinate value is a first preset value in the first outer contour line; The calculation module is used to calculate the first expected value of the x-coordinate of the first pixel. The calculation module is also used to calculate the first variance between the x-coordinate value of each first pixel and the first expected value; If the largest first variance is greater than the preset width threshold, the pixel values ​​of the pixels whose y-coordinate values ​​are less than the first preset value in the first outer contour of the target image are modified to the preset pixel values ​​to obtain the pixel information of the outer contour of the lane line in the identified target image. The calculation module is also used to calculate the second position information of the lane line fitted based on the first position information, and the error between the second position information and the first position information. The segmentation module is used to segment the outer contour line into a preset number of segments when the error is greater than or equal to a preset error threshold. The fitting module is used to fit the outer contour line using the weighted least squares method based on the position information of the pixel points of each segment line and the preset weight factor configured for each segment line, so as to obtain the first target lane line.

9. An electronic device, characterized in that, The device includes: a processor and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the method for determining lane lines as described in any one of claims 1-7.

10. A computer storage medium, characterized in that, The computer storage medium stores computer program instructions, which, when executed by a processor, implement the method for determining lane lines as described in any one of claims 1-7.

11. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device causes the electronic device to perform the method for determining lane lines as described in any one of claims 1-7.

Citation Information

Patent Citations

  • Lane line detection method based on feature point piecewise linear fitting

    CN104008387A

  • Big data image processing method and system based on safe area

    CN112132036A

  • Lane line detection method, distance measurement method and corresponding device

    CN112949398A