A method and system for segmented fitting of lane lines

Abstract

This invention provides a method and system for lane line segmentation and fitting. First, lane lines are extracted from the acquired image using a semantic segmentation model, resulting in lane line segmentation. Then, connected component calculations are performed on these results, and key points are sampled. Next, the Hough transform line detection algorithm is used to select key points that conform to a straight-line relationship, fitting multiple straight lane lines. Unselected key points belong to the curve portion. The positional changes of adjacent key points in the curve portion are then used to represent the first derivative of the curve, and the critical points where the direction of the first derivative changes are identified as the boundary points of the curve, achieving segmented representation of the curve. Subsequently, a cubic polynomial is used to refit the key points of the curve. Finally, the multiple segmented lane lines obtained are stitched together to form multiple complete lane lines. This invention ensures that each segmented lane line can be effectively fitted, thereby improving the accuracy of lane line recognition in roads.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a method and system for lane line segmentation fitting. Background Technology

[0002] In recent years, Autonomous Driving Systems (ADS) have developed rapidly, and the era of intelligent driving is dawning. ADS primarily utilizes various sensors on the vehicle to perceive the surrounding environment and acquire road and target information. Among the road information, lane markings can regulate the current vehicle's driving rules and help determine the driving intentions of surrounding vehicles, playing a crucial role in subsequent planning and control.

[0003] Current lane detection methods mainly involve segmenting road information to obtain lane feature points and then directly fitting the lane lines using polynomials. However, considering the expressive power of polynomials and to avoid Runge's phenomenon, cubic polynomials are often used for fitting in practice.

[0004] However, in real-world roads, lane markings vary widely, and the fitting capability of cubic polynomials is limited, making it impossible to accurately fit lane markings. A common approach is to artificially divide the lane markings into multiple simple lane line segments and use multiple mathematical models to fit each segment separately. However, determining the dividing points is a crucial issue. Another method involves dividing the lane markings into three equal parts based on distance and fitting each part separately. But these segmentation methods are too simplistic and fail to ensure that each lane line segment is smooth enough to be effectively fitted by a cubic polynomial, thus falling short of the requirement for accurate lane marking description and requiring improvement in recognition accuracy. Summary of the Invention

[0005] This invention provides a lane line segmentation fitting method and system, which can segment lane lines according to their own characteristics, ensuring that each segmented lane line can be effectively fitted, thereby achieving an accurate description of lane lines in the road, improving recognition accuracy, providing reliable perception information for subsequent vehicle planning and control, and realizing a safer autonomous driving system.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A lane line segmentation fitting method includes the following steps:

[0008] Step S1: Acquire images during the vehicle's movement;

[0009] Step S2: Use a semantic segmentation model to extract the lane lines of the current frame image to obtain the lane line segmentation results;

[0010] Step S3: Convert the segmentation result into a binary image, and then obtain the corresponding connected regions by scanning;

[0011] Step S4: Sample the connected regions to obtain key points of the lane lines;

[0012] Step S5: Use the Hough transform line detection algorithm to select key points that conform to the straight line relationship and fit multiple straight lane lines, while the unselected key points belong to the curve part.

[0013] Step S6: Use the positional changes of adjacent key points of the curve to represent the first derivative of the curve, and find the critical point of the change in the direction of the first derivative as the boundary point of the curve to realize the segmented representation of the curve.

[0014] Step S7: Fit the key points in each curve segment using a cubic polynomial to obtain multiple curved lane lines;

[0015] Step S8: The multiple straight lane lines and curved lane lines obtained from the segmentation are stitched together to form multiple complete lane lines in the current frame image.

[0016] This invention, based on the characteristics of lane lines, first uses the Hough Transform straight line detection algorithm to eliminate straight sections, achieving initial lane line segmentation. To ensure accurate fitting of curved sections, the remaining curved sections are further segmented. Decomposition points are automatically selected based on the direction of the first derivative of the curve, achieving segmentation of the curved section. After segmentation, a cubic polynomial is used for refitting. Based on the fitted segmented lane line information, the segmented lane lines are spliced ​​together to form multiple complete lane lines, facilitating the use of lane line information in subsequent planning and control. Through this method, this invention ensures that each segmented lane line can be effectively fitted, thereby improving the recognition accuracy of lane lines in roads.

[0017] Furthermore, when detecting the next frame image, steps S2-S4 are executed to obtain the lane line key points of the next frame image, and then it is determined whether the lane line key points of the next frame image belong to the existing segmented lane lines of the previous frame image.

[0018] For key points that do not belong to the existing segmented lane lines in the previous frame image, execute steps S5-S7 to obtain multiple new straight lane lines and curved lane lines.

[0019] For key points that belong to the lane lines that are already segmented in the previous frame image, the straight lane lines and curved lane lines obtained from the previous frame image are filtered and updated in real time using the extended Kalman filter algorithm.

[0020] Finally, the new straight lane lines and curved lane lines are spliced ​​together with the filtered straight lane lines and curved lane lines to form multiple complete new lane lines.

[0021] For newly observed lane line key points during vehicle movement, this invention first determines whether they belong to existing segmented lane lines. If they do not belong to existing segmented lane lines, the invention segments the lane lines according to the method described in previous steps. If they belong to existing segmented lane lines, the lane line information is updated using the Extended Kalman Filter (EKF) algorithm combined with the currently observed lane line key points to achieve real-time lane line tracking. By comparing the presence of existing key points in consecutive frames, the recognition process is streamlined, and the existing key points are filtered and updated using EKF, seamlessly connecting lane line recognition between consecutive frames, effectively improving recognition accuracy and efficiency.

[0022] Furthermore, in step S2, a semantic segmentation model based on a convolutional neural network is used to extract lane lines from the image. By classifying whether the pixel position in the image belongs to a lane line, the segmentation result of the lane line is obtained.

[0023] The recognition process here is based on a semantic segmentation model trained on a standard image containing lane lines. When the test image is input into the model, the model performs feature extraction, classification and other processes, and finally outputs the segmented region containing lane lines.

[0024] Furthermore, in step S4, the specific process of sampling the connected region to obtain the key points of the lane line is as follows:

[0025] First, calculate the normal vector at the boundary point of each connected region. Then, calculate the intersection of the single normal vector of the current boundary point with the boundary of the opposite connected region. Next, calculate the average value of the current boundary point and the intersection point. Use this average value as the corresponding key point and perform inverse transmission transformation on the image to obtain the position information of the key point in the world coordinate system.

[0026] Furthermore, in step S5, the Hough transform line detection algorithm is used to detect the key points belonging to the straight section of the lane line, and the fitting equation of the straight section is obtained:

[0027] y = kx + m

[0028] Where k represents the slope between the lane line and the world coordinate system, and m represents the distance between the lane line and the vehicle;

[0029] By fitting multiple key points that conform to a straight-line relationship using the above formula, multiple straight lane lines can be obtained.

[0030] Furthermore, in step S7, the cubic polynomial is as follows:

[0031] y = a + bx + cx 2 +dx 3

[0032] Where a represents the distance between the current lane line and the vehicle, b represents the slope between the lane line and the world coordinate system, and c and d represent the curvature and rate of change of curvature of the lane line, respectively.

[0033] By fitting multiple key points on the segmented curve using the above formula, multiple corresponding curved lane lines can be obtained.

[0034] Furthermore, in step S8, the specific process of stitching together the multiple straight and curved lane lines obtained from the segmentation to form multiple complete lane lines in the current frame image is as follows:

[0035] The splicing process is divided into two cases;

[0036] If the lane lines obtained from the current segment are lane lines connected under the same connected domain, they are directly determined to be components of the same lane line, and the endpoints are directly connected.

[0037] If the lane lines obtained from the current segment are not connected to lane lines in the same connected domain, the endpoints of the segmented lane lines are extended to the endpoints of other lane lines, and the distances between the two extended lane lines at the endpoints are compared. Considering the smoothness of the lane lines, the first derivatives of the two lane lines at this position need to be further compared. If both satisfy the set threshold conditions, they are determined to be components of the same lane line, and the endpoints are directly connected.

[0038] Furthermore, for key points that do not belong to the existing segmented lane lines in the previous frame image, steps S5-S7 are executed to obtain multiple new straight lane lines and curved lane lines. Then, the new lane lines obtained after segmentation are initialized according to their straight or curved type. The initialization process is as follows:

[0039] For the straight line part, the initialization parameters in the line fitting process are directly obtained when using the Hough transform line detection algorithm for line detection.

[0040] For the curved section, the least squares method is used in conjunction with the collected key point information of the lane lines to perform cubic polynomial fitting, thereby initializing the fitting parameters.

[0041] Furthermore, for key points belonging to lane lines already segmented in the previous frame image, the process of real-time filtering and updating of the straight and curved lane lines obtained from the previous frame image using the extended Kalman filter algorithm is as follows:

[0042] For a straight lane, its mathematical expression is y = kx + m. Taking the slope k and offset m as state variables, given the vehicle's speed v, the vehicle's heading angle offset Δγ, and the travel interval t, then in the world coordinate system, the slope of the lane remains unchanged, while the offset changes by -Δs. y Therefore, we get:

[0043] The state equation is:

[0044]

[0045] The observation equation is:

[0046] y = kx + m

[0047] The Jacobian matrix of the corresponding observation equation is:

[0048]

[0049] For curved lane markings, the mathematical expression is y = a + bx + cx. 2 +dx 3 Taking the polynomial coefficients as state variables, given the vehicle's speed v, heading angle Δγ, travel interval t, and change in heading s, the total deviation is -Δs. x The changes in the vehicle's position in the world coordinate system before and after the update are as follows:

[0050] Δs x =vtsin(γ)

[0051] Δs y =vtcos(γ)

[0052] x' world =x world +Δs x

[0053] y' world =y world +Δs y

[0054] The state equation is:

[0055]

[0056] The observation equation is:

[0057] y = a + bx + cx 2 +dx 3

[0058] The Jacobian matrix of the corresponding observation equation is:

[0059]

[0060] Where, Δs x Δs y Let x represent the distances the vehicle travels along the x and y directions within time t, respectively. world 、x' world These represent the vehicle's position in the x-direction and y-direction in the world coordinate system before and after the update, respectively. world y' world These represent the vehicle's position in the y-direction in the world coordinate system before and after the update.

[0061] The present invention also provides a lane line segmentation fitting system, which is used to implement the above-mentioned lane line segmentation fitting method. The system includes:

[0062] The acquisition module is used to acquire images during the vehicle's movement.

[0063] The segmentation module is used to extract lane lines from the current frame image using a semantic segmentation model, and obtain the lane line segmentation results.

[0064] The sampling module is used to convert the segmentation results into a binary map, then obtain the corresponding connected regions by scanning, and sample the connected regions to obtain the key points of the lane lines;

[0065] The straight line detection module uses the Hough transform straight line detection algorithm to select key points that conform to the straight line relationship and fits them to obtain multiple straight lane lines;

[0066] The curve detection module uses the positional changes of adjacent key points in the curve to represent the first derivative of the curve, and finds the critical point of the change in the direction of the first derivative as the boundary point of the curve to realize the segmented representation of the curve. The key points in each segment of the curve are fitted with a cubic polynomial to obtain multiple curved lane lines.

[0067] The stitching module stitches together multiple straight and curved lane lines obtained from the segmentation to form multiple complete lane lines in the current frame image;

[0068] The filtering module combines the key points of segmented lane lines with the extended Kalman filter algorithm to perform real-time filtering and updating of straight and curved lane lines.

[0069] The processor communicates with the aforementioned modules to perform data processing, transmission, and storage.

[0070] The beneficial effects of this invention are as follows:

[0071] This invention utilizes acquired image information, first performing semantic segmentation and sampling to obtain key points of lane lines, and then modeling the lane lines based on these key points. In terms of modeling, the Hough transform straight line detection algorithm is first used to detect the straight sections of the lane lines and fit multiple straight lane lines, achieving preliminary segmentation of the lane lines and avoiding the influence of straight sections on the fitting of curved sections. Then, the positional changes of the key points are used to represent the first derivative of the curve, and the critical points where the direction of the first derivative changes are found as the boundary points of the curve, achieving segmented representation of the curve. Next, a cubic polynomial is used to model the key points of the curved sections, thus obtaining multiple curved lane lines. Here, the curve is differentiated and segmented, effectively avoiding situations where the curve is too complex to fit, ensuring that each segmented lane line can be effectively fitted. Finally, the obtained straight and curved lane lines are spliced ​​together to obtain multiple complete lane lines. This invention can accurately identify and describe lane lines in roads, improving recognition accuracy and providing reliable perception information for subsequent vehicle planning and control, enabling a safer autonomous driving system. Attached Figure Description

[0072] Figure 1 This is a flowchart illustrating a lane line segmentation fitting method according to the present invention.

[0073] Figure 2 A schematic diagram of sampling key points of the lane lines;

[0074] Figure 3 A schematic diagram for determining the boundary point between a straight line and a curve;

[0075] Figure 4 This is a schematic diagram of a cubic polynomial model;

[0076] Figure 5 A schematic diagram for determining the boundary points between curves;

[0077] Figure 6 This is a schematic diagram of lane line splicing;

[0078] Figure 7 This is a flowchart illustrating the process of processing the next frame of the image.

[0079] Figure 8 A schematic diagram of the state transition for extended Kalman filtering of a curve;

[0080] Figure 9 This is a schematic diagram of the structure of a lane line segmentation fitting system according to the present invention. Detailed Implementation

[0081] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0082] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0083] Example 1:

[0084] In real-world urban roads, numerous complex and curved lane lines exist. Furthermore, greater perception distance is crucial for the safety of autonomous driving systems, and longer, more distant lane lines imply greater complexity in their description. These issues present significant challenges to lane line modeling. Therefore, this embodiment proposes an adaptive lane line segmentation fitting method and system to achieve lane line fitting in complex scenarios.

[0085] like Figures 1-6 As shown in the figure, this embodiment provides a lane line segmentation fitting method, which includes the following steps:

[0086] Step S1: Acquire images during the vehicle's movement;

[0087] Step S2: Use a semantic segmentation model to extract the lane lines of the current frame image to obtain the lane line segmentation results;

[0088] Step S3: Convert the segmentation result into a binary image, and then obtain the corresponding connected regions by scanning;

[0089] Step S4: Sample the connected regions to obtain key points of the lane lines;

[0090] Step S5: Use the Hough transform line detection algorithm to select key points that conform to the straight line relationship and fit multiple straight lane lines, while the unselected key points belong to the curve part.

[0091] Step S6: Use the positional changes of adjacent key points of the curve to represent the first derivative of the curve, and find the critical point of the change in the direction of the first derivative as the boundary point of the curve to realize the segmented representation of the curve.

[0092] Step S7: Fit the key points in each curve segment using a cubic polynomial to obtain multiple curved lane lines;

[0093] Step S8: The multiple straight lane lines and curved lane lines obtained from the segmentation are stitched together to form multiple complete lane lines in the current frame image.

[0094] In step S1, this embodiment can use an onboard camera or photography device to collect image information of the vehicle in real time while it is driving, so that the vehicle can obtain images containing lane lines in real time during the autonomous driving process.

[0095] In step S2, this embodiment uses a semantic segmentation model based on a convolutional neural network to extract lane lines from the image. By classifying whether a pixel in the image belongs to a lane line, the segmentation result of the lane lines is obtained. The recognition process here is based on a semantic segmentation model trained on a standard image containing lane lines. When the test image is input into the model, the model performs feature extraction, classification, and other processes, and finally outputs the segmented region containing lane lines.

[0096] After obtaining the segmented regions, in step S3, this embodiment calculates the connected regions. First, the segmentation results are converted into a binary map based on whether the pixels belong to the lane line category, and then the corresponding connected regions are obtained by scanning.

[0097] In step S4, key points are selected based on the obtained connected regions. To ensure uniform sampling along the lane direction, as follows... Figure 2 As shown, the specific process of sampling the connected region to obtain the key points of the lane line in this embodiment is as follows:

[0098] First, calculate the normal vector at each boundary point of the connected region. Then, calculate the intersection of the single normal vector of the current boundary point with the boundary of the opposite connected region. Next, calculate the average value of the current boundary point and the intersection point, and use this average value as the corresponding key point. Figure 2 (Hollow circle), and perform inverse transmission transformation on the image to obtain the position information of key points in the world coordinate system.

[0099] In real-world road scenarios, lane lines most commonly take the form of straight lines. This type of lane line is simple to model and can be modeled separately from curves with more complex mathematical relationships, preventing the straight line portion from affecting the curve fitting effect. Therefore, after obtaining the key points, in step S5 of this embodiment, the straight line portion is detected. The Hough transform straight line detection algorithm is used to detect the key points belonging to the straight line portion of the lane line, and the fitting equation for the straight line portion is obtained:

[0100] y = kx + m

[0101] Where k represents the slope between the lane line and the world coordinate system, and m represents the distance between the lane line and the vehicle;

[0102] By fitting multiple key points that conform to a straight-line relationship using the above formula, multiple straight lane lines can be obtained.

[0103] This invention utilizes the Hough transform line detection algorithm to detect key points that conform to a linear relationship. Figure 3 Select a solid circle and obtain the corresponding mathematical description y = kx + m, such as... Figure 3 As shown, the unselected key points belong to the curve portion. Figure 3 (Hollow circle), the endpoints of the straight line are the boundary points between the curve and the straight line.

[0104] After obtaining the key points of the curve, this invention uses a cubic polynomial to model the curve. However, the expressive power of the cubic polynomial is limited and cannot effectively reproduce the true situation of the lane lines. To solve this problem, step S6 of this invention further segments the curve to avoid the curve being too complex and unable to be fitted. A typical case where the cubic polynomial cannot fit the curve is when there is one x for multiple y in the corresponding coordinate system, such as... Figure 5 As shown, for this type of curve, there must be a series of points where the direction of the first derivative of the curve changes before and after them. The direction of the first derivative of the curve between these points remains consistent, which can ensure the monotonicity of the curve, so that each x corresponds to only one y value, avoiding the problem of cubic curve failure. Therefore, step S6 of the present invention uses the positional change of key points before and after to represent the first derivative of the curve, and finds the critical point of the change in the direction of the first derivative as the boundary point of the curve to realize the segmented representation of the curve.

[0105] like Figure 4 As shown, in step S7, the cubic polynomial is as follows:

[0106] y = a + bx + cx 2 +dx 3

[0107] Where a represents the distance between the current lane line and the vehicle, b represents the slope between the lane line and the world coordinate system, and c and d represent the curvature and rate of change of curvature of the lane line, respectively.

[0108] By fitting multiple key points on the segmented curve using the above formula, multiple corresponding curved lane lines can be obtained.

[0109] In step S8 of this embodiment, the specific process of stitching together the multiple straight lane lines and curved lane lines obtained from the above segmentation to form multiple complete lane lines in the current frame image is as follows:

[0110] There are two main ways to splice segmented lane lines;

[0111] If the lane lines obtained from the current segment are lane lines connected under the same connected domain, they are directly determined to be components of the same lane line, and the endpoints are directly connected.

[0112] If the lane lines obtained from the current segment are not connected by lane lines within the same connected domain, such as Figure 6 As shown, the endpoints of the segmented lane lines are extended to the endpoints of other lane lines, and the distances between the two extended lane lines at the endpoints are compared. Considering the smoothness of the lane lines, the first derivatives of the two lane lines at that position need to be further compared. If both meet the set threshold conditions, they are determined to be components of the same lane line, and the endpoints are directly connected.

[0113] Since the vehicle acquires images while in motion, each frame is continuously checked. When checking the next frame, steps S2-S4 are executed to obtain the lane line key points of the next frame. Then, it is determined whether the lane line key points of the next frame belong to the existing segmented lane lines of the previous frame. Figure 7 As shown;

[0114] For key points that do not belong to the existing segmented lane lines in the previous frame image, execute steps S5-S7 to obtain multiple new straight lane lines and curved lane lines.

[0115] For key points that belong to the lane lines that are already segmented in the previous frame image, the straight lane lines and curved lane lines obtained from the previous frame image are filtered and updated in real time using the extended Kalman filter algorithm.

[0116] Finally, the new straight lane lines and curved lane lines are spliced ​​together with the filtered straight lane lines and curved lane lines to form multiple complete new lane lines.

[0117] For newly observed lane line key points during vehicle movement, this invention first determines whether they belong to existing segmented lane lines. If they do not belong to existing segmented lane lines, the invention segments the lane lines according to the method described in previous steps. If they belong to existing segmented lane lines, the lane line information is updated using the Extended Kalman Filter (EKF) algorithm combined with the currently observed lane line key points to achieve real-time lane line tracking. By comparing the presence of existing key points in consecutive frames, the recognition process is streamlined, and the existing key points are filtered and updated using EKF, seamlessly connecting lane line recognition between consecutive frames, effectively improving recognition accuracy and efficiency.

[0118] For key points that do not belong to the previous frame image and already have segmented lane lines, steps S5-S7 are executed to obtain multiple new straight lane lines and curved lane lines. Then, the new lane lines obtained after segmentation are initialized according to their straight or curved type. The initialization process is as follows:

[0119] For the straight line part, when using the Hough transform line detection algorithm to detect the line, the initialization parameters k and m of the fitting equation y = kx + m in the line fitting process are directly obtained;

[0120] For the curved portion, the least squares method is used in conjunction with the collected key point information of the lane lines to perform cubic polynomial fitting, thereby achieving a cubic polynomial y = a + bx + cx. 2 +dx 3 Initialization of parameters a, b, c, and d.

[0121] In this embodiment, to make the lane line information more stable, for key points belonging to the lane lines already segmented in the previous frame image, the present invention uses the extended Kalman filter algorithm to perform real-time filtering and updating of the straight and curved lane lines obtained from the previous frame image. The specific process is as follows:

[0122] For a straight lane, its mathematical expression is y = kx + m. Taking the slope k and offset m as state variables, given the vehicle's speed v, the vehicle's heading angle offset Δγ, and the travel interval t, then in the world coordinate system, the slope of the lane remains unchanged, while the offset changes by -Δs. y Therefore, we get:

[0123] The state equation is:

[0124]

[0125] The observation equation is:

[0126] y = kx + m

[0127] The Jacobian matrix of the corresponding observation equation is:

[0128]

[0129] For curved lane markings, the mathematical expression is y = a + bx + cx. 2 +dx 3 Taking the polynomial coefficients as state variables, given the vehicle's speed v, heading angle Δγ, travel interval t, and change in heading s, the total deviation is -Δs. x The changes in the vehicle's position in the world coordinate system before and after the update are as follows:

[0130] Δs x =vtsin(γ)

[0131] Δs y =vtcos(γ)

[0132] x' world =x world +Δs x

[0133] y' world =y world +Δs y

[0134] The state equation is:

[0135]

[0136] The observation equation is:

[0137] y = a + bx + cx 2 +dx 3

[0138] The Jacobian matrix of the corresponding observation equation is:

[0139]

[0140] Where, Δs x Δs y Let x represent the distances the vehicle travels along the x and y directions within time t, respectively. world 、x' world These represent the vehicle's position in the x-direction and y-direction in the world coordinate system before and after the update, respectively. world y' world These represent the vehicle's position in the y-direction in the world coordinate system before and after the update.

[0141] like Figure 9 As shown, the present invention also provides a lane line segmentation fitting system, which is used to implement the above-mentioned lane line segmentation fitting method. The system includes:

[0142] The acquisition module is used to acquire images during the vehicle's movement.

[0143] The segmentation module is used to extract lane lines from the current frame image using a semantic segmentation model, and obtain the lane line segmentation results.

[0144] The sampling module is used to convert the segmentation results into a binary map, then obtain the corresponding connected regions by scanning, and sample the connected regions to obtain the key points of the lane lines;

[0145] The straight line detection module uses the Hough transform straight line detection algorithm to select key points that conform to the straight line relationship and fits them to obtain multiple straight lane lines;

[0146] The curve detection module uses the positional changes of adjacent key points in the curve to represent the first derivative of the curve, and finds the critical point of the change in the direction of the first derivative as the boundary point of the curve to realize the segmented representation of the curve. The key points in each segment of the curve are fitted with a cubic polynomial to obtain multiple curved lane lines.

[0147] The stitching module stitches together multiple straight and curved lane lines obtained from the segmentation to form multiple complete lane lines in the current frame image;

[0148] The filtering module combines the key points of segmented lane lines with the extended Kalman filter algorithm to perform real-time filtering and updates for straight and curved lane lines.

[0149] The processor communicates with the aforementioned modules to perform data processing, transmission, and storage.

[0150] This invention utilizes acquired image information to first extract key points of the lane lines. Then, through inverse transmission transformation, the position information of these key points in the world coordinate system is obtained, and the lane lines are modeled based on these key points. In terms of modeling, this invention first uses the Hough transform line detection algorithm to detect the straight sections of the lane lines and obtains the fitting equation y = kx + m for these straight sections, achieving preliminary segmentation of the lane lines and avoiding the influence of straight sections on the fitting of curved sections. Furthermore, considering that the representation of straight sections is relatively simple, detecting the straight sections of the lane lines first can also reduce the overall complexity of lane line fitting to some extent, thus reducing the fitting time.

[0151] Then, to ensure accurate fitting of the curve portion, the lane line curve needs to be further segmented. Here, this invention automatically selects decomposition points based on the directional change of the first derivative of the lane line, ensuring that each curve segment has monotonicity, thus automatically dividing the complex curve into multiple simpler curves. For the segmented curves, this scheme uses a cubic polynomial y = a + bx + cx. 2 +dx 3 Perform fitting.

[0152] Finally, based on the fitted segmented lane line information, the present invention splices the segmented lane lines to form multiple complete lane lines, so as to facilitate the use of lane line information in subsequent planning and control.

[0153] Furthermore, considering the noise interference generated when vehicles are moving, this invention uses the Extended Kalman Filter (EKF) algorithm to update the lane lines in real time when detecting the next frame of the lane line image, ensuring that the optimal estimate of the lane lines is obtained.

[0154] This invention can accurately identify and describe lane lines in the road, improving recognition accuracy and providing reliable perception information for subsequent vehicle planning and control, thus enabling a safer autonomous driving system.

[0155] The above embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention.

Claims

1. A method for segmented fitting of lane lines, characterized in that, Includes the following steps: Step S1: Acquire images during the vehicle's movement; Step S2: Use a semantic segmentation model to extract the lane lines of the current frame image to obtain the lane line segmentation results; Step S3: Convert the segmentation result into a binary image, and then obtain the corresponding connected regions by scanning; Step S4: Sample the connected regions to obtain lane line key points. The specific process is as follows: First, calculate the normal vector at the boundary point of each connected region. Then, calculate the intersection of the single normal vector of the current boundary point with the boundary of the opposite connected region. Next, calculate the average value of the current boundary point and the intersection point. Use this average value as the corresponding key point and perform inverse transmission transformation on the image to obtain the position information of the key point in the world coordinate system. Step S5: Use the Hough transform line detection algorithm to select key points that conform to the straight line relationship and fit multiple straight lane lines, while the unselected key points belong to the curve part. Step S6: Use the positional changes of adjacent key points of the curve to represent the first derivative of the curve, and find the critical point of the change in the direction of the first derivative as the boundary point of the curve to realize the segmented representation of the curve. Step S7: Fit the key points in each curve segment using a cubic polynomial to obtain multiple curved lane lines; Step S8: The multiple straight lane lines and curved lane lines obtained from the segmentation are stitched together to form multiple complete lane lines in the current frame image. The specific process is as follows: The splicing process is divided into two cases; If the lane lines obtained from the current segment are lane lines connected under the same connected domain, they are directly determined to be components of the same lane line, and the endpoints are directly connected. If the lane lines obtained from the current segment are not connected to lane lines in the same connected domain, the endpoints of the segmented lane lines are extended to the endpoints of other lane lines, and the distances between the two extended lane lines at the endpoints are compared. Considering the smoothness of the lane lines, the first derivatives of the two lane lines at this position need to be further compared. If both satisfy the set threshold conditions, they are determined to be components of the same lane line, and the endpoints are directly connected.

2. The lane line segmentation fitting method according to claim 1, characterized in that, When detecting the next frame image, execute steps S2-S4 to obtain the lane line key points of the next frame image, and then determine whether the lane line key points of the next frame image belong to the existing segmented lane lines of the previous frame image. For key points that do not belong to the existing segmented lane lines in the previous frame image, execute steps S5-S7 to obtain multiple new straight lane lines and curved lane lines. For key points that belong to the lane lines that are already segmented in the previous frame image, the straight lane lines and curved lane lines obtained from the previous frame image are filtered and updated in real time using the extended Kalman filter algorithm. Finally, the new straight lane lines and curved lane lines are spliced ​​together with the filtered straight lane lines and curved lane lines to form multiple complete new lane lines.

3. The lane line segmentation fitting method according to claim 1, characterized in that, In step S2, a semantic segmentation model based on a convolutional neural network is used to extract lane lines from the image. By classifying whether the pixel position in the image belongs to a lane line, the segmentation result of the lane lines is obtained.

4. The lane line segmentation fitting method according to claim 1, characterized in that, In step S5, the Hough transform line detection algorithm is used to detect key points belonging to the straight section of the lane line, and the fitting equation of the straight section is obtained: in This represents the slope between the lane line and the world coordinate system. Indicates the distance between the lane line and the vehicle; By fitting multiple key points that conform to a straight-line relationship using the above formula, multiple straight lane lines can be obtained.

5. The lane line segmentation fitting method according to claim 1, characterized in that, In step S7, the cubic polynomial is as follows: in Indicates the current distance between the lane line and the vehicle. This represents the slope between the lane line and the world coordinate system. These represent the curvature and rate of change of curvature of the lane lines, respectively. By fitting multiple key points on the segmented curve using the above formula, multiple corresponding curved lane lines can be obtained.

6. The lane line segmentation fitting method according to claim 2, characterized in that, For key points that do not belong to the existing segmented lane lines in the previous frame, steps S5-S7 are executed to obtain multiple new straight lane lines and curved lane lines. The newly obtained lane lines after segmentation are then initialized according to their straight or curved type. The initialization process is as follows: For the straight line part, the initialization parameters in the line fitting process are directly obtained when using the Hough transform line detection algorithm for line detection. For the curved section, the least squares method is used in conjunction with the collected key point information of the lane lines to perform cubic polynomial fitting, thereby initializing the fitting parameters.

7. The lane line segmentation fitting method according to claim 2, characterized in that, The process of real-time filtering and updating of straight and curved lane lines obtained from the previous frame image using the extended Kalman filter algorithm for key points belonging to lane lines that already have segmented lines in the previous frame image is as follows: For a straight lane line, its mathematical expression is: Take the slope and offset The speed of the vehicle is known as a state variable. The vehicle's heading angle deflection is and driving interval In the world coordinate system, the slope of the lane line remains unchanged, while the offset changes as follows: Therefore, we get: The state equation is: The observation equation is: The Jacobian matrix of the corresponding observation equation is: For curved lane markings, the mathematical expression is: Taking the polynomial coefficients as state variables, the vehicle speed is known. The vehicle's heading angle deflection is The vehicle travel interval is The offset changes to The changes in the vehicle's position in the world coordinate system before and after the update are as follows: The state equation is: The observation equation is: The Jacobian matrix of the corresponding observation equation is: in, , In time Inner vehicle along direction and Distance traveled in the direction, , The vehicles are positioned in the world coordinate system before and after the update. The position of direction, , The vehicles are positioned in the world coordinate system before and after the update. The location of the direction.

8. A lane line segmentation fitting system, characterized in that, This system is used to implement a lane line segmentation fitting method as described in any one of claims 1-7, and the system includes: The acquisition module is used to acquire images during the vehicle's movement. The segmentation module is used to extract lane lines from the current frame image using a semantic segmentation model, and obtain the lane line segmentation results. The sampling module is used to convert the segmentation results into a binary map, then obtain the corresponding connected regions by scanning, and sample the connected regions to obtain the key points of the lane lines; The straight line detection module uses the Hough transform straight line detection algorithm to select key points that conform to the straight line relationship and fits them to obtain multiple straight lane lines; The curve detection module uses the positional changes of adjacent key points in the curve to represent the first derivative of the curve, and finds the critical point where the direction of the first derivative changes as the boundary point of the curve to realize the segmented representation of the curve. The key points in each segment of the curve are fitted with a cubic polynomial to obtain multiple curved lane lines. The stitching module stitches together multiple straight and curved lane lines obtained from the segmentation to form multiple complete lane lines in the current frame image; The filtering module combines the key points of segmented lane lines with the extended Kalman filter algorithm to perform real-time filtering and updating of straight and curved lane lines. The processor communicates with the aforementioned modules to perform data processing, transmission, and storage.

Images