Lane detection method and device, electronic equipment, storage medium and program product

Abstract

This application provides a lane detection method, apparatus, electronic device, storage medium, and program product, relating to the interdisciplinary fields of intelligent driving and computer vision. The method includes: inputting an image to be detected into a lane detection model to obtain a predicted lane and its left lane line, right lane line, and center line; the lane detection model is trained using images annotated with combinations of the left lane line, right lane line, and center line of the same lane as training samples, and using geometric constraints between lane lines and / or morphological constraints of lane lines as loss functions. The method provided in this application annotates the left lane line, right lane line, and center line of the same lane in the training samples as combinations, achieving an accurate representation of the overall spatial structure of the lane; using geometric constraints between lane lines and morphological constraints of lane lines as loss functions enables the model to perceive the rationality of the lane's geometric structure; thus improving the accuracy of lane detection.

Description

Technical Field

[0001] This application relates to the field of intelligent driving and computer vision intersection technology, and in particular to a lane detection method, device, electronic device, storage medium and program product. Background Technology

[0002] In advanced autonomous driving systems, while high-precision maps can provide prior road information, they suffer from inherent drawbacks such as delayed updates, limited coverage, and high maintenance costs, making it difficult to support safe driving on unknown roads, temporary construction zones, rural roads, or areas not covered by maps. Therefore, developing real-time perception capabilities that do not rely on high-precision maps has become an industry consensus. Among these, accurate and robust perception of lane-level road structures is a core prerequisite for achieving vehicle lateral control, path planning, and behavioral decision-making.

[0003] Therefore, improving the accuracy of lane detection and providing more reliable and safer perception input for autonomous driving systems that do not rely on high-precision maps has become a pressing technical problem for the industry. Summary of the Invention

[0004] This application provides a lane detection method, apparatus, electronic device, storage medium, and program product to address the technical problem of how to improve the accuracy of lane detection and provide more reliable and safer perception input for autonomous driving systems that do not rely on high-precision maps.

[0005] This application provides a lane detection method, including: The image to be detected is input into the lane detection model, and the lane detection model outputs the predicted lane in the image to be detected, as well as the left lane line, right lane line and center line of the predicted lane; The lane detection model is trained using images labeled with the left lane line, right lane line, and center line of the same lane as a combination as training samples, and using the geometric relationship constraints between lane lines and / or the morphological constraints of lane lines in the training samples as loss functions.

[0006] In some embodiments, the lane detection model is trained based on the following steps: The left lane line, right lane line, and center line of the same lane in each image are labeled as a combination to generate training samples; Based on the geometric relationship constraints between lane lines and / or the morphological constraints of lane lines in the training samples, a loss function is constructed. The initial model is trained based on the training samples and the loss function to obtain the lane detection model.

[0007] In some embodiments, the geometric constraints between lane lines include: The spatial distance between the left and right lane lines of the same lane is greater than the minimum lane width threshold and less than the maximum lane width threshold.

[0008] In some embodiments, the step of using the geometric relationship constraints between lane lines and / or the morphological constraints of lane lines in the training samples as the loss function includes: Determine the spatial distance between corresponding points on the left and right lane lines of the same lane as predicted; If the spatial distance between any corresponding points is less than the minimum lane width threshold, a first penalty loss proportional to the difference between the spatial distance and the minimum lane width threshold is generated. If the spatial distance between any corresponding points is greater than the maximum lane width threshold, a second penalty loss proportional to the difference between the spatial distance and the maximum lane width threshold is generated. The loss function is constructed based on the first penalty loss and the second penalty loss.

[0009] In some embodiments, the shape constraints of each lane line include: The point sequence of each lane line maintains second-order smoothness at at least one scale.

[0010] In some embodiments, the step of using the geometric relationship constraints between lane lines and / or the morphological constraints of lane lines in the training samples as the loss function includes: Determine at least one scale; Based on the at least one scale, select multiple sets of three points that meet the requirements of the at least one scale from the point sequence of each predicted lane line; The linear deviation of each group is determined based on the coordinates of three points in each group. The linear deviations of each group are summed and normalized to determine the second-order smoothing loss of at least one scale. The loss function is constructed based on the calculated weights of the at least one scale and the second-order smoothing loss of the at least one scale.

[0011] In some embodiments, the step of inputting the image to be detected into a lane detection model to obtain the predicted lane in the image to be detected, as well as the left lane line, right lane line, and center line of the predicted lane, output by the lane detection model, includes: The image to be detected is input into the lane detection model, and the predicted lane in the image to be detected and the offset of the left lane line, right lane line and center line of the predicted lane are obtained from the output of the lane detection model. The centerline of the predicted lane is determined based on the offsets of the left lane line, right lane line, and centerline of the predicted lane.

[0012] In some embodiments, the method further includes: Based on the confidence level of each predicted lane, multiple candidate predicted lanes are selected. Based on the confidence level of each candidate predicted lane and the planar region overlap between each candidate predicted lane, non-maximum suppression is performed on the multiple candidate predicted lanes to determine the lane detection result in the image to be detected.

[0013] In some embodiments, the step of performing non-maximum suppression on the plurality of candidate predicted lanes based on the confidence level of each candidate predicted lane and the planar region overlap between each candidate predicted lane, and determining the lane detection result in the image to be detected, includes: Based on the confidence scores of each candidate predicted lane, the candidate predicted lanes are sorted in descending order to generate a candidate list. During the current iteration, the candidate predicted lane with the highest current confidence is selected from the candidate list, the planar region overlap between the remaining candidate predicted lanes and the candidate predicted lane with the highest current confidence is calculated, the remaining candidate predicted lanes with a planar region overlap exceeding a preset overlap threshold are deleted, and the candidate predicted lane with the highest current confidence is added to the result list. If the candidate list is empty, the candidate predicted lanes in the result list are determined as the lane detection results in the image to be detected.

[0014] This application provides a lane detection device, comprising: The detection module is used to input the image to be detected into the lane detection model and obtain the predicted lane in the image to be detected, as well as the left lane line, right lane line and center line of the predicted lane, output by the lane detection model. The lane detection model is trained using images labeled with the left lane line, right lane line, and center line of the same lane as a combination as training samples, and using the geometric relationship constraints between lane lines and / or the morphological constraints of lane lines in the training samples as loss functions.

[0015] This application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the lane detection method.

[0016] This application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the lane detection method described above.

[0017] This application provides a computer program product, including a computer program that, when executed by a processor, implements the lane detection method.

[0018] The lane detection method, apparatus, electronic device, storage medium, and program product provided in this application annotate the left lane line, right lane line, and center line of the same lane in the training samples as a combination to achieve an accurate representation of the overall spatial structure of the lane. By using the geometric relationship constraints between lane lines and / or the morphological constraints of each lane line in the training samples as the loss function, the final lane detection model has the ability to perceive the rationality of the lane geometry and can output lane detection results that are more geometrically consistent and physically reasonable. This improves the accuracy of lane detection and provides more reliable and safer perception input for autonomous driving systems that do not rely on high-precision maps. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the lane detection method provided in this application.

[0022] Figure 2 This is a schematic diagram of the lane structure annotation based on geometry guidance provided in this application.

[0023] Figure 3 This is a flowchart illustrating the model training method provided in this application.

[0024] Figure 4 This is a flowchart illustrating the predictive lane post-processing method provided in this application.

[0025] Figure 5 This is a structural schematic diagram of the lane detection device provided in this application.

[0026] Figure 6 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps, units, or modules is not necessarily limited to those explicitly listed, but may include other steps, units, or modules not explicitly listed or inherent to such processes, methods, products, or devices.

[0029] Lane detection methods in related technologies can be broadly categorized into traditional image processing methods and data-driven methods based on deep learning. Traditional methods rely on edge detection, Hough transform, and multinomial fitting, which are often sensitive to factors such as lighting, occlusion, and wear, and lack robustness. With the increasing application of deep learning in visual tasks, more and more research is using models such as convolutional neural networks and Transformers to predict multi-point sets or parameterized forms of lane lines to improve detection accuracy.

[0030] However, these techniques still have the following shortcomings: (1) Most datasets adopt the independent labeling mode for each lane line, making it difficult for the model to learn the geometric structure of the real lane; (2) Most deep learning models directly regress the point set or polynomial parameters of a lane line, resulting in insufficient stability; (3) The loss function is mostly concentrated on point-level L1 loss (absolute value loss) / L2 loss (squared loss) fitting, and the model is prone to problems such as width abnormality, line jitter, and local discontinuity; (4) Sorting and suppressing line segments based on confidence level can easily lead to lane duplication or missed detection.

[0031] In order to address the shortcomings of related technologies, Figure 1 This is a flowchart illustrating the lane detection method provided in this application, as shown below. Figure 1 As shown, the method includes step 110.

[0032] Step 110: Input the image to be detected into the lane detection model to obtain the predicted lane in the image to be detected, as well as the left lane line, right lane line and center line of the predicted lane; The lane detection model is trained using images labeled with the left lane line, right lane line, and center line of the same lane as a combination as training samples, and the loss function is the geometric relationship constraints between lane lines and / or the morphological constraints of each lane line in the training samples.

[0033] Specifically, the lane detection method provided in this application is executed by a lane detection device or system. This device can be implemented in software, such as a lane detection program; or it can be a device that executes the lane detection method, such as a terminal, computer, or server.

[0034] Lane detection models can be end-to-end deep learning-based models. After training, this model receives an image as input and directly outputs one or more predicted lanes within the image. The model can employ common deep neural network architectures, such as Convolutional Neural Network (CNN)-based architectures, Transformer-based architectures, or a combination of both. The model may include a backbone network for extracting deep features from the input image, and one or more detection heads for decoding features and predicting lane line information.

[0035] The input to the lane detection model is the image to be detected. This image can be a single frame captured by an onboard camera or a frame from a video stream. The camera can be installed in front of, behind, or to the side of the vehicle to cover different driving scenarios. The image to be detected can also be a pre-processed image, such as one that has undergone distortion correction, image enhancement, or has been converted to a bird's-eye view through inverse perspective transformation.

[0036] The output of the lane detection model is a predicted lane. Unlike related technologies that typically output scattered lane segments, the predicted lane in this embodiment is a structured combination. Specifically, each predicted lane explicitly contains three parts: a left lane line, a right lane line, and a center line. These three lines are logically bound together to form a complete lane region description. Each lane line (left, right, or center line) can be represented in various forms, such as a sequence ordered along the lane direction. A point set consisting of points ,in , For point The horizontal coordinate, For point The vertical coordinate; it can also be represented as the parameters of a parametric curve, such as a polynomial curve or a Bézier curve.

[0037] This application improves the training method of the lane detection model, ensuring that the model can learn the inherent structure of the lanes, thereby outputting geometrically reasonable and consistent prediction results during inference. The core of this training process lies in two aspects: the labeling method of training samples and the design method of the loss function.

[0038] From the perspective of the annotation method of the training samples, unlike the method in related technologies that annotates each visible lane line as an independent individual, the embodiments of this application adopt the method of annotating the left lane line, right lane line and center line of the same lane in the image as a combination.

[0039] Figure 2 This is a schematic diagram of the lane structure annotation based on geometry guidance provided in this application, such as... Figure 2 As shown, a lane on a road is labeled as a complete geometric structure. Each lane consists of a pair of left and right lane lines, and the corresponding center line is explicitly labeled. Furthermore, the left lane line, right lane line, and center line belonging to the same lane are bound together to form lane-level labeling units. In this labeling method, a lane can be represented as a triple. ,in Indicates the left lane line. Indicates the right lane line. This annotation method represents the center line. It elevates the originally independent lane line annotations to lane-level structured annotations with clear geometric semantics. From the data source, it provides supervision information for the model to learn the pairing relationship between the left and right lane lines and their geometric dependence on the center line, so that the model no longer treats them as irrelevant individuals during training.

[0040] From the perspective of loss function design, the loss function measures the difference between the model's prediction and the ground truth labeling, and guides the optimization direction of the model parameters. The design goal of the loss function in this embodiment is not only to require the lane lines predicted by the model to be close to the ground truth labeling in location, but also to require the prediction results to meet specific geometric constraints. These geometric constraints can specifically include geometric relationship constraints between lane lines, as well as morphological constraints on each lane line. The model's loss function is constructed using these geometric constraints.

[0041] Geometric relationship constraints primarily focus on the reasonableness of the relationships between different lane lines (i.e., left, right, and center lines) within a predicted lane. For example, in the real world, the left and right lane lines of the same lane will almost never intersect, and the distance between them (i.e., lane width) will remain within a reasonable range. Therefore, this constraint can be designed as a penalty term; when the model predicts that the left and right lane lines intersect, or that the distance between them is too narrow or too wide, the loss function will produce a large value, thus forcing the model to adjust its parameters in subsequent training to avoid producing such physically inconsistent predictions.

[0042] Morphological constraints primarily focus on the reasonableness of the shape of individual lane lines (whether left, right, or center lines). For example, real lane lines are typically smooth, continuous curves. Therefore, this morphological constraint can be designed as a penalty term; when the model predicts lane lines with unnatural sharp angles, violent jitters, or breaks, the loss function will produce a larger value. This guides the model to learn to generate curves with smoother shapes and more continuous curvature changes, conforming to the morphological characteristics of real roads.

[0043] By incorporating these geometric constraints into the overall loss function, the lane detection model is forced to learn to conform to these inherent geometric rules while learning to fit the labeled data. This enables its predictive ability to go beyond simple pixel-level imitation and to develop an understanding of the rationality of lane structures.

[0044] By combining the above-mentioned annotation method for training samples and the design method of loss function, when performing lane detection, inputting the image to be detected into the trained lane detection model can obtain a structurally complete predicted lane containing the left lane line, right lane line and center line.

[0045] The lane detection method provided in this application annotates the left lane line, right lane line, and center line of the same lane in the training samples as a combination, thereby achieving an accurate representation of the overall spatial structure of the lane. Using the geometric relationship constraints between lane lines and / or the morphological constraints of each lane line in the training samples as the loss function, the final lane detection model possesses the ability to perceive the rationality of the lane's geometric structure, enabling it to output lane detection results that are geometrically more consistent and physically more reasonable. This improves the accuracy of lane detection and provides more reliable and safer perception input for autonomous driving systems that do not rely on high-precision maps.

[0046] It should be noted that each implementation method of this application can be freely combined, rearranged, or executed individually, and does not need to rely on or depend on a fixed execution order.

[0047] In some embodiments, Figure 3 This is a flowchart illustrating the model training method provided in this application, as shown below. Figure 3 As shown, the lane detection model is trained based on the following steps: Step 310: Label the left lane line, right lane line and center line of the same lane in each image as a combination to generate training samples; Step 320: Construct a loss function based on the geometric relationship constraints between lane lines in the training samples and / or the morphological constraints of each lane line; Step 330: Train the initial model based on the training samples and loss function to obtain the lane detection model.

[0048] Specifically, the training process of the lane detection model mainly includes three stages: generating structured training samples, constructing a loss function with geometric constraints, and iteratively optimizing the initial model.

[0049] In the stage of generating structured training samples: A large amount of raw road image data can be collected through the vehicle-mounted data acquisition platform under various road environments (such as urban roads, highways, and rural roads), different lighting conditions (such as daytime, nighttime, and backlighting), and weather conditions to form a raw image set.

[0050] For each image in the original image set, the following combined annotation operation is performed by a human annotator or with the help of a semi-automatic annotation tool: (1) Identify areas in the image that logically belong to the same lane; (2) For each identified lane, accurately mark its left boundary line (i.e., the left lane line) and right boundary line (i.e., the right lane line). Each boundary line is marked as a series of two-dimensional coordinate points arranged in an orderly manner along the direction of vehicle travel.

[0051] (3) After marking the left and right lane lines, explicitly generate the center line corresponding to the lane based on these two lines. For example, for each pair of points that correspond in the vertical direction on the left and right lane lines, calculate their midpoint as the corresponding point on the center line. By performing this operation on all pairs of corresponding points, a complete center line can be generated.

[0052] (4) Bind the labeled left lane line, right lane line and generated center line in the data structure to form a structured labeling unit, i.e. a triple. The complete labeling of an image is the set of triples corresponding to all lane units in the image.

[0053] Through the above steps, each training sample generated in the end contains an original image and one or more structured triplet annotation combinations.

[0054] In the stage of constructing the loss function that includes geometric constraints: The loss function is typically composed of multiple weighted components. In addition to conventional fitting losses (such as L1 or L2 losses used to measure the distance between predicted points and ground truth points), embodiments of this application also introduce the following constraint losses.

[0055] (1) Constructing a loss term based on geometric relationship constraints: This constraint aims to ensure that the relationship between the lines within a predicted lane unit is reasonable. For example, a lane width loss term can be constructed. When calculating this loss term, the distance between the left and right lane lines predicted by the model is checked point by point. When the distance deviates from a preset reasonable range, the value of the loss term will increase accordingly, thereby penalizing such unreasonable predictions during model training.

[0056] (2) Constructing a loss term based on morphological constraints: This constraint aims to ensure that the lane lines predicted by the model are smooth and continuous. For example, a smoothness loss term can be constructed. This loss term is achieved by calculating the curvature or second-order difference between adjacent points on the lane line. If the predicted lane line has sharp bends or high-frequency jitter, the calculated curvature or second-order difference will be large, leading to an increase in the value of the loss term, thereby guiding the model to generate curves with more natural shapes.

[0057] Ultimately, the total loss function can be a weighted sum of the fitting loss and one or more of the constraint losses mentioned above.

[0058] During the iterative optimization and training phase of the initial model: Before training begins, an initial model needs to be prepared. This model can be a model with a deep neural network architecture but whose parameters have not yet been optimized. Its parameters can be randomly initialized, or, to speed up convergence, the parameters of a model pre-trained on a large general image dataset (such as ImageNet) can be used as initial values.

[0059] The training samples and loss function are applied to iteratively optimize the initial model. After iterative optimization training, the parameters of the initial model are sufficiently optimized, enabling it not only to identify the location of lane lines but also to understand and obey the inherent geometry of the lanes. At this point, the trained initial model becomes the final lane detection model.

[0060] The lane detection method provided in this application ensures that the final lane detection model has the ability to perceive the rationality of lane geometry through the whole process design from data annotation and loss function design to model iterative optimization, and can output lane detection results that are more geometrically consistent and physically reasonable.

[0061] In some embodiments, the geometric constraints between lane lines include: The spatial distance between the left and right lane lines of the same lane is greater than the minimum lane width threshold and less than the maximum lane width threshold.

[0062] Specifically, the spatial distance between the left and right lane lines of the same lane is the lane width. In real-world road scenarios, while lane widths may vary depending on road grade and design specifications, they typically remain within a relatively fixed range that conforms to engineering standards. Utilizing this strong prior knowledge to constrain the model's predictive behavior can improve the reasonableness of lane detection results.

[0063] The minimum lane width threshold defines the minimum acceptable width of a lane. This value is set based on traffic engineering standards and common sense about driving. For example, even on the narrowest rural roads or temporary construction zones, the lane width needs to ensure safe passage for ordinary vehicles, and therefore is unlikely to be less than a certain limit. In one specific embodiment, the minimum lane width threshold can be set to 2.5 meters. Any lane width less than this should be considered unreasonable or dangerous.

[0064] The maximum lane width threshold defines the maximum acceptable width of a lane. Similarly, this value is derived from road design standards. For example, standard highway or urban arterial road lane widths are typically between 3.5 and 3.75 meters, and even wider ramps or special lanes have an upper limit on their width. In a specific implementation, the maximum lane width threshold can be set to 5.0 meters. Any predicted lanes significantly exceeding this width likely indicate that the model has incorrectly matched non-adjacent lane lines together, or that incorrect associations have occurred in scenarios such as lane splitting.

[0065] The lane detection method provided in this application, by introducing a lane width range for geometric relationship constraints, can suppress abnormal prediction situations such as intersections, excessive narrowness, excessive width, or inconsistent shape drift between left and right lane lines, greatly improving the physical rationality and reliability of lane detection results.

[0066] In some embodiments, the loss function is the geometric relationship constraints between lane lines in the training samples and / or the morphological constraints of each lane line, including: Determine the spatial distance between corresponding points on the left and right lane lines of the same lane as predicted; If the spatial distance between any corresponding points is less than the minimum lane width threshold, a first penalty loss is generated that is proportional to the difference between the spatial distance and the minimum lane width threshold. If the spatial distance between any corresponding points is greater than the maximum lane width threshold, a second penalty loss is generated that is proportional to the difference between the spatial distance and the maximum lane width threshold. A loss function is constructed based on the first penalty loss and the second penalty loss.

[0067] Specifically, taking a single iteration of model training as an example, for an input image, the lane detection model outputs one or more predicted lanes. For the same predicted lane, it includes predicted left and right lane lines. Both lines are ordered along the lane direction. It consists of an ordered set of points.

[0068] Calculate the lane width point by point along the lane lines, which is the spatial distance between corresponding points on the left and right lane lines. For example, from... arrive For each index position, extract the point on the left lane line and the point on the right lane line; these two points form a corresponding pair. Then, calculate the spatial distance between them. . For the first lane on the left The horizontal coordinates of the points For the first lane on the right The horizontal coordinates of each point.

[0069] The spacing between the left and right lane lines in the same lane is greater than the minimum lane width threshold. And less than the maximum lane width threshold , can be represented as: .

[0070] If the spatial distance between any corresponding points is less than the minimum lane width threshold, that is... Then, the first penalty loss needs to be calculated. The first penalty loss is used to penalize cases where the predicted lane width is too narrow.

[0071] In one specific embodiment, the first penalty loss is the difference between the spatial distance and the minimum lane width threshold. It is directly proportional. That is, the greater the deviation of the predicted lane width from the minimum lane width threshold (the predicted lane width is too narrow), the greater the first penalty loss.

[0072] If the spatial distance between any corresponding points is greater than the maximum lane width threshold, that is... Then, a second penalty loss needs to be calculated. The second penalty loss is used to penalize cases where the predicted lane width is too wide.

[0073] In one specific embodiment, the second penalty loss is the difference between the spatial distance and the maximum lane width threshold. It is directly proportional. That is, the greater the deviation of the predicted lane width from the maximum lane width threshold (the predicted lane width is too wide), the greater the second penalty loss.

[0074] Finally, the first penalty loss and the second penalty loss are added as calculation terms to the total loss function.

[0075] The lane detection method provided in this application constrains the spatial distance between the left and right lane lines predicted by the model to meet preset minimum lane width thresholds and maximum lane width thresholds. When the predicted distance between the left and right lane lines exceeds a reasonable range, the corresponding prediction result will be penalized, thereby guiding the model to learn a lane width distribution that conforms to the real road structure.

[0076] In some embodiments, the shape constraints of each lane line include: The point sequence of each lane line maintains second-order smoothness at at least one scale.

[0077] Specifically, in a discrete sequence of points, second-order smoothing can be intuitively understood as the curvature of the point sequence changing gradually. Specifically for lane lines, second-order smoothing represents: (1) Local continuity: the position of any point on the lane line should roughly form a straight line or a smooth arc with its two adjacent points, rather than forming a sharp corner; (2) Overall curvature consistency: when observed over a larger area, the curvature trend of the lane line should be gradual. For example, when entering a curve from a straight line, the curvature gradually increases, rather than abruptly.

[0078] Scale refers to the spacing between points when sampling a point sequence. Smoothing constraints at a single scale may be limited. For example, second-order smoothing constraints at smaller scales are used to restrict local continuity between adjacent points; second-order smoothing constraints at larger scales are used to suppress curvature fluctuations over long distances, thereby effectively reducing overall lane shape jitter.

[0079] Therefore, embodiments of this application can select multiple scales to apply second-order smoothing constraints to the predicted point sequences of the left lane line, right lane line, and center line, while constraining the local continuity and overall curvature consistency of the lane lines, so that the direction of second-order geometric change of the point sequence at different scales is consistent with the actual annotation.

[0080] The lane detection method provided in this application applies second-order smoothing constraints to lane lines at multiple scales, which can effectively solve the problems of jitter, spikes and unnatural curvature in model prediction results, and significantly improve the visual quality and temporal stability of lane line prediction results.

[0081] In some embodiments, the loss function is the geometric relationship constraints between lane lines in the training samples and / or the morphological constraints of each lane line, including: Determine at least one scale; Based on at least one scale, select multiple sets of three points that meet at least one scale requirement from the point sequence of each predicted lane line; The linear deviation of each group is determined based on the coordinates of three points in each group. Sum and normalize the linear deviations of each group to determine the second-order smoothing loss at at least one scale. A loss function is constructed based on weights calculated at at least one scale and second-order smoothing loss at at least one scale.

[0082] Specifically, scale The sampling interval used to calculate linear deviation on a point sequence is defined. Multiple scales constituting a set can be determined. ,For example .

[0083] When the scale takes a small value, for example This corresponds to checking the collinearity of each point with its two directly adjacent points before and after it. It is mainly used to constrain local continuity and suppress local high-frequency noise and sharp inflection points.

[0084] When the scale takes a large value, for example This corresponds to checking the collinearity of each point with its adjacent points at intervals of several points. It is mainly used to constrain the overall curvature consistency and suppress overall shape jitter.

[0085] By combining multiple scales, the micro and macro morphology of lane lines can be constrained simultaneously, thereby achieving comprehensive smooth control.

[0086] For scale Multiple sets of points satisfying the scale are selected from the predicted point sequences of each lane line. The three points required can be represented as { , , } With the center point, and According to the scale Two reference points are determined before and after. To ensure the index doesn't go out of bounds, the center point... index satisfy .

[0087] The linear deviation of each group is determined by the coordinates of three points in each group, and can be expressed as follows: .

[0088] The linear deviations of each group are summed and normalized to determine the second-order smoothing loss at each scale. Based on the calculated weights of each scale and the second-order smoothing loss at each scale, the total second-order smoothing loss can be obtained. This can be expressed as a formula: .

[0089] in, For scale The calculated weights.

[0090] The total second-order smoothing loss can be added to the total loss function as a calculation term.

[0091] The lane detection method provided in this application achieves comprehensive and smooth control over the local details and global trends of lane lines by constraining the shape of the predicted lanes at different scales. This makes the predicted lane lines closer to the real and smooth road markings in shape, thereby effectively suppressing problems such as downstream control system jitter caused by unstable model prediction.

[0092] In some embodiments, the image to be detected is input into a lane detection model to obtain the predicted lane in the image to be detected, as well as the left lane line, right lane line, and center line of the predicted lane, output by the lane detection model, including: Input the image to be detected into the lane detection model to obtain the predicted lane in the image to be detected, as well as the offset of the left lane line, right lane line and center line of the predicted lane. The centerline of the predicted lane is determined based on the offsets of the left lane line, right lane line, and centerline of the predicted lane.

[0093] Specifically, it is not only difficult to allow the lane detection model to independently predict three seemingly unrelated curves (left, right, and center line), but it is also difficult to guarantee the inherent geometric relationship between the three. The embodiments of this application adopt a binding modeling method. Instead of directly predicting the center line, the model calculates and generates the center line by predicting the center line offset and combining it with the predicted left and right lane lines. This ensures a high degree of consistency and geometric rationality of the three lines at the model structure level.

[0094] When the image to be detected is fed into the trained lane detection model, the model first predicts the left and right lane lines, and then further predicts the offset of the center line relative to the center lines of the left and right lane lines. The center line is obtained by averaging the corresponding points of the left and right lane lines and then adding the predicted center line offset, which can be expressed by the formula: .

[0095] in, For the first lane on the left The horizontal coordinates of the points For the first lane on the right The horizontal coordinates of each point; To represent the centerline offset predicted by the model; For the center line The horizontal coordinates of each point.

[0096] The lane detection method provided in this application improves the accuracy of center line prediction and the rationality of the overall lane structure by "predicting the left and right lane lines and binding them to generate the center line".

[0097] In some embodiments, Figure 4 This is a flowchart illustrating the predictive lane post-processing method provided in this application, as shown below. Figure 4 As shown, the method also includes: Step 410: Based on the confidence level of each predicted lane, select multiple candidate predicted lanes; Step 420: Based on the confidence of each candidate predicted lane and the overlap of the planar regions between each candidate predicted lane, non-maximum suppression is performed on multiple candidate predicted lanes to determine the lane detection result in the image to be detected.

[0098] Specifically, during lane detection model inference, in addition to outputting the geometric information of each predicted lane (i.e., left lane line, right lane line, and center line), the lane detection model typically also outputs a confidence score for each predicted lane unit. This confidence score is a value between 0 and 1, indicating how likely the model is that the prediction result is a real lane (prediction score).

[0099] A confidence threshold can be set. All predicted lanes are filtered based on the confidence threshold and the confidence score of each predicted lane. If the confidence score of any predicted lane is greater than or equal to the confidence threshold, it indicates that the predicted lane is likely a real lane and is retained as a candidate predicted lane. If the confidence score of any predicted lane is less than the confidence threshold, it indicates that the predicted lane is likely not a real lane and is discarded.

[0100] After initial screening, the original, potentially massive set of predictions was reduced to a smaller set of candidate lane predictions with higher average quality.

[0101] The lane plane region refers to each candidate predicted lane. Since it includes both left and right lane lines, it can be viewed as defining a strip-shaped planar region. This region can be represented as a polygon, with its vertices formed by the sequence of points from the left and right lane lines. Planar region overlap measures the degree to which two candidate predicted lanes overlap spatially. This overlap can be quantified by calculating the Intersection over Union (IoU) of the lane plane regions defined by the two lanes. The IoU ranges from [0, 1], with a larger value indicating that the two candidate predicted lanes are spatially closer and more likely to describe the same real lane.

[0102] Using lanes as the basic unit, non-maximum suppression can be further applied based on the degree of overlap between different predicted lanes, thereby obtaining the final lane detection result while ensuring the integrity of the lane structure.

[0103] The lane detection method provided in this application uses lanes as the basic unit to filter prediction results. It can filter redundant or repeated lane prediction results while ensuring the integrity of the lane structure, thereby improving the accuracy of lane detection and providing more reliable and safer perception input for autonomous driving systems that do not rely on high-precision maps.

[0104] In some embodiments, based on the confidence level of each candidate predicted lane and the planar region overlap between each candidate predicted lane, non-maximum suppression is performed on multiple candidate predicted lanes to determine the lane detection result in the image to be detected, including: Based on the confidence scores of each candidate predicted lane, the candidate predicted lanes are sorted in descending order to generate a candidate list. In the current iteration, the candidate prediction lane with the highest current confidence is selected from the candidate list, the planar region overlap between the remaining candidate prediction lanes and the candidate prediction lane with the highest current confidence is calculated, the remaining candidate prediction lanes with a planar region overlap exceeding a preset overlap threshold are deleted, and the candidate prediction lane with the highest current confidence is added to the results list. If the candidate list is empty, the candidate predicted lanes in the results list are determined as the lane detection results in the image to be detected.

[0105] Specifically, candidate lanes can be sorted in descending order based on their confidence level to generate a candidate list.

[0106] Nonmaximum suppression is an iterative computation process. You can first initialize an empty list of results.

[0107] In the current iteration, the candidate lane with the highest current confidence is selected from the candidate list. Then, all the remaining candidate lanes in the candidate list are traversed, and the planar region overlap between each remaining candidate lane and the candidate lane with the highest current confidence is calculated.

[0108] If the overlap of the planar regions exceeds a preset overlap threshold, it indicates that the remaining candidate predicted lanes and the candidate predicted lane with the highest confidence are very likely describing the same physical lane. Based on the principle of non-maximum suppression (retaining the highest confidence and suppressing the others), the remaining candidate predicted lanes are removed from the candidate list, and the candidate predicted lane with the highest confidence is added to the results list.

[0109] If the overlap of the planar regions does not exceed the preset overlap threshold, it means that the remaining candidate predicted lanes may describe different, adjacent lanes than the candidate predicted lane with the highest confidence. The remaining candidate predicted lanes are retained in the candidate list.

[0110] Repeat the above iterative process until the candidate list is empty.

[0111] The final results list contains all filtered, non-overlapping predicted lanes with high confidence levels. These predicted lanes represent the lane detection results in the image to be detected.

[0112] The lane detection method provided in this application improves the accuracy of lane detection through non-maximum suppression, providing more reliable and safer perception input for autonomous driving systems that do not rely on high-precision maps.

[0113] The apparatus provided in the embodiments of this application is described below. The apparatus described below can be referred to in correspondence with the method described above.

[0114] Figure 5 This is a structural schematic diagram of the lane detection device provided in this application, as shown below. Figure 5 As shown, the device includes: The detection module 510 is used to input the image to be detected into the lane detection model and obtain the predicted lane, the left lane line, the right lane line and the center line of the predicted lane in the image to be detected output by the lane detection model. The lane detection model is trained using images labeled with the left lane line, right lane line, and center line of the same lane as a combination as training samples, and the loss function is the geometric relationship constraints between lane lines and / or the morphological constraints of each lane line in the training samples.

[0115] The lane detection device provided in this application annotates the left lane line, right lane line, and center line of the same lane in the training samples as a combination, thereby achieving an accurate representation of the overall spatial structure of the lane. Using the geometric relationship constraints between lane lines and / or the morphological constraints of each lane line in the training samples as the loss function, the final lane detection model possesses the ability to perceive the rationality of the lane's geometric structure, enabling it to output lane detection results that are geometrically more consistent and physically more reasonable. This improves the accuracy of lane detection and provides more reliable and safer perception input for autonomous driving systems that do not rely on high-precision maps.

[0116] Figure 6 This is a schematic diagram of the structure of the electronic device provided in this application, such as... Figure 6As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communications bus 640, wherein the processor, communications interface, and memory communicate with each other via the communications bus. The processor can invoke logical commands stored in the memory to execute the methods described in the above embodiments, for example: The image to be detected is input into the lane detection model, and the lane detection model outputs the predicted lane in the image to be detected, as well as the left lane line, right lane line and center line of the predicted lane. The lane detection model is trained using images labeled with the left lane line, right lane line and center line of the same lane as a combination as training samples, and the geometric relationship constraints between lane lines and / or the morphological constraints of each lane line in the training samples as loss functions.

[0117] Furthermore, when the logical commands in the aforementioned memory can be implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several commands to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0118] The processor in the electronic device provided in this application embodiment can call logical instructions in the memory to implement the above method. Its specific implementation method is the same as the aforementioned method implementation method and can achieve the same beneficial effect, which will not be repeated here.

[0119] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the methods provided in the above embodiments.

[0120] The specific implementation method is the same as the aforementioned method implementation method and can achieve the same beneficial effects, so it will not be repeated here.

[0121] This application provides a computer program product, including a computer program that, when executed by a processor, implements the method described above.

[0122] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0123] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A lane detection method characterized by, include: The image to be detected is input into the lane detection model, and the lane detection model outputs the predicted lane in the image to be detected, as well as the left lane line, right lane line and center line of the predicted lane; The lane detection model is trained using images labeled with the left lane line, right lane line, and center line of the same lane as a combination as training samples, and using the geometric relationship constraints between lane lines and / or the morphological constraints of lane lines in the training samples as loss functions.

2. The lane detection method according to claim 1, characterized by, The lane detection model is trained based on the following steps: The left lane line, right lane line, and center line of the same lane in each image are labeled as a combination to generate training samples; Based on the geometric relationship constraints between lane lines and / or the morphological constraints of lane lines in the training samples, a loss function is constructed. The initial model is trained based on the training samples and the loss function to obtain the lane detection model.

3. The lane detection method according to claim 1, characterized by, The geometric constraints between lane lines include: The spatial distance between the left and right lane lines of the same lane is greater than the minimum lane width threshold and less than the maximum lane width threshold.

4. The lane detection method according to claim 3, characterized by, The loss function, which uses the geometric relationship constraints between lane lines and / or the morphological constraints of lane lines in the training samples, includes: Determine the spatial distance between corresponding points on the left and right lane lines of the same lane as predicted; If the spatial distance between any corresponding points is less than the minimum lane width threshold, a first penalty loss proportional to the difference between the spatial distance and the minimum lane width threshold is generated. If the spatial distance between any corresponding points is greater than the maximum lane width threshold, a second penalty loss proportional to the difference between the spatial distance and the maximum lane width threshold is generated. The loss function is constructed based on the first penalty loss and the second penalty loss.

5. The lane detection method according to claim 1, characterized by, The shape constraints of each lane line include: The point sequence of each lane line maintains second-order smoothness at at least one scale.

6. The lane detection method according to claim 5, characterized by, The loss function, which uses the geometric relationship constraints between lane lines and / or the morphological constraints of lane lines in the training samples, includes: Determine at least one scale; Based on the at least one scale, select multiple sets of three points that meet the requirements of the at least one scale from the point sequence of each predicted lane line; The linear deviation of each group is determined based on the coordinates of three points in each group. The linear deviations of each group are summed and normalized to determine the second-order smoothing loss of at least one scale. The loss function is constructed based on the calculated weights of the at least one scale and the second-order smoothing loss of the at least one scale.

7. The lane detection method according to claim 1, characterized by, The step of inputting the image to be detected into the lane detection model and obtaining the predicted lane in the image to be detected, as well as the left lane line, right lane line, and center line of the predicted lane, as output by the lane detection model, includes: The image to be detected is input into the lane detection model, and the predicted lane in the image to be detected and the offset of the left lane line, right lane line and center line of the predicted lane are obtained from the output of the lane detection model. The centerline of the predicted lane is determined based on the offsets of the left lane line, right lane line, and centerline of the predicted lane.

8. The lane detection method of claim 1, wherein, The method further includes: Based on the confidence level of each predicted lane, multiple candidate predicted lanes are selected. Based on the confidence level of each candidate predicted lane and the planar region overlap between each candidate predicted lane, non-maximum suppression is performed on the multiple candidate predicted lanes to determine the lane detection result in the image to be detected.

9. The lane detection method according to claim 8, characterized by, The process of performing non-maximum suppression on the multiple candidate predicted lanes based on the confidence level of each candidate predicted lane and the planar region overlap between each candidate predicted lane, and determining the lane detection result in the image to be detected, includes: Based on the confidence scores of each candidate predicted lane, the candidate predicted lanes are sorted in descending order to generate a candidate list. During the current iteration, the candidate predicted lane with the highest current confidence is selected from the candidate list, the planar region overlap between the remaining candidate predicted lanes and the candidate predicted lane with the highest current confidence is calculated, the remaining candidate predicted lanes with a planar region overlap exceeding a preset overlap threshold are deleted, and the candidate predicted lane with the highest current confidence is added to the result list. If the candidate list is empty, the candidate predicted lanes in the result list are determined as the lane detection results in the image to be detected.

10. A lane detection device characterized by comprising: include: The detection module is used to input the image to be detected into the lane detection model and obtain the predicted lane in the image to be detected, as well as the left lane line, right lane line and center line of the predicted lane, output by the lane detection model. The lane detection model is trained using images labeled with the left lane line, right lane line, and center line of the same lane as a combination as training samples, and using the geometric relationship constraints between lane lines and / or the morphological constraints of lane lines in the training samples as loss functions.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the lane detection method according to any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the lane detection method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the lane detection method according to any one of claims 1 to 9.

Images