A lane line labeling method, electronic device and medium

Abstract

This invention discloses a lane line annotation method. First, it acquires the point and surface results of lane line detection separately. Then, it fuses the point and surface results using a multi-model approach to obtain a fused point set. Next, it tracks the fused point set and selects consecutive frames for tracking and fusion to obtain a merged point set. Finally, it performs fitting based on the merged point set and uses the fitting result as the lane line annotation result. This method automates lane line annotation and significantly improves its accuracy. Furthermore, it can quickly and accurately generate a large amount of sample data, providing a training set for lane line detection models and increasing the efficiency of developing new lane line detection models.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a lane marking method, electronic device, and medium. Background Technology

[0002] In recent years, with the increasing application of artificial intelligence technology in the driving field, research on autonomous driving technology and Advanced Driver Assistance Systems (ADAS) has received significant attention from both academia and industry. One of the most challenging tasks in autonomous driving systems is traffic scene understanding, which involves identifying and understanding the environment surrounding an object through various sensors and control modules. For safety reasons, to avoid harm to humans, autonomous vehicles should be aware of their surroundings, including the positions of other vehicles and their own. The more predictable the vehicle's movement, the safer it is; therefore, understanding the accurate conditions of each lane is crucial for autonomous vehicles. Lane detection technology can distinguish and locate markings on the road and pinpoint the exact shape of each lane in a traffic scene. It is essential in intelligent vehicle vision navigation systems and is a core task in modern assisted and autonomous driving systems. Lane detection supports decisions such as lane keeping, centering, and lane changing, as well as functions such as lane departure warnings. It plays a vital role in applications such as real-time vehicle positioning, route planning, lane keeping assist, and adaptive cruise control, helping intelligent vehicles better locate themselves and drive more safely.

[0003] Since lane markings are typically represented as sets of line segments without thickness, traditional lane marking methods require complex post-processing after performing pixel-level dense prediction. To address this deficiency, deep learning algorithms are currently widely used for lane detection. However, deep learning-based lane detection often requires a large amount of lane line sample data for training, and the size of the training data directly determines the model's detection performance. Existing open-source lane line datasets are usually insufficient to meet all real-world scenario requirements. Therefore, in practical applications, it is often necessary to generate and label training datasets before providing them to the model for learning. Manually labeling sample data is time-consuming and labor-intensive, and even interactive labeling based on deep learning requires human intervention. This not only affects algorithm efficiency but also significantly increases the cost of developing the algorithm. Summary of the Invention

[0004] To address some or all of the problems in the existing technology, the first aspect of this invention provides a lane line annotation method that employs a multi-model fusion strategy and incorporates tracking time-series information to automatically annotate lane lines based on an existing deep learning lane line detection model. The lane line annotation method includes:

[0005] Obtain the point and surface results of lane line detection respectively;

[0006] The point and surface results are fused using a multi-model approach to obtain a fused point set.

[0007] The fused point set is tracked, and consecutive frame results are selected for tracking and fusion to obtain the merged point set; and

[0008] The merged point set is fitted, and the fitting result is used as the lane line labeling result.

[0009] Furthermore, the point results are obtained through detection using a Cross Layer Refinement Network (CLRNet) model.

[0010] Furthermore, the surface result is obtained through a panoramic perception system (You Only Look Once for Panoptic Driving Perception, YOLOP).

[0011] Furthermore, the fusion of the point and surface results includes position fusion and orientation fusion.

[0012] Furthermore, the location fusion includes:

[0013] The background in the vanishing point cropping result and the surface result based on the lane line vanishing point; and

[0014] The positions of the points in each point set of the point result are compared with the binary map of the corresponding lane lines in the surface result. If the number of points in the point set that overlap with the binary map exceeds a threshold, the point set is retained; otherwise, the point set is deleted.

[0015] Furthermore, the threshold is one-half of the total number of points in the point set.

[0016] Furthermore, the directional fusion includes:

[0017] The connected regions of all binary map lane lines in the surface result are refined into lane lines with a width of one pixel to obtain instantiated lane lines;

[0018] The instantiated lane lines are uniformly broken to obtain a discrete set of lane points;

[0019] The curvature of each lane line in the discrete lane point set and the point result is calculated and compared. If the difference between the two results is less than a threshold, the corresponding lane line is retained; otherwise, the corresponding lane line is deleted.

[0020] Furthermore, the curvature calculation of the lane lines includes:

[0021] Calculate the curvature between adjacent points on the lane line and sum them up to obtain the curvature of the lane line.

[0022] Furthermore, the merged point set is obtained according to the following steps:

[0023] The fusion point set was tracked using a Kalman filter; and

[0024] The tracking results are fused with the detection results of the next frame, including:

[0025] On the same lane line, calculate the distance between points in the tracking result and points in the detection result. If the distance is less than a preset value, merge the points based on the detection result; otherwise, discard the corresponding points.

[0026] In N consecutive frames, if M or more tracking or detection results include a uniform lane line, then the lane line is filled into the other results in the N consecutive frames where the lane line was not detected, according to the tracking algorithm, where M > N / 2; and

[0027] If the number of lane lines included in consecutive K frames is different, the K-frame result is discarded. If the number of lane lines included in the K frames is the same, the middle frame is retained as the merged result, where K... <N。

[0028] Furthermore, a polynomial fitting is employed to fit the merged point set.

[0029] Based on the lane marking method described above, a second aspect of the present invention provides an electronic device for the lane marking method, comprising a memory and a processor, wherein the memory is configured to store a computer program that executes the lane marking method described above when the processor is running.

[0030] A third aspect of the present invention also provides a computer-readable storage medium for computing resource allocation, which stores a computer program that, when run on a processor, executes the lane marking method as described above.

[0031] This invention provides a lane line labeling method that, based on existing deep learning lane line detection models, employs a multi-model fusion strategy and incorporates tracking time-series information to label lane lines. This method automates lane line labeling and significantly improves the accuracy of lane line labeling. Consequently, it can quickly and accurately generate a large amount of sample data to provide a training set for lane line detection models, thereby increasing the efficiency of developing new lane line detection models. Attached Figure Description

[0032] To further illustrate the above and other advantages and features of the various embodiments of the present invention, a more specific description of the various embodiments of the present invention will be presented with reference to the accompanying drawings. It is to be understood that these drawings depict only typical embodiments of the invention and are therefore not intended to limit its scope. In the drawings, identical or corresponding parts will be indicated by identical or similar reference numerals for clarity.

[0033] Figure 1 The diagram shows a flowchart of a lane marking method according to an embodiment of the present invention. Detailed Implementation

[0034] In the following description, the invention is described with reference to various embodiments. However, those skilled in the art will recognize that the embodiments may be practiced without one or more specific details or in conjunction with other alternatives and / or additional methods or components. In other instances, well-known structures or operations are not shown or described in detail so as not to obscure the inventive points of the invention. Similarly, for illustrative purposes, specific numbers and configurations are set forth to provide a comprehensive understanding of embodiments of the invention. However, the invention is not limited to these specific details.

[0035] In this specification, references to "an embodiment" or "this embodiment" mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of the invention. The phrase "in one embodiment" appearing throughout this specification does not necessarily refer to the same embodiment in all instances.

[0036] It should be noted that the embodiments of the present invention describe the method steps in a specific order; however, this is only for illustrating the specific embodiment and not for limiting the order of the steps. On the contrary, in different embodiments of the present invention, the order of the steps can be adjusted according to actual needs.

[0037] To quickly and accurately label lane lines and obtain a large amount of sample data for training lane line detection models, thereby improving the model's detection efficiency and accuracy, this invention provides a lane line labeling method. This method employs a multi-model fusion strategy combined with tracking temporal information, utilizing existing deep learning lane line detection models to more accurately label lane lines. Due to its automation and high accuracy, it can provide a large amount of sample data, thus increasing the efficiency of developing new lane line detection models.

[0038] The present invention will be further described below with reference to the accompanying drawings of the embodiments.

[0039] Figure 1 The diagram shows a flowchart of a lane marking method according to an embodiment of the present invention.

[0040] like Figure 1As shown, a lane marking method includes:

[0041] First, in step 111, lane line point results are obtained. The lane line point results are detected using a first lane line detection model. In one embodiment of the invention, the point results are obtained through a Cross Layer Refinement Network (CLRNet) model. Specifically, the CLRNet first uses high-level semantic features to detect lane lines and perform coarse localization, then performs precise localization of lane lines based on low-level semantic features, and collects more global semantic information through ROIGather to establish the relationship between ROI lane features and the entire feature map, thereby resolving some visually invisible lane lines. It should be understood that in other embodiments of the invention, lane line detection modules with other common structures can also be used to detect lane line point results.

[0042] Simultaneously, in step 112, lane line and surface results are obtained. A second lane line detection model is used to detect the lane line and surface results. In one embodiment of the invention, the surface results are obtained through a panoramic perception system (You Only Look Once for Panoptic Driving Perception, YOLOP). The YOLOP panoramic perception system uses a semantic segmentation algorithm to achieve lane line detection. It consists of an encoder for feature extraction and three decoders for specific tasks, capable of simultaneously performing traffic target detection, drivable area segmentation, and lane detection. The encoder consists of a backbone network and a fusion module (Neck). The backbone network extracts features from the input image; for example, a classic image classification network can be used as the backbone network. The Neck fuses the features generated by the backbone network; for example, it may include a Spatial Pyramid Pooling (SPP) module and a Feature Pyramid Network (FPN) module. The SPP generates and fuses features at different scales, while the FPN fuses features at different semantic levels, thereby generating information containing multiple scales and semantic levels. The decoders correspond to the three tasks described above. If the FPN layer of size (W / 8, H / 8, 256) is fed to the corresponding lane segmentation decoder, and the output feature map is restored to its initial size (W, H, 2) through the above processing, it can represent the probability that each pixel in the input image is a lane line or the background, thereby realizing lane line detection. It should be understood that in other embodiments of the present invention, lane line detection modules with other common structures can also be used to detect the surface results of lane lines.

[0043] Next, in step 102, multi-model fusion is performed. The point results and surface results are fused using multi-model fusion to obtain a fused point set. Lane line detection results, such as those from the CLRNet model, are output as point sets, a commonly used lane line label format. However, while the CLRNet model has higher overall detection accuracy than models like the YOLOP model, its recall is lower. Therefore, in one embodiment of this invention, point results are prioritized and surface results are used as supplementary methods during multi-model fusion. Specifically, in one embodiment of this invention, the multi-model fusion includes two levels: position fusion and orientation fusion. In one embodiment of this invention, the position fusion includes:

[0044] First, based on the vanishing point of the lane lines, redundant background is removed from the point and surface results; and...

[0045] Next, the positions of the points in each point set of the point results are compared with the binary images of the corresponding lane lines in the surface results. If the number of points in the point set that overlap with the binary image exceeds a threshold, the point set is retained; otherwise, the point set is deleted. In one embodiment of the present invention, the threshold can be set to, for example, half of the total number of points in the corresponding point set. That is, when more than half of the points in a point set overlap with the surface detection result, the point set for lane line detection is retained. Furthermore, in one embodiment of the present invention, before the comparison, the number of lane lines in the point results and the surface results are determined separately. Specifically, the number of lane lines in the surface results is determined based on the instantiation of the semantic segmentation result. The instantiation of the semantic segmentation result is obtained by refining the connected regions of all lane lines in the binary image of the surface results into lane lines with a width of one pixel. The number of lane lines in the point results is determined by the number of point sets, where one point set represents one lane line.

[0046] In one embodiment of the present invention, the orientation fusion is performed based on the result of position fusion and includes:

[0047] First, the connected regions of all binary map lane lines in the surface result are refined into lane lines with a width of one pixel to obtain instantiated lane lines;

[0048] Next, the instantiated lane lines are uniformly broken to obtain a discrete set of lane points. This discrete set of lane points, like the point result, is uniformly distributed along the vertical direction of the image.

[0049] Finally, calculate and compare the curvatures of each lane line in the discrete lane point set and the point result respectively. If the difference between the two results is less than the threshold, retain the corresponding lane line; otherwise, delete the corresponding lane line. In one embodiment of the present invention, the curvature of the lane line is obtained by accumulating the curvatures between each adjacent point on the lane line. In one embodiment of the present invention, the curvature of the corresponding lane line can also be obtained by only calculating the curvature based on the overlapping or adjacent points between the discrete lane point set and the point result and then accumulating it;

[0050] Next, in step 103, tracking and fusion. Track the fused point set, and select consecutive frame results for tracking and fusion to obtain a merged point set. In one embodiment of the present invention, after multi-model fusion, use Kalman filtering to track the fused point set obtained in step 102, select consecutive frame results for tracking and fusion, and fuse the tracked result with the result detected in the next frame. In one embodiment of the present invention, the tracking and fusion is achieved through the following fusion strategy:

[0051] First, merging of points on the lane line. Specifically, on the same lane line, calculate the distance between the points in the tracking result and the points in the detection result. If the distance is less than the preset value, merge based on the detection result; otherwise, discard the corresponding points;

[0052] Second, merging of lane lines. Specifically, in consecutive N frames, if M frames or more of the tracking or detection results include the same lane line, fill the lane line into the results of the other frames in the consecutive N frames where the lane line is not detected according to the tracking algorithm, where M > N / 2. For example, the value of N can be 5 and the value of M can be 3; and

[0053] Third, screening based on the number of lane lines. Specifically, in consecutive adjacent K frames, if the number of lane lines included is different, discard the results of the K frames. If the number of lane lines included in the K frames is the same, retain the middle frame as the merged result, where

[0054] K < N. For example, the value of K can be 3; and

[0055] Finally, in step 104, fitting and annotation. Fit based on the merged point set, and use the fitting result as the annotation result of the lane line. In one embodiment of the present invention, based on the merged point set, polynomial fitting is used to fit the points on the lane line to make it smoother. The final fitting result is saved in a txt format text as the annotation result of the lane line. Each line contains the point set data of each lane line, where (x, y) are the coordinates of the points, and each point is separated by a space.

[0056] Based on the lane marking method described above, the present invention also provides an electronic device for the lane marking method, which includes a memory and a processor, wherein the memory is configured to store a computer program that executes the lane marking method described above when the processor is running.

[0057] The present invention also provides a computer-readable storage medium for computing resource allocation, which stores a computer program that, when run on a processor, executes the lane marking method as described above.

[0058] Although various embodiments of the invention have been described above, it should be understood that they are presented by way of example only and not as limitations. It will be apparent to those skilled in the art that various combinations, modifications, and alterations can be made without departing from the spirit and scope of the invention. Therefore, the breadth and scope of the invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should be defined solely by the appended claims and their equivalents.

Claims

1. A method for marking lane lines, characterized in that, Including the following steps: Obtain the point and surface results of lane line detection respectively; The point and surface results are fused using a multi-model approach to obtain a fused point set. The position fusion includes cropping the background from the point and surface results based on the vanishing points of the lane lines, and comparing the positions of the points in each point set of the point results with the binary images of the corresponding lane lines in the surface results. If the number of points in the point set that overlap with the binary image exceeds a threshold, the point set is retained; otherwise, the point set is deleted. The direction fusion includes refining the connected regions of all lane lines in the binary image of the surface results into lane lines of single-pixel width to obtain instantiated lane lines, uniformly breaking the instantiated lane lines to obtain discrete lane point sets, calculating the curvature between each adjacent point on each lane line in the discrete lane point set and the point results, and accumulating the curvature to obtain the curvature of each lane line in the discrete lane point set and the point results. The curvature of the discrete lane point set and the point results is compared. If the difference between the two results is less than a threshold, the corresponding lane line is retained; otherwise, the corresponding lane line is deleted. The fusion point set is tracked using Kalman filtering. On the same lane line, the distance between points in the tracking result and points in the detection result is calculated. If the distance is less than a preset value, the detection result is used as the basis for merging; otherwise, the corresponding point is discarded. In N consecutive frames, if M or more tracking or detection results include the same lane line, the lane line is filled into the results of other undetected lane lines in the N consecutive frames according to the tracking algorithm. In K consecutive adjacent frames, if the number of lane lines included is different, the K frame results are discarded. If the number of lane lines included in the K frames is the same, the middle frame is retained as the merging result.<N,M> N / 2, yielding the merged point set; and The merged point set is fitted, and the fitting result is used as the lane line labeling result.

2. The lane marking method as described in claim 1, characterized in that, The point results are obtained through detection using a cross-layer refinement network model; and / or The surface results are obtained through a panoramic perception system.

3. The lane marking method as described in claim 1, characterized in that, The threshold is one-half of the total number of points in the point set.

4. The lane marking method as described in claim 1, characterized in that, Polynomial fitting is used to fit the merged point set.

5. An electronic device for lane line detection, characterized in that, The system includes a memory and a processor, wherein the memory is configured to store a computer program that executes the lane marking method as described in any one of claims 1 to 4 when the processor is running.

6. A computer-readable storage medium for computing resource allocation, characterized in that, The device contains a computer program that, when run on a processor, executes the lane marking method as described in any one of claims 1 to 4.

Images