Lane marking methods, devices, electronic equipment and storage media

Abstract

This disclosure relates to a lane line annotation method, apparatus, electronic device, and storage medium. It involves acquiring multiple frames of images and the pose data of the acquisition device at the time of acquisition for each frame. Each frame is then inspected to obtain lane line points. For each lane line point, at least two sets of lane line point cloud data are obtained based on the lane line points in adjacent frames. These at least two sets of point cloud data are then fused to obtain a fused point cloud of lane lines for each frame. The annotated lane lines in each frame are then obtained based on this fused point cloud. Compared to existing technologies, this disclosure, by fusing at least two sets of point clouds of lane lines in each frame to obtain a fused point cloud of lane lines for each frame, can fill in missed lane lines and discard falsely detected lane lines, improving lane line detection accuracy and thus enhancing lane line annotation performance.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a lane marking method, apparatus, electronic device, and storage medium. Background Technology

[0002] Lane detection plays a crucial role in autonomous driving and driver assistance systems. Training a good lane detection network often requires a large amount of labeled data. Even with a sufficient amount of labeled data, detection models trained on previously labeled data may encounter new problems when faced with new scenarios, thus requiring human annotation of new images. However, manual annotation is often inefficient, time-consuming, and prone to errors.

[0003] Therefore, related technologies often utilize high-precision maps to identify marked lane lines and then reproject the lane line markings onto the image using sensor location information, thus obtaining the lane line annotations. This method leverages the high positioning accuracy and rich lane line annotation information of maps, providing annotation information for training lane line detection models and eliminating the cost of manual annotation.

[0004] However, in scenarios where there are no high-precision maps, it is time-consuming and laborious to re-collect data and create high-precision maps. The equipment cost for data collection and the production cost of high-precision maps are high. Moreover, existing annotation methods have the problem that the annotation results reprojected back to the image cannot be aligned with the lane lines, resulting in poor lane line annotation effects. Summary of the Invention

[0005] To address the aforementioned technical problems, this disclosure provides a lane marking method, apparatus, electronic device, and storage medium to reduce lane marking costs and improve lane marking effectiveness.

[0006] In a first aspect, embodiments of this disclosure provide a lane marking method, the method comprising:

[0007] Acquire multiple frames of images and the pose data of the acquisition device when each frame of the multiple frames of images is acquired;

[0008] Each frame of the multi-frame image is detected to obtain the lane line points of each frame;

[0009] For each frame of the image, based on the lane line points of its adjacent frame images, at least two sets of lane line point cloud data of the frame image are obtained, and the at least two sets of point cloud data are fused to obtain the fused point cloud of the lane lines of each frame image.

[0010] The lane lines marked in each frame of the image are obtained by fusing the point cloud of the lane lines in each frame of the image.

[0011] In some embodiments, the pose data of the acquisition device includes at least the position, pitch angle, yaw angle, and roll angle of the acquisition device.

[0012] In some embodiments, detecting each frame of the multi-frame images to obtain lane line points for each frame includes:

[0013] Each frame of the multi-frame image is input into the lane line detection model, and the lane line points of each frame are obtained through the lane line detection model.

[0014] In some embodiments, the lane detection model is trained through the following steps:

[0015] Acquire multiple sample images and the actual lane line points in each sample image;

[0016] The initial lane detection model is trained based on the multi-frame sample images to obtain the lane line point detection results for each frame sample image.

[0017] Based on the actual lane line points in each sample image, the loss calculation is performed on the lane line point detection results of each sample image to obtain the loss calculation result of the initial lane line detection model.

[0018] Based on the loss calculation results of the initial lane detection model, update the model parameters of the initial lane detection model;

[0019] If the accuracy of the updated lane detection model is greater than a preset threshold, then the lane detection model training is complete.

[0020] In some embodiments, for each frame of image, obtaining at least two sets of lane line point cloud data for that frame based on lane line points in adjacent frames, and fusing the at least two sets of point cloud data to obtain a fused point cloud of lane lines for each frame, includes:

[0021] Based on the multiple frames of images and the pose data of the acquisition device when each frame of image is acquired, the external parameter change data of the vehicle is obtained.

[0022] For each frame of the image, at least two sets of point cloud data are obtained based on the vehicle's extrinsic parameter change data and the lane line points of each frame of the image. The at least two sets of point cloud data are then fused to obtain the fused point cloud of the lane lines in each frame of the image.

[0023] In some embodiments, for each frame of an image, at least two sets of point cloud data are obtained based on the vehicle's extrinsic parameter variation data and the lane line points of each frame of the image. The at least two sets of point cloud data are then fused to obtain a fused point cloud of the lane lines for each frame of the image, including:

[0024] The lane line points of each frame image are projected onto the bird's-eye view to obtain the first point cloud of the lane lines of each frame image;

[0025] For the multi-frame image, the next frame image is projected onto the bird's-eye view of the previous frame image based on the vehicle's external parameter change data to obtain the second point cloud of the lane line in each frame image.

[0026] Based on the first point cloud of lane lines and the second point cloud of lane lines in each frame of the image, the lane lines of each frame of the image are fused to obtain the first fused point cloud of lane lines in each frame of the image.

[0027] For the multi-frame images, the previous frame image is projected onto the bird's-eye view of the next frame image based on the vehicle's external parameter change data to obtain the lane line third point cloud of each frame image.

[0028] Based on the first point cloud of lane lines and the third point cloud of lane lines in each frame of the image, the lane lines of each frame of the image are fused to obtain the second fused point cloud of lane lines in each frame of the image.

[0029] The lane lines of each frame are fused based on the first fused point cloud and the second fused point cloud of the lane lines of each frame to obtain the fused point cloud of the lane lines of each frame.

[0030] In some embodiments, obtaining the lane lines marked in each frame image by fusing the point cloud of lane lines in each frame image includes:

[0031] Project the fused point cloud of lane lines from each frame onto each frame of the image;

[0032] The fused point cloud of lane lines in each frame of the image is sampled to obtain discrete points in each frame of the image;

[0033] The discrete points in each frame of the image are filtered to obtain the lane lines marked in each frame of the image.

[0034] Secondly, embodiments of this disclosure provide a lane marking device, comprising:

[0035] The acquisition module is used to acquire multiple frames of images and the pose data of the acquisition device when each frame of the multiple frames of images is acquired;

[0036] The detection module is used to detect each frame of the multi-frame images to obtain the lane line points of each frame image;

[0037] The fusion module is used to obtain at least two sets of lane line point cloud data for each frame of the image based on the lane line points of its adjacent frame images, and to fuse the at least two sets of point cloud data to obtain the fused point cloud of lane lines for each frame of the image.

[0038] The annotation module is used to obtain the annotated lane lines in each frame of the image based on the fused point cloud of the lane lines in each frame of the image.

[0039] Thirdly, embodiments of this disclosure provide an electronic device, including:

[0040] Memory;

[0041] Processor; and

[0042] Computer programs;

[0043] The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in the first aspect.

[0044] Fourthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method as described in the first aspect.

[0045] Fifthly, embodiments of this disclosure also provide a computer program product comprising a computer program or instructions that, when executed by a processor, implement the method described in the first aspect.

[0046] The lane line annotation method, apparatus, electronic device, and storage medium provided in this disclosure acquire multiple frames of images and the pose data of the acquisition device when each frame of the multiple frames is acquired. Each frame of the multiple frames is detected to obtain lane line points. For each lane line point in each frame, at least two sets of lane line point cloud data are obtained based on the lane line points in adjacent frames. The at least two sets of point cloud data are fused to obtain a fused point cloud of lane lines in each frame. The annotated lane lines in each frame are obtained based on the fused point cloud of lane lines in each frame. Compared to existing technologies, this disclosure, by fusing at least two sets of point clouds of lane lines in each frame to obtain a fused point cloud of lane lines in each frame, can fill in missed lane lines and discard falsely detected lane lines, improving lane line detection accuracy. It does not require high-precision maps, reducing lane line annotation costs. Furthermore, the lane lines in each frame are annotated based on the fused point cloud of lane lines in each frame to obtain annotation results, improving the lane line annotation effect. Attached Figure Description

[0047] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0048] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart of the lane marking method provided in the embodiments of this disclosure;

[0050] Figure 2 A flowchart of a lane marking method provided in another embodiment of this disclosure;

[0051] Figure 3 A flowchart of a lane marking method provided in another embodiment of this disclosure;

[0052] Figure 4 This is a schematic diagram of the lane marking device provided in an embodiment of the present disclosure;

[0053] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure;

[0054] Figure 6 This is a schematic diagram of lane line point cloud provided in an embodiment of the present disclosure;

[0055] Figure 7This is a schematic diagram of the lane line point cloud fusion process provided in an embodiment of this disclosure. Detailed Implementation

[0056] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0057] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0058] Lane detection plays a crucial role in autonomous driving and driver assistance systems. Training a good lane detection network often requires a large amount of labeled data. Even with a sufficient amount of labeled data, detection models trained on previously labeled data may encounter new problems when faced with new scenarios, thus requiring human annotation of new images. However, manual annotation is often inefficient, time-consuming, and prone to errors.

[0059] Therefore, related technologies often utilize high-precision maps to identify marked lane lines and then reproject the lane line markings onto the image using sensor location information, thus obtaining the lane line annotations. This method leverages the high positioning accuracy and rich lane line annotation information of maps, providing annotation information for training lane line detection models and eliminating the cost of manual annotation.

[0060] However, in scenarios where there are no high-precision maps, it is time-consuming and laborious to re-collect data and create high-precision maps. The equipment cost for data collection and the production cost of high-precision maps are high. Moreover, existing annotation methods have the problem that the annotation results reprojected back to the image cannot be aligned with the lane lines, resulting in poor lane line annotation effects.

[0061] To address this issue, this disclosure provides a lane marking method, which will be described below with reference to specific embodiments.

[0062] Figure 1 This is a flowchart illustrating a lane marking method provided in an embodiment of this disclosure. This method can be applied to electronic devices and to scenarios involving lane marking, reducing lane marking costs and improving marking effectiveness. It is understood that the lane marking method provided in this disclosure can also be applied to other scenarios.

[0063] The following is about Figure 1The lane marking method shown here will be introduced. This method includes the following steps:

[0064] S101. Acquire multiple frames of images and the pose data of the acquisition device when each frame of the multiple frames of images is acquired.

[0065] For example, an electronic device acquires multiple frames of images, as well as the pose data of the acquisition device when each frame of the multiple frames is acquired. The multiple frames of images are acquired by the acquisition device. In some embodiments, the pose data of the acquisition device includes at least the position, pitch angle, yaw angle, and roll angle of the acquisition device, and the position of the acquisition device can be three-dimensional coordinates.

[0066] S102. Detect each frame of the multi-frame image to obtain the lane line points of each frame.

[0067] After acquiring multiple frames of images, the electronic device detects each frame to obtain lane line points for each frame. For example, by building an offline high-precision lane line detection network, the position of the lane lines in each image is inferred, and the valid lane line point clouds in the images are selected.

[0068] S103. For the lane line points of each frame image, based on the lane line points of its adjacent frame images, obtain at least two sets of lane line point cloud data for that frame image, and fuse the at least two sets of point cloud data to obtain the fused point cloud of the lane lines of each frame image.

[0069] For each frame of an image, the electronic device obtains at least two sets of lane line point cloud data for that frame based on the lane line points of its adjacent frames. These at least two sets of point cloud data are then fused to obtain a fused point cloud of lane lines for each frame. For example, at least one set of point cloud data is obtained based on the pose data of the acquisition device when each frame is captured, and another set of adjacent frame point cloud data is obtained based on the lane line points of each frame. These at least two sets of point cloud data are then fused to obtain the fused point cloud of lane lines for each frame. In other words, fusing a set of original point cloud data obtained from the lane line points of each frame with at least one set of transformed point cloud data from adjacent frames obtained based on the pose data of the acquisition device when each frame is captured to obtain the fused point cloud of lane lines for each frame can fill in missed lane lines and discard falsely detected lane lines, thus improving the accuracy of lane line detection.

[0070] S104. Obtain the lane lines marked in each frame of the image by fusing the point cloud of the lane lines in each frame of the image.

[0071] After obtaining the fused point cloud of lane lines for each frame of the image, the electronic device obtains the labeled lane lines in each frame of the image based on the fused point cloud of lane lines. The labeled lane lines include some discrete points that constitute the lane lines, as well as the type of lane lines. The lane line type can be a single dashed line, a single solid line, double solid lines, double dashed lines, etc.

[0072] This disclosure, through acquiring multiple frames of images and the pose data of the acquisition device at the time of acquisition of each frame, detects lane line points in each frame to obtain lane line points. For each lane line point, based on the lane line points in adjacent frames, at least two sets of lane line point cloud data are obtained. These at least two sets of point cloud data are fused to obtain a fused point cloud of lane lines in each frame. The lane lines labeled in each frame are then obtained based on this fused point cloud. Compared to existing technologies, this disclosure, by fusing at least two sets of point clouds of lane lines in each frame to obtain a fused point cloud of lane lines, can fill in missed lane lines and discard falsely detected lane lines, improving lane line detection accuracy. It does not require high-precision maps, reducing lane line labeling costs. Furthermore, the lane lines in each frame are labeled based on the fused point cloud of lane lines to obtain labeling results, improving lane line labeling effectiveness.

[0073] Figure 2 A flowchart of a lane marking method provided in another embodiment of this disclosure is shown below. Figure 2 As shown, the method includes the following steps:

[0074] S201. Acquire multiple frames of images and the pose data of the acquisition device when each frame of the multiple frames of images is acquired.

[0075] Specifically, the implementation process and principle of S201 and S101 are the same, and will not be repeated here.

[0076] S202. Input each frame of the multi-frame images into the lane line detection model, and obtain the lane line points of each frame image through the lane line detection model.

[0077] In this step, each frame of the multi-frame image is sequentially input into the lane line detection model, and the lane line points of each frame of the image are obtained through the lane line detection model.

[0078] In some embodiments, the lane detection model in S202 is trained using S601, S602, S603, S604, and S605:

[0079] S601. Obtain multiple sample images and the actual lane line points in each sample image;

[0080] S602. The initial lane line detection model is trained based on the multi-frame sample images to obtain the lane line point detection results for each frame sample image.

[0081] S603. Based on the real lane line points in each frame of the sample image, perform loss calculation on the lane line point detection results of each frame of the sample image to obtain the loss calculation result of the initial lane line detection model.

[0082] S604. Based on the loss calculation results of the initial lane detection model, update the model parameters of the initial lane detection model;

[0083] By continuously updating the model parameters of the initial lane detection model based on the loss calculation results, the accuracy of the lane detection model can be improved.

[0084] S605. If the accuracy of the updated lane detection model is greater than a preset threshold, then the lane detection model training is complete.

[0085] If the accuracy of the updated lane detection model is greater than a preset threshold, then the lane detection model has been trained and can be used for lane point cloud detection.

[0086] S203. Based on the multiple frames of images and the pose data of the acquisition device when each frame of image is acquired, obtain the external parameter change data of the vehicle.

[0087] Since the acquisition device is installed on the vehicle, the position and angle relationship between the vehicle and the acquisition device are fixed. The electronic device can obtain the external parameter change data of the vehicle based on the multiple frames of images and the pose data of the acquisition device when each frame of image is acquired.

[0088] S204. For lane line points in each frame of an image, at least two sets of point cloud data are obtained based on the external parameter change data of the vehicle and the lane line points in each frame of the image. The at least two sets of point cloud data are fused to obtain the fused point cloud of the lane lines in each frame of the image.

[0089] After obtaining the vehicle's extrinsic parameter change data, the electronic device obtains at least two sets of point cloud data for each frame of the image based on the vehicle's extrinsic parameter change data and the lane line points of each frame of the image. The electronic device then fuses the at least two sets of point cloud data to obtain the fused point cloud of the lane lines in each frame of the image.

[0090] S205. Project the fused point cloud of lane lines from each frame of the image onto each frame of the image.

[0091] After obtaining the fused point cloud of lane lines for each frame of the image, the electronic device projects the fused point cloud of lane lines from each frame of the image onto each frame of the image.

[0092] S206. Sample the fused point cloud of lane lines in each frame of the image to obtain discrete points in each frame of the image.

[0093] Furthermore, the fused point cloud of the lane lines in each frame of the image is sampled (for example, a point is fitted every 10 pixels in the y direction) to obtain discrete points in each frame of the image.

[0094] S207. Filter the discrete points in each frame of the image to obtain the lane lines marked in each frame of the image.

[0095] After obtaining the discrete points in each frame of the image, the electronic device filters the discrete points in each frame of the image to obtain the lane lines marked in each frame of the image.

[0096] This embodiment of the disclosure acquires multiple frames of images and the pose data of the acquisition device when each frame of the multiple frames is acquired. Each frame of the multiple frames is input into a lane line detection model, and lane line points are obtained through the lane line detection model. Further, based on the multiple frames of images and the pose data of the acquisition device when each frame is acquired, extrinsic parameter variation data of the vehicle is obtained. For each lane line point in each frame, at least two sets of point cloud data are obtained based on the vehicle's extrinsic parameter variation data and the lane line points in each frame. These at least two sets of point cloud data are fused to obtain a fused point cloud of the lane lines in each frame. Then, the fused point cloud of the lane lines in each frame is projected onto each frame, and the fused point cloud of the lane lines in each frame is sampled to obtain discrete points in each frame. These discrete points are then filtered to obtain the lane lines marked in each frame. Compared to existing technologies, this disclosure obtains vehicle extrinsic parameter change data based on the multiple frames of images and the pose data of the acquisition device when each frame of image is acquired. For lane line points in each frame of image, at least two sets of point cloud data are obtained based on the vehicle extrinsic parameter change data and the lane line points in each frame of image. The at least two sets of point cloud data are fused to obtain the fused point cloud of lane lines in each frame of image. This can fill in missed lane lines and discard falsely detected lane lines, thereby improving lane line detection accuracy. It does not require high-precision maps, reducing lane line annotation costs. Then, the fused point cloud of lane lines in each frame of image is projected onto each frame of image. The fused point cloud of lane lines in each frame of image is sampled to obtain discrete points in each frame of image. The discrete points in each frame of image are filtered to obtain the annotated lane lines in each frame of image, further improving the lane line annotation effect.

[0097] Figure 3 A flowchart of a lane marking method provided in another embodiment of this disclosure is shown below. Figure 3 As shown, the method includes the following steps:

[0098] S301. Acquire multiple frames of images and the pose data of the acquisition device when each frame of the multiple frames of images is acquired.

[0099] Specifically, the implementation process and principle of S301 and S101 are the same, and will not be repeated here.

[0100] S302. Detect each frame of the multi-frame image to obtain the lane line points of each frame.

[0101] Specifically, the implementation process and principle of S302 and S102 are the same, and will not be repeated here.

[0102] S303. Based on the multi-frame images and the pose data of the acquisition device when each frame image is acquired, obtain the external parameter change data of the vehicle.

[0103] Specifically, the implementation process and principle of S303 and S203 are the same, and will not be repeated here.

[0104] S304. Project the lane line points of each frame image onto the bird's-eye view to obtain the first point cloud of the lane lines of each frame image.

[0105] The electronic device projects the lane line points detected on each frame of the image onto a bird's-eye view to obtain the first point cloud of the lane lines in each frame of the image.

[0106] S305. For the multi-frame images, based on the vehicle's external parameter change data, the next frame image is projected onto the bird's-eye view of the previous frame image to obtain the second point cloud of the lane line in each frame image.

[0107] In this step, for the multiple frames of images, the next frame of images is projected onto the bird's-eye view of the previous frame of images based on the external parameter change data of the vehicle, so as to obtain the second point cloud of the lane line in each frame of images.

[0108] For the multiple frames of images, projecting the subsequent frame onto the bird's-eye view of the previous frame can start from the Nth frame (the last frame). Based on the external parameter change data, the lane line point cloud on the Nth frame image is projected onto the bird's-eye view of the previous frame to obtain the lane line point cloud P' projected from the Nth frame to the (N-1)th frame. N-1 P' was analyzed using interpolation. N-1 For each lane line in the image, points are taken at equal intervals (e.g., 0-80 meters, fitting a point every 1 meter); for the previous frame's P... N-1The point cloud is subjected to the same fitting process. The first point cloud data (original point cloud data) P of the (N-1)th frame is calculated. N-1 The points on each lane line in the image are projected from the Nth frame to the lane line point cloud P' in the (N-1)th frame. N-1 The distance between points on each lane is considered to be on the same lane if the distance between two points is less than a preset threshold. Specifically, it involves calculating the distance between points on two lanes, such as... Figure 6 As shown, the first point cloud data in frame N-1 contains lane line 1 and lane line 2. The lane line point cloud P' projected from frame N to frame N-1 is... N-1 There are lane lines 1' and 2'. Lane line 1 has points A, B, and C, and lane line 2 has points D, E, and F. Calculate the relationship between point A of lane line 1 in the first point cloud data of frame N-1 and the lane line point cloud P' projected from frame N to frame N-1. N-1 The distances between points a, b, and c on lane line 1' and points d, e, and f on lane line 2' are calculated. If the distance between point A and point a is less than a preset threshold, lane line 1 and lane line 1' are considered to have a corresponding relationship, and points A and a are considered to be two corresponding points. Similarly, the distances between the remaining points in the first point cloud data of frame N-1 and points a, b, c, d, e, and f are calculated. The remaining points are points B, C, D, E, and F. Based on this, two lane lines with a corresponding relationship can be identified, and corresponding points on the lane lines with a corresponding relationship can also be identified.

[0109] For any point cloud point Pn, assign a confidence level Sn to each point in Pn based on its distance from the vehicle body. The farther away the point is, the lower the confidence level. Here, n represents any point in the point cloud.

[0110] For two points that have a corresponding relationship, the confidence level of the point on the (N-1)th frame is S. N-1 The lane line point cloud P' projected from frame N to frame N-1 N-1 Confidence S' of corresponding points on N-1 =1-S N-1 .like Figure 7 As shown, the circular black dots represent the first point cloud of lane lines in the (N-1)th frame image, and the triangles represent the lane line point cloud projected from the Nth frame onto the (N-1)th frame image. Further fusing the two sets of point cloud data yields the second point cloud: as shown... Figure 7 As shown, region 1 represents the intersecting part, and the value of the point on the second point cloud of the lane line obtained from the intersecting part is P^. N-1 =S N-1 *P N-1 +(1-S N-1 )*P' N-1Region 2 represents the non-overlapping area. Points from the (N-1)th frame in Region 2 are retained and added to the second point cloud of the lane line, i.e., P^ N-1 =P N-1 The points projected from the Nth frame in region 2 onto the lane line point cloud of the (N-1)th frame are added to the second lane line point cloud, i.e., P^ N-1 =P' N-1 Delete points X and Y (which are falsely detected points) in region 2 that are far from the lane lines. Frame by frame, fuse the points to obtain the second point cloud P^ of the lane lines in each frame, and record which lane line in each frame is composed of which lane line detected in which frame. This allows for the filling in of missed lane lines and the discarding of falsely detected lane lines.

[0111] It should be noted that the lane line point cloud of each frame is obtained by projecting the lane line point cloud from the next frame onto the bird's-eye view of the previous frame. Since there is no (N+1)th frame after the Nth frame, the aforementioned projection and fusion process cannot be performed on the Nth frame. There are three processing methods: 1) Do not fuse the lane lines of the Nth frame; 2) Replace P' with a blank frame. N Perform fusion; 3) Set the lane lines of the Nth frame image to P' by default. N To integrate.

[0112] In some embodiments, images of missed and false detections are input into the lane line detection model, and the model is fine-tuned to optimize detection performance.

[0113] S306. Based on the first point cloud of lane lines and the second point cloud of lane lines in each frame of the image, the lane lines of each frame of the image are fused to obtain the first fused point cloud of lane lines in each frame of the image.

[0114] The electronic device fuses the lane lines in each frame of an image based on the first and second point clouds of lane lines. Specifically, it compares the first and second point clouds of lane lines in each frame to achieve lane line fusion, resulting in a first fused point cloud of lane lines for each frame. Because lane lines are mutually verified, missed lane lines can be identified and falsely detected lane lines can be discarded, and lane line categories can be corrected. Lane line categories can include single dashed lines, single solid lines, double solid lines, double dashed lines, etc.

[0115] S307. For the multi-frame images, based on the vehicle's external parameter change data, the previous frame image is projected onto the bird's-eye view of the next frame image to obtain the lane line third point cloud of each frame image.

[0116] In this step, for the multiple frames of images, the previous frame image is projected onto the bird's-eye view of the next frame image based on the external parameter change data of the vehicle, so as to obtain the third point cloud of the lane line in each frame image.

[0117] For the multi-frame image, the previous frame image is projected onto the bird's-eye view of the next frame image. Starting from the first frame (starting frame), the lane line point cloud on the first frame image is projected onto the bird's-eye view of the next frame based on the external parameter change data to obtain the lane line point cloud P'2 projected from the first frame to the second frame. The same operation is performed, and its implementation principle and method are the same as S305, so it will not be described in detail here.

[0118] It should be noted that the lane line point cloud on the previous frame image is projected onto the bird's-eye view of the next frame to obtain the lane line point cloud P' of each frame image. Since there is no 0 frame image before the 1st frame image, the projection and fusion process cannot be performed on the 1st frame. There are three processing methods: 1) Do not fuse the lane lines of the 1st frame image; 2) Use a blank frame to replace P'1 for fusion; 3) Blend the lane lines of the 1st frame image by default as P'1.

[0119] S308. Based on the first point cloud of lane lines and the third point cloud of lane lines in each frame of the image, the lane lines of each frame of the image are fused to obtain the second fused point cloud of lane lines in each frame of the image.

[0120] The electronic device fuses the lane lines in each frame of the image based on the first and third point clouds of the lane lines. Specifically, it compares the first and third point clouds of the lane lines in each frame to achieve lane line fusion, resulting in a second fused point cloud of the lane lines in each frame. Because the lane lines are mutually verified, it can further supplement missed lane lines and discard falsely detected lane lines, and correct the lane line category. Lane line categories can include single dashed lines, single solid lines, double solid lines, double dashed lines, etc.

[0121] S309. Based on the first fused point cloud of the lane lines in each frame of the image and the second fused point cloud of the lane lines in each frame of the image, the lane lines of each frame of the image are fused to obtain the fused point cloud of the lane lines in each frame of the image.

[0122] The electronic device fuses the lane lines of each frame of the image based on a first fused point cloud and a second fused point cloud of the lane lines. Specifically, it fuses the lane lines by comparing the first and second fused point clouds of the lane lines in each frame, resulting in a fused point cloud of the lane lines for each frame. Through mutual verification of the lane lines, it further fills in any missed lane lines and discards any falsely detected lane lines, and corrects the lane line categories. Lane line categories can include single dashed lines, single solid lines, double solid lines, double dashed lines, etc.

[0123] S310. Based on the fused point cloud of lane lines in each frame of the image, the lane lines marked in each frame of the image are obtained.

[0124] Specifically, the implementation process and principle of S310 and S104 are the same, and will not be repeated here.

[0125] This embodiment of the disclosure acquires multiple frames of images and the pose data of the acquisition device when each frame of the multiple frames is acquired, and detects each frame of the multiple frames to obtain lane line points for each frame. Further, based on the multiple frames of images and the pose data of the acquisition device when each frame is acquired, extrinsic parameter change data of the vehicle is obtained. The lane line points of each frame are projected onto a bird's-eye view to obtain a first point cloud of lane lines for each frame. For the multiple frames of images, based on the extrinsic parameter change data of the vehicle, the subsequent frame is projected onto the bird's-eye view of the previous frame to obtain a second point cloud of lane lines for each frame. Based on the first and second point clouds of lane lines for each frame, the lane lines of each frame are fused to obtain a first fused point cloud of lane lines for each frame. For the multiple image frames, based on the vehicle's extrinsic parameter change data, the previous image frame is projected onto the bird's-eye view of the next image frame to obtain the third point cloud of lane lines for each frame. Based on the first and third point clouds of lane lines for each frame, the lane lines of each frame are fused to obtain a second fused point cloud of lane lines. Then, based on the first and second fused point clouds of lane lines for each frame, the lane lines of each frame are fused to obtain a fused point cloud of lane lines. Finally, the lane lines labeled in each frame are obtained based on the fused point clouds of lane lines. Compared to existing technologies, this disclosure does not require high-precision maps, reducing lane line labeling costs. Because the fused point cloud of lane lines for each frame is obtained through multiple lane line fusions, lane lines can be mutually verified, further supplementing missed lane lines and discarding falsely detected lane lines, and correcting lane line categories, thereby improving lane line labeling effectiveness.

[0126] Figure 4 This is a schematic diagram of the lane marking device provided in this embodiment. The lane marking device can be an electronic device as described in the above embodiment, or it can be a component or assembly within that electronic device. The lane marking device provided in this embodiment can execute the processing flow provided in the lane marking method embodiment, such as... Figure 4As shown, the lane marking device 40 includes: an acquisition module 41, a detection module 42, a fusion module 43, and a marking module 44; wherein, the acquisition module 41 is used to acquire multiple frames of images and the pose data of the acquisition device when each frame of the multiple frames of images is acquired; the detection module 42 is used to detect each frame of the multiple frames of images to obtain lane line points of each frame of images; the fusion module 43 is used to obtain at least two sets of lane line point cloud data of each frame of images based on the lane line points of its adjacent frames, and to fuse the at least two sets of point cloud data to obtain the fused point cloud of lane lines of each frame of images; the marking module 44 is used to obtain the marked lane lines in each frame of images based on the fused point cloud of lane lines of each frame of images.

[0127] Optionally, the pose data of the acquisition device includes at least the position, pitch angle, yaw angle, and roll angle of the acquisition device.

[0128] Optionally, when the detection module 42 detects each frame of the multi-frame images to obtain the lane line points of each frame, it is specifically used to: input each frame of the multi-frame images into the lane line detection model, and obtain the lane line points of each frame through the lane line detection model.

[0129] Optionally, the lane detection model is trained through the following steps: acquiring multiple sample images and the actual lane line points in each sample image; training the initial lane detection model based on the multiple sample images to obtain the lane line point detection results for each sample image; performing loss calculation on the lane line point detection results for each sample image based on the actual lane line points in each sample image to obtain the loss calculation result of the initial lane detection model; updating the model parameters of the initial lane detection model based on the loss calculation result of the initial lane detection model; if the accuracy of the updated lane detection model is greater than a preset threshold, then the lane detection model training is complete.

[0130] Optionally, when the fusion module 43 obtains at least two sets of lane line point cloud data for each frame of image based on the lane line points of its adjacent frame images, and fuses the at least two sets of point cloud data to obtain the fused point cloud of lane lines for each frame of image, it is specifically used to: obtain vehicle extrinsic parameter change data based on the multiple frames of images and the pose data of the acquisition device when each frame of image is acquired; for each frame of image, obtain at least two sets of point cloud data based on the vehicle extrinsic parameter change data and the lane line points of each frame of image, and fuse the at least two sets of point cloud data to obtain the fused point cloud of lane lines for each frame of image.

[0131] Optionally, when the fusion module 43 obtains at least two sets of point cloud data based on the vehicle's extrinsic parameter change data and the lane line points of each frame of image, and fuses the at least two sets of point cloud data to obtain the fused point cloud of the lane lines of each frame of image, it is specifically used to: project the lane line points of each frame of image onto a bird's-eye view to obtain the first point cloud of the lane lines of each frame of image; for the multiple frames of image, project the next frame of image onto the bird's-eye view of the previous frame of image based on the vehicle's extrinsic parameter change data to obtain the second point cloud of the lane lines of each frame of image; and based on the first point cloud of the lane lines and the second point cloud of the lane lines of each frame of image, perform the fusion of the lane line points of each frame of image on the fused point cloud of the lane lines of each frame of image. The lane lines of each frame of the image are fused to obtain a first fused point cloud of the lane lines in each frame of the image; for the multiple frames of the image, the previous frame of the image is projected onto the bird's-eye view of the next frame of the image based on the external parameter change data of the vehicle to obtain a third point cloud of the lane lines in each frame of the image; based on the first point cloud and the third point cloud of the lane lines in each frame of the image, the lane lines of each frame of the image are fused to obtain a second fused point cloud of the lane lines in each frame of the image; based on the first fused point cloud and the second fused point cloud of the lane lines in each frame of the image, the lane lines of each frame of the image are fused to obtain a fused point cloud of the lane lines in each frame of the image.

[0132] Optionally, when the annotation module 44 obtains the annotated lane lines in each frame image based on the fused point cloud of lane lines in each frame image, it is specifically used to: project the fused point cloud of lane lines in each frame image onto each frame image; sample the fused point cloud of lane lines in each frame image to obtain discrete points in each frame image; and filter the discrete points in each frame image to obtain the annotated lane lines in each frame image.

[0133] Figure 4 The lane marking device shown in the embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and will not be repeated here.

[0134] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. The electronic device can be the electronic device described in the above embodiments. The electronic device provided in this disclosure can execute the processing flow provided in the lane marking method embodiments, such as… Figure 5 As shown, the electronic device 50 includes: a memory 51, a processor 52, a computer program, and a communication interface 53; wherein the computer program is stored in the memory 51 and configured to be executed by the processor 52 using the lane marking method described above.

[0135] In addition, this disclosure also provides a computer-readable storage medium storing a computer program thereon, which is executed by a processor to implement the lane marking method described in the above embodiments.

[0136] Furthermore, this disclosure also provides a computer program product, which includes a computer program or instructions that, when executed by a processor, implement the lane marking method described above.

[0137] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0138] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0139] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0140] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0141] Acquire multiple frames of images and the pose data of the acquisition device when each frame of the multiple frames of images is acquired;

[0142] Each frame of the multi-frame image is detected to obtain the lane line points of each frame;

[0143] For each frame of the image, based on the lane line points of its adjacent frame images, at least two sets of lane line point cloud data of the frame image are obtained, and the at least two sets of point cloud data are fused to obtain the fused point cloud of the lane lines of each frame image.

[0144] The lane lines marked in each frame of the image are obtained by fusing the point cloud of the lane lines in each frame of the image.

[0145] In addition, the electronic device can also perform other steps in the lane marking method described above.

[0146] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0147] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0148] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0149] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0150] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0151] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0152] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for marking lane lines, characterized in that, The method includes: Acquire multiple frames of images and the pose data of the acquisition device when each frame of the multiple frames is acquired; the pose data of the acquisition device includes at least the position, pitch angle, yaw angle, and roll angle of the acquisition device; Each frame of the multi-frame image is detected to obtain the lane line points of each frame; For each frame of the image, based on the lane line points of its adjacent frame images, at least two sets of lane line point cloud data of the frame image are obtained, and the at least two sets of point cloud data are fused to obtain the fused point cloud of the lane lines of each frame image. The lane lines marked in each frame of the image are obtained by fusing the point cloud of the lane lines in each frame of the image. Specifically, for each frame of image, based on the lane line points in adjacent frames, at least two sets of lane line point cloud data for that frame are obtained. These at least two sets of point cloud data are then fused to obtain a fused point cloud of lane lines for each frame, including: Based on the multiple frames of images and the pose data of the acquisition device when each frame of image is acquired, the external parameter change data of the vehicle is obtained. For lane line points in each frame of an image, at least two sets of point cloud data are obtained based on the external parameter change data of the vehicle and the lane line points in each frame of an image. The at least two sets of point cloud data are fused to obtain the fused point cloud of the lane lines in each frame of an image. For each frame of the image, at least two sets of point cloud data are obtained based on the vehicle's extrinsic parameter change data and the lane line points of each frame of the image. These at least two sets of point cloud data are then fused to obtain a fused point cloud of the lane lines for each frame of the image, including: The lane line points of each frame image are projected onto the bird's-eye view to obtain the first point cloud of the lane lines of each frame image; For the multi-frame image, the next frame image is projected onto the bird's-eye view of the previous frame image based on the vehicle's external parameter change data to obtain the second point cloud of the lane line in each frame image. Based on the first point cloud of lane lines and the second point cloud of lane lines in each frame of the image, the lane lines of each frame of the image are fused to obtain the first fused point cloud of lane lines in each frame of the image. For the multi-frame images, the previous frame image is projected onto the bird's-eye view of the next frame image based on the vehicle's external parameter change data to obtain the lane line third point cloud of each frame image. Based on the first point cloud of lane lines and the third point cloud of lane lines in each frame of the image, the lane lines of each frame of the image are fused to obtain the second fused point cloud of lane lines in each frame of the image. The lane lines of each frame are fused based on the first fused point cloud and the second fused point cloud of the lane lines of each frame to obtain the fused point cloud of the lane lines of each frame.

2. The method according to claim 1, characterized in that, The step of detecting each frame of the multi-frame images to obtain the lane line points of each frame includes: Each frame of the multi-frame image is input into the lane line detection model, and the lane line points of each frame are obtained through the lane line detection model.

3. The method according to claim 2, characterized in that, The lane detection model is trained through the following steps: Acquire multiple sample images and the actual lane line points in each sample image; The initial lane detection model is trained based on the multi-frame sample images to obtain the lane line point detection results for each frame sample image. Based on the actual lane line points in each sample image, the loss calculation is performed on the lane line point detection results of each sample image to obtain the loss calculation result of the initial lane line detection model. Based on the loss calculation results of the initial lane detection model, update the model parameters of the initial lane detection model; If the accuracy of the updated lane detection model is greater than a preset threshold, then the lane detection model training is complete.

4. The method according to claim 1, characterized in that, The process of obtaining the lane lines marked in each frame of the image by fusing point clouds of lane lines includes: Project the fused point cloud of lane lines from each frame onto each frame of the image; The fused point cloud of lane lines in each frame of the image is sampled to obtain discrete points in each frame of the image; The discrete points in each frame of the image are filtered to obtain the lane lines marked in each frame of the image.

5. A lane marking device, characterized in that, The device includes: The acquisition module is used to acquire multiple frames of images and the pose data of the acquisition device when each frame of the multiple frames of images is acquired; the pose data of the acquisition device includes at least the position, pitch angle, yaw angle and roll angle of the acquisition device; The detection module is used to detect each frame of the multi-frame images to obtain the lane line points of each frame image; The fusion module is used to obtain at least two sets of lane line point cloud data for each frame of the image based on the lane line points of its adjacent frame images, and to fuse the at least two sets of point cloud data to obtain the fused point cloud of lane lines for each frame of the image. The annotation module is used to obtain the annotated lane lines in each frame of the image based on the fused point cloud of the lane lines in each frame of the image; The fusion module is specifically used to obtain the vehicle's extrinsic parameter change data based on the multi-frame images and the pose data of the acquisition device when each frame image is acquired; for the lane line points of each frame image, at least two sets of point cloud data are obtained based on the vehicle's extrinsic parameter change data and the lane line points of each frame image, and the at least two sets of point cloud data are fused to obtain the fused point cloud of the lane lines of each frame image. For each frame of the image, at least two sets of point cloud data are obtained based on the vehicle's extrinsic parameter change data and the lane line points of each frame of the image. These at least two sets of point cloud data are then fused to obtain a fused point cloud of the lane lines for each frame of the image, including: The lane line points of each frame image are projected onto the bird's-eye view to obtain the first point cloud of the lane lines of each frame image; For the multi-frame image, the next frame image is projected onto the bird's-eye view of the previous frame image based on the vehicle's external parameter change data to obtain the second point cloud of the lane line in each frame image. Based on the first point cloud of lane lines and the second point cloud of lane lines in each frame of the image, the lane lines of each frame of the image are fused to obtain the first fused point cloud of lane lines in each frame of the image. For the multi-frame images, the previous frame image is projected onto the bird's-eye view of the next frame image based on the vehicle's external parameter change data to obtain the lane line third point cloud of each frame image. Based on the first point cloud of lane lines and the third point cloud of lane lines in each frame of the image, the lane lines of each frame of the image are fused to obtain the second fused point cloud of lane lines in each frame of the image. The lane lines of each frame are fused based on the first fused point cloud and the second fused point cloud of the lane lines of each frame to obtain the fused point cloud of the lane lines of each frame.

6. An electronic device, characterized in that, include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-4.

Images